Last week I got some complaints about user profile information not being synchronized to other sites in a SharePoint 2007 environment. After some investigation I found the following error in the ULS log:
<date> <time> OWSTIMER.EXE (0x0864) 0x08A4 SharePoint Portal Server User Profiles 8zzy Medium Aborting sweepsynch for guid instance <GUID> due to null or non-online content database
[Solution]
After configuring the database as Ready again, the profile information synchronization resumed again.
[Background information]
If you put you content database off-line, SharePoint is still able to access it but it will not create new site collections. Because were creating new site collections, which we wanted created in a specific database we put all databases (except for that specific one) on off-line. As it turned out, the downside of this is that SharePoint is unable to synchronized the user profile information to databases that are off-line.
Wednesday, November 26, 2008
Monday, November 24, 2008
[MOSS2007/WSSv3] Messages about missing resource files in SharePoint ULS logs
[ISSUE]
The following messages appear a lot in the ULS logs of SharePoint:
11/24/2008 10:06:58.66 w3wp.exe (0x07F4) 0x1550 Windows SharePoint Services General 72kg High #20015: Cannot open "Proj.en-US.resx": no such file or folder.
11/24/2008 10:06:58.66 w3wp.exe (0x07F4) 0x1550 Windows SharePoint Services General 72kg High (#2: Cannot open "Proj.en-US.resx": no such file or folder.
[SOLUTION]
This is caused by the fact that the resource file is not present with a specific English tag in its name. To solve this issue do the following
The following messages appear a lot in the ULS logs of SharePoint:
11/24/2008 10:06:58.66 w3wp.exe (0x07F4) 0x1550 Windows SharePoint Services General 72kg High #20015: Cannot open "Proj.en-US.resx": no such file or folder.
11/24/2008 10:06:58.66 w3wp.exe (0x07F4) 0x1550 Windows SharePoint Services General 72kg High (#2: Cannot open "Proj.en-US.resx": no such file or folder.
[SOLUTION]
This is caused by the fact that the resource file is not present with a specific English tag in its name. To solve this issue do the following
- Open the directory C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\Resources
- The mentioned files are not in there, but the proj.resx is
- Open a command prompt and browse to the same directory
- Run the following commands:
copy avm.resx avm.en-US.resx
copy compproc.resx compproc.en-US.resx
copy ep.resx ep.en-US.resx
copy er.resx er.en-US.resx
copy hlpdsk.resx hlpdsk.en-US.resx
copy itteam.resx itteam.en-US.resx
copy proj.resx proj.en-US.resx
copy pws.resx pws.en-US.resx
copy rer.resx rer.en-US.resx
copy tsa.resx tsa.en-US.resx - This will copy the default resource files to the English version resource files and the messages will disappear.
Monday, November 17, 2008
[MOSS2007/WSSv3] Kerberos, CNAMEs and SharePoint
When trying to configure Kerberos on the environment, I ran into the following issue:
[ISSUE]
When opening the SharePoint site from another machine, Internet Explorer logs on with NTLM anyway. When opening the site from the server, a logon prompt appears which won't accept any credentials. After three times, an Unauthorized message is shown.
[TROUBLESHOOTING]
I had all Service Principal Names configured as described in Configure Kerberos authentication (Office SharePoint Server). To get more information I enabled Kerberos logging (How to enable Kerberos event logging), which resulted in a lot of events in the System log:
[SOLUTION]
After using Network Monitor v3.2, I found that the Kerberos traffic from the server was requesting a different Service Principal Name than the url of the SharePoint site. The error which was also listed in the eventlog was returned by AD. The SPN that was send, contained the CNAME value instead of the original URL. Using Google, I found the following article:
The Problem with Kerberos Delegation (Section: "Confused Yet?")
This article states the following:
After changing our CNAME records to A records, all Kerberos issues were resolved.
In other words:
Do not use CNAMEs when using Kerberos authentication!!!
[ISSUE]
When opening the SharePoint site from another machine, Internet Explorer logs on with NTLM anyway. When opening the site from the server, a logon prompt appears which won't accept any credentials. After three times, an Unauthorized message is shown.
[TROUBLESHOOTING]
I had all Service Principal Names configured as described in Configure Kerberos authentication (Office SharePoint Server). To get more information I enabled Kerberos logging (How to enable Kerberos event logging), which resulted in a lot of events in the System log:
Source: Kerberos
Type: Error
Event: 3
Message: "A Kerberos Error Message was received:
on logon session <user>
Client Time:
Server Time: 12:5:19.0000 10/30/2008 Z
Error Code: 0x34 KRB_ERR_RESPONSE_TOO_BIG
Extended Error:
Client Realm:
Client Name:
Server Realm: <domain>
Server Name: krbtgt/<domain>
Target Name: krbtgt/<domain>@<domain>
Error Text:
File: e
Line: 6c0
Error Data is in record data.
Type: Error
Event: 3
Message: "A Kerberos Error Message was received:
on logon session <user>
Client Time:
Server Time: 12:5:19.0000 10/30/2008 Z
Error Code: 0x34 KRB_ERR_RESPONSE_TOO_BIG
Extended Error:
Client Realm:
Client Name:
Server Realm: <domain>
Server Name: krbtgt/<domain>
Target Name: krbtgt/<domain>@<domain>
Error Text:
File: e
Line: 6c0
Error Data is in record data.
[SOLUTION]
After using Network Monitor v3.2, I found that the Kerberos traffic from the server was requesting a different Service Principal Name than the url of the SharePoint site. The error which was also listed in the eventlog was returned by AD. The SPN that was send, contained the CNAME value instead of the original URL. Using Google, I found the following article:
The Problem with Kerberos Delegation (Section: "Confused Yet?")
This article states the following:
- Computer running IIS: ws.identitychaos.com (WS)
- Service Account hosting the Central Admin app pool: idchaos\svc.wss.farm
- Host Header for Central Admin: spca.identitychaos.com
- SPN's registered to idchaos\svc.wss.farm
- HTTP/spca
- HTTP/spca.identitychaos.com
Using a CNAME, when IE requests the SPN, spca.identitychaos.com resolves to ws.identitychaos.com (the alias resolves to the host) so IE asks AD, "tell me which account has an SPN registration of HTTP/ws.identitychaos.com" and it gets the WS computer account, not the service account. How do you fix it? You use A records instead of CNAME's.
After changing our CNAME records to A records, all Kerberos issues were resolved.
In other words:
Do not use CNAMEs when using Kerberos authentication!!!
Wednesday, November 12, 2008
[MOSS2007] Search issues due to corrupt SSL certificate
After installing MOSS2007 which has Service Pack 1, the Infrastructure Update and the August Cumulative Update slipstreamed I ran into the following issue:
My environment consists of one Index server and two Search servers. When opening the new Search Administration page, it is supposed to list the status of all of the index and search servers. Unfortunately the Search servers were listed as Error. After doing some more investigation I found the following error appear in the application log of both search servers each minute:
When I tried to open the SearchAdmin.asmx page (http://<index or search server name>/SearchAdmin.asmx), I get a page cannot be displayed.
This issue is caused by a corrupt SSL certificate on the "Office Web Services" web site. To solve this issue, you have to create a new certificate. You can use the SelfSSL tool, included in the IIS resource kit.
Run the following command on all search and index servers:
selfssl.exe /N:CN=<server> /K:1024 /V:9999 /S:951338967 /P:56738
My environment consists of one Index server and two Search servers. When opening the new Search Administration page, it is supposed to list the status of all of the index and search servers. Unfortunately the Search servers were listed as Error. After doing some more investigation I found the following error appear in the application log of both search servers each minute:
Event: 6482
Type: Error
Source: Office SharePoint Server
Category:Office Server Shared Services
Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (aa8a83f6-ff42-4be3-ae6c-c1cb2205cc8f).
Reason: The underlying connection was closed: An unexpected error occurred on a send.
Techinal Support Details:
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)
at Microsoft.Office.Server.Search.Administration.SearchApi..ctor(WellKnownSearchCatalogs catalog, SearchSharedApplication application)
at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Type: Error
Source: Office SharePoint Server
Category:Office Server Shared Services
Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (aa8a83f6-ff42-4be3-ae6c-c1cb2205cc8f).
Reason: The underlying connection was closed: An unexpected error occurred on a send.
Techinal Support Details:
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)
at Microsoft.Office.Server.Search.Administration.SearchApi..ctor(WellKnownSearchCatalogs catalog, SearchSharedApplication application)
at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
When I tried to open the SearchAdmin.asmx page (http://<index or search server name>/SearchAdmin.asmx), I get a page cannot be displayed.
This issue is caused by a corrupt SSL certificate on the "Office Web Services" web site. To solve this issue, you have to create a new certificate. You can use the SelfSSL tool, included in the IIS resource kit.
Run the following command on all search and index servers:
selfssl.exe /N:CN=<server> /K:1024 /V:9999 /S:951338967 /P:56738
Thursday, November 06, 2008
[MOSS2007] Crawl issue after installation .NET Framework 3.5 SP1
[ISSUE]
When the .NET Framework 3.5 SP1 has been installed, it is possible that you get some Access Denied error messages when crawling your MOSS sites. If you're indexer is set to crawl itself, via the Central Admin setting, or manual hosts modification, you might be running into this issue. You will see a combination of errors in your event logs, ULS logs and crawl logs that match the following:
Crawl Log:Event Type: Warning
Event Source: Office Server Search
Event Category: Gatherer
Event ID: 2436
Date: 10/22/2008
Time: 10:02:55 AM
User: N/A
Computer: MOSS2007
Description:The start address <http://portal/> cannot be crawled.
Context: Application 'SharedServices1', Catalog 'Portal_Content'
Details:Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content. (0x80041205)
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
ULS Logs: (verbose)10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common MS Search Indexing 7hoo Verbose GetVirtualServerPolicy is about to call WS for virtual server, strVirtualServerUrl http://portal/10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable ***** Couldn't retrieve server http://portal/ policy, hr = 80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:548
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable ****** Server portal security initialization failed, hr = 80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:317
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::InitURLType: Return error to caller, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3acc.cxx Line:1542
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::InitURLType fails, Url http://portal/, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3acc.cxx Line:184
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::Init fails, Url http://portal/, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3handler.cxx Line:243
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Handler::CreateAccessorExB: Return error to caller, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3handler.cxx Line:261
Security Logs:Logon Failure:
Reason: An error occurred during logon
User Name: mosscrawl
Domain: SPI
Logon Type: 3
Logon Process: Èù8
Authentication Package: NTLM
Workstation Name: MOSS2007
Status code: 0xC000006D
Substatus code: 0x0
Caller User Name: -
Caller Domain: -
Caller Logon ID: -
Caller Process ID: -
Transited Services: -
Source Network Address: XX.XX.XX.XX
Source Port: 1417
[SOLUTION]
Perform the steps mentioned in the KB article below to fix this issue:
http://support.microsoft.com/kb/896861
Source: Ron Grzywacz's Blog
When the .NET Framework 3.5 SP1 has been installed, it is possible that you get some Access Denied error messages when crawling your MOSS sites. If you're indexer is set to crawl itself, via the Central Admin setting, or manual hosts modification, you might be running into this issue. You will see a combination of errors in your event logs, ULS logs and crawl logs that match the following:
Crawl Log:Event Type: Warning
Event Source: Office Server Search
Event Category: Gatherer
Event ID: 2436
Date: 10/22/2008
Time: 10:02:55 AM
User: N/A
Computer: MOSS2007
Description:The start address <http://portal/> cannot be crawled.
Context: Application 'SharedServices1', Catalog 'Portal_Content'
Details:Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content. (0x80041205)
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
ULS Logs: (verbose)10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common MS Search Indexing 7hoo Verbose GetVirtualServerPolicy is about to call WS for virtual server, strVirtualServerUrl http://portal/10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable ***** Couldn't retrieve server http://portal/ policy, hr = 80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:548
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable ****** Server portal security initialization failed, hr = 80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:317
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::InitURLType: Return error to caller, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3acc.cxx Line:1542
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::InitURLType fails, Url http://portal/, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3acc.cxx Line:184
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::Init fails, Url http://portal/, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3handler.cxx Line:243
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Handler::CreateAccessorExB: Return error to caller, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3handler.cxx Line:261
Security Logs:Logon Failure:
Reason: An error occurred during logon
User Name: mosscrawl
Domain: SPI
Logon Type: 3
Logon Process: Èù8
Authentication Package: NTLM
Workstation Name: MOSS2007
Status code: 0xC000006D
Substatus code: 0x0
Caller User Name: -
Caller Domain: -
Caller Logon ID: -
Caller Process ID: -
Transited Services: -
Source Network Address: XX.XX.XX.XX
Source Port: 1417
[SOLUTION]
Perform the steps mentioned in the KB article below to fix this issue:
http://support.microsoft.com/kb/896861
Source: Ron Grzywacz's Blog
Wednesday, November 05, 2008
[MOSS2007/WSSv3] Login box when a normal user logs on, but not with admin
[ISSUE]
A logon box constantly appears when a users is trying to log onto a SharePoint site. Whatever the user enters, he cannot get onto SharePoint.
[CAUSE]
The environment on which we experienced this behaviour was locked down according to security guidelines. One of these guidelines was to strip the "Read & Execute" permissions on the C:\Windows folder from the local users group. The .NET Framework requires Execute permissions on the C:\Windows directory to operate properly and because SharePoint relies on .NET, users were prompted for credentials.
[SOLUTION]
Add the Execute permissions for the Users group on the C:\Windows folder or restore the permissions on the C:\Windows directory to default:
- Open a command prompt.
- On a domain controller, run
secedit /configure /db c:\windows\temp\seceditsv.sdb /cfg "c:\windows\security\templates\DC security.inf" /log c:\windows\temp\seceditsv.log
- On a non-domain controller, run
secedit /configure /db c:\windows\temp\seceditsv.sdb /cfg "c:\windows\security\templates\setup security.inf" /log c:\windows\temp\seceditsv.log
A logon box constantly appears when a users is trying to log onto a SharePoint site. Whatever the user enters, he cannot get onto SharePoint.
[CAUSE]
The environment on which we experienced this behaviour was locked down according to security guidelines. One of these guidelines was to strip the "Read & Execute" permissions on the C:\Windows folder from the local users group. The .NET Framework requires Execute permissions on the C:\Windows directory to operate properly and because SharePoint relies on .NET, users were prompted for credentials.
[SOLUTION]
Add the Execute permissions for the Users group on the C:\Windows folder or restore the permissions on the C:\Windows directory to default:
- Open a command prompt.
- On a domain controller, run
secedit /configure /db c:\windows\temp\seceditsv.sdb /cfg "c:\windows\security\templates\DC security.inf" /log c:\windows\temp\seceditsv.log
- On a non-domain controller, run
secedit /configure /db c:\windows\temp\seceditsv.sdb /cfg "c:\windows\security\templates\setup security.inf" /log c:\windows\temp\seceditsv.log
Subscribe to:
Posts (Atom)