[SP ALL] Opening a web service is returning a 401.1 "Access Denied" error

[ISSUE]
Yesterday I was asked to assist in troubleshooting an issue with a SharePoint web service. The SharePoint indexing process failed to work properly for just one web application. Some investigation revealed that the indexer was unable to open the sitedata.asmx web service. When trying to open the same web service via IE, I was prompted for credentials however whatever credentials were entered, after three attempts an "Access Denied" page (401.1 error) was shown.

[SOLUTION]
Unfortunately Process Monitor didn't reveal anything and I noticed that the sitedata web service wasn't the only web service that failed. After some troubleshooting I found out that the cause was in the web config:

The "remove verb *.asmx" line was placed after the "add verb *.asmx" line in the httphandler setting, essentially removing the configuration after adding it. For example:

<add verb="*" path="*.asmx" validate="false" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
....
<remove verb="*" path="*.asmx" />

After correcting this by placing the remove line in front of the add line, all web services started working just fine!

<remove verb="*" path="*.asmx" />
....
<add verb="*" path="*.asmx" validate="false" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />

[EXPLANATION]
Why is this happening? By first placing the remove line, you make sure any declarations that are done in other (global) configuration files are made void. That way you know for sure that no conflicts will occur between two configurations. However after removing the asmx httphandler, you have to declare it again, else SharePoint (or better yet IIS) does not know how to handle the asmx file. The confusing part for this issue is that it will display an authentication prompt to the user, without it actually being an authentication issue.

[SP2010] SharePoint Timer service crashes constantly

[SYMPTOMS]
I tried to retract a solution, but the status remained "Retracting" and never changed. After some investigation I found out that the SharePoint Timer service on one of the servers crashed every couple of minutes. The event log showed the following errors:

Log Name: System
Source: Service Control Manager
Date: 3/31/2011 9:24:55 AM
Event ID: 7024
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: [Server name]
Description:
The SharePoint 2010 Timer service terminated with service-specific error %%-2147467259.

and

Log Name: System
Source: Service Control Manager
Date: 3/31/2011 9:24:55 AM
Event ID: 7031
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: [Server name]
Description:
The SharePoint 2010 Timer service terminated unexpectedly. It has done this 3 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service.

The ULS log showed the following errors:

- The timer service could not initialize its configuration, please check the configuration database. Will retry later.

- Exiting the process because the timer could not be initialized after multiple attempts.
- The timer service is stopping

[CAUSE]
After searching the Internet I found one article where someone explained that this was caused by the fact that the Configuration Cache directory (C:\ProgramData\Microsoft\SharePoint\Config) did not contain a folder with the farm GUID as the name. After checking the configuration cache folder, that folder was indeed missing.

I then remembered I had to clear the configuration cache last week because the implementation of the February 2011 Cumulative Update failed during the Configuration Wizard step. Clearing the configuration cache fixed this issue. As it turned out, I was a little too enthousiastic with deleting the folders :-)

[Resolution]

1. Open the Registry Editor
2. Browse to HKLM > SOFTWARE > Microsoft > Shared Tools > Web Server Extensions > 14.0 > Secure > ConfigDB
3. Copy the value in the property "Id"
4. Browse to folder C:\ProgramData\Microsoft\SharePoint\Config and create a folder with the name of the previously copied value
5. Restart the SharePoint Timer service
6. The folder should be populated with XML files within a minute.

[MOSS/WSSv3] Site Settings not visible, even when I am a site collection admin

[ISSUE]
Last week, I had to perform some restore activities for a site collection that became corrupt. When I was done, users reported that permissions were gone. After logging on I found out that I wasn't able to open the Site Settings. The Site Actions > Site Settings menu option was gone?!?!?!?

[SOLUTION]
After some troubleshooting I found out that during earlier steps, the site collection accidentally was set to Read-Only lock. That is why I wasn't shown any admin screens. When the lock was lifted, all started working again!

[MOSS] Unknown Error when opening AreaNavigationSettings.aspx

[ISSUE]
Last week one of my colleagues contacted me about an issue he encountered. When trying to open the navigation settings page (/_layouts/AreaNavigationSettings.aspx), he encountered an "Unknown Error" (don't you just love those nice and clear errors). When opening the default page, the Quick Launch navigation on that page was also displaying errors:




[TROUBLESHOOTING]
When using the CustomErrors=Off and StackTrace=True options in the web.config, we were able to retrieve a "Object reference not set to an instance of an object" message. In the ULS logs, we found the following errors:

• Unable to retrieve the CachedObject that this ProxySiteMapNode references: /sites/test/sites/183/Pages/testpage.aspx
• PortalSiteMapProvider was unable to fetch children for node at URL: , message: Object reference not set to an instance of an object

[SOLUTION]
The quick launch navigation was changed manually and navigation items were added. Looking a little further, I found that testpage.aspx didn't exist anymore in the Pages library. Probably a user had deleted this page, causing this issue. When I recreated the testpage.aspx, the Quick Launch navigation started working again and the Navigation Settings page didn't show the error anymore.

[MOSS/WSSv3] Error while deploying a solution

[ISSUE]
Last week I encountered an issue while deploying a solution. I received the following error "This operation uses the SharePoint Administration service (spadmin), which could not be contacted. If the service is stopped or disabled, start it and try the operation again."

[SITUATION]
The environment I was working on consisted out of three servers, one web front end, one application and one database server. The application server is running the Central Administration role. The above error was thrown on the web front end server.

[CAUSE]
This issue was caused by a bug in the .NET Framework v2.0. After deploying this update, the issue was solved and the deployment ran fine.

SharePoint 2010 RC installation bug

The Release Candidate of SharePoint 2010 has an issue with the installation. When installing on a domain controller, you will end up with SharePoint 2010 installed in a Stand-Alone setup. This setup is using SQL Express 2008, which is not supported on a domain controller. Now I hear you say "why are you installing on a DC" and you are right, you don't want SharePoint installed on a DC.......in a production situation. In my case I am testing on a single VM, with all the roles on one server, including the DC role.

Quote MS: "In the Release Candidate build, we recommend you do not to install on a Domain Controller. While the Setup User Interface does the correct thing by suppressing the Standalone option, after the installation is done and the RC is fully installed, it is actually a Standalone install and not a farm install. Although you are not technically blocked, the install will not work since SQL Express 2008 is not supported on a Domain Controller."

Because SQL Express 2008 is not supported on a DC, if the installation detects you are installing on a DC, the selection page where you can choose between a Stand-Alone and Complete installation is removed. Unfortunately the Stand-Alone option is the default selected option, causing you to end up with a Stand-Alone installation anyways and no way to get around this via the GUI :-)

Solution:
The best way to install SharePoint 2010 RC on a domain controller is by using an scripted installation.

1. Extract the installation file by executing <filename> /extract:<folder name>
2. Open a Windows Explorer and create a file called config.xml.
3. Open the config.xml file and paste in the part below
4. Change the <PID KEY> into your own product key and the install folder
5. Save the file
6. Start a command prompt and browse to the folder
7. Run "setup /config config.xml"
8. SharePoint 2010 RC is now being installed as a Complete installation!

Have fun SharePointing!


Config.xml:

<Configuration>
<Package Id="sts">
<Setting Id="LAUNCHEDFROMSETUPSTS" Value="Yes"/>
</Package>

<Package Id="spswfe">
<Setting Id="SETUPCALLED" Value="1"/>
<Setting Id="OFFICESERVERPREMIUM" Value="1" />
</Package>

<Logging Type="verbose" Path="%temp%" Template="SharePoint Server Setup(*).log"/>
<PIDKEY Value="<PID KEY>" />
<Setting Id="SERVERROLE" Value="APPLICATION"/>
<Setting Id="USINGUIINSTALLMODE" Value="0"/>
<Setting Id="SETUP_REBOOT" Value="Never" />
<Setting Id="SETUPTYPE" Value="CLEAN_INSTALL"/>
<INSTALLLOCATION Value="D:\Program Files\Microsoft SharePoint" />
<Display Level="Basic" CompletionNotice="Yes" AcceptEULA="Yes" />
</Configuration>

[MOSS/WSSv3] Error "The detection failed, this can be due to a corrupted installation database"

[ISSUE]
When trying to install a SharePoint Cumulative Update I encountered the following error, directly after the update detection:

The detection failed, this can be due to a corrupted installation database

[CAUSE]
The installation files in the C:\Windows\Installer folder were deleted. These files are required to install updates.

[BACKGROUND]
When an application or update of an application is installed, the installation files are stored in the C:\Windows\Installer folder. When another update is installed, the updates checks the Installer folder and retrieves install information and other required components from that folder. So never delete files in the C:\Windows\Installer folder......in our case someone unfortunately did, resulting in the earlier mentioned error :-(

[SOLUTION]
On the MS Forums I found the solution for this issue. Doug Chandler has found out a way of copying the missing files from a working server to the broken server and running the Office Diagnostics tool to fix the issue. The only difference I encountered was that when I ran the Office Diagnostics tool, the tool required the installation packages of SharePoint and all of the language packs. By default it check the same folders as used during installation, so if you have these files in another directory it is going to ask you for the specific folder for all packages (148 in my case).

[MOSS] Event 5214 - The EXECUTE permission was denied on the object 'proc_FetchDocForUpdate'

[ISSUE]
The following event was occuring on our environment quite often:

Event Type: Error
Event Source: Windows SharePoint Services 3
Event Category: Database
Event ID: 5214
Date: 11/19/2009
Time: 4:28:54 PM
User: N/A
Computer: [SERVER]
Description:
Insufficient SQL database permissions for user '[account]' in database 'SharePoint_AdminContent_3995bd54-8091-4157-b162-8aaaf7116355' on SQL Server instance '[SQL SERVER]'. Additional error information from SQL Server is included below.

The EXECUTE permission was denied on the object 'proc_FetchDocForUpdate, database 'SharePoint_AdminContent_3995bd54-8091-4157-b162-8aaaf7116355', schema 'dbo'.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

[SOLUTION]
To solve this issue, perform the following steps:

• Open the SQL Management Studio
• Browse to the database in question, in our case "'SharePoint_AdminContent_3995bd54-8091-4157-b162-8aaaf7116355"
• Open the database and then Security > Roles > Database Roles
• In the right part of the window, right click the WSS_Content_Application_Pools role and click Properties
• Select the menu option "Securables"
• Click "Add"
• Select "Specific objects" and click "OK"
• Click "Object Types", select "Stored Procedures" and click "OK"
• Add the following stored procedures: proc_FetchDocForUpdate, proc_GetWebMetaInfo, proc_UpdateDirtyDocument, proc_UpdateListItem
• Click "OK" to add these stored procedures
• Select the added stored procedures and select "Execute" in the "Grant" column.
• Click "Add" once more
• Select "Specific objects" and click "OK"
• Click "Object Types", select "Views" and click "OK"
• Add the following view: UserData
• Click "OK" to add this view
• Select the added view and select "Select" in the "Grant" column.
• Click "OK" to complete

[MOSS/WSSv3] Error 1387 when removing users from Farm Administrators group

[ISSUE]
Today I tried to remove the user accounts from the Farm Administrators group, that have left the company a while ago. Unfortunately I was unable to do so, SharePoint presented me with an error 1387.

[CAUSE]
Using Google I ran into the following blog post Unable to remove user from SharePoint Farm Administrators group : Error 1387. Here Tim was talking about the fact that the accounts were deleted and SharePoint was performing some kind of check.

[SOLUTION]
After temporarily recreating the accounts of the users, I was able to delete the accounts from the group successfully!

[MOSS2007/WSSv3] "The expected version of the product was not found on the system" while installing update

[ISSUE]
I just tried to install the June Cumulative Update on a test environment. After the "Running detection" step, I got the following error:

"The expected version of the product was not found on the system"

[CAUSE]
Some troubleshooting revealed that we had installed SP2 for WSS and MOSS on the environment, but not for the installed language packs. After we installed SP2 for the WSS and MOSS Language Packs, the update installed just fine.

[MOSS/WSSv3] Error "Unable to get the private bytes memory limit for the W3WP process."

[ISSUE]
On the environment of a customer we encountered a lot of errors in the event log, which also caused some performance issues. The message was:
"Unable to get the private bytes memory limit for the W3WP process. The ASP.NET cache will be unable to limit its memory use, which may lead to a process restart. Error 0x80070005".

[CAUSE]
The message is caused by a known issue with insufficient permissions in your IIS metabase. The metabase ACL's on the target server did not include the IIS_WPG group on the following two nodes of the metabase (IIS_WPG is in both ACL's on a clean install):
- W3SVC/AppPools
- W3SVC/Filters

[SOLUTION]
Download the MetaACL utility from KB267904. When you have downloaded and installed the program, run the vbs via the following command:

cscript metaacl.vbs IIS://Localhost/W3SVC/AppPools IIS_WPG RE

The path is case sensitive - type exactly as above; after you run this command restart the IIS services and see if this corrects the problem.

Source

[MOSS/WSSv3] Large log files with default logging options

[ISSUE]
When you configured the SharePoint logging as default, it is very much possible that the logs are filled with "Preserving template record with id....." messages. You would expect these messages only to be logged when logging is set to Verbose mode.

[SOLUTION]
The question how to solve this issue has been asked a lot on the Internet, for example on the MS Forums. Unfortunately nobody had a real answer. But since a couple of months, Microsoft fixed this issue. The April Cumulative Update now contains a fix for this issue:

When you set the least critical event to report in the Event log to ERROR, and you set the least critical event to report to the trace log to MEDIUM, the following messages are logged in the Unified Logging Service (ULS) logs:
Preserving template record with size…
Deleting template record with size…
However, you only expect these ULS messages to appear if the logging level for General is set to Verbose.

Just install this update and you are good to go!!

[MOSS/WSSv3] Bug in STSADM MergeContentDBS command

With Service Pack 1, Microsoft introduced the STSAdm command "Mergecontentdbs". With this command you can move site collections between databases. Unfortunately I recently ran into a bug in this command:

When your site collection contains multi valued columns, it is possible that the data in these columns will be gone after migration.

This issue is confirmed by Microsoft and according to them fixed in the April Cumulative Update. To prevent this issue from occuring, install Service Pack 2 and the April Cumulative Update on your environment.

[MOSS2007] Unexplainable errors on the server desktop

[ISSUE]
A while ago we received some unexplainable errors on the servers desktop. A popup window would appear with the Title "Error" and three buttons "Abort, Retry and Ignore". That was all the info we got. When the popup was shown, IIS stopped responding all together until one of the buttons was clicked.

We had to call in the assistance of Microsoft and after numerous troubleshooting sessions we tracked down the issue to the Search component of SharePoint.

[CAUSE]
A user has uploaded a picture of himself to a picture library. He wanted to use that picture as his profile picture, so he copied the URL of the page (not of the picture) and managed to somehow paste this into the ProfilePicture URL field.

Unfortunately the picture library had spaces in the URL and when opening a picture, SharePoint always places the location where the user came from in the URL. Because the library contains spaces, were these URL double encoded:
Space: %20
Percentage sign: %25
Double encoded space: %2520




This caused the user to paste a double encoded URL into his ProfilePicture field. As it turns out, SharePoint throws an assertion error when it is requested to return this value, for example when searching for that specific person.

[RESOLUTION]
After discovering the issue, the Microsoft engineers checked internally and found out that coincidentally the issue was fixed in MOSS Service Pack 2. They redesigned the assertion handling in this service pack, fixing the issue.

Before we implemented Service Pack 2 on our environment, we changed the value is the users profile to a correct value and ran another crawl. Fortunately we never saw the issue ever again.

[WSSv3\MOSS] Shortcoming in stsadm MergeContentDbs - does not support multiple SQL instances

In our environment we have multiple SQL instances that host the content databases. Just tried to copy a site collection from one content database on one SQL instance to another content database on a different SQL instance. Unfortunately I got the following message:

"The databases need to be on the same database server in order to combine them"

It looks like the command does not support multiple SQL instances, which would be a real shortcoming.

[MOSS2007] Import connections only shows a few domain controllers

[ISSUE]
At my current project we experienced the the issue that only one domain controller was listed when trying to configure SharePoint to use a specific domain controller (Shared Services /ssp/admin/_layouts/EditDSServer.aspx?dn=<domain name>).

[EXTRA INFORMATION]
The specific domain has about 75+ domain controllers world wide, so the fact that SharePoint lists only one is something strange. Because the listed domain controller isn't the closed one, SharePoint will always generate WAN traffic and imports will be slower.

[CAUSE]
After a long period of troubleshooting we discovered that the Active Directory guys had deleted all _ldap and _kerberos DNS entries (except for the one domain controller) in _tcp.dc._msdcs.. They have done this to make sure that computers that log on from an unmanaged site (which IP address is not configured in the AD subnets) always use the central AD server instead of randomly choose one and generating unnecessary WAN traffic. As soon as we added some extra _ldap DNS keys, these popped up in the list.

[SOLUTION]
We added the _ldap and _kerberos DNS entries for all domain controller in our 2nd datacenter. We are now able to select the closest domain controller.

[MOSS2007] Output caching error for _layouts/images path (Event 5785)

[ISSUE]
On our SharePoint environment we receive the following message very often in the Application event log:

Event Type: Error
Event Source: Office SharePoint Server
Event Category: Publishing Cache
Event ID: 5785
Date: [date]
Time: [time]
User: N/A
Computer: [server name]
Description:Unable to connect publishing custom string handler for output caching. IIS Instance Id is '[IIS web ID]', Url is 'http://[domain name]/_layouts/images/[image name].gif'.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

[CAUSE]
The reason why this is logged is that SharePoint tries to cache the file mentioned. Because the _layouts/images folder is not considered a SharePoint managed path it is not able to do so.

[SOLUTION]
To fix this issue:
- Open the web config for the mentioned web application.
- Search for the <location path="''_layouts/images"> section
- Add the following text to the section (just before </system.web>)
<httpmodules>
<remove name="PublishingHttpModule">
</httpmodules>

Source: MS Forums article, last post

[MOSS2007] Error while opening Site Columns page (/_layouts/mngfields.aspx)

[ISSUE]
When you open the Site Columns page (/_layouts/mngfields.aspx, Site Settings > Site Columns) SharePoint throws an "Unknown Error" at you.

When you change the CustomErrors=Off and CallStack=true values in the web config, SharePoint shows you the following error:
Object reference not set to an instance of an object.
at Microsoft.SharePoint.ApplicationPages.FieldListRenderer.Render(HtmlTextWriter output)
at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
at System.Web.UI.Control.RenderControl(HtmlTextWriter writer)
at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)

[CAUSE]
This issue is caused by the fact that a feature creates site columns using the incorrect casing in the url. For example:
http://<site url>/sites/Test instead of http://<site url>/sites/test
http://<site url>/Sites/test instead of http://<site url>/sites/test
source

[SOLUTION]
To determine which site columns are causing this issue:

• Open a sub site and open Site Columns of that site
• To the left of each column, the location where they have been created is shown
• Hover over links pointing to the parent site and check URL to which the link is pointing
• One or more URLs will have incorrect casing.

To solve this issue, you either have to change your feature so that it uses the internal URL which SharePoint is using. In my case, the feature that caused this issue was the "Office SharePoint Server 2007 Enterprise Site Collection Feature". In other words, some code I could not change.

I found an article on the Internet where they mentioned changing the site url in the database. I know database changes are not supported by Microsoft, but there was no other way to change this.

• Open the Content Database
• Open the table ContentTypes
• Check the column Scopes, you will find some URLs with the correct casing and some with incorrect casing. Change the casing of the incorrect records.

[MOSS2007] Profile synchronization does not work for databases which are off-line/stopped

Last week I got some complaints about user profile information not being synchronized to other sites in a SharePoint 2007 environment. After some investigation I found the following error in the ULS log:
<date> <time> OWSTIMER.EXE (0x0864) 0x08A4 SharePoint Portal Server User Profiles 8zzy Medium Aborting sweepsynch for guid instance <GUID> due to null or non-online content database

[Solution]
After configuring the database as Ready again, the profile information synchronization resumed again.

[Background information]
If you put you content database off-line, SharePoint is still able to access it but it will not create new site collections. Because were creating new site collections, which we wanted created in a specific database we put all databases (except for that specific one) on off-line. As it turned out, the downside of this is that SharePoint is unable to synchronized the user profile information to databases that are off-line.

[MOSS2007/WSSv3] Messages about missing resource files in SharePoint ULS logs

[ISSUE]
The following messages appear a lot in the ULS logs of SharePoint:
11/24/2008 10:06:58.66 w3wp.exe (0x07F4) 0x1550 Windows SharePoint Services General 72kg High #20015: Cannot open "Proj.en-US.resx": no such file or folder.
11/24/2008 10:06:58.66 w3wp.exe (0x07F4) 0x1550 Windows SharePoint Services General 72kg High (#2: Cannot open "Proj.en-US.resx": no such file or folder.

[SOLUTION]
This is caused by the fact that the resource file is not present with a specific English tag in its name. To solve this issue do the following

• Open the directory C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\Resources
• The mentioned files are not in there, but the proj.resx is
• Open a command prompt and browse to the same directory
• Run the following commands:
  copy avm.resx avm.en-US.resx
  copy compproc.resx compproc.en-US.resx
  copy ep.resx ep.en-US.resx
  copy er.resx er.en-US.resx
  copy hlpdsk.resx hlpdsk.en-US.resx
  copy itteam.resx itteam.en-US.resx
  copy proj.resx proj.en-US.resx
  copy pws.resx pws.en-US.resx
  copy rer.resx rer.en-US.resx
  copy tsa.resx tsa.en-US.resx
• This will copy the default resource files to the English version resource files and the messages will disappear.