[MOSS2007] Search issues due to corrupt SSL certificate

After installing MOSS2007 which has Service Pack 1, the Infrastructure Update and the August Cumulative Update slipstreamed I ran into the following issue:

My environment consists of one Index server and two Search servers. When opening the new Search Administration page, it is supposed to list the status of all of the index and search servers. Unfortunately the Search servers were listed as Error. After doing some more investigation I found the following error appear in the application log of both search servers each minute:

Event: 6482
Type: Error
Source: Office SharePoint Server
Category:Office Server Shared Services

Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (aa8a83f6-ff42-4be3-ae6c-c1cb2205cc8f).

Reason: The underlying connection was closed: An unexpected error occurred on a send.

Techinal Support Details:
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)
at Microsoft.Office.Server.Search.Administration.SearchApi..ctor(WellKnownSearchCatalogs catalog, SearchSharedApplication application)
at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

When I tried to open the SearchAdmin.asmx page (http://<index or search server name>/SearchAdmin.asmx), I get a page cannot be displayed.

This issue is caused by a corrupt SSL certificate on the "Office Web Services" web site. To solve this issue, you have to create a new certificate. You can use the SelfSSL tool, included in the IIS resource kit.

Run the following command on all search and index servers:
selfssl.exe /N:CN=<server> /K:1024 /V:9999 /S:951338967 /P:56738

[MOSS2007] Crawl issue after installation .NET Framework 3.5 SP1

[ISSUE]
When the .NET Framework 3.5 SP1 has been installed, it is possible that you get some Access Denied error messages when crawling your MOSS sites. If you're indexer is set to crawl itself, via the Central Admin setting, or manual hosts modification, you might be running into this issue. You will see a combination of errors in your event logs, ULS logs and crawl logs that match the following:

Crawl Log:Event Type: Warning
Event Source: Office Server Search
Event Category: Gatherer
Event ID: 2436
Date: 10/22/2008
Time: 10:02:55 AM
User: N/A
Computer: MOSS2007
Description:The start address <http://portal/> cannot be crawled.

Context: Application 'SharedServices1', Catalog 'Portal_Content'
Details:Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content. (0x80041205)
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

ULS Logs: (verbose)10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common MS Search Indexing 7hoo Verbose GetVirtualServerPolicy is about to call WS for virtual server, strVirtualServerUrl http://portal/10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable ***** Couldn't retrieve server http://portal/ policy, hr = 80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:548
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable ****** Server portal security initialization failed, hr = 80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:317
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::InitURLType: Return error to caller, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3acc.cxx Line:1542
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::InitURLType fails, Url http://portal/, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3acc.cxx Line:184
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::Init fails, Url http://portal/, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3handler.cxx Line:243
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Handler::CreateAccessorExB: Return error to caller, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3handler.cxx Line:261

Security Logs:Logon Failure:
Reason: An error occurred during logon
User Name: mosscrawl
Domain: SPI
Logon Type: 3
Logon Process: Èù8
Authentication Package: NTLM
Workstation Name: MOSS2007
Status code: 0xC000006D
Substatus code: 0x0
Caller User Name: -
Caller Domain: -
Caller Logon ID: -
Caller Process ID: -
Transited Services: -
Source Network Address: XX.XX.XX.XX
Source Port: 1417

[SOLUTION]
Perform the steps mentioned in the KB article below to fix this issue:
http://support.microsoft.com/kb/896861
Source: Ron Grzywacz's Blog

[MOSS2007/WSSv3] Login box when a normal user logs on, but not with admin

[ISSUE]
A logon box constantly appears when a users is trying to log onto a SharePoint site. Whatever the user enters, he cannot get onto SharePoint.

[CAUSE]
The environment on which we experienced this behaviour was locked down according to security guidelines. One of these guidelines was to strip the "Read & Execute" permissions on the C:\Windows folder from the local users group. The .NET Framework requires Execute permissions on the C:\Windows directory to operate properly and because SharePoint relies on .NET, users were prompted for credentials.

[SOLUTION]
Add the Execute permissions for the Users group on the C:\Windows folder or restore the permissions on the C:\Windows directory to default:
- Open a command prompt.
- On a domain controller, run
secedit /configure /db c:\windows\temp\seceditsv.sdb /cfg "c:\windows\security\templates\DC security.inf" /log c:\windows\temp\seceditsv.log
- On a non-domain controller, run
secedit /configure /db c:\windows\temp\seceditsv.sdb /cfg "c:\windows\security\templates\setup security.inf" /log c:\windows\temp\seceditsv.log

[MOSS2007/WSSv3] Using a clustered SQL named instance behind a firewall in SharePoint

[Issue]
For a customer I have been working on a new installation of SharePoint 2007. The environment has high availability requirements, so all components need to be redundant. The customer environment already contained a clustered SQL environment which we could use for SharePoint. We just needed to install a new SQL named instance, dedicated for the SharePoint 2007 installation.

For extra security, the SQL cluster was placed in a separate database VLAN, which was secured by a firewall. To enable communication between the SharePoint 2007 servers and the SQL named instance we opened UDP port 1434 and TCP port 4000 (port used by the instance).

After installing SharePoint 2007 on one server, I ran the SharePoint Products and Technologies Configuration Wizard to create a new farm/Configuration Database. When I used the <server>\<instance> format, SharePoint reported that it could not connect to the instance. It reported that the instance was not running, the server wasn’t a SQL server or a firewall was blocking the traffic.

[Reason]
After troubleshooting on the SharePoint and SQL servers, I turned to the firewall guys. With their troubleshooting tools we found out that:

• SharePoint connected to virtual SQL cluster name (ip 10.10.10.3) using the UDP protocol over port 1434 to determine the port of the SQL instance (see “Background Information”)


• The firewall allowed the traffic to the virtual SQL cluster name


• The server replied on the physical node ip address (ip 10.10.10.1) back to the server


• Because the server registered an outbound connection on the virtual SQL cluster name, it does not allow a reply on a different ip address and blocked the reply

Due to this behavior the SharePoint Config Wizard could not determine the port used by the instance and therefore could not connect to the instance.

After using Google we ran into the following article:
Unable to connect to a SQL Server named instance on a cluster

This issue is caused by the fact that the SQL Browser is not “cluster-aware”. It listens to all ip addresses of the server and replies always on the ip addresses of the physical node instead of on the ip address on which it received the request. The solution according to the blog post: Use SQL Server 2008 in which it is solved…….no real option at the moment.

[Solution]
To solve this issue, we created a ODBC System DSN which is configured to connect to the correct port directly and preventing the discovery process from happening:

• Start > Settings > Control Panel > Administrative Tools > Data Sources (ODBC)


• Tab System DSN


• Click Add


• Select SQL Server


• Enter the name you would like the DSN to have


• Enter a description


• Enter the name of the server


• Click Next


• Click “Client Configuration”
  Clear the “Dynamically determine port” checkbox and enter the port number of the SQL instance


• Click Next


• The connection is now checked. Click Finish to save the DSN

Now use the name of the DSN as the SQL server name.

[Background information]
When a named instance is used, in most cases that instance is running on a port which is chosen randomly at installation time (customizable afterwards). Any processes that would like to use that specific instance first connect to the SQL Browser (using UDP over port 1434), which supplies the port that the instance is running on. The process then connects to the specific port.

[MOSS2007] Error in search configuration after installing Infrastructure Update

The infrastructure update introduces a Search Administration page on the Shared Services. When creating a slipstream installation, this page is automatically created when creating the Shared Services. But when you install the Infrastructure Update on an existing environment, this page is not automatically created. It does change the breadcrumb trail on the Search pages of the Shared Services site, which cause some error messages. You cannot open the “Search Administration” page (http://<ssp_url>/ssp/admin/searchadministration.aspx).

To solve this, run the following commands:
stsadm.exe -o installfeature -name S2SearchAdmin
stsadm.exe -o activatefeature -name S2SearchAdmin -url http://<ssp_url>/ssp/admin -force

Usefull and Free SharePoint Tools (Part 2)

A while ago I posted a list of tools I ran into and found very usefull. Here is part 2: More usefull tools :-)

SharePoint Inspector
Using the SharePoint Inspector you can browse through the SharePoint object model, can see objects composing its structure, get their properties by reflection. This cool can be very useful when you want to check if your code does what it should do. You can use also some advanced features like activate/deactivate SharePoint features, add/remove event receivers, manage your recycle bin.

Link: SharePoint Inspector

SharePoint Manager
The SharePoint Manager 2007 is a SharePoint object model explorer. It enables you to browse every site on the local farm and view every property. It also enables you to change the properties (at your own risk). It is a very powerfull tool for developers that like to know what the SharePoint holds of secrets.

Link: SharePoint Manager

MS IT Site Delete Capture
Every had the unpleasant question of users if you could restore a site or site collection they accidentally deleted? The only way to do this by default is to perform a restore to a separate environment and use stsadm to extract the site and restore into your environment again. Even though SharePoint 2007 does have a recycle bin, this does not capture site or site collection deletions.

After you installed the Microsoft IT Site Delete Capture Feature 1.0 SharePoint administrators can intercept both site/web delete requests and archive the site/web to a resource local to the web front-end computer or UNC path before the site/web is removed from the configuration and content databases.

The Microsoft IT Site Delete Capture Feature 1.0 also exposes functionality allowing SharePoint administrators to send e-mail messages to the end-user indicating the site has been archived and deleted, any failure in the event receiver will generate an e-mail message to the end-user indicating that the site/web has not been deleted. The message format, text, and language are stored in a flexible, culture-independent extensible markup language configuration file.

Link: MS IT Site Delete Capture

SharePoint 2007 Test Data Population Tool
The SharePoint 2007 Test Data Population Tool is a capacity planning and performance testing tool that populates data for testing SharePoint deployments. The SharePoint 2007 Test Data Population Tool is available as a command-line executable program that extracts information about how to populate the server from an XML configuration file.

Use this tool to populate test data for SharePoint capacity planning. Also included are some sample test scripts that can be used for performance and load testing SharePoint and Excel Services with the test data. These tools are being provided as is and there is no ongoing support for these tools.

Link: SharePoint 2007 Test Data Population Tool

SharePoint Logging Spy
A lot of SharePoint 2007 environments consist of multiple servers. Each server is producing logs and in case of issues you always have to dig through multiple server logs. The SharePoint Logging Spy allows you real time diagnostics of multiple servers in a SharePoint farm using a single console view.

SharePoint Logging Spy is a real time diagnostic application for MOSS 2007 which allows a SharePoint administrator to view (and save to disk) the ULS Log & Event log entries from multiple machines in a SharePoint farm through a single console. NOTE - There is a known issue with Event Log Entries not being displayed from remote machines. SharePoint Logging Spy also allows the administrator to view the historic ULS logs on the farm using the same interface.
SharePoint Logging Spy also has a treeview which allows a developer or administrator to view the state of the current SharePoint enviroment. The Plug-in architechture allows for extensibility of the core application.

The application also provides drill downs for information regarding the thread or process which generated the log entries which are reported in the main screen. The thread and process information is collected on demand (when you click the menu) and is NOT historical therefore it is feasable that the process/thread may no longer exist when you ask for the information and you will get an empty result set.
To easy configuration there is a discovery wizard which will discover on-line servers in the farm and allow selective monitoring to begin.

Link: SharePoint Logging Spy

SharePoint SUSHI
SUSHI is a powerful, user-friendly SharePoint application enabling you to accomplish common SharePoint administrative and development tasks. You can think of SUSHI as a Swiss army knife for SharePoint.

SUSHI = SharePoint Utility with a Smart, Helpful Interface

SUSHI Tools
Administration
Security Reports: View all sites and lists a user has access to (VERY USEFULL!!!)
Profile Images Import: Upload Profile Images and Apply them to User Profiles.
Backup: Backup a site. Restore: Restore site from backup.
Email Test: Send a test email to make sure outbound email is set up correctly.

Lists
Copy View: Copy a view from one SharePoint list to another. Meta Data: Get detailed information about metadata in a list. Move metadata from one column to another for all items in a list. And more.
Bulk List: Creation Create a large number of subsites based on a site template.
Bulk Site Columns: Create Site Columns. Import Documents: Upload entire file structures from a file share into a document library.

Sites
Bulk Site Creation: Create a large number of subsites based on a site template.
Themes: Apply a Theme to many sites at once.

Link: SharePoint SUSHI

[MOSS2007/WSSv3] Technical Library content available as downloadable CHM file

Just recently Microsoft released the entire Technical Library for Windows SharePoint Services v3 and Office SharePoint Server 2007 in Compiled Help format (CHM) to be downloaded from their site:

Available here:
- Windows SharePoint Services 3.0 Technical Library in Compiled Help format
- Office SharePoint Server 2007 Technical Library in Compiled Help format

[MOSS2007] Error during editing navigation items

[ISSUE]
When trying to edit the navigation items in a site, I get the following error message:
"The page has been modified by another author on <time>"
.....even when I am 100% nobody is changing the navigation of this site.

The reason why I was editing the navigation was because some items were having some strange URL's:
/_layouts/viewlsts.aspx?BaseType=0?BaseType=0?BaseType=0?BaseType=0

[CAUSE]
After browsing on the Internet I found that this had been caused because I used the Content and Structure Manager to move a site. SharePoint has an issue, which is causing the behaviour above.

[SOLUTION]
To solve this issue, do the following:

• Remove all of the Document Libraries and lists from the navigation by:
• Open the list/library you want to remove from the quick launch
• Click Settings > List Settings (or Library Settings)
• Click on Title, description and navigation
• Choose No for Display this list on the Quick Launch?
• Remove all corrupted navigation items
• Readd the Lists/Libraries

[MORE INFO]
Technet Forum
Windows Tech
Blog Paul Gavin

[MOSS2007/WSSv3] SharePoint 2007 and Console Applications [UPDATED]

During a project, the developers created a console application which they wanted to schedule in Task Scheduler. This turned out to be a challenge because using console applications with SharePoint 2007 requires permissions on three locations. The service account will need to have the following permissions:

1.) Permissions on the SharePoint databases

• Open SQL Management Studio
• Go to Security > Logins
• Right click Logins and select New Login
• Enter account name and open User Mappings tab
• Select the master database, permissions public
• Select the config database (default SharePoint_Config), permissions public and WSS_Content_Application_Pools
• Select the content database (default format WSS_Content_), permissions public and db_owner

2.) Permissions on the SharePoint server

• Add the service account to the local administrator group (See update below)

3.) Permissions on the SharePoint site

• Grant the service account the required permissions in the SharePoint site
• For example: add to Site Owners group

[UPDATE] The service account doesn't need to be local admin on the server. You can use the following permissions as well:

• "Logon as batch job" in Local Security Policy
• Member of the local WSS_ADMIN_WPG group
• Grant the local group BATCH read permissions to C:\Windows\System32\cmd.exe to make sure the scheduled task can run batch scripts.

[MOSS2007] Error "Cannot complete this action. Please try again."

Over the past month I have been working on a major incident on our production SharePoint 2007 environment.

ISSUE:

When opening a sub site in a site collection by entering it url without default.aspx file, SharePoint would show an HTTP 500 error (Internal Server error). When you try to open the default.aspx file, SharePoint would display the error "Cannot complete this action. Please try again.". The symptoms were similar to:
http://support.microsoft.com/default.aspx?scid=kb;EN-US;937038


CAUSE:
About a month ago a site administrator of a sub site noticed that the NT AUTHORITY\AUTHENTICATED USERS group was listed in the site members group. In other words, everybody who was authenticated in the domain or trusted domains had Contribute permissions. As you can imagine this was not supposed to happen, so he tried to delete the AUTHENTICATED USERS group from the members group. What the user exactly did and what happened is not really clear, but the the result was that the sub site was no longer reachable.

STEPS TAKEN:
After searching for a solution for about a day, we could not figure it out and called in the support of Microsoft. Unfortunately they also weren't very sure how to solve it, but did mentioned the possibility of installing SP1 which might solve the issue. On the other hand, the also feared that installing SP1 would fail because of the broken site. To be sure we prefered to test the installation of SP1 first. In the meantime, the users couldn't access their data and we had to fix that first.

The suggestion of Microsoft was to use stsadm -o export/import. We would restore a known good copy of the database to a restore environment, export the sub site tree, copy it to the production environment and import on top of the broken sub site. Unfortunately this did not work, we received the same error. The alternative was to restore to a new sub site. The import started fine, but after about seven hours stsadm crashed! Also a second import crashed......now what.

We had bought MetaLogix Migration Manager to migrate content from file shares, ASP/HTML websites and other SharePoint environments into our new central MOSS2007 environment. So we gave that tool a go. Fortunately that worked. After several hours, the data was available again for the users.

FIX THE PRODUCTION ENVIRONMENT:
I wanted to test the installation of SP1 on a copy of that site collection. I tried to use stsadm -o backup/restore to copy the production site collection and to a restore environment. The backup went fine, but restore failed after 8 to 10 hours :-(

I then tried to create a backup of the entire database, restore that to the restore database server and attach to an empty web application. Fortunately that worked and I had an environment to test SP1 on. The installation of SP1 completed successfully and we now know that we can install SP1 without any issues on the environment.

KEY INFO:
The entire site collection was about 48GB of size. The sub site tree was about 25GB. Moveing the data using the default stsadm tools, did not work. It looks like stsadm is having some issues with large sites.

According to Microsoft Product Support, SP1 contains stored procedures which check and fix security issues. They weren't sure if those stored procedures would fix our issue. Obviously it did :-)