[MOSS2007] PowerShell Library - Start Incremental crawl

[DESCRIPTION]
SharePoint 2007 does not support crawl schedules larger than 24 hours. So if you want to schedule a crawl to run every 36 hours, by default there is no way to do this.

At a customer we had a very large environment, which had a huge amount of changes on a daily bases. A crawl during the weekend took just a few minutes, but during the week this increased to 32 hours. Because Microsoft also recommends not to start a crawl when the previous crawl is still running (even though it cannot run two instances of the same crawl, SharePoint does something that can break the index before it detects the crawl is already running), I have created a script that does this for me.

The script checks if a crawl is running and if so quits. If a crawl is not running, it starts another incremental crawl. By scheduling this script every 15 minutes, I can make sure the crawl is never stopped for more than 15 minutes.

How to use the script:

1. Download the PowerShell script
2. (If necessary) Change the Content Source name
3. Run or schedule the script

[MOSS2007] Crawling schedule alternative

[ISSUE]
On a customer environment, we were having some issues with the default search scheduler. On this environment we currently have more than 4.8 million items in the index and migrations are still happening. This means that incremental crawls run for several hours and sometimes even more than 24 hours.

By default it is only possible to schedule it once a day maximum (so not once every two days) and according to a Microsoft engineer, it is not advised to run a crawl when a previous crawl is still running. Somehow that can result in a corrupt SSP. Configuring a schedule to run each 15 minutes is therefore not an option.

[SOLUTION]
To solve this issue, I have created a PowerShell script. This script checks if a crawl is running and if not, starts a new incremental crawl. I have scheduled this script to run every 15 minutes.

[System.Reflection.Assembly]::Load("Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c") | out-null
[System.Reflection.Assembly]::Load("Microsoft.Office.Server, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c") | out-null
[System.Reflection.Assembly]::Load("Microsoft.Office.Server.Search, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c") | out-null

$serverContext = [Microsoft.Office.Server.ServerContext]::Default
$context = [Microsoft.Office.Server.Search.Administration.SearchContext]::GetContext($serverContext)

$sspcontent = new-object Microsoft.Office.Server.Search.Administration.Content($context)
$sspContentSources = $sspcontent.ContentSources

foreach ($cs in $sspContentSources)
{
  if ($cs.Name -eq "Local Office SharePoint Server sites")
  {
    Write-Host "NAME: ", $cs.Name, " - ", $cs.CrawlStatus
    if ($cs.CrawlStatus -eq [Microsoft.Office.Server.Search.Administration.CrawlStatus]::Idle)
    {
      Write-Host "Starting Incremental crawl"
      $cs.StartIncrementalCrawl();
    }
    else
    {
        Write-Host "Crawl running"
    }
  }
}

[MOSS2007] Unexplainable errors on the server desktop

[ISSUE]
A while ago we received some unexplainable errors on the servers desktop. A popup window would appear with the Title "Error" and three buttons "Abort, Retry and Ignore". That was all the info we got. When the popup was shown, IIS stopped responding all together until one of the buttons was clicked.

We had to call in the assistance of Microsoft and after numerous troubleshooting sessions we tracked down the issue to the Search component of SharePoint.

[CAUSE]
A user has uploaded a picture of himself to a picture library. He wanted to use that picture as his profile picture, so he copied the URL of the page (not of the picture) and managed to somehow paste this into the ProfilePicture URL field.

Unfortunately the picture library had spaces in the URL and when opening a picture, SharePoint always places the location where the user came from in the URL. Because the library contains spaces, were these URL double encoded:
Space: %20
Percentage sign: %25
Double encoded space: %2520




This caused the user to paste a double encoded URL into his ProfilePicture field. As it turns out, SharePoint throws an assertion error when it is requested to return this value, for example when searching for that specific person.

[RESOLUTION]
After discovering the issue, the Microsoft engineers checked internally and found out that coincidentally the issue was fixed in MOSS Service Pack 2. They redesigned the assertion handling in this service pack, fixing the issue.

Before we implemented Service Pack 2 on our environment, we changed the value is the users profile to a correct value and ran another crawl. Fortunately we never saw the issue ever again.

[MOSS2007] SharePoint removes entries from hosts file with multiple entries on one line

[SITUATION]
The SharePoint environment consists of three servers, a web front end, an index/central admin and a database server. According to Microsoft Best Practices, we have activated the Web Application role on the index server as well and configured the index server to use the local server. SharePoint does this by modifying the HOSTS file.

[ISSUE]
Last week a colleague noticed that some entries in the hosts file were periodically removed. After some investigation, he found out that this only happens with entries where multiple hostnames are linked to one IP address, for example:

• 127.0.0.1<tab>server.domain.intra<tab>server

All entries that had only one server name in each line, remained in the hosts file.

[SOLUTION]
To get around this issue we changed the following line:

• 127.0.0.1<tab>server.domain.intra<tab>server

into:

• 127.0.0.1<tab>server.domain.intra
• 127.0.0.1<tab>server

After this change, SharePoint left the entries alone.

[MOSS2007] Search issues due to corrupt SSL certificate

After installing MOSS2007 which has Service Pack 1, the Infrastructure Update and the August Cumulative Update slipstreamed I ran into the following issue:

My environment consists of one Index server and two Search servers. When opening the new Search Administration page, it is supposed to list the status of all of the index and search servers. Unfortunately the Search servers were listed as Error. After doing some more investigation I found the following error appear in the application log of both search servers each minute:

Event: 6482
Type: Error
Source: Office SharePoint Server
Category:Office Server Shared Services

Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (aa8a83f6-ff42-4be3-ae6c-c1cb2205cc8f).

Reason: The underlying connection was closed: An unexpected error occurred on a send.

Techinal Support Details:
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)
at Microsoft.Office.Server.Search.Administration.SearchApi..ctor(WellKnownSearchCatalogs catalog, SearchSharedApplication application)
at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

When I tried to open the SearchAdmin.asmx page (http://<index or search server name>/SearchAdmin.asmx), I get a page cannot be displayed.

This issue is caused by a corrupt SSL certificate on the "Office Web Services" web site. To solve this issue, you have to create a new certificate. You can use the SelfSSL tool, included in the IIS resource kit.

Run the following command on all search and index servers:
selfssl.exe /N:CN=<server> /K:1024 /V:9999 /S:951338967 /P:56738

[MOSS2007] Crawl issue after installation .NET Framework 3.5 SP1

[ISSUE]
When the .NET Framework 3.5 SP1 has been installed, it is possible that you get some Access Denied error messages when crawling your MOSS sites. If you're indexer is set to crawl itself, via the Central Admin setting, or manual hosts modification, you might be running into this issue. You will see a combination of errors in your event logs, ULS logs and crawl logs that match the following:

Crawl Log:Event Type: Warning
Event Source: Office Server Search
Event Category: Gatherer
Event ID: 2436
Date: 10/22/2008
Time: 10:02:55 AM
User: N/A
Computer: MOSS2007
Description:The start address <http://portal/> cannot be crawled.

Context: Application 'SharedServices1', Catalog 'Portal_Content'
Details:Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content. (0x80041205)
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

ULS Logs: (verbose)10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common MS Search Indexing 7hoo Verbose GetVirtualServerPolicy is about to call WS for virtual server, strVirtualServerUrl http://portal/10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable ***** Couldn't retrieve server http://portal/ policy, hr = 80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:548
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable ****** Server portal security initialization failed, hr = 80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:317
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::InitURLType: Return error to caller, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3acc.cxx Line:1542
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::InitURLType fails, Url http://portal/, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3acc.cxx Line:184
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Accessor::Init fails, Url http://portal/, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3handler.cxx Line:243
10/22/2008 10:59:04.23 mssdmn.exe (0x1420) 0x151C Search Server Common PHSts 0 Monitorable CSTS3Handler::CreateAccessorExB: Return error to caller, hr=80041205 - File:d:\office\source\search\search\gather\protocols\sts3\sts3handler.cxx Line:261

Security Logs:Logon Failure:
Reason: An error occurred during logon
User Name: mosscrawl
Domain: SPI
Logon Type: 3
Logon Process: Èù8
Authentication Package: NTLM
Workstation Name: MOSS2007
Status code: 0xC000006D
Substatus code: 0x0
Caller User Name: -
Caller Domain: -
Caller Logon ID: -
Caller Process ID: -
Transited Services: -
Source Network Address: XX.XX.XX.XX
Source Port: 1417

[SOLUTION]
Perform the steps mentioned in the KB article below to fix this issue:
http://support.microsoft.com/kb/896861
Source: Ron Grzywacz's Blog

[MOSS2007] PDF and SharePoint 2007

Just like in SharePoint 2003, SharePoint 2007 does not show the pdf icon and is not able to index these files. But fortunately Microsoft made it possible to reconfigure SharePoint yourself and upload an icon and install a PDF iFilter. Adobe created an iFilter to accomplish this.

SharePoint 2003 only came as a 32-bit version, but with SharePoint 2007 SharePoint can be installed as a 64-bit version as well. And here is the catch……Adobe has not released a 64-bit version of its iFilter yet and it looks like it won’t be available for a very long time (come on Adobe!!).

To solve this issue you have several options:

1. Reinstall the server with 32-bit Windows and reinstall SharePoint…….ok, not really an option.
2. Not index PDF files at all…….ok, also not an option.
3. Beg Adobe to create a 64-bit version of their iFilter…….ok, you have something to do for about 5 minutes but it won’t solve your problem. But if you have nothing to do……go ahead: Submit Product Enhancement
4. Try to trick MOSS into accepting the 32-bit version of the iFilter…….uhm, do you really want to?

But fortunately there is a fifth solution!! A company called FoxIT released a alternative PDF iFilter............and they created a 64-bit version as well…..and it is free!! What more do you want?? Great, isn’t it :-)

So go ahead and download the 32-bit version or the 64-bit version

Happy indexing!!

Error when trying to create a crawl schedule for a content source

When you try to configure a crawl schedule for a content source, you will receive the error message:
Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)

This is a known issue and Microsoft has published a solution on their site: KB926959

Search and Index Sizing and Planning - Real world data from MSW

Source: Blog Joel Oleson

"I've heard such a huge variety of guidance around Index sizing. This is a topic that will have a huge swath, so its very important to understand your data or to be conservative. If you have the ability to resize later or add larger disks you may find this data compelling.

In the capacity planning document you'll see we recommend 30% of disk for Index. What does that mean? Well, since now we have a search db, the index edb file, and SSP database it can be confusing. Reading the response from Sam from Microsoft's internal/Intranet deployment you can see how mileage really does vary. If the 12TB deployment which is currently being indexed were to have 30% of disk just for the size of the index on disk, the file would be 3TB! Currently the index file on disk is 83GB, but the search database is 243GB. This is with 19.4 million documents indexed. Since the recommendation is to have 2X the size for the index file for swapping it out on the query server, a planner would say that they should plan for 60% on disk or over 6TB. This would be quite a loss given the current size or "real world" size is 83GB. They currently have 300GB allocated for that drive and since it's on a SAN they can grow the disk if needed. Obviously with a heavy records management repository or a page heavy site will have different results, so be sure to understand your content.

My tip is don't over plan or let this kill your design on this one. The recommendation has gone from 50% to 30% over time, maybe you've even seen 10%. My recommendation is to understand your data. Remember that the content of audio, video, archives, ZIPs, PDFs, MDB, MPP, MSG, VSD, GIF, JPG, PSD, CAD, WAV, MSI, EXE, and hundreds of file types are not indexed by default. You have to add Ifilters for the files that are not indexed by default (which is a pretty decent list (most Office file types and text based formats) and you should be selective about what you add since many of them are not multi threaded. The other consideration is file size. If files larger than 16MB are not indexed by default, then the biggest files taking up the largest size on disk won't be indexed. When you have a 15MB PPT, how much of it is even indexed? Maybe 100K worth (if you have verbose notes)?

Our indexes are larger than they were in SPS 2003, but my recommendation is to be conservative at 10% with the expectation that you'll really see something around 1-5%. Don't underestimate the Search Database though. In the MSW farm, that database sees the most action in terms of writes. It is the most actively written to database in the farm from what I hear. Makes sense understanding that it is the property store. Although I've almost completely gone to RAID 5 in all my planning for minimizing cost, I do recommend a RAID 0+1 drive for the search and config database, and RAID 0+1 for the transaction logs.

Message from Sam...

The real-world data today is that we are indexing 12TB of Sharepoint content worldwide + an unknown amount of non-Sharepoint content from our Redmond SSP. The numbers look like this:

Number of documents indexed: 19.4 million Size of search database: 243GB Size of index on disk: 83GB

Thus you could consider the amount of disk used to be about 326GB. Assuming a 14TB total corpus say (just a guess, really) then the real-world data would indicate 2.33%. Of course this is very much a 'mileage may vary' exercise as everyone's document mix is different."