When Spotlight leaves you in the dark.
Spotlight, in the right circumstance, can be a life-saver. But what happens when Spotlight goes bad. When it's running as a service on Mac OS X Server, it's not only a danger to the network share points, (and your ability to find, well, anything) it's also a danger to the server itself. The daemon that makes Spotlight work is called mdworker, and it creates a hidden folder at the root of every volume attached to the system to store the info about the data on that particular volume. The "index" is built by watching the fsevents API. mdworker asks fsevents whats new, and then ads the changes to the index for that particular disk, (this over simplified, but for this post it gets the point across) . But, like most things comprised entirely of ones and zeros, these data stores are susceptible to corruption.
With the stage set, lets journey into the dark underbelly of Spotlight and mdworker.
So a client calls me and says, "spotlight on the network shares is no longer working." And I replied, "well, you are running server 10.5. I am surprised it ever worked." For those of you who may not know, Spotlight on 10.5 server could be a disaster in waiting. Not only did it not work half of the time, it's inability to index correctly, could cause the mdworker daemon to hog up the CPU, and in some cases crash the server. So needless to say I was not surprised to get the call.
Then 10.6 is released, and the spotlight débâcle has been addressed, and all is well in the world of mdworker and fsevents. So the same client calls and says, "I need Spotlight, lets upgrade the server." So we upgrade, and migrate the server… and spotlight is still dead.
After some very blue language on my part, I sat down at the keyboard and started hunting. The easy way to fix a misbehaving Spotlight index (most of the time) is a straight forward process. Open the Spotlight pane of System Preferences, and set every Disk on the system to Private (you use the plus button, then select the disk in the drop down dialogue). After a restart, undo the Privacy on the disks and Spotlight should re-index. But on server it a different kettle of fish. First open the Terminal.app and run the fallowing command to disable mdworker: sudo mdutil –Ei off / Enter the password, and mdworker will stop indexing.
Now the fun part. Move to the root of the startup disk by typing: cd / and hit Return. Now we are going to remove the index file, by typing: sudo rm –R .Spotlight.V100 and hit Return, enter then password. Type: ls –al and hit return to confirm that the index is gone. Now navigate to the root of every Disk attached to the sysytem and repeat. (cd /Volumes/diskname)
On a client system, after rm-ing the index files on the disks you would restart mdworker and this would be the end of it. But on OS X Server it's only the first half. Now we need to move to the root folder of every Share Point and remove the .Spotlight.V100 file (use the cd command to navigate to the folders).
Why you ask. Why does Spotlight put an index file in the Shared folders. Well it's for network searches, the indexes announce themselves to mdworker on your client system when you connect to the server, allowing you search the Share Points as if they are local volumes.
So now that we have rm-ed every last one of the pesky Spotlight.V100 files, we need to restart mdworker. At the prompt type: sudo mdutil –i on / and mdworker should reindex every thing in the /Volumes folder, well as the Share Points.
And that should be it, it's happy searching from then on. Hope this helps if you are have a Spotlight issue, or maybe just helped you understand how Spotlight works, in the deep dark pit that is the UNIX underbelly of Mac OS X.
Thanks for spending some time with us and command-prompt.com.