When Spotlight leaves you in the dark.

By Seth On January 31, 2010 · In Tech Journal

Spot­light, in the right cir­cum­stance, can be a life-saver. But what hap­pens when Spot­light goes bad. When it's run­ning as a ser­vice on Mac OS X Server, it's not only a dan­ger to the net­work share points, (and your abil­ity to find, well, any­thing) it's also a dan­ger to the server itself. The dae­mon that makes Spot­light work is called mdworker, and it cre­ates a hid­den folder at the root of every vol­ume attached to the sys­tem to store the info about the data on that par­tic­u­lar vol­ume. The "index" is built by watch­ing the fsev­ents API. mdworker asks fsev­ents whats new, and then ads the changes to the index for that par­tic­u­lar disk, (this over sim­pli­fied, but for this post it gets the point across) .  But, like most things com­prised entirely of ones and zeros, these data stores are sus­cep­ti­ble to corruption.

With the stage set, lets jour­ney into the dark under­belly of Spot­light and mdworker.

So a client calls me and says, "spot­light on the net­work shares is no longer work­ing." And I replied, "well, you are run­ning server 10.5. I am sur­prised it ever worked." For those of you who may not know, Spot­light on 10.5 server could be a dis­as­ter in wait­ing. Not only did it not work half of the time, it's inabil­ity to index cor­rectly, could cause the mdworker dae­mon to hog up the CPU, and in some cases crash the server. So need­less to say I was not sur­prised to get the call.

Then 10.6 is released, and the spot­light débâ­cle has been addressed, and all is well in the world of mdworker and fsev­ents. So the same client calls and says, "I need Spot­light, lets upgrade the server." So we upgrade, and migrate the server…  and spot­light is still dead.

After some very blue lan­guage on my part, I sat down at the key­board and started hunt­ing. The easy way to fix a mis­be­hav­ing Spot­light index (most of the time) is a straight for­ward process. Open the Spot­light pane of Sys­tem Pref­er­ences, and set every Disk on the sys­tem to Pri­vate (you use the plus but­ton, then select the disk in the drop down dia­logue). After a restart, undo the Pri­vacy on the disks and Spot­light should re-index. But on server it a dif­fer­ent ket­tle of fish. First open the Terminal.app and run the fal­low­ing com­mand to dis­able mdworker:  sudo mdu­til –Ei off  / Enter the pass­word, and mdworker will stop indexing.

Now the fun part. Move to the root of the startup disk by typ­ing:  cd / and hit Return. Now we are going to remove the index file, by typ­ing:   sudo rm –R .Spotlight.V100 and hit Return, enter then pass­word. Type:   ls –al and hit return to con­firm that the index is gone. Now nav­i­gate to the root of every Disk attached to the sysytem and repeat. (cd /Volumes/diskname)

On a client sys­tem, after rm-ing the index files on the disks you would restart mdworker and this would be the end of it. But on OS X Server it's only the first half. Now we need to move to the root folder of every Share Point and remove the .Spotlight.V100 file (use the cd com­mand to nav­i­gate to the folders).

Why you ask. Why does Spot­light put an index file in the Shared fold­ers. Well it's for net­work searches, the indexes announce them­selves to mdworker on your client sys­tem when you con­nect to the server, allow­ing you search the Share Points as if they are local volumes.

So now that we have rm-ed every last one of the pesky Spotlight.V100 files, we need to restart mdworker. At the prompt type:   sudo mdu­til –i on  / and mdworker should rein­dex every thing in the /Volumes folder, well as the Share Points.

And that should be it, it's happy search­ing from then on. Hope this helps if you are have a Spot­light issue, or maybe just helped you under­stand how Spot­light works, in the deep dark pit that is the UNIX under­belly of Mac OS X.

Thanks for spend­ing some time with us and com​mand​-prompt​.com.

–Seth