Most of the hacks I post here are about getting things done. Today I’m going to write about avoiding obstacles that might stop you from getting things done. Namely, I’m going to case-study debugging a sluggish OSX machine.
THE SCENARIO
My boss approached me about a weird slowdown he was having where his machine was slowing to a crawl. Trying to move the cursor was taking a lifetime to move a couple of inches. There wasn’t a beachball cursor indicating an application was sucking up resources and I was able to launch programs, just very S L O W L Y.
So, what did we do?
I’ve done this stuff for a while and you start to get a sense of what seems like the usual suspects when you’re debugging an OSX problem. I’m going to run through the process I use and tell you about what we found out.
THE PROCESS
Step One.
We back up a lot of data at Primal Screen. We do both SD and HD video and have about 10 Terabytes of online storage and 5 Terabytes for near-line archive storage, combine that with individual client machine backups, and we, unfortunately, end up backing up during business hours. So when I hear about a mid-day slowdown, the first thing my “spidey-sense” goes off about is back-up. We use Retrospect from a central server so I did a quick check to see if it was running the client backup on Doug’s machine. Nope.
On to step two.
I also knew that Doug had recently installed Tiger(OSX 10.4) on his laptop. We are relatively late adopters on software for production machines and we’ve had enough problems with Tiger that it still isn’t universally adopted. One of the “features” we struggle with is Spotlight. Spotlight is a great idea. It indexes your data and allows for quick context sensitive searches, but it’s definitely a version 1.0 technology. The interface is a bit wonky and you can sometimes run into the next problem I suspected, Spotlight indexing. Spotlight needs to “index” your computer to gather all of the information that it uses. Indexing means that spotlight goes through the folders on drives that are searched by Spotlight and gathers the metadata from your files so that you can search it. While this is happening your machine can slow down and become unresponsive. So that’s the next place we look.
If you open Applications/Utilities/Activity Monitor (or for the geeky who like the command line, via Applications/Utilities/terminal top -o cpu), the process mds is active when Spotlight is either indexing or searching. The mdimport process is also active during indexing. If we were having Spotlight problems, those processes would be active and using excess CPU cycles. Nope, in this case.
If this turns out to be your problem, a great resource is available at thexlab.com on various ways of stopping and altering Spotlight indexing.
This is also the step where we would look for other programs taking too many CPU cycles. Sometimes you’ll have a spinning beachball cursor and not be sure of the source. Here’s where you’d check that.
On to step three.
THE SOLUTION
We now take a stop at Applications/Utilities/Console. The console is where applications send data about their current state. You’ll find errors, status and crash information here. The GUI hides a lot of interesting information from you. Some of it is a bit geeky and extraneous but you can pick up a lot of info and understanding about your computer by checking out the console.
So, we open it and I notice these entries….
SoundGrinderCMPlugInFactory-I-Debug AllocCMPlugInType-I-Debug SoundGrinderCMPlugInQueryInterface-I-Debug SoundGrinderCMPlugInAddRef-I-Debug SoundGrinderCMPlugInRelease-I-Debug SoundGrinderCMPlugInExamineContext-I-Debug SoundGrinderCMPlugInPostMenuCleanup-I-Debug
This looks promising…
hmmm, a CMPlugin is throwing debug info. A CM Plugin is a Contextual Menu plugin. Contextual Menus appear when you Control-click or Right-click (yeah, even macs work with two button mice these days) in a “context” that allows their use. Ok, but what is Sound Grinder?
Ahh, yes. Sound Grinder is an audio conversion utility we were checking out to convert some audio clips. Doug had removed the program when he decided to use a different app. But why was I still getting debug info? Sure enough, in /Library/Contextual Menu Items/ a SoundGrinderCMPlugIn.plugin file.
So, we’re definitely suspicious of this file. Let’s remove it. Another quick search for “uninstalling Sound Grinder” revealed:
To uninstall Sound Grinder you can simply drag the Sound Grinder folder to the trash, and trash the plugin located at:/Library/Contextual Menu Items/SoundGrinderCMPlugIn.plugin. If you wish to delete the preferences, look in the preference folder for a file called "com.monkeytools.soundgrinder.plist", and drag to the trash.
We removed the plugin and that cleared up the problem. BTW, Sound Grinder seems like a very nice program and I’m not sure exactly why I was having the slowdown problems that we found related to the CMPlugin, although it probably didn’t help that we had yanked most of the rest of the program from the expected location.
I do think a nice universal OSX uninstall utility for programs that install things outside of the application bundle would be a good thing. I’ve got bits and pieces of no longer used detritus all over the place.
REFERENCE
So, in this case we caught our problem in the triage stage. Sometimes, you don’t find your problem here though. Where to next? I’ve had great success with Jaguar, Panther and Tiger Cache Cleaner from Northern Softworks. It runs a bunch of standard maintenance and utility scripts that tend to ferret out corrupt caches and permissions problems. For disk problems, I recommend DiskWarrior and avoid Norton
(mainly for it’s insistence in leaving bits and pieces of itself everywhere, go figure). I’ve also been greatly assisted by John Gruber and his Daring Fireball site. His rundown of the Software Update process is a great piece of info and his font cache problem exposé is at least vaguely the inspiration for this post.
Hope you’ve find this helpful, and here’s hoping your avoiding these gotcha problems in your workflow.
Mike Curtis said
I’d read that Retrospect went to hell after the OS X version came out – do you have an opinion on that? I used to use it with AIT tapes and had various issues and difficulties with it.
I’d heard that BRU was the way to go now with LTO-3 tapes. Any opinion on that?
-mike
Dale Bradshaw said
Hi Mike,
Primal Screen has a heavy investment in Retrospect, liiterally hundreds of terabytes of data, so my experience might be colored somewhat by how far we are down the road with the app. In general, we get it to work nicely in our workflow both for backup of servers and remote clients and archiving of jobs. The OSX transition was a tough one, but there is currently a robust code base that is pretty stable, at least in this install.
Would I recommend Retrospect for somebody starting fresh? Yes, with some caveats.
Retropsect is probably not ideal for small shops with a flat project structure. The interface can be confusing and you really need to spend a lot of time with it to be fluent in the operation of the workflow. Primal has a person who practically lives in the app and he’s pretty amazing about fast fluid operation.
emc2/Insignia formerly Dantz has an “enterprisey” approach to software. They have a lot of jargon and even do things like calling “small and medium businesses” SMB’s. They also own some key intellectual property in the backup space and have hampered innovation in backup. In other words, they aren’t my favorite actors in the business.
That said, I can only recommend what I know and I know both the strengths and weaknesses of Retrospect and I chose to stay with the product.
BTW, LT0-3 tapes rock.
Best,
Dale
Dale said
hmm, it DOES look like we’ll be taking a look at alternatives to Retropsect. Here’s my follow up post about this
captain said
I would recommend becoming proficient in rsync and crontab. You can fully configure EXACTLY HOW you want things backed up, and WHEN you want them to do it. If you don’t have complex backup needs CCC (Carbon Copy Cloner) will give you a pretty GUI interface to rsync (albeit, their own hacked version of an old rsync executable, which apparently allows non-standard, and undocumented, flags… that worries me)
There is no substitute for unix administration via the terminal! 😉