So apparently the visualization stuff in Banshee has been disabled since it's a CPU hog. I don't know why I didn't notice it during testing (and I did check) but there seems to be a five-fold increase in CPU utilization with the visualization pipeline running. For me, this is an increase from 1-3% to 10-12% on dual-core 2.6GHz amd64.
After ruling out a few things I discovered the root cause. In OpenVP, PCM and spectrum data are represented as 32-bit floats, which means that the pipeline needs to convert whatever the audio format is into this one. Along the way it resamples the stream too, to provide a consistent frame rate of consistent-size slices. 512 samples 60 times per second is 30720 samples/sec. If anyone can show me a song found in the wild encoded at this rate I'll give you... well, nothing, since I'm a college student and couldn't afford to give you this pencil I have sitting on my desk.
Anywho, the conversion to float cannot be avoided but the resampling can be. By setting it to the more common 44100Hz sample rate CPU usage dropped to 4-6%. It's pretty cheap to convert formats and throw duplicate data around, but interpolating data is a lot more expensive.
At some point the vis pipeline will be smart enough to split music into chunks of a size depending on the current sample rate. Until then this patch should be good enough. (I hope.)