Various "worrywarts" have been concerned for long time about too-smart-for-their-own-good phones being used to spy on naive users using microphones and/or cameras, with GPS location thrown in for free. If you have a smart phone, it could be doing it right now. Don't bother switching it "off", because unless you take the battery out and discharge all capacitors it could still be recording and then upload the swag at the next available opportunity. I'm not saying it's actually doing this, but it "has the technology" and unless your phone has 100% free-as-in-liberty software that you compiled and installed yourself, there's no way to tell if it's doing this or not.
Here's a recent item from Forbes, not usually dismissed as a long-haired hippy tinfoil-hat publication: http://www.forbes.com/sites/adriankingsleyhughes/2012/10/03/how-your-android-smartphone-could-be-used-to-spy-on-you/
FLOSS and compiling yourself is insufficient. One would also need to examine all the source code of all the software and whatever binary bootstrap code is used to initially compile the compiler, and one would need a trusted platform on which to do the examination and compilation. Then one would need to trust that the loading of the software is "secure" and even that the hardware/firmware has no security flaws.
(By the way, software can be licensed in a way that allows use of source code and use of 3rd party patches without being Free/Open Source. If unlicensed use can be controlled or is a minor concern, proprietary software can even have the source be freely distributable--with only licensees having the rights to compiler and/or use the software.)
Given the practical need for a web of trust, compiling the software oneself is probably not that helpful. If one cannot trust the hardware vendor not to have pre-installed and unremovable (or just secret) malware, how can one trust that the software one installs will actually control the hardware?
A camera can be fairly effectively covered (if one knows where it is/they are). The microphone might be more difficult to physically incapacitate.
Similar issues have already existed with personal computers. Many laptops have built-in webcams and microphones.
Because of its size, a cell phone would be somewhat easier to drop into a "privacy box" (with optional RF blocking) than a laptop. Such a box could even provide a charging function.
Sorry, not even compiling everything yourself will guarantee that no malware or spyware is infesting your device. See the Wikipedia article on Backdoors, http://en.wikipedia.org/wiki/Backdoor_(computing).
I recently got a passport card and it comes with a metalized sleeve to shield it from RFID spies. I could see something like that being needed for smart phones.
Maybe the phone covers that just about everyone buys should have shutters to cover the camera lens when not in use. Something to muffle the sound going into the mic would be appropriate as well.
Finally, you really want to make sure it's completely off and inaccessible a Faraday cage could be used as a cover over the cover. Sounds like a good Kickstarter project.
As I was starting to read your comment and saw "metallized sleeve," it occurred to me that this might be a great product idea. Then I read to the end and I see you already came to the same conclusion :)
Errr....most people consider the ability of a smartphone to RECEIVE calls or other push updates one of its more important features, so putting it in a metallic sleeve is never going to be a popular option outside of those who shop the kitchen aisle for their headwear.
Since RFID is remotely powered, and can be activated from a distance much further than the few inches its proponents claim, it is a totally different class of potential security issue than anything else on a smartphone. Even with the battery removed RFID can be activated from at least 10 feet away with some inexpensive gear that will fit inconspicuously in a briefcase or backpack.
Hey Max, I know this is off-topic, but I couldn't find your earlier column (much earlier) about great Sci Fi novels that should be made into movies. Ender's Game is coming to theaters Nov. 1 of this year. Thought you would be interested.