The great Robert Fogel, sadly recently departed, noted in his discussion about slavery that the system was, for the most part, very efficient. One point he liked to emphasise is that people are so used to the notion of efficiency being applied to good ends that they don't consider the alternative possibility - efficiency as applied to evil in fact produces monstrous outcomes.
A similar tension exists in the way people understand the spy services. Whenever you see Hollywood depictions of the CIA (or just shadowy agents of some secret department, standing in for the CIA), 90-odd percent of the time they are displayed as having a sinister level of competence in their ability to pull off evil actions. They're everywhere, they see everything, and they can hunt you down. Of course, Jason Bourne wins in the end, but you're not left in any doubt that most of the time, the government gets its way.
This view eventually permeates a large amount of social thinking on the matter. Consider the 9/11 truthers - according to their claims, the government managed to organise a massive conspiracy to plant all sorts of explosives inside two skyscrapers, demolish them with people inside, make it look like planes were crashing into them on live television, and blame the Muslims. All while keeping this totally under wraps, except for the keen eyes of the producers of 'Loose Change'.
You could spend hours debating with these clowns about whether fire can actually melt steel, but it seems you might get much further by simply noting, 'Have you been to the DMV recently? What impression did that give you about the competence of the average government employee?'
This is the default Shylock rule - when you're thinking about the government, assume it will be run by folks at the DMV. Can you trust the government to clean up after Katrina? Think the DMV. Is it likely that the next round of financial regulation will prevent the next housing crisis? Think the DMV. Can you create police SWAT teams all over the country in rinky dink places and not have them consistently raiding the wrong houses while looking for marijuana? Think the DMV.
But... what about the CIA? Surely, if competence exists anywhere, it must exist there, right? When it really counts, when the chips are down, these guys are the pros, and they wouldn't screw it up?
Except, you know, with the Iraqi weapons of mass destruction.
The point of the DMV rule is not that it's always right. It's just that it tends to be a fairly good predictor of what's actually going to happen.
I wrote a while ago about the fact that before this latest news broke, you tended to read a lot of stories about Chinese hackers tooling on the US - hacking into Google, diverting all internet traffic to China, that kind of thing. You'd very rarely hear about any US operations - the only exceptions were cases like Stuxnet which accidentally got released into the wild. In other words, you'd hear about the good side of the program (the US is releasing a computer virus to screw up Iranian centrifuges for enriching uranium) at the same time as the bad (this wasn't meant to be in the papers, meaning the virus got found out).
When I didn't read anything about US Cyber operations, I applied the DMV principle and assumed that these clowns just didn't know what they were doing. But the alternative was always that they were so good at what they were doing that you never heard about it. Since, of course, you weren't meant to.
As it turns out, the NSA has been spying on Americans like J Edgar Hoover on a dirt-digging mission to cover up for his cross-dressing proclivities.
Say what you will about the ethics of these programs (and I tend to be considerably wary of them) - they don't seem incompetent. They seem scarily competent. They seem like The Bourne Identity, when in reality I was expecting a cross between Fawlty Towers and Yes Minister (except with everyone being like the Minister).
The bad news is that this the NSA seems to play extraordinarily fast and loose with the 4th amendment, and has enormous power to spy on American citizens in a way that would make the Founding Fathers spin in their graves faster than a virus-ridden Iranian centrifuge.
The good news is that for the fraction of the things the NSA does which are likely beneficial to the country (and even the most jaded skeptic would probably admit that this fraction is non-zero), they can hopefully apply the same level of competence.
And if you can actually get the privacy destroying parts of the NSA's work removed (which, sadly, you probably can't), this whole imbroglio might actually be good news.
In other words, the government may be negative NPV, but it's not pure evil. So jacking up competence will at least have some effects on the revenues side of the ledger.