Data Mining, Innumeracy, and the Odds on Everything

Data mining, in the form of the discredited Total Information Awareness program or its renamed-but-not-fundamentally-different successor, code-named "Basketball", has been in the news periodically as the latest and sexiest tool in the War on Terra^tm, but does it really work?

Short answer: No.

Bruce Schneier over at Wired News calls it out for those of us who still hold onto an irrational belief in science and mathematics. Take a look at his commentary and see for yourself why data mining will never be able to smoke out a single terrorist.

Excerpt:

Let's look at some numbers. We'll be optimistic -- we'll assume the system has a one in 100 false-positive rate (99 percent accurate), and a one in 1,000 false-negative rate (99.9 percent accurate). Assume 1 trillion possible indicators to sift through: that's about 10 events -- e-mails, phone calls, purchases, web destinations, whatever -- per person in the United States per day. Also assume that 10 of them are actually terrorists plotting.
This unrealistically accurate system will generate 1 billion false alarms for every real terrorist plot it uncovers. Every day of every year, the police will have to investigate 27 million potential plots in order to find the one real terrorist plot per month. Raise that false-positive accuracy to an absurd 99.9999 percent and you're still chasing 2,750 false alarms per day -- but that will inevitably raise your false negatives, and you're going to miss some of those 10 real plots.
This isn't anything new. In statistics, it's called the "base rate fallacy"... [link added]

It's all in the numbers, people, it's in the odds, it's in the critical analysis of what the numbers mean -- and what they don't mean -- and since the Repugnican anti-intellectual anti-education schoolyard-bully Bushevik apparatchiks took control of the educational framework of this country they have been going out of their way for years to dumb down the populace in every field of science, including mathematics. It shouldn't come as a suprise that Joe Sixpack and his buddies just don't get it. They can't.

Innumeracy is what it's called, and its consequences are as bad as those of illiteracy, and in many ways even more pernicious. Those who can't wrap their minds around even the simplest comparative numerical concepts are primary victims for any shell-game con-artist Ponzi scheme that comes along. And the BFEE has plenty of them, from the oxymoronic destroy-Social-Security-in-order-to-save-it reform package to the Orwellian-named deficit-reduction budget plan -- they can with full confidence make pretty much any outlandish claim they want, swear that the statistics back them up, and the poor fuckers in the Moron-American voting bloc will go along with it because they are incapable of running the numbers for themselves.

Pathetic, really. You'd think they'd teach basic statistics and mathematical thinking in the schools at an early age, but since that would contribute to creating and sustaining the very thing the BFEE doesn't want, an informed and educated electorate that is capable of logical thinking and critical anaylsis, odds are it's not going to happen.