Saturday, March 11, 2006

Data Mining, Innumeracy, and the Odds on Everything

Data mining, in the form of the discredited Total Information Awareness program or its renamed-but-not-fundamentally-different successor, code-named "Basketball", has been in the news periodically as the latest and sexiest tool in the War on Terratm, but does it really work?

Short answer: No.

Bruce Schneier over at Wired News calls it out for those of us who still hold onto an irrational belief in science and mathematics. Take a look at his commentary and see for yourself why data mining will never be able to smoke out a single terrorist.


Let's look at some numbers. We'll be optimistic -- we'll assume the system has a one in 100 false-positive rate (99 percent accurate), and a one in 1,000 false-negative rate (99.9 percent accurate). Assume 1 trillion possible indicators to sift through: that's about 10 events -- e-mails, phone calls, purchases, web destinations, whatever -- per person in the United States per day. Also assume that 10 of them are actually terrorists plotting.
This unrealistically accurate system will generate 1 billion false alarms for every real terrorist plot it uncovers. Every day of every year, the police will have to investigate 27 million potential plots in order to find the one real terrorist plot per month. Raise that false-positive accuracy to an absurd 99.9999 percent and you're still chasing 2,750 false alarms per day -- but that will inevitably raise your false negatives, and you're going to miss some of those 10 real plots.
This isn't anything new. In statistics, it's called the "base rate fallacy"... [link added]
It's all in the numbers, people, it's in the odds, it's in the critical analysis of what the numbers mean -- and what they don't mean -- and since the Repugnican anti-intellectual anti-education schoolyard-bully Bushevik apparatchiks took control of the educational framework of this country they have been going out of their way for years to dumb down the populace in every field of science, including mathematics. It shouldn't come as a suprise that Joe Sixpack and his buddies just don't get it. They can't.

Innumeracy is what it's called, and its consequences are as bad as those of illiteracy, and in many ways even more pernicious. Those who can't wrap their minds around even the simplest comparative numerical concepts are primary victims for any shell-game con-artist Ponzi scheme that comes along. And the BFEE has plenty of them, from the oxymoronic destroy-Social-Security-in-order-to-save-it reform package to the Orwellian-named deficit-reduction budget plan -- they can with full confidence make pretty much any outlandish claim they want, swear that the statistics back them up, and the poor fuckers in the Moron-American voting bloc will go along with it because they are incapable of running the numbers for themselves.

Pathetic, really. You'd think they'd teach basic statistics and mathematical thinking in the schools at an early age, but since that would contribute to creating and sustaining the very thing the BFEE doesn't want, an informed and educated electorate that is capable of logical thinking and critical anaylsis, odds are it's not going to happen.


Granny said...

I'm not great on statistics but I do understand what you just explained.

Re South Dakota (I didn't want to join anonymous any longer thanks), they don't care about coat hangers and knitting needles. If some woman dies as a result their attitude is it serves her right. It has always been thus. It's all about punishment; not about saving babies.

Anonymous said...

No truer words were ever written. This is the same "keep the populace dumb and incapable of critical thinking while the crooked politicians rob the treasury blind", that has been going on for years in Louisiana. The CRIMINALLY CORRUPT REPUGNICANS have taken it national.

Anonymous said...

Three classes that should be mandatory in every school are Logic, Mathematical Thinking and Basic Statistics.
Without a working knowledge of the ideas and functions contained in those classes, a person is doomed to failure in life and will indeed be fair game for any unscrupulous charlatan who comes along.
It's no wonder that Americans carry an average credit card debt in the neighborhood of $15,000. They can't figure out that they are being gouged by the banks to the tune of billions a year in the high interest rates.