January 22, 2013: It’s now possible to predict the future. Not surprisingly, no one in the intelligence business is eager to talk about this. This seemingly magical ability has been a long time coming. For those in the data analysis business, it comes as no surprise. Over the last half century there has been a fundamental shift in how intelligence is collected and analyzed. It started with computers becoming large enough, and affordable enough, to analyze huge quantities of data that previously had been less effectively sorted through by hand. Then came spy satellites and credit cards, which enabled huge quantities of data on people to be collected. Telephone records got computerized, as did all sorts of other activities. More data of human activity, when analyzed, yields evidence of why people acted as they did and how they will act in the near future.

What this flood of data provided was opportunities to invent data mining and predictive analysis systems. For example, in the last decade this made it possible for the U.S. forces in Iraq and Afghanistan to more accurately predict where roadside bombs and enemy leaders were, as well as the location of enemy weapon storage sites, smuggling routes, and bomb making workshops. This was done using predictive analysis, which collects huge quantities of information and uses data mining and other tools to analyze it for patterns, which reveal things the enemy is trying to hide. 

The enemy finds this sort of thing very annoying. A sniper or smart bomb is something an Islamic terrorist can understand. Well, okay, the smart bombs smack of magic but these intel tools are incomprehensible to most everyone. Yet everyone in the United States is touched by these tools, every day. Businesses use data mining and predictive analysis to see what their customers and competitors are going to do next. Fortunately snipers and smart bombs are not involved. 

As more data is collected more things can be predicted. For example, Wall Street has long (okay, about three decades) used systems that analyzed the news for combinations of events that would predict future trends in financial markets. Intelligence agencies have caught up with this sort of thing and figured out how to apply it to the predicting diplomatic and media trends, especially those that trigger violent street demonstrations, terrorism, and the sudden overthrow of governments.

Oddly enough, the basic ideas behind these new intelligence tools (data mining and predictive analysis) were actually invented over a century ago as part of the development of junk mail. Who knew? Now these tools predict what the enemy is going to do. For decades the statistical tools used to determine who to send junk mail to (so the sender would make a profit) were not much use to the military. Then came cheaper and more powerful computers and the development of data mining and analysis tools. This made a big difference because the more data you have to work with, the easier it is to predict things. This has been known for over a century.

By 2008, with thousand dollar laptop computers equipped with hundred gigabyte (or more) hard drives, you could put large amounts of data in one place, do the calculations, and make accurate predictions. This wasn't possible 40 years ago, when a 75 megabyte hard drive cost $45,000 and the computer doing the calculations cost even more than that. You also didn't have digital photography (more data you can store for analysis) or a lot of data, in general, stored electronically. It's all different today. That thousand gigabyte hard drive (holding over 10,000 times more data than the $45,000 drive of yore) cost less than a hundred bucks. The laptop running the analysis software would have qualified as a supercomputer a decade ago. Back then there were theories of how data analysis could predict things. Now all those theories are being put to the test and many have worked. 

In the last six years intel analysts have realized how powerful their tools are. And for those who studied math, statistics, or business in college they know the power of data mining because it has become a very popular business tool. In places like Iraq and Afghanistan, lots of data is being collected all the time. It was some local data mining that led to the capture of Saddam Hussein, the death of al Qaeda-in-Iraq leader Zarqawi, and al Qaeda head Osama bin Laden. Over a hundred senior (team leader and up) al Qaeda terrorists have been killed or captured in Iraq using these techniques. The same thing is happening now in Afghanistan.

Data mining is basically simple in concept. In any large body of data you will find patterns. Even if the bad guys are trying to avoid establishing patterns, they will anyway. It's human nature and only the most attentive pros can avoid this trap. Some trends are more reliable than others but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of thousands of U.S. troops by giving them warning of where roadside bombs and ambushes are likely to be or where the bad guys are hiding out. Similarly, when data was taken off the site of a terrorist leader's death, it often consisted only of names, addresses, and other tidbits. But with the vast databases of names, addresses, and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data can generate dozens of raids and even more leads. The enemy tried to adapt to all this and did to a certain extent. But the predictive analysis moves faster than the opposition can change and adapt. The only effective defense is a new enemy strategy, one that's a break with past practices. This sort of thing is very rare and not easily done. Even so, the predictive analysis eventually sorts it all out. 

Speed has always been an advantage in combat but, until recently, rarely something intelligence analysis was noted for. This is no longer the case. Predictive analysis is something the troops depend on, not only for tips on what to avoid but for names and places to go after.  Israeli intelligence taught the Americans how to study terrorist organizations and identify key leaders and technical specialists. These people became the key targets and that tactic enabled Israel to defeat the Palestinian terror campaign eight years ago. But this was done with old-fashioned police work and a network of informers inside Palestinian communities. The new computerized systems move data collection and analysis into the 21st century, using technology and concepts that many police departments are using to good effect. But being able to speak to the system, and have it understand what you are looking for, raises the intel game to a whole new level.