June 17, 2013: The Somali pirates are having a very bad year and it’s all because of math. Pirate attacks were down 70 percent last year and it has been over a year since the pirates captured a large ship (that they could get several million dollars ransom for). The usual reasons given for this shift are better security measures on the large merchant ships and more effective patrolling by the international anti-piracy task force operating off Somalia. But there’s an even more important reason that gets little mention: math, data mining, and predictive analysis. These items resulted in PARS (Piracy Attack Risk Surface), a statistical model of pirate behavior as modified by weather, shipping traffic patterns, and whatever inexplicable things the pirates have been doing lately. This showed the anti-piracy task force which areas to watch most carefully and warned merchant ships what areas they should be most alert in. As a result, pirate attacks dropped from a peak of 181 in 2009 to 32 last year. That’s an 83 percent drop and that trend continues. Most importantly, it’s become extremely difficult for the pirates to get close to a likely target without the ship they are after spotting them and speeding away. Worse, an anti-piracy task force aircraft or ship (often both) tends to show up at the same time.

Some pirate gangs have gone bankrupt and others have just shut down because the prospects of taking any high-value ships have seemingly evaporated. The Somali and Arab businessmen who provide negotiating and other services have told the pirates about these new tools the anti-piracy task force is using, and some of the more educated pirate leaders (some are businessmen) understand what’s going on and none of them have yet come up with any way to defeat tools like PARS (and several similar ones also created to defeat the pirates).

This use of data mining and predictive analysis for military purposes has been around for decades, and in the last decade it has become much more commonplace and implemented more rapidly. This made it possible for the U.S. forces in Iraq and Afghanistan to more accurately predict where roadside bombs and enemy leaders were, as well as the location of enemy weapon storage sites, smuggling routes, and bomb making workshops. This was done using predictive analysis, which collects huge quantities of information and uses data mining and other tools to analyze it for patterns, which reveal things the enemy is trying to hide.

When it came to the pirates, the naval intelligence analysts quickly found that weather was a major factor in where and when the pirates could go. Data could be collected from fishermen (and foreign fishing fleets) that operated off Somalia to find out the conditions that made pirate operations impractical. Weather satellites and easily available sensors made it possible to create a real-time map of what areas off the Somali coast were hospitable to pirate operations and when. This enabled anti-piracy ships and aircraft to narrow their patrol and search areas. Over the last two years the pirates got the impression there were a lot more warships and aircraft watching when, in fact, the number of ships and aircraft did not change that much.

Those on the receiving end of math weapons find this sort of thing very annoying. A sniper or smart bomb is something most people can understand. Well, okay, the smart bombs smack of magic but these intel tools are incomprehensible to most everyone. Yet everyone in the United States is touched by these tools, every day. Businesses use data mining and predictive analysis to see what their customers and competitors are going to do next. Pirates, snipers, and people planting smart bombs are not much different. People tend to behave in predictable ways, if only you have the math and computing power to tease out the details.

As more data is collected more things can be predicted. For example, Wall Street has long (okay, about three decades) used systems that analyzed the news for combinations of events that would predict future trends in financial markets. Intelligence agencies have caught up with this sort of thing and figured out how to apply it to the predicting diplomatic and media trends, especially those that trigger violent street demonstrations, terrorism, and the sudden overthrow of governments.

Oddly enough, the basic ideas behind these new intelligence tools (data mining and predictive analysis) were actually invented over a century ago, as part of the development of junk mail. Who knew? Now these tools predict what the enemy is going to do. For decades the statistical tools used to determine who to send junk mail to (so the sender would make a profit) were not much use to the military because implementation was cumbersome and time-consuming. Then came cheaper and more powerful computers and the development of data mining and analysis tools. This made a big difference because the more data you have to work with, the easier it is to predict things. This has been known for over a century. What then is need is the ability to crunch the numbers quickly.

These days, with thousand dollar laptop computers equipped with a terabyte size hard drive, you could put large amounts of data in one place, do the calculations, and make accurate predictions and do it all while at the front line. This wasn't possible 40 years ago, when a 75 megabyte hard drive cost $60,000 and the computer doing the calculations cost even more than that. You also didn't have digital photography (more data you can store for analysis) or a lot of data, in general, stored electronically. It's all different today. That thousand gigabyte hard drive (holding over 10,000 times more data than the $60,000 drive of yore) cost less than a hundred bucks. The laptop running the analysis software would have qualified as a supercomputer a decade ago. Back then there were theories of how data analysis could predict things. Now all those theories are being put to the test and many have worked.

In the last decade intel analysts have realized how powerful their tools are. And for those who studied math, statistics, or business in college they know the power of data mining because it has become a very popular business tool. In places like Somalia and Afghanistan, a lot of data is being collected all the time. It was some local data mining that led to the capture of Saddam Hussein, the death of al Qaeda-in-Iraq leader Zarqawi and al Qaeda head Osama bin Laden. Hundreds of senior (team leader and up) al Islamic terrorists have been killed or captured (mainly in Iraq and Afghanistan) using these techniques. The same thing is happening now in Somalia with pirates.

Data mining is basically simple in concept. In any large body of data you will find patterns. Even if the bad guys are trying to avoid establishing patterns, they will anyway. Its human nature and only the most attentive pros can avoid this trap. Some trends are more reliable than others but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of thousands of U.S. troops by giving them warning of where roadside bombs and ambushes are likely to be or where the bad guys are hiding out. Similarly, when data was taken off the site of a terrorist leader's death, it often consisted only of names, addresses, and other tidbits. But with the vast databases of names, addresses, and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data can generate even more leads and dozens of new raids. The enemy tried to adapt to all this and did to a certain extent. But the predictive analysis moves faster than the opposition can change and adapt. The only effective defense is a new enemy strategy, one that's a break with past practices. This sort of thing is very rare and not easily done. Even so, the predictive analysis eventually sorts it all out.

Speed has always been an advantage in combat but, until recently, rarely something intelligence analysis was noted for. This is no longer the case. Predictive analysis is something the troops depend on, not only for tips on what to avoid but for names and places to go after. Predicting where Somali pirates will be is simply the latest application of something Israeli intelligence experts developed to find Palestinian terrorists seeking to kill Israeli civilians after 2000 (when the Palestinians rejected a peace deal and launched a terror campaign). The Israelis taught the Americans how to study terrorist organizations and identify key leaders and technical specialists. These people became the key targets, and that tactic enabled Israel to defeat the Palestinian terror campaign eight years ago. But this was done with old-fashioned police work and a network of informers inside Palestinian communities. The new computerized systems move data collection and analysis into the 21st century, using technology and concepts that many police departments are using to good effect. But being able to speak to the system, and have it understand what you are looking for, raises the intel game to a whole new level.

The current outcry over using data mining to monitor actual or potential terrorists is another example of how little public knowledge there is of these techniques, even though they have been used on a large scale for over a century. Politicians know about this stuff, as campaigning, especially in the United States, has been quick to capitalize on data mining and predicting what voters are going to do before the voters do. Alas, the media is more inclined to demonize any technology they don’t understand rather than try to explain. That’s an easier and more profitable approach.