November 22, 2013: The U.S. Air Force recently announced that, from now on, officers selected for promotion to sensitive (as in having to do with nuclear weapons) jobs would be subject to an Internet search to seek out any bad habits the candidates had not already admitted to. This came after a general in charge of air force nuclear weapons was recently dismissed after it was discovered he had a gambling problem. What was really surprising about this, at least to personnel (“human resources”) managers outside the military, was that the air force was not already using this very powerful technique to check up on candidates for critical jobs. Civilian firms had been using Internet search for over five years. What this was really all about was the resistance within the military to OSINT (Open Source Intelligence). Over the last few decades the U.S. intelligence community has resisted recognizing the importance of OSINT, despite the enthusiastic acceptance of Internet-based OSINT by so many individual military personnel and commercial information gatherers since 2000. Over the last few years OSINT has been gradually accepted throughout the military for intelligence work. Many intelligence professionals now consider OSINT a crucial tool and one that can only grow in usefulness.

The Internet has made OSINT a really, really huge source of useful intelligence. It's not just the millions of gigabytes of information that are placed on the net but the even more voluminous masses of message board postings, blogs, emails, and IMs (instant messaging) that reveal what the culture and groups are currently thinking. It was corporate intelligence practitioners who alerted the government intel people to the growing usefulness of Internet based data. Corporations have developed, over the last few decades, a keen interest in gathering intel on competitors, new markets, and all manner of things that might affect them (like evaluating job candidates). The Internet has made this a much more useful and affordable exercise, especially since corporations are less likely to break the law when gathering intel, or have access to the powerful legal tools available to government investigators and analysts for accessing data that is usually kept from the public.

For years corporate intel specialists were concerned that government agencies, especially the CIA, were not taking sufficient advantage of OSINT. Part of the problem was cultural. The intelligence agencies have always been proud of their special intel tools, like spy satellites, electronic listening stations, and spy networks. Most of these things are unique to government intelligence operations. People who use this stuff tend to look down on a bunch of geeks who simply troll the web. Even when the geeks kept coming up with valuable stuff, they didn't get any respect from the military. On the commercial side it was different and for the military it all began to change after September 11, 2001, when many intelligence specialists, who were reservists, were called to active duty. Many of these men and women worked in BI (Business Intelligence, sometimes called corporate spying) and brought with them a respectful attitude towards OSINT and spectacular (to the government intel people) ability to use it in a military environment.

Before long many junior members of the intel agencies were using OSINT more frequently. Then it was pointed out that there was growing evidence that some foreign countries were exploiting OSINT (especially the Internet) more effectively than the United States. No foreign intel agency will admit to this, but there are indications that some nations are mining the Internet quite intensively and effectively. Data mining is a heavily used commercial tool that the U.S. intel agencies have used, but now they have adopted the corporate techniques of plowing through vast quantities of unclassified data and often finding gold. This has scary implications for military organizations that do not scour the Internet for embarrassing data about officers up for promotion. If unfriendly nations find this dirt they can use it to blackmail officers in other nations and force them to become spies.

An example of the official acceptance of OSINT in the United States occurred in 2012, when the U.S. Army issued a manual, Army Techniques Publication 2-22.9, which detailed how to use open source (mainly searching the Internet) intelligence most effectively. This was the kind of OSINT troops had been using for over a decade. The publication of ATP 22.9 was a way for the senior army leadership to say, "message received and understood." ATP 22.9, despite all the useful tips it contained, didn't go far in helping the many soldiers already using the Internet but it was be useful in convincing their bosses that a lot of useful stuff can be obtained from the Internet.

The government and military intel community has the money and software chops to screen and analyze huge quantities of data on the Internet, both text and pictures. Despite all of these resources, the intel behemoths continue to get overtaken by civilian amateurs. A large factor in this was the appearance of Google Earth and other commercial satellite photo sources. This revolutionized military intelligence, and the way news on military affairs is developed and spread. Case in point was details on the transformation of the Chinese armed forces, and the activities of the North Korean military. Both China and North Korea have long been very secretive about military affairs. But the appearance of Google Earth (originally as Earth View) a decade ago changed everything. By putting so much satellite photography at the disposal of so many people, in such an easy-to-use fashion, unexpected discoveries were made.

People soon discovered that if they had a high-speed Internet connection, they could use Google Earth to find satellite photos of all sorts of interesting stuff. This was especially true of the "Forbidden Kingdoms" (China, Russia North Korea, and a few others). While the CIA and the military has had access to satellite photos of these countries since the 1960s, little of it was shown to the public. Now that so many people can examine these lower resolution, civilian satellite images many have gone over vast stretches of the Forbidden Kingdoms and found things that were newsworthy and never reported before. Things like new military bases, test sites for new weapons, and the new weapons themselves. The open discussion of these findings, most of them already known to the large national intel agencies, brought forth insights and analysis that was often superior to what the much smaller number of professional analysts were capable of. Another example of “the wisdom of the crowd.”

Technically, the countries in question can request that Google not show these classified military facilities. But in making that request they point out where the classified operation is. So far a lot of this stuff is just there to find. And users find it. This is called "crowdsourcing" (where large numbers of people accomplish impressive feats of research or analysis because they can quickly mobilize and get to the task via the Internet). The U.S. military will not say that they appreciate the work done via crowdsourcing, but individual analysts and intelligence officials have made it known, unofficially, that crowdsourcing is another useful tool that unexpectedly came their way via the Internet.