Cognitive traps for intelligence analysis
Encyclopedia
Intelligence analysis
is plagued by many of the cognitive traps also encountered in other disciplines. The first systematic study of the specific pitfalls lying between an intelligence analyst and clear thinking was carried out by Dick Heuer.
According to Heuer, these traps may be rooted either in the analyst's organizational culture or his or her own personality.
, first studied in Gestalt psychology and in relation to the subject-object problem
.
Experienced analysts may recognize that they have fallen prey to mirror-imaging if they discover that they are unwilling to examine variants of what they consider most reasonable in light of their personal frame of reference. Less-perceptive analysts affected by this trap may regard legitimate objections as a personal attack, rather than looking beyond ego to the merits of the question. Peer review (especially by people from a different background) can be a wise safeguard. Organizational culture can also create traps which render individual analysts unwilling to challenge acknowledged experts in the group.
Another trap, target fixation
, has an analogy in aviation: it occurs when pilots become so intent on delivering their ordnance that they lose sight of the big picture and crash into the target. This is a more basic human tendency than many realize. Analysts may fixate on one hypothesis, looking only at evidence that is consistent with their preconceptions and ignoring other relevant views. The desire for rapid closure is another form of idea fixation.
"Familiarity with terrorist methods, repeated attacks against U.S. facilities overseas, combined with indications that the continental United States was at the top of the terrorist target list might have alerted us that we were in peril of a significant attack. And yet, for reasons those who study intelligence failure
will find familiar, 9/11 fits very much into the norm of surprise caused by a breakdown of intelligence warning." The breakdown happened, in part, because there was poor information-sharing among analysts (in different FBI offices, for example). At a conceptual level, US intelligence knew that al-Qaida actions almost always involve multiple, near-simultaneous attacks; however, the FBI did not assimilate piecemeal information on oddly behaving foreign flight-training students into this context.
On the day of the hijackings (under tremendous time pressure), no analyst associated the multiple hijackings with the multiple-attack signature of al-Qaeda
. The failure to conceive that a major attack could occur within the US left the country unprepared. For example, irregularities detected by the Federal Aviation Administration
and North American Air Defense Command did not flow into a center where analysts could consolidate this information and (ideally) collate it with earlier reports of odd behavior among certain pilot trainees, or the possibility of hijacked airliners being used as weapons.
Inappropriate analogies are yet another cognitive trap. Though analogies may be extremely useful they can become dangerous when forced, or when they are based on assumptions of cultural or contextual equivalence. Avoiding such analogies is difficult when analysts are merely unconscious of differences between their own context and that of others; it becomes extremely difficult when they are unaware that important knowledge is missing. Difficulties associated with admitting one's ignorance are an additional barrier to avoiding such traps. Such ignorance can take the form of insufficient study: a lack of factual information or understanding; an inability to mesh new facts with old; or a simple denial of conflicting facts.
, whereby a group heavily invested in a particular collection technology ignores valid information from other sources (functional specialization). It was a Soviet tendency to value HUMINT (HUMan INTelligence), gathered from espionage, above all other sources; the Soviet OSINT was forced to go outside the state intelligence organization in developing the USA (later USA-Canada) Institute of the Soviet Academy of Sciences.
Another specialization problem may come as a result of security compartmentalization. An analytic team with unique access to a source may overemphasize that source's significance. This can be a major problem with long-term HUMINT relationships, in which partners develop personal bonds.
Groups (like individual analysts) can also reject evidence which contradicts prior conclusions. When this happens it is often difficult to assess whether the inclusion of certain analysts in the group was the thoughtful application of deliberately contrarian "red teams", or the politicized insertion of ideologues to militate for a certain policy. Monopolization of the information flow (as caused by the latter) has also been termed "stovepiping", by analogy with intelligence-collection disciplines.
The social anthropologist Edward T. Hall illustrated one such conflict
with an example from the American Southwest. "Anglo" drivers became infuriated when "Hispanic" traffic police would cite them for going one mile an hour over the speed limit, although a Hispanic judge would later dismiss the charge. "Hispanic" drivers, on the other hand, were convinced that "Anglo" judges were unfair because they would not dismiss charges due to extenuating circumstances.
Both cultures were rational with regard to law enforcement and the adjudication of charges; indeed, both believed that one of the two had to be flexible and the other had to be formal. However, in the Anglo culture it was the police who had discretion with regard to issuing speeding tickets, while the court was expected to stay within the letter of the law. In the Hispanic culture the police were expected to be strict, but the courts would balance the situation. There was a fundamental misunderstanding; both sides were ethnocentric, and both incorrectly assumed the other culture was a mirror image of itself. In this example denial of rationality was the result in both cultures, yet each was acting rationally within its own value set.
In a subsequent interview, Hall spoke widely about intercultural communication.
He summed up years of study with the statement, "I spent years trying to figure out how to select people to go overseas. This is the secret. You have to know how to make a friend. And that is it!"
To make a friend, one has to understand the culture of the potential friend, one's own culture, and how things which are rational in one may not translate to the other. Key questions are:
Hall asserted:
Proportionality bias assumes that small things in one culture are small in every culture. In reality, cultures prioritize differently. In Western (especially northern European) culture, time schedules are important; being late can be a major discourtesy. Waiting one's turn is the cultural norm, and failing to stand in line is a cultural failing. "Honor killing" seems bizarre in some cultures, but is an accepted part of others.
Even within a culture, however, individuals remain individual. Presumption of unitary action by organizations is another trap. In Japanese culture the lines of authority are very clear, but the senior individual will also seek consensus. American negotiators may push for quick decisions, but the Japanese need to build consensus first; once it exists, they may execute faster than Americans.
In like manner, no analyst in US Navy force protection conceived of an Arleigh Burke class destroyer
destroyer such as the being attacked with a small suicide boat—a boat much like those the Japanese planned to use extensively against invasion forces during World War II.
In the Pacific Theater of World War II
, the Japanese seemed to believe that their language was so complex that even if their cryptosystems such as PURPLE were broken, outsiders would not really understand the content. That was not strictly true, but it was sufficiently so that there were cases where even the intended recipients did not clearly understand the writer's intent.
On the other side, the US Navy assumed that ships anchored in the shallow waters of Pearl Harbor
were safe from torpedo
attack, even though the British had demonstrated the feasibility of shallow-water torpedo attacks in the 1940 Battle of Taranto
.
Even if intelligence services had credited the 9/11
conspirators with the organizational capacity necessary to hijack four airliners simultaneously, no one would have suspected that the hijackers' weapon of choice would be the box-cutter.
Likewise, the US Navy underestimated the danger of suicide boats in harbor, and set rules of engagement
that allowed an unidentified boat to sail into the USS Cole without being warned off or taken under fire. An Arleigh Burke class destroyer
is one of the most powerful warships ever built, yet US security policies did not protect the docked USS Cole.
would react to situations in the same manner as they themselves. Similarly, in the run-up to the Gulf War, there was a serious misapprehension—independent of politically motivated intelligence manipulation—that Saddam Hussein
would view the situation vis-à-vis Kuwait as the State Department and White House viewed it.
Opposing countries are not monolithic, even within their governments. There can be bureaucratic competition, which becomes associated with different ideas. Some dictators (such as Hitler and Stalin) were known for creating internal dissension, so that only the leader was in complete control. A current issue, which analysts understand but politicians may not (or may want to exploit by playing on domestic fears), is the actual political and power structure of Iran
; one must not equate the power of Iran's president with that of the president of the United States.
Opponents are not always rational. They may have a greater risk tolerance than one's own country. Maintaining the illusion of a WMD threat appears to have been one of Saddam Hussein's survival strategies. Returning again to the Iranian example, an apparently irrational statement from President Mahmoud Ahmadinejad would not carry the weight of a similar statement by Supreme Leader
Ali Khamenei
. Analysts sometimes assume that the opponent is all-wise and knows all of your side's weaknesses. Despite this danger, opponents are unlikely to act according to one's best-case scenario; they may take the worst-case approach, to which one is most vulnerable.
Of all modern militaries, the Russians treat strategic deception
(or, in their word, maskirovka; this goes beyond our phrase to include deception, operational security and concealment) as an integral part of all planning. The highest levels of command are involved.
Bacon wrote further that The counterattack by the Steppe Front stunned the Germans.
The opponent may try to overload one's analytical capabilities
as a gambit for those preparing the intelligence budget, and for those agencies where the fast track to promotion is in data collection; one's own side may produce so much raw data that the analyst is overwhelmed, even without enemy assistance.
Intelligence analysis
Intelligence analysis is the process of taking known information about situations and entities of strategic, operational, or tactical importance, characterizing the known, and, with appropriate statements of probability, the future actions in those situations and by those entities...
is plagued by many of the cognitive traps also encountered in other disciplines. The first systematic study of the specific pitfalls lying between an intelligence analyst and clear thinking was carried out by Dick Heuer.
According to Heuer, these traps may be rooted either in the analyst's organizational culture or his or her own personality.
Types
The most common personality trap, known as mirror-imaging, is the analysts' assumption that the people being studied think like the analysts themselves. An important variation is to confuse actual subjects with one's information or images about them, as the sort of apple one eats and the ideas and issues it may raise. It poses a dilemma for the scientific method in general, since science uses information and theory to represent complex natural systems as if theoretical constructs might be in control of indefinable natural processes. An inability to distinguish subjects from what one is thinking about them is also studied under the subject of functional fixednessFunctional fixedness
Functional fixedness is a cognitive bias that limits a person to using an object only in the way it is traditionally used. The concept of functional fixedness originated in Gestalt Psychology, a movement in psychology that emphasizes holistic processing...
, first studied in Gestalt psychology and in relation to the subject-object problem
Subject-object problem
The subject–object problem, a longstanding philosophical issue, is concerned with the analysis of human experience, and of what within experience is "subjective" and what is "objective."...
.
Experienced analysts may recognize that they have fallen prey to mirror-imaging if they discover that they are unwilling to examine variants of what they consider most reasonable in light of their personal frame of reference. Less-perceptive analysts affected by this trap may regard legitimate objections as a personal attack, rather than looking beyond ego to the merits of the question. Peer review (especially by people from a different background) can be a wise safeguard. Organizational culture can also create traps which render individual analysts unwilling to challenge acknowledged experts in the group.
Another trap, target fixation
Target fixation
Target fixation is a process by which the brain is focused so intently on an observed object that awareness of other obstacles or hazards can diminish. Also, in an avoidance scenario, the observer can become so fixated on the target that they will forget to take the necessary action to avoid it,...
, has an analogy in aviation: it occurs when pilots become so intent on delivering their ordnance that they lose sight of the big picture and crash into the target. This is a more basic human tendency than many realize. Analysts may fixate on one hypothesis, looking only at evidence that is consistent with their preconceptions and ignoring other relevant views. The desire for rapid closure is another form of idea fixation.
"Familiarity with terrorist methods, repeated attacks against U.S. facilities overseas, combined with indications that the continental United States was at the top of the terrorist target list might have alerted us that we were in peril of a significant attack. And yet, for reasons those who study intelligence failure
Intelligence Failure
Intelligence Failure is the eighth studio album of the actor Viggo Mortensen and seventh with the avant-garde guitarist Buckethead, released on 2005. The album combines traditional and original compositions, as well as clips of public speeches, most notably of George W. Bush and members of his...
will find familiar, 9/11 fits very much into the norm of surprise caused by a breakdown of intelligence warning." The breakdown happened, in part, because there was poor information-sharing among analysts (in different FBI offices, for example). At a conceptual level, US intelligence knew that al-Qaida actions almost always involve multiple, near-simultaneous attacks; however, the FBI did not assimilate piecemeal information on oddly behaving foreign flight-training students into this context.
On the day of the hijackings (under tremendous time pressure), no analyst associated the multiple hijackings with the multiple-attack signature of al-Qaeda
Al-Qaeda
Al-Qaeda is a global broad-based militant Islamist terrorist organization founded by Osama bin Laden sometime between August 1988 and late 1989. It operates as a network comprising both a multinational, stateless army and a radical Sunni Muslim movement calling for global Jihad...
. The failure to conceive that a major attack could occur within the US left the country unprepared. For example, irregularities detected by the Federal Aviation Administration
Federal Aviation Administration
The Federal Aviation Administration is the national aviation authority of the United States. An agency of the United States Department of Transportation, it has authority to regulate and oversee all aspects of civil aviation in the U.S...
and North American Air Defense Command did not flow into a center where analysts could consolidate this information and (ideally) collate it with earlier reports of odd behavior among certain pilot trainees, or the possibility of hijacked airliners being used as weapons.
Inappropriate analogies are yet another cognitive trap. Though analogies may be extremely useful they can become dangerous when forced, or when they are based on assumptions of cultural or contextual equivalence. Avoiding such analogies is difficult when analysts are merely unconscious of differences between their own context and that of others; it becomes extremely difficult when they are unaware that important knowledge is missing. Difficulties associated with admitting one's ignorance are an additional barrier to avoiding such traps. Such ignorance can take the form of insufficient study: a lack of factual information or understanding; an inability to mesh new facts with old; or a simple denial of conflicting facts.
Organizational culture
Even extremely creative thinkers may find it difficult to gain support within their organization. Often more concerned with appearances, managers may suppress conflict born of creativity in favor of the status quo. A special case of stereotyping is stovepipingStovepiping
Stovepiping is a metaphorical term which recalls a stovepipe's function as an isolated vertical conduit, and has been used, in the context of intelligence, to describe several ways in which raw intelligence information may be presented without proper context...
, whereby a group heavily invested in a particular collection technology ignores valid information from other sources (functional specialization). It was a Soviet tendency to value HUMINT (HUMan INTelligence), gathered from espionage, above all other sources; the Soviet OSINT was forced to go outside the state intelligence organization in developing the USA (later USA-Canada) Institute of the Soviet Academy of Sciences.
Another specialization problem may come as a result of security compartmentalization. An analytic team with unique access to a source may overemphasize that source's significance. This can be a major problem with long-term HUMINT relationships, in which partners develop personal bonds.
Groups (like individual analysts) can also reject evidence which contradicts prior conclusions. When this happens it is often difficult to assess whether the inclusion of certain analysts in the group was the thoughtful application of deliberately contrarian "red teams", or the politicized insertion of ideologues to militate for a certain policy. Monopolization of the information flow (as caused by the latter) has also been termed "stovepiping", by analogy with intelligence-collection disciplines.
The "other culture"
There are many levels at which one can misunderstand another culture, be it that of an organization or a country. One frequently encountered trap is the rational-actor hypothesis, which ascribes rational behavior to the other side—according to a definition of rationality from one's own culture.The social anthropologist Edward T. Hall illustrated one such conflict
with an example from the American Southwest. "Anglo" drivers became infuriated when "Hispanic" traffic police would cite them for going one mile an hour over the speed limit, although a Hispanic judge would later dismiss the charge. "Hispanic" drivers, on the other hand, were convinced that "Anglo" judges were unfair because they would not dismiss charges due to extenuating circumstances.
Both cultures were rational with regard to law enforcement and the adjudication of charges; indeed, both believed that one of the two had to be flexible and the other had to be formal. However, in the Anglo culture it was the police who had discretion with regard to issuing speeding tickets, while the court was expected to stay within the letter of the law. In the Hispanic culture the police were expected to be strict, but the courts would balance the situation. There was a fundamental misunderstanding; both sides were ethnocentric, and both incorrectly assumed the other culture was a mirror image of itself. In this example denial of rationality was the result in both cultures, yet each was acting rationally within its own value set.
In a subsequent interview, Hall spoke widely about intercultural communication.
He summed up years of study with the statement, "I spent years trying to figure out how to select people to go overseas. This is the secret. You have to know how to make a friend. And that is it!"
To make a friend, one has to understand the culture of the potential friend, one's own culture, and how things which are rational in one may not translate to the other. Key questions are:
- What is culture?
- How is an individual unique within a culture?
Hall asserted:
Proportionality bias assumes that small things in one culture are small in every culture. In reality, cultures prioritize differently. In Western (especially northern European) culture, time schedules are important; being late can be a major discourtesy. Waiting one's turn is the cultural norm, and failing to stand in line is a cultural failing. "Honor killing" seems bizarre in some cultures, but is an accepted part of others.
Even within a culture, however, individuals remain individual. Presumption of unitary action by organizations is another trap. In Japanese culture the lines of authority are very clear, but the senior individual will also seek consensus. American negotiators may push for quick decisions, but the Japanese need to build consensus first; once it exists, they may execute faster than Americans.
The "other side" is different
The analyst's country (or organization) is not identical to that of their opponent. One error is to mirror-image the opposition, assuming it will act the same as your country and culture would under the same circumstances. "It seemed inconceivable to the U.S. planners in 1941 that the Japanese would be so foolish to attack a power whose resources so exceeded those of Japan, thus virtually guaranteeing defeat".In like manner, no analyst in US Navy force protection conceived of an Arleigh Burke class destroyer
Arleigh Burke class destroyer
The Arleigh Burke class of guided missile destroyers is the United States Navy's first class of destroyer built around the Aegis combat system and the SPY-1D multi-function phased array radar. The class is named for Admiral Arleigh "31-Knot" Burke, the most famous American destroyer officer of...
destroyer such as the being attacked with a small suicide boat—a boat much like those the Japanese planned to use extensively against invasion forces during World War II.
The "other side" makes different technological assumptions
An opponent's cultural framework affects its approach to technology. This complicates the task of one's own analysts in assessing the opponent's resources, how they may be used and defining intelligence targets accordingly. Mirror-imaging—committing to a set of common assumptions rather than challenging those assumptions—has figured in numerous intelligence failures.In the Pacific Theater of World War II
Pacific War
The Pacific War, also sometimes called the Asia-Pacific War refers broadly to the parts of World War II that took place in the Pacific Ocean, its islands, and in East Asia, then called the Far East...
, the Japanese seemed to believe that their language was so complex that even if their cryptosystems such as PURPLE were broken, outsiders would not really understand the content. That was not strictly true, but it was sufficiently so that there were cases where even the intended recipients did not clearly understand the writer's intent.
Pearl Harbor
Pearl Harbor, known to Hawaiians as Puuloa, is a lagoon harbor on the island of Oahu, Hawaii, west of Honolulu. Much of the harbor and surrounding lands is a United States Navy deep-water naval base. It is also the headquarters of the U.S. Pacific Fleet...
were safe from torpedo
Torpedo
The modern torpedo is a self-propelled missile weapon with an explosive warhead, launched above or below the water surface, propelled underwater towards a target, and designed to detonate either on contact with it or in proximity to it.The term torpedo was originally employed for...
attack, even though the British had demonstrated the feasibility of shallow-water torpedo attacks in the 1940 Battle of Taranto
Battle of Taranto
The naval Battle of Taranto took place on the night of 11–12 November 1940 during the Second World War. The Royal Navy launched the first all-aircraft ship-to-ship naval attack in history, flying a small number of obsolescent biplane torpedo bombers from an aircraft carrier in the Mediterranean Sea...
.
Even if intelligence services had credited the 9/11
September 11, 2001 attacks
The September 11 attacks The September 11 attacks The September 11 attacks (also referred to as September 11, September 11th or 9/119/11 is pronounced "nine eleven". The slash is not part of the pronunciation...
conspirators with the organizational capacity necessary to hijack four airliners simultaneously, no one would have suspected that the hijackers' weapon of choice would be the box-cutter.
Rules of engagement
Rules of Engagement refers to those responses that are permitted in the employment of military personnel during operations or in the course of their duties. These rules of engagement are determined by the legal framework within which these duties are being carried out...
that allowed an unidentified boat to sail into the USS Cole without being warned off or taken under fire. An Arleigh Burke class destroyer
Arleigh Burke class destroyer
The Arleigh Burke class of guided missile destroyers is the United States Navy's first class of destroyer built around the Aegis combat system and the SPY-1D multi-function phased array radar. The class is named for Admiral Arleigh "31-Knot" Burke, the most famous American destroyer officer of...
is one of the most powerful warships ever built, yet US security policies did not protect the docked USS Cole.
The "other side" doesn't make decisions as you do
Mirror-imaging can be a major problem for policymakers, as well as analysts. During the Vietnam War, Johnson and McNamara assumed that Ho Chi MinhHo Chi Minh
Hồ Chí Minh , born Nguyễn Sinh Cung and also known as Nguyễn Ái Quốc, was a Vietnamese Marxist-Leninist revolutionary leader who was prime minister and president of the Democratic Republic of Vietnam...
would react to situations in the same manner as they themselves. Similarly, in the run-up to the Gulf War, there was a serious misapprehension—independent of politically motivated intelligence manipulation—that Saddam Hussein
Saddam Hussein
Saddam Hussein Abd al-Majid al-Tikriti was the fifth President of Iraq, serving in this capacity from 16 July 1979 until 9 April 2003...
would view the situation vis-à-vis Kuwait as the State Department and White House viewed it.
Opposing countries are not monolithic, even within their governments. There can be bureaucratic competition, which becomes associated with different ideas. Some dictators (such as Hitler and Stalin) were known for creating internal dissension, so that only the leader was in complete control. A current issue, which analysts understand but politicians may not (or may want to exploit by playing on domestic fears), is the actual political and power structure of Iran
Politics of Iran
The politics of Iran take place in a framework of theocracy guided by an Islamist ideology. The December 1979 constitution, and its 1989 amendment, define the political, economic, and social order of the Islamic Republic of Iran, declaring that Shi'a Islam of the Twelver school of thought is...
; one must not equate the power of Iran's president with that of the president of the United States.
Opponents are not always rational. They may have a greater risk tolerance than one's own country. Maintaining the illusion of a WMD threat appears to have been one of Saddam Hussein's survival strategies. Returning again to the Iranian example, an apparently irrational statement from President Mahmoud Ahmadinejad would not carry the weight of a similar statement by Supreme Leader
Supreme leader
A supreme leader typically refers to a figure in the highest leadership position of an entity, group, organization, or state, who exercises strong or all-powerful authority over it. In religion, the supreme leader or supreme leaders is God or Gods...
Ali Khamenei
Ali Khamenei
Ayatollah Seyed Ali Hoseyni Khāmene’i is the Supreme Leader of Iran and the figurative head of the Muslim conservative establishment in Iran and Twelver Shi'a marja...
. Analysts sometimes assume that the opponent is all-wise and knows all of your side's weaknesses. Despite this danger, opponents are unlikely to act according to one's best-case scenario; they may take the worst-case approach, to which one is most vulnerable.
The "other side" may be trying to confuse you
The analysts' job is to form hypotheses; however, they should also be prepared to reexamine them repeatedly in light of new information instead of searching for evidence buttressing one favored theory. They must remember that the enemy may be deliberately deceiving them with information which seems plausible to the enemy. Donald Bacon observed that "the most successful deception stories were apparently as reasonable as the truth. Allied strategic deception, as well as Soviet deception in support of the operations at Stalingrad, Kursk, and the 1944 summer offensive, all exploited German leadership’s preexisting beliefs and were, therefore, incredibly effective." Theories that Hitler thought implausible were not accepted. Western deception staffs alternated "ambiguous" and "misleading" deceptions; the former intended simply to confuse analysts, and the latter to make one false alternative especially likely.Of all modern militaries, the Russians treat strategic deception
Deception
Deception, beguilement, deceit, bluff, mystification, bad faith, and subterfuge are acts to propagate beliefs that are not true, or not the whole truth . Deception can involve dissimulation, propaganda, and sleight of hand. It can employ distraction, camouflage or concealment...
(or, in their word, maskirovka; this goes beyond our phrase to include deception, operational security and concealment) as an integral part of all planning. The highest levels of command are involved.
The opponent may try to overload one's analytical capabilities
as a gambit for those preparing the intelligence budget, and for those agencies where the fast track to promotion is in data collection; one's own side may produce so much raw data that the analyst is overwhelmed, even without enemy assistance.