Why do intelligence agencies get things wrong? A whole catalog of factors would have to be produced to answer this question. At the top of list is the sheer difficulty of the work. Trying to piece together information about an adversary operating in secret is an inherently difficult challenge. In the face of deception, denial, and uncertainty, it is understandable that analysts at a place like the CIA sometimes get things wrong.

But one of the more common pitfalls that intelligence analysts face is their own preconceived ideas. It is remarkable how powerful a force these can be. Perhaps this is one factor explaining the bizarre language of the recent National Intelligence Estimate (NIE) flatly declaring that Iran had shut down its nuclear-weapons program in 2003 even as the same document presents evidence that the most critical aspects of a nuclear program–the uranium-enrichment process–is humming along at steady clip at Natanz.

If this is an instance of intelligence officers clinging desperately to their ideas in the face of evidence to the contrary, it would not be the first time in the history of the CIA. A fascinating case concerns the question of whether the USSR was supporting international terrorism in the 1970’s and 80’s.

In 1981, Secretary of State Alexander Haig publicly and controversially asserted that the USSR was behind terrorist actions around the world. It was only after this statement that he asked the intelligence community to produce an NIE assessing his claim. This, of course, was backward; public statements by high-ranking officials should follow intelligence, not the other way around.

In any event, the task of producing the estimate fell to the Soviet division of the CIA. The full story is told in Robert Gates’s indispensable 1996 memoir, From the Shadows. “The first draft by the analysts proved beyond a shadow of a doubt that Haig had exaggerated the Soviet role — that the Soviet did not organize or direct international terrorism.” The NIE stated, in Gates’s summary:

that the Soviets disapproved of terrorism, discouraged the killing of innocents by groups they trained and supported, did not help free-lance third-world terrorist groups like the Abu Nidal organization, and under no circumstances did Moscow support the nihilist terrorist groups of Western Europe — the Red Brigades, the Red Army Faction [RAF], and so on. It cited Soviet public condemnations of such groups and carefully described the distinctions the Soviets made between national liberation groups or insurgencies and groups involved in out-and-out terrorism.

This estimate made its way for approval to Bill Casey, Reagan’s CIA director, who found it thinly sourced, improperly framed, and tendentiously argued: it had been too “narrowly focused on whether the Soviets exercised direct operational control of terrorist groups” and in Casey’s view “‘had the air of a lawyer’s plea’ that an indictment should issue because there was not enough evidence to prove the case beyond a reasonable doubt.”

Disappointed by the quality of the NIE, Casey sent it back for redrafting, this time not by CIA’s Soviet division but by the Defense Intelligence Agency, the Pentagon’s intelligence arm. The document that emerged after a prolonged interagency wrangle was more nuanced than the original CIA draft. On the crucial question of whether the USSR had supported the nihilist terrorist groups it reported that the evidence was “thin and contradictory,” but also:

that some individuals in such groups had been trained by Soviet friends and allies that also provided them with weapons and safe transit. It also observed that the Soviets had often publicly condemned such groups and considered them uncontrollable adventurers whose activities on occasion undermined Soviet objectives It noted that some such nihilistic terrorists had found refuge in Eastern Europe.

How well did either estimate — the ultra-cautious CIA one, and the cautious DIA one — hold up?

A decade later, after Communism collapsed and the archives opened, the full picture became clear, and it was now obvious, writes Gates, that both CIA and DIA had been far wide of the mark:

[w]e found out that the East Europeans (especially the East Germans) indeed not only had provided sanctuary for West European “nihilist” terrorists, but had trained, armed, and funded many of them. (For example, during the late 1970’s — early 80’s, the East German Stasi (intelligence service) supplied the West German Red Army Faction with weapons, training, false documentation, and money. The training and weapons were put to use in the RAF car-bomb attack against Ramstein Air Force Base in West Germany on August 31, 1981, which injured seventeen people. The same group was also involved in the unsuccessful rocket attack against the car of General Frederick Krosen in Heidelberg in September 1981.) It was inconceivable that the Soviets, and especially the KGB, which had these governments thoroughly penetrated, did not know and allow (if not encourage) these activities to continue. . . .

We also learned in March 1985 about a Soviet effort to target U.S. servicemen in West Germany for terrorist attacks that shocked us all. According to information from Soviet sources, Soviet agents had been assigned the task of locating dead-drop sites — places for information being transmitted to and from agents — inside bars and restaurants near American military installations in West German cities. The purpose of these sites, however was not for dead drops, but for hiding explosive devices that would be set off in a way to make them look like terrorist attacks. The sites included behind vending machines, in a ventilation cavity under a sink, in a bathroom stall over the windowsill, on a wooden beam over a lavatory, under the bottom of a paper-towel dispense, and so on. CIA  checked out fourteen of these reported sits and confirmed the existence of all but one, just as reported. And every location was filled with U.S. servicemen or dependents or was known to be frequented by U.S. and NATO servicemen. We later concluded that the targeting had been done in 1983, probably in connection with the very aggressive Soviet campaign against deployment of the INF missiles.

How was all this missed? The widespread conviction within the agency that the Soviet leaders would not do such violent things led analysts to rule out the possibility. “The same analysts who complained constantly,” writes Gates, “about the lack of good human intelligence on Soviet activities in effect argued that the absence of such reporting proved their case.” In other words, systematic bias led the CIA to produce an estimate that was the diametric reversal of reality.

If that sounds familiar, it is.