In his great book On War, Clausewitz wrote little about secret intelligence. What he did say was not complimentary. "Many intelligence reports in war are contradictory; even more are false, and most are uncertain. This is true of all intelligence but even more so in the heat of battle." In fact, Clausewitz was wrong. Wellington had just shown in the Peninsula war that good intelligence helps you to win stunning victories over superior forces; and in Belgium that, if your intelligence is inadequate, you risk being humbugged by Napoleon.
Still, we can sympathise with Clausewitz. The sensationalism with which the newspapers in Britain treat anything to do with secret intelligence makes rational discussion almost impossible. Press reports about the recent Mitrokhin affair left the public with no sense of how much damage the Russians had done to British interests-or, indeed, how much damage Mitrokhin's defection with some old KGB files had done to theirs. The only interesting thing to emerge from the whole farrago was the curious fact that one grandmother thought that spying for the Soviet Union was a good way of helping ordinary Russians.
Secrets are like sex. We all suspect that others get more than we do. Journalists and novelists have a natural and generously indulged interest in blowing up the importance of the subject. Their audience, the public, finds morbid fascination in plumbing the heights and the depths of human behaviour as revealed in the secret world: heroism, betrayal and, indeed, sex in all shapes and forms. Then there is the enchantment of simple things, the gadgetry of espionage: secret inks, dead letter boxes, microdot cameras. Politicians, public, press and novelists are all seduced by the glamour of the secret world.
The feverish interest continues, even though the cold war is over. Secret agents on both sides compete to publish their memoirs. Academic studies abound. So do websites with such titles as "Spy News on Line." The CIA, the British Security Service (M15), and the Government Communications Headquarters (GCHQ)-the successor to Bletchley Park which cracked the German Enigma code during the war-all have their own websites, too. Only Britain's Secret Intelligence Service (M16) has kept aloof from cyberspace. But the penalty is that conspiracy theorists and hackers fill the gap with their own versions.
For the intelligence agencies themselves, secrecy is an indispensable cloak. Without secrecy they cannot operate. However much-and however rightly-parliaments may press for oversight, you cannot-by definition-have secret agencies which operate in the full glare of public scrutiny. But the agencies are certainly not beyond exploiting their mantle of secrecy. It helps to impress their opponents and it is good for recruitment. It also makes it harder for finance ministries to demand value for money. Moreover, it ensures ready access to national leaders who are as willing as the public to be seduced by the glamour of secrecy and who have the additional excitement of feeling-as the public can never feel-that they know things no one else does. And the sense of being an initiate is a great comfort to the spies themselves. If you cannot tell your friends and relatives what you do all day, it is a comfort to know that you belong to a secret band of brothers, bound together by arcane knowledge. The cloak of secrecy is attractive for another reason. Even in a democracy, it is a useful argument against critics: "All you ever hear about are our failures. Our successes are too secret to be revealed."
The essence of Clausewitz's criticism is that it is just when you most need good intelligence that things go most wrong. The stress of war increases both the unreliability of the intelligence you get and the mendacity of the people who provide it. All wars generate irrational fears and paranoia. The cold war had the extra edge of hysteria which is typical of ideological wars. It was no accident that the Doctor's Plot in Moscow coincided with the McCarthy witch-hunts. We find it easy to believe in the unseen enemy who gives us someone else to blame for our own failures. At various times the Pope, the Freemasons, the communists, the Jews, the KGB, the CIA have all played this role. Some years ago a brilliant television thriller, A Very British Coup, portrayed Whitehall's rather innocuous Joint Intelligence Committee as the very heart of darkness of the British establishment. An academic recently claimed that the head of M16 was more influential than the Chancellor of the Exchequer and the governor of the Bank of England. These are the fantasies of paranoia.
Of course, all of us need as much reliable information as we can absorb and use. But most of that information is openly available. AJP Taylor once said that the foreign office had no secrets. Since the arrival of the worldwide web, that is almost literally true. Almost everything you need to know is out there somewhere if you know where to look. Indeed, because of the tortuous methods by which it is gathered, secret information is harder to collect and harder to check than information obtained openly. It is even more likely to be distorted by special interests. One bit of information, whether ingeniously gathered by secret means or taken from the internet, is not intrinsically more valuable than another. It is either true or it is false. It either arrives in time or it doesn't. It is either understood or misunderstood by those whose job it is to assess. And it is either exploited or ignored by those who have the responsibility to act.
This said, all governments do need information-however fragmentary and unreliable-about their enemies, their rivals and sometimes, indeed, their friends; information which those enemies, rivals and friends are determined that they should not have. This is why the history of spying goes back to the dawn of time, long before Joshua sent his two agents to hide with the harlot Rahab (sex, even then) while they spied out the city of Jericho. Governments need military intelligence in order to assess the military capabilities of their potential enemy; and they need political intelligence in order to assess his intentions. Since the first world war, governments have also used technical means: aerial reconnaissance, nowadays conducted from satellites; and signals intelligence, nowadays also conducted largely with the aid of satellites backed by sophisticated computers. But even machines must be supplemented from time to time by unreliable human observers. A pedestrian in Belgrade could have directed you to the Chinese embassy, even if the CIA couldn't.
Even the best intelligence must be subjected to five tests. Accurate? Relevant? Properly interpreted? Timely? Used effectively? Secret intelligence-like open information-often fails one or more of these tests. For example, the intelligence produced from Enigma was usually objective and accurate: it consisted of documentary evidence brilliantly extracted from the heart of the German war machine. But it was often difficult to interpret and use. In 1943 and 1944, those who favoured a second front read the intelligence as meaning that the Germans would be unable to transfer any significant force from Russia to the Atlantic Wall to counter an assault. Prudent British generals insisted on a certainty, well beyond what secret intelligence was capable of providing, that their German opponents would be too weak to throw the Allies back into the sea. The matter was resolved by judgement and politics, not by secret information. The Americans' Venona project rivalled Enigma in intellectual ingenuity. It broke the KGB's wartime codes and showed how successful the Russians had been in penetrating the Allies' atomic bomb project and other operations. But most of the information Venona produced was too sensitive or too late to be used, and most of the Soviet agents it unmasked escaped or died before they could be prosecuted.
Even the best secret intelligence is subject to abuse. The massive US intelligence assault on the Soviet Union led in the 1950s to overestimates of Soviet military strength which Eisenhower called "imaginative creations of irresponsibility." The military on both sides had an institutional interest in bidding up the presumed strength of their opponents. The annual assessments of Soviet power produced by Cap Weinberger's Pentagon were terrifying. They drove up defence expenditure in a most satisfying manner. We now know that they were much exaggerated. But the bias is not surprising, perhaps not even reprehensible. Those who are responsible for a nation's security-the generals, admirals, airmen and spies-are bound to go for the worst case analysis. After all it is they, not the politicians, who will be shot on their own quarterdecks if things go wrong.
A reliable understanding of what drives other people is as hard to come by in public life as it is in private. Western governments understood little more than the intelligent outside observer about the motivations of Stalin, Hitler, or indeed Saddam Hussein and Milosevic. In 1969, Britain's Joint Intelligence Committee admitted, in a top secret paper on "The fundamental factors influencing the formulation of Soviet foreign policy and the principles which now appear to underlie it," that it could draw on no relevant intelligence. The analysis may be none the worse for lacking a secret infrastructure. Gossip, prejudice and professional bias play an even greater role in the evaluation of motives and intentions than they do in the evaluation of an opponent's military machine, where at least some of the facts are quantifiable. By the 1960s it was already becoming clear that the Soviet Union was a military giant, but an economic and political pygmy. But as late as 1988, some-including some with direct access to western intelligence-were arguing that Gorbachev's perestroika was another plot to bamboozle the west. From time to time a spy may achieve a real coup: he may find the secret plan, or worm his way into the confidence of the dictator's mistress. But that is no guarantee that the dictator will do what he has boasted about in bed. People change their minds and act irrationally.
In the end, the problem is simple. The future has one charming characteristic: we do not know what it will contain. Intelligence agencies are no more able to predict the unpredictable than astrologers. In August 1991 the Moscow coup came upon Britain's Joint Intelligence Committee (and indeed upon me, then British ambassador in Moscow) "out of a blue sky." A couple of years later, the CIA predicted a war within six months between Russia and Ukraine: the prediction leaked, and was spread all over the world's newspapers. Five years later, the two countries ratified a friendship agreement. But "intelligence failure" is neither unexpected nor particularly blameworthy. Like the rest of us, secret agencies can only deal in best judgements.
the cold war was the apogee of the spying business. But there may be less than meets the eye even to the achievements of those heroic days. As the files open, scholarly work is beginning to put this period into perspective. The intelligence agencies on both sides demonstrated remarkable ingenuity. On both sides there were brave men and villains. They not only gathered information, but tried to influence events. Each agency boasted of its"agents of influence." But the record of both sides was uneven. In 1953 the CIA succeeded in overthrowing Prime Minister Mussadeq of Iran, after he nationalised western oil interests. But the CIA bungled the invasion of Cuba and its plans to assassinate Castro collapsed in comedy. Both the KGB and the CIA had their covert successes in Afghanistan. But the KGB failed to subvert west European countries in the decades after the war, despite the presence of large communist parties in France and Italy. They placed spies in high places in Bonn, but never impeded West Germany from joining Nato or hugely outperforming East Germany. And the Soviet imposition of communist governments in east Europe owed much more to the Red Army than to the KGB.
Most of the spy novels of those days were about the long war between the intelligence agencies themselves-the battle between Karla and Smiley. It sometimes seemed as if the energies of both sides were concentrated more on penetrating the other, than on discovering information of use to their governments. In the autumn of 1991, the sight of one western intelligence chief after another coming to Moscow to meet his former enemies was almost comic. Nostalgia among the professionals for the old days is still tangible.
Of course the secret agencies on both sides did provide valuable intelligence to their governments. The information helped in the management of crises such as Cuba in 1962. Occasionally they offered a useful back channel for negotiation. But the greatest contribution which the covert war made to undermining the Soviet Union was the invention of the reconnaissance satellite. This robbed the Soviet Union of the veil of secrecy which it had used to discipline its people and to hide its weaknesses. In the end it was the KGB, not the CIA, which contributed most to the collapse of the Soviet Union. The KGB's domestic task of preventing the Soviet people from having independent thoughts and uncontrolled relations with the outside world impeded innovation and reform without which the Soviet Union could not survive. The KGB's muddled performance during the 1991 coup succeeded in accelerating the collapse of the system it was supposed to defend.
So, we should not exaggerate the role of the secret struggle in the ebb and flow of the east-west conflict. The Soviet Union collapsed for two reasons. First, the US had the will, and the economic and military muscle, to win the cold war against the Soviet Union as it had won the hot war against Germany and Japan. Second, the Soviet political, economic and social system was unviable and incapable of reform. The Soviet attempt to steal military technology from the west was a sign of its inability to compete even in the military sphere where it seemed strongest.
the intelligence agencies and their supporters know that things have changed dramatically and that they now have to justify their existence as never before. They argue that we still need to know the capabilities of a potential enemy, lest we should find ourselves having to fight him-as we have done recently in the Falklands, Iraq, Bosnia and Kosovo. Moreover, they argue that there is a new enemy-terrorists, drug smugglers and dealers in weapons of mass destruction. Only through secret operations can we penetrate their secrets and disrupt their activities. As our enemies use modern technology with ever greater sophistication, we need to make an even greater effort than before, to counter them.
These arguments are justified. But they give no guidance about the kind of money that the taxpayer needs to spend to achieve such objectives. Nor do they provide any guarantee against the fraud, waste and abuse which always flourish where daylight is not allowed to penetrate. Finance ministries have tried to impose a simple test. If the secret information is so valuable, they say, why not make the consumer in other government departments pay for the bits that they use? Then we would know what was worthwhile and what was not. This approach does not commend itself to the ministries, nor to the agencies themselves. Secret expenditures in all the contending countries fell substantially when the cold war ended. They are now creeping up again. The British still spend substantially more than their continental partners. This, no doubt, reflects our greater competence in these matters. It is also part of our subscription to the "special relationship" with the US, whose substance-such as it is-lies above all in close military and intelligence cooperation. Whether we shall want to continue that relationship and retain the full measure of those expensive competencies is a question of national will and national self-image. Things could change as Britain adjusts to the uncertain post-Kosovo Europe of the next millennium.
But the larger question is: how can democracies control not only the spending, but also the activities of agencies which are meant to operate in secret? How can they ensure that the agencies act in the general interest, rather than pursuing their own agendas? Most people in most countries think that spying on foreigners is acceptable. But governments also run spies at home: security services to guard against the enemy within. People are more ambivalent about these. And for the secret policemen the temptation is always to set themselves up as the arbiters of what is and what is not a threat to the state. They know, because they have access to arcane knowledge which is closed even to their political masters. They know that traitors can be found even in the seat of government itself. In open as well as totalitarian states, the secret policemen can come to regard themselves as the guardians of the state-above the law, the leader or the party. Even today there are Russian secret policemen-serving officers-who believe that they can justify the horrors of the past, including the 1937 purges, as a necessary defence of the state.
In Britain, improbably enough, it was Margaret Thatcher's government which first began to tackle these questions in a systematic way. In 1985 it set up an independent tribunal to deal with complaints by the public about eavesdropping. In 1993 details about the three intelligence agencies-the Secret Intelligence Service (SIS); the Security Service; and the GCHQ, the body which intercepts the secret communications of foreigners-were finally published. The heads of the agencies were named and glossy pamphlets were issued. The hitherto "secret vote" was published-aggregated to the point of non-communication. The SIS and GCHQ were given legal status for the first time, lagging a few years behind the Security Service, whose head, Stella Rimington, the first woman to run a secret agency, had already become a media figure. A system of parliamentary oversight was created, modelled on the arrangements which had existed in the US Congress for many years. A commissioner was appointed, with wide powers to call for the most secret documents and to hear complaints from the public about alleged abuses by the agencies. Civil libertarians say that these arrangements are not perfect. Of course they are not. Because of the necessary secrecy in which the agencies operate, there will always be an inner circle into which the outsider may not penetrate. But the new arrangements are a decent first step.
Those who agree with Clausewitz argue that the huge sums spent by governments on secrets would be better spent on new hospitals. With the end of the cold war, they say, the spying profession should be allowed to fade away. This goes too far. We still need the intelligencers and their agencies. The task of the British public is to ensure that they remain competent, relevant, under control and reasonably cheap. It is not a simple task. A less sensational public debate would be a good start.
