Does the modern centre-left have an economic theory? It has come, reluctantly, to acknowledge the primacy of the market, but demands intervention to reduce inequality of outcomes and to improve equality of opportunity. The most articulate rationale for this economic philosophy is the doctrine of market failure. This notion was the centrepiece of Gordon Brown's most extended exposition of economic philosophy, a speech to the Social Market Foundation in 2003. The argument was then amplified in a book published a year later, entitled Microeconomic Reform in Britain. (The status of this volume, part economic treatise, part political tract, is curious. The author is given as "HM Treasury," and the editors are Ed Balls, Brown's right-hand man, and now in the cabinet as children, schools and families secretary; Joe Grice, a career civil servant; and Gus O'Donnell, then permanent secretary at the treasury and now cabinet secretary—with a foreword by Brown himself.) The market failure doctrine described in Microeconomic Reform has been the guiding principle of Brown's treasury. Sophisticated lobbyists have learnt to frame arguments in terms of market failure. The thesis also wins wide acceptance among economists, and is influential in Brussels as well as in Whitehall. In this essay I explain the content and origins of the market failure doctrine, why it cannot provide the economic philosophy which the moderate left is seeking and what the outline of an alternative might look like.
The central claims of the centre-left remain valid. Unaided, markets do not give acceptable outcomes to the provision of education and pensions, transport and health, because in these spheres choices are unavoidably political, in the sense that such choices are and should be based on collective decisions about the nature of society, not simply on the self-interested decisions of individuals. Nor is there in these areas, or in general, a dichotomy between the economic sphere and the political: far from being in opposition to the market, social and political dimensions of conduct are central to an understanding of how markets work. The doctrine of market failure fails to accommodate these considerations, and by conceding too much to market fundamentalists, loses both intellectual coherence and political resonance. The market failure doctrine is based on an imperfect understanding of why markets succeed, not just why they fail, and hence provides a misleading guide not only to when government intervention is appropriate, but also to the ways in which market forces can improve the operation of the public sector.
The market failure doctrine concludes that where these conditions are met there is no efficiency justification for state intervention in markets. Conversely, violations of these assumptions—monopolies, asymmetries of information, externalities and public goods (in the economists' sense)—provide a rationale for such interference. But once you start looking, it is possible to find minor violations of these assumptions everywhere. So, the argument goes, market failure must be measured against "state failure"—the difficulty government may encounter in achieving its own objectives.
Of course, politics and policymaking are too complex for any single doctrine—including the market failure doctrine—to encapsulate. And what politicians do is not always the same as what they say they do. But what they say they do frequently has substantial influence, direct and indirect, on what they do. And the redressing of market failure is what the Labour government says it does in economic policy.
To understand the basis of the market failure argument, a short excursion in economic theory is required. General claims about the efficiency of markets have been made for centuries—Adam Smith's "invisible hand" remark is often interpreted in this way. But only with the development of a more mathematical approach to economics in the 1950s were the claims of market efficiency precisely formulated through what came to be known as the "fundamental theorems of welfare economics."
The model that was developed frames the determination of economic policy as an optimisation problem. The objective is to maximise a "social welfare function," an aggregation of individual choices, and the fundamental theorems show that this objective can be achieved by a competitive market so long as the government gets the initial distribution of resources right. The conclusion that many economists of the centre-left have drawn from this model is that production and efficiency can be left to profit-maximising firms, while equity and allocation should be the responsibility of a government which oversees the distribution and redistribution of resources. Fairness is secured by the tax and benefit system, not by control of the means of production.
Along with this notion of separation between production (a job for the private sector) and distribution (a job for government) comes a strong emphasis on incentives. Individuals and companies are assumed to have identifiable, self-interested objectives, and the aim of policy is to devise structures to ensure that the pursuit of these self-interested objectives is consistent with wider social goals.
The common use of the term "market failure" goes back to a 1958 article by Francis Bator, a mathematical economist who later worked as an aide to Lyndon B Johnson. Bator emphasised monopoly, externalities and public goods as common violations of the assumptions of the fundamental theorems of welfare economics. Information asymmetry was added later after a revolution in economic thinking in the 1970s, which emphasised the role of imperfect information.
The level of abstraction implicit in this model is high. There is nothing intrinsically wrong with this —scientific method relies on abstraction. But care is required, in economics as in science, in connecting between models and the external world. Ordinarily, a failure of correspondence between the assumptions of a model and the world is a problem for the model, not for the world. When it turned out that light did not in fact travel in straight lines, no one suggested that the solution was that those tiresome deviations from Newtonian mechanics should be straightened out. And in the real world a variety of social and economic institutions—reputation, trust, co-operative relationships—have evolved to manage issues such as information asymmetry and externalities. Thus, these "market failures" are not failures of the market itself, but failures of a particular model of the market.
Basic philosophical differences divide left and right. What priority should be given to claims of individual rights and private property relative to those of solidarity and social justice? Is the public interest more than an aggregation of private interests? The model that underpins the market failure doctrine seems to answer these questions, and others, in the right's favour. A particular philosophy is inherent in the mathematics. The model takes individual preferences as given, along with personal resources and property rights, and sees social welfare as an aggregation of individual preferences. The primacy of material incentives as determinants of economic behaviour is not a prediction of the model, but an assumption.
These issues quickly become apparent in the application of the framework to the most pressing of policy issues—the management of health and education. Gordon Brown's commitment to a tax-financed NHS is well known. But the market failure doctrine requires that this preference be expressed in the language of market failure. This is what is done in Brown's SMF speech and in Microeconomic Reform. The principal market failure identified is information asymmetry: "people are unable to predict their future healthcare needs"; "consumers may lack sufficient information to make optimal choices"; there are "asymmetries of information between insurers and healthcare providers." While these statements are largely true, they do not explain how such problems are better resolved by Britain's publicly funded, publicly provided NHS than in the mixed systems of most other European countries. More importantly, they have nothing to do with the real reasons most people—including Brown—support a publicly funded NHS. These reasons begin with concepts like compassion and fairness rather than information asymmetry.
If illness were truly an unpredictable event, then private medical insurance would work better than it does. The major informational problem in medical insurance is not that consumers do not know their future healthcare needs, but that insurers know relatively well which categories are and are not likely to prove expensive. Private insurance is fine for healthy employees but not for the old and the poor. And while information asymmetry is endemic in health provision—if we did not think the doctor knew more than we do, why would we visit the surgery?—medicine has worked out effective social institutions to manage inequalities of information in the provision of complex products. Centuries of ritual and experience are designed to maintain, mostly justifiably, the faith of the patient in the doctor.
The rationale of public intervention is different. If individuals decide that they will not make provision for the emergency medical treatment of themselves and their families, we will not respect these choices and we will not let people die in the street. Staff and patients seek a caring NHS, in which they believe that the motives of providers are largely uninfluenced by commercial considerations.
Equality of provision matters more in healthcare than in the provision of most other commodities. These reasons explain why healthcare is almost completely collectivised in all developed countries outside the US and largely collectivised even there. And none of them have much to do with a conventional taxonomy of market failures. They are all the product of a widely shared political consensus about the nature of society which cannot find expression through individual choices in the marketplace.
The notion that some economic choices are essentially collective, and cannot be described as a summation of personal preferences, strikes at the heart of the market failure doctrine. But the problems of defining objectives go wider. While it is at least possible, though I think mistaken, to argue that the output of a health system might be measured purely by health outcomes—mortality and morbidity—the output of an educational system is plainly multidimensional: average attainment levels are important but not the only indicator that matters. Microeconomic Reform acknowledges the multiplicity of educational goals: "Imparting basic skills to children, preparing them for work, instilling ideas of citizenship, and fostering emotional growth. Although they are not contradictory, these goals compete for resources." From this complexity, the argument runs, the economist as social engineer must find something to prioritise.
Up to a point. Imagine the debate which would establish those priorities. I think that emotional intelligence is crucial; you emphasise workplace skills. How do we resolve our disagreement? By democratic vote? Or perhaps these objectives should receive weights: 40 per cent to one, 30 per cent to another. If we can only describe all the objectives, and agree on the weights, we might construct an index of educational attainment. But no one imagines that policy is, or could be, determined in this way. The multiple objectives of education are neither compatible nor conflicting, but ultimately incommensurable. The term incommensurability is anathema to economists, for whom it is an article of faith that there are always trade-offs. But what if, for example, the people who say that they are not willing to put a value on human life—as expressed through public spending—are telling the truth? Behind every consistent choice, the theory goes, lies a clear expression of priorities, even if we are not consciously aware of it. But, as every politician must quickly recognise, consistency is not always a feature of either individual decisions or collective choices. Perhaps the real inconsistency is that of the economist, who insists on consumer sovereignty but rejects individual judgements that are not compatible with the stable, well-defined structure of underlying preferences which is the basis of the fundamental theorems of welfare economics.
The market failure model supposes that a social welfare function can be defined, and that it should be the basis of policy decisions. The existence of such a function requires that not just for individuals, but for society as a whole, there exists a coherent, consistent preference ordering which politicians must first divine and then maximise. But this formulation is a misunderstanding of the nature of choice in a democratic society. There is no plausible means of constructing such a preference ordering—not by the aggregation of individual preference, nor by the imposition of some idea of a general will—only a process of mediation between often compatible but sometimes conflicting views on specific policies. The modern political context is one in which a balance must be struck between different purposes. Sometimes we have moved too far in one direction, sometimes too far in another. Education policy might have emphasised emotional growth too much at the expense of preparing students for work. A decade from now, these issues will no doubt look different. The politician who enters this messy debate with clear priorities is a figure with whom one can sympathise, but whose instincts are properly restrained by the institutions of a pluralist democracy.
If the economic theory of the centre-left must accommodate collective choices, it must also acknowledge collective actions. But just as the market failure model interprets collective choice in terms of the aggregation of individual choices, so it interprets collective actions as the aggregation of individual actions. But armies, production lines, hospitals, railways and schools function because people work as teams. And in appraising the performance of teams, issues of incentives are relevant, but only part of the story. There is a large middle ground between the excessively utopian belief that everyone will do the right thing once they perceive what it is, and the excessively cynical assumption that everyone is principally driven by the prospect of personal financial gain. It is in that middle ground that both the public and private sectors of modern economies function.
People have many motives for working—money, of course, but also pride in the job, the respect of colleagues, family and friends. Employment contracts are most successful when they make use of all these motivations. Taylorism in the workplace failed: even in the automobile industry, crude incentive systems proved unsuccessful and were abandoned because workers had no concern for the quality of the product. Purely instrumental motivation created dismal workplace relations, and negotiation over the terms of the bonus schemes proved interminable.
And in the last decade, Taylorism has failed in the boardroom too, for essentially similar reasons. The elaboration of complex incentive systems for top managers distorted motivation—among both those who benefited from the schemes and those who did not—and attracted into executive positions the relatively small number of people for whom personal greed was indeed an overriding motivation.
People make careers in education and medicine at least in part for non-materialistic reasons, and we want it to be that way. The motives of those who deliver treatment matter, because what we want from a health system is not simply pills and surgery, but the ministrations of people who care. Much medical treatment is not very cost-effective in terms of health outcomes, and a high proportion of total spending goes on the last year of life: but the economic approach to health policy, which measures output only in terms of "quality-adjusted life years" gained, fails to recognise that even in matters of life and death we care about processes as well as outcomes. The provision and receipt of healthcare is a social activity. The misapprehensions that put financial incentives at centre stage explain how, as in the recent mishandling of GP contracts, the state has spent so much more on health services only to reduce the morale of those who work in them.
The market failure doctrine, dedicated to reforming the world in line with the model, has led to inappropriate policies for public sector reform. It has led to policies that burden consumers with information they do not want and cannot understand, instead of ensuring that the trust they wish to place in their suppliers is well founded. And it has led to systems of contracting or decentralisation that fail to recognise that people value freedom of action for its own sake, and not just for associated financial incentives. The attraction of fundholding to some GPs, or of school budgets to many headteachers, is not just that they might make a few bob out of it, but also that these mechanisms give them greater autonomy. The market failure approach directs our attention to secondary issues of information asymmetry and fails to recognise the more fundamental reasons—the primacy of collective decisions and collective action—that make unmoderated market forces inappropriate means of providing health or education.
Moreover, the policy relevance of the market failure doctrine is limited by another, perhaps more subtle, problem of connection between the model and the world. The resources and endowments which economic agents trade are not given by nature, as the model implies, but established through social process. In the simple models of exchange I used to teach, apples grow in my garden, pears in yours, and since I want some pears and you some apples, there is room for mutually beneficial trade. The tax and benefit system might take some of my apples, or give me your pears, but there is no room in this model for argument about either the definition of apples or pears or about our initial entitlements.
But few exchanges in modern economies have this simplicity. The nature, as well as the distribution, of property rights is a matter of social construction and political debate. The majority of contested economic policy issues reflect disputes about the nature of entitlements, or they occur when parties look to government to fill in the implicit terms of imperfectly specified contracts. Arguments about whether goods are in fact apples or pears, and whether they grew in my garden or yours, are a major part of political debate. Below is a list of recent policy issues, each of which concerns either the nature of the default position—what rights do individuals have when they begin the market process of exchange?—or the content of complex relationships, in which it has proved impossible to specify every contingency in advance, and one or both parties looks to a political process to resolve their entitlement.
Top-up fees: what rights do individuals have to higher education, and on what terms? Problematic issues in retail financial services: what were the implied terms of endowment mortgages, Equitable Life policies, defined contribution pension schemes, and what responsibilities does government assume for a regulatory agency which is inept in monitoring the solvency of financial institutions? The funding of long-term care of the elderly: if the costs associated with uncertainties of medical treatment are largely socialised, how does this judgement apply to costs associated with longevity in general, or to abnormally costly longevity? Executive remuneration and corporate governance: what is the proper balance of rights and responsibilities between managers and other stakeholders in large corporations?
Not one of these questions, which are typical of everyday debate in microeconomic policy, can be resolved either by leaving it to the market or identifying a market failure from the conventional list. Each arises because property rights are not an external "given," but socially constructed. Once this is acknowledged, the apparent power of the fundamental theorems of welfare economics dissolves. If government is called on to define entitlements, or to regulate agreements which have not worked out as planned—and this is a large part of what economic policy is about—then there is no alternative to pragmatic, issue-specific assessment of how best to achieve the simultaneous goals of efficiency and equity, and the attempt to separate these objectives must fail. The economist who brings to these questions his standard toolkit of individual rationality and market failure, is not wrong, merely irrelevant.
Yet the most serious weakness of the market failure doctrine is that its model provides not just an inadequate account of how markets fail, but also of how they succeed. The model provides a partial explanation of one of the most striking features of market economies—their capacity to achieve co-ordination without a co-ordinator. It is remarkable, and wildly counterintuitive, not only that the question "Who is in charge of the supply of bread to New York?" has no answer, but that the supply of bread to New York is better managed by a system in which there is no answer than by one in which there is.
But the endemic deficits and surpluses of a planned economy which is unable to co-ordinate large quantities of information in a rapidly changing environment are only part of the explanation for the failure of central planning. Similarly, with the suppression of incentives; in reality, few societies have offered as wide a range of incentives as the Soviet Union, from the privileges of the nomenklatura to the perils of the gulag. The main failing of planned economies was that they could not accommodate the flexibility needed to cope with an uncertain future. This issue is not one the market failure doctrine can easily recognise, since the underlying model does not recognise innovation and uncertainty, except in trivial ways.
If the partial genius of market economies lies in their capacity to achieve co-ordination without a co-ordinator, the greater genius lies in their ability to innovate and adapt in an environment of uncertainty and change. The sustained achievement of market economies comes from their pace of innovation—in products, technology and organisation—derived from the ability of market systems to undertake small-scale experiment, to watch the results, to mimic what works and discard what doesn't.
This is the mechanism which I describe as "disciplined pluralism" in my book The Truth about Markets—a mechanism which allows a multiplicity of small-scale experiments, and in which the successful experiment is quickly imitated while the unsuccessful quickly folds. Such disciplined pluralism is an inherently inefficient process. It relies on constant displays of irrational optimism, and most of its experiments fail. Rationalist bureaucracies detest such pluralism. They are run by micromanagers, people at the centre who feel able to set clear priorities in education and believe that the information asymmetries inherent in healthcare can be addressed by nationalising its provision.
The historic error of old Labour has been to conflate the need for collective choices and collective action with central direction and political control. A successful team achieves common goals via co-operative activity effected through individual decisions. This process, neither market nor hierarchy, characterises the effective sports team or school, workshop or military unit. The trick that disciplined pluralism—decentralised choices with accountability—achieves is to replicate that combination of free choice and co-ordinated outcome throughout the economic system. That insight—the economics of Friedrich Hayek (concerned with the dynamic capacity of a market economy to experiment and innovate) rather than of Milton Friedman (concerned to promote the allocative efficiency of competitive markets to attack all kinds of state intervention)—is the lesson the left needs to learn from the right.
Failure to understand this has been New Labour's economic mistake. There is something of the zeal of the convert in the embrace of the market. Taken as a whole, market economies have proved far more effective than planned economies. But this does not imply that every individual market outcome is superior to every individual planned outcome.
A centre-left economic policy should recognise the primacy of the market neither as an ideal nor a necessary evil, but as the best pragmatic solution to a wide range of economic problems. Efficient production and allocation is a mixture of individual and collective choice and action. The shape of our transport infrastructure needs to be a political decision, but the railways need to be run by professional managers accountable to customers rather than political appointees responsible to ministers. Major inequalities in health provision need to be addressed, but ministers cannot know and should not decide which treatments should be provided. Education requires public funding, but no one has the capacity, or should have the power, to "set clear priorities."
The market is a tool, not a fount of wisdom. Earnings are not a measure of desert. On balance, open capital markets are beneficial, but that does not mean that every foreign takeover benefits the British economy. Free trade is massively superior to its alternative, but entails job losses and wage reductions for specific groups unless policies alleviate the consequences. Mergers generally have more to do with management egos and investment banking fees than a better allocation of economic resources. Markets don't do the basic research, or the training that isn't job-specific, on which the innovative capacity of economic systems depends.
The list of market failures is a guide to some common problems in economic policy. But it is no more than that, and it does not provide a proper account of the limits of markets, or their strengths. The notion that the boundaries of markets can be precisely defined by regulation and contract has been common to both the New Labour left and the Tory right in the last decade. But a precise definition is not always possible. There is, and should be, a large grey area in which disciplined pluralism, which is the strength of market organisation, coexists with political control.
In that world, educational goals are not determined either by central state direction, or by the simple aggregation of individual choices. Both are relevant and neither is sufficient. Multiple goals emerge, are different across different parts of the educational system, and evolve over time, through an interaction between those who provide the service and those who pay for it. In a similar way, medical treatment must be managed through trust relationships between politicians, professionals and patients. The reason for providing choice of provider in health and education is not that consumers see inherent virtue in choice, but that the ability of consumers to exercise choice raises standards. There is no wide disagreement about what constitutes a good school or hospital: but the most effective means of getting a good school is to be able to reject a bad one, and individual choice is far more effective in achieving this than collective choice—exit trumps voice. The reason for promoting competition between providers is not that conflict is better than co-operation, but that recognising success and failure is indispensable to innovation and imitation.
Incentives matter; but it is not only individual incentives which matter, and the failure to recognise this has given us both corrupt corporate bosses and demoralised public sector workers. So long as market organisation is equated with individual greed and jingling cash registers, the limits to markets will in practice be set by the determination of the public to keep them away from those areas of human activity—like health and education—that matter most to them. And that would, indeed, be a market failure.
