Long live the database state

Brian Jarman, emeritus professor of general practice at Imperial College, is a kindly looking man. He is famous in medicine for designing the statistical model that was used to pay GPs (the Jarman Index); leading the campaign to keep Barts Hospital open; and, 15 years ago and perhaps most notably, devising a way of predicting how likely patients are to die in hospital. That analysis revealed that death rates in England, even when controlled for variables like a patient's age or class, differed by up to 76 per cent.

Death rates are not universally accepted as the best way to measure a good hospital. But tragic events keep bringing the issue back onto the agenda. In 2001, the Kennedy inquiry into children's heart surgery at the Bristol Royal Infirmary found that up to 35 babies may have died unnecessarily during the early 1990s. The inquiry called for more surveillance of hospital performance. This year just such an analysis uncovered another scandal at Stafford General Hospital, where receptionists rather than nurses were deciding which patients needed urgent treatment when they came to A&E. The healthcare regulator found that 400 lives may have been lost; deaths which smarter analysis of data might have prevented.

Yet the spread of such analysis in our health service has been painfully slow. Because Jarman's original methodology used data taken from confidential patient records, the secretary of state for health had to give formal consent for its publication. This was refused for years. Politicians of both parties were nervous: how would the public respond to evidence that the NHS is a dangerous postcode lottery? Only in 2001 did health secretary Alan Milburn take a different view and authorise the publication of the hospital standardised mortality ratio (HSMR) in the Good Hospital Guide. (My company, Dr Foster, publishes this guide and provides other data about the quality of local health services.) Since then, better use of data has slowly begun to change the ways the NHS is run. Walsall hospital, for instance, had the highest death rate in 2001, recording 1,080 deaths when it should have had only around 830. By 2004 it had improved its performance dramatically through a number of innovations in clinical practice and probably saves more than 275 lives a year as a result.



Despite such early successes, we still don't routinely know how good our doctors are—or, for that matter, our teachers or social workers. Ten years ago, we had national hospital data. Today, that is still all we have. No similar data is published on GPs or social care, and precious little on whether health is improved by the NHS. One senior NHS manager told a meeting in London recently that, in her view, a third of GPs in this country were good, a third average, and a third so bad "I wouldn't send my dog to see them." The problem is that we don't know which third is which. Yet much of the data needed to find out already exists. The problem is that it is not used.

Every time we interact with a public service we leave a record—a medical report, perhaps, or an exam result. Those records should be the lifeblood our public services. If shared and analysed—securely—they can help services to improve the quality of their performance (see "Good Data Improves Health," p42). They can also help to prevent problems: identifying, for instance, those at risk from diabetes or even child abuse. And the economic downturn brings new urgency: if public services don't improve productivity they will soon be unaffordable. Both Gordon Brown and David Cameron seem to support the view that data sharing is essential for leaner, fitter government. But in practice such data remains unused, and our public services remain unsafe and inefficient.

Overcoming this problem means taking on the powerful civil liberties lobby, which is against data sharing of almost any kind. One of its leaders is Ross Anderson, professor of security engineering at Cambridge and chair of the Foundation for Information Policy Research. In 2004, the student newspaper Varsity named him Cambridge's "most powerful person." Earlier this year he co-authored an influential report, "Database State," (Joseph Rowntree Reform Trust) which argued that for the government to hold "information on every aspect of our lives" is a real threat to civil liberties.

Anderson has an encyclopaedic knowledge of government IT systems and the danger, in his view, that they pose. His central thesis is that most public sector databases are insecure, probably illegal, and infringe human rights. Even if one accepts that doctors must have access to electronic patient records, Anderson thinks the data should not be centrally held in databases or shared between public services. It is true that government departments and ministers have lost such data. But there has yet to be a case, according to John Suffolk, the government's chief information officer, in which this data loss has harmed a single person. Such incompetence cannot be excused, but it is not Anderson's main argument against data sharing. His position is not just that confidential data should not be shared, but that anonymised data should only be shared in exceptional circumstances, and that data sharing should not occur unless people choose to opt in.

When a patient goes into hospital, a record is kept at the end of their bed and on a computer in the hospital. Once they are discharged (or have died), this record is entered into the local computer system. Each month this data is anonymised—all the things that identify the patient are stripped off—and transferred to a central database. The same kind of data exists in all our public services. But to be useful such databases must be able to compare all available records. Thousands of people each year go for a knee replacement; very rarely, a patient dies as a result. Any such death warrants careful investigation. But if that patient had opted out of the NHS system of analysing its records even anonymously (as Anderson wants), the hospital would be incapable of investigating the individual case, and no one would be able to compare the efficiency of knee replacement surgery in general.

Why does Anderson think that patients must give their explicit consent to providing even anonymous data? He cites this possibility: find a celebrity in a magazine who is in hospital with a rare condition, hunt through the anonymous data for such admissions, then phone the hospitals on a pretext and you might locate the famous person. But there has never been a case in Britain in which someone given access to anonymous patient information has used it to intrude into someone's life—if such a case existed we would all know about it.

Underneath this debate lies a clear tension between two competing social goods: the desire to defend civil liberties and the need for better public services. Anderson's strong arguments for the former risk undermining the latter. Yet the small risks of a government holding data on citizens are greatly outweighed by the potential benefits. Certainly, the public take a more mixed view than Anderson. On the one hand, 62 per cent want public services to share individuals' information "so that they can get a quicker, more personalised and efficient service." On the other, as Ben Page of pollsters Ipsos-Mori notes, 71 per cent also say they don't trust government to maintain their privacy. "The operative word is government," Page told me. "People don't like government, but they do trust public services and they want them to get better." Most of us already trust private companies with astonishing amounts of our data—much more intimate, in some ways, than the public sector would ever need. Still, Anderson's position strikes a chord. The 20th century is packed with examples of states abusing data to persecute minorities or run programmes of social eugenics. Many big centrally commissioned IT projects have been inept. And many doctors would agree with his claim that: "In medicine, you have to make sure that the systems are responsive to needs. This is not like a McDonald's franchise. Medicine is so complex that you can't [have a "one size fits all" electronic medical record]."

Yet without basic information lives will continue to be lost in our hospitals and public money wasted on inefficient services. There is another very powerful argument, too, for systematic use of anonymised personal data in public services: to target scarce resources at the people who need them most and reduce widening social inequalities.

***

Richard Webber more or less invented the art of population profiling—a technique that uses data to analyse our attitudes and behaviours and predict what we might do in the future. This is known as "customer insight" and it is dependent on analysis derived from personal data. Webber, now a visiting professor at King's College, London started his quest to give public services a sound evidence base in the 1970s when he worked at a social research unit in Liverpool. He discovered that there are some things that we share with people who live in similar neighbourhoods (such as susceptibility to disease or owning similar cars) and others we do not (such as criminal behaviour or aptitude for different sports). He likes to show people photos taken at random in different streets, which he has sorted into "herds" by the data. It is eerie how similar some are, from choice of houses to the style of patios.

Webber's model finds herds by linking people and postcodes. When Margaret Thatcher closed his research unit he moved into the private sector to work for Experian, the credit-checking agency. Here, he invented Mosaic, a system incorporating anonymised data from more than 400 sources which generate different postcode maps that predict people's behaviour—will this "herd" buy designer clothes or save more than another? Mosaic's 61 herds—from captains of industry to white van man—allow supermarkets to target stock to local populations, and let Saga write tailored letters to every man and woman in the country as they turn 50. It has revolutionised the retail and direct marketing industries.

In the same way, public services should be able to predict who is most likely to want to give up smoking, be at risk of diabetes or play truant from school. Armed with such data the public sector could intervene earlier, prevent problems and ultimately save money. In the US this approach is common. Jonathan Lord, former director of the US health insurance company Humana, says it is the equivalent of "laser-guided weapons, minimising collateral waste of public resources." In Britain it is largely ignored, despite the fact that a decade of heavy investment hasn't really helped public services to reduce social inequality. Today infant mortality rates in parts of Birmingham remain worse than in Bangladesh. You are three times more likely to be admitted for emphysema if you live in Tyneside than in Dorset. Modern public services still don't improve the quality of life evenly. And we should be looking to data to help find a solution.

Today some public service entrepreneurs are experimenting with these approaches. Birmingham East and North primary care trust, for instance, is working with my company to build a model of its population to predict local health behaviours. The idea is that such data can help the NHS stop those at risk of chronic illnesses actually contracting them. Others like Northampton PCT are working with private healthcare providers like United Healthcare and Bupa to develop similar approaches. Such early intervention should reduce the need for expensive hospital services later on. Research by Tower Hamlets council recently discovered that local Bengalis with minor injuries tended to go to hospital rather than visit their GP. Further investigation revealed that this was because many Bengalis preferred to see a clinician in a white coat. An advertising campaign promoting the competence of GPs helped to reduce attendance rates at A&E by 6.4 per cent.

Such preventive measures are also hindered because local public services aren't allowed to share data between themselves. Patient records, for instance, aren't linked to the benefits data. As a result local authorities do not know which GPs commonly sign people off work. Yet if it was known that one GP had, for instance, an unusual number of patients claiming incapacity benefit because of mild mental illness, the health service could work with that practice to introduce more effective workplace health schemes to help people back to work—or design better mental health services. Currently nobody knows which patients also claim benefits—so nothing is done. Indeed, legislation prohibits data-sharing of this sort.

Free school meals are another example. At present, a parent whose child qualifies must apply—a process that takes weeks because local authorities aren't allowed automatic access to the benefit records that confirm eligibility. Such parents could only benefit from an online service which, like a credit card application, was able to check their status instantly. It is just this lack of data sharing that, in general, is holding up our government's ability to exploit digital technology and make public services more personalised. Ultimately, we should be aiming to link as much local public service data as possible—crime, education, benefits, health and so on—to identify those who need help. If the health service operated as effectively as the direct marketing sector, it might well survive for another half century on the resources that it has.

***

Data sharing has become an increasingly political issue. In early July, the Times ran a front-page story under the headline "Google or Microsoft could hold NHS patient records say Tories," reporting Conservative plans to allow patients to store their medical records with private companies, not the NHS. But the article also implied that David Cameron was considering requiring patients to choose to "opt-in" to such records—in other words, to give people a veto on state access to their personal data. Senior Tories downplayed the story but it seems that the party is debating this behind the scenes. The Tories say that they want to put more information in the hands of citizens, ending the public sector's de facto monopoly of public sector data. That is the right direction to travel. But we must not abandon national, comparable records that help managers analyse and compare performance. Cameron himself recently called for an "information revolution" in health, saying that he wants the NHS to be able to answer questions like: "Where can my mother get the best treatment for Alzheimers?" These questions can only be answered if there is a universal national patient record in the NHS.

If the next government, of whichever party, wants a better public sector it must encourage more use of personal data; not less. What should be done? Data sharing must be made easier, first by removing the legislative obstacles to sharing government databases. The government should also pledge to publish as much new anonymised data as possible—most importantly by providing access to data on doctors and giving the NHS access to data on benefits. More should be done to encourage businesses and charities to turn this data into things that people can use—from websites rating GPs to maps of local crime. And the government should be ready to take on those lobby groups, such as the British Medical Association, that have stood in the way of using data to rate public services. People should be allowed to share their personal data with whom they wish, be it a small charity or a giant like Google. But no one who uses a public service should be allowed to opt out of sharing their records. Nor can people rely on their record being anonymised— at the moment sexual health services can be anonymous, and as a result there are almost no measures of performance in that sector.

If we want to have good public services, we are going to have to trust them with our data; and if our public services want us to pay for them, they will have to show us that they are using our data effectively and securely. But armed with the type of data created by Brian Jarman, and the tools built by Richard Webber, we can build safer, cheaper public services that know their users better. In October 2007, Gordon Brown said: "A great prize of the information age is that by sharing information across the public sector—responsibly, transparently but also swiftly—we can now deliver personalised services for millions of people." He was right. It's time he—or a future Tory government—delivered it.