The Trolley Problem always struck me as a little dubious. First posed in its modern form in 1967 as a conundrum in ethical philosophy, it generally runs something like this. You see a runaway trolley (of lethal size) rolling down a track towards five prone people, all of whom will die if it hits. But there is a lever that, if pulled, will divert the trolley onto a side track where just a single person lies in its path. Do you pull the lever?
There are now many variations. You could stop the trolley instead by pushing a fat man off a bridge into its path. You could just press a button which activates a mechanism that pushes him over. The fat man is a fiend who has put these people in peril. The person on the side track is a young child. And so on. There are all sorts of potential problems with these thought-experiments—can they be mapped onto real-world dilemmas? Can there be a meaningful optimum among so many variables?—but nonetheless they are popular in moral philosophy. The point here is to see whether there is any ethical calculus that can tell you what to do.
Yet at a talk I chaired recently by Yuval Noah Harari, today's trending futurologist and author of the phenomenally successful book Sapiens, he argued that new technologies may take philosophical questions like this out of the cerebral sphere of the hypothetical and into the very real and harsh arena of the marketplace. “Even if you say in theory, yes this is the right thing to do, when you actually find yourself in that real-life situation, you very often behave in a completely different way,” Harari began. But if you are programming a particular decision into a machine, that is then precisely how the machine will behave come what may.
And this is what we are now having to do. For driverless cars, the Trolley Problem is no longer hypothetical. Faced with an impending and unavoidable accident, what should it do? What if the only options are to kill the driver and passengers or to kill pedestrians? “This is [now] a practical question of engineering, not a theoretical question of philosophy,” said Harari.
His answer? “Given that we live in a capitalist, neoliberal society, I guess somebody up there in Google or Toyota will say, let’s just leave it up to market forces. We’ll design and build two cars: the Toyota Altruist and the Toyota Egoist. And the customers will decide.” The audience laughed, but he was only half-joking.
Now researchers at the Massachusetts Institute of Technology’s Media Lab, a hothouse of wild and inventive ideas, have created a machine to help them. It is called the Moral Machine, and its purpose is to sample a wide spectrum of global public opinion for ethical guidance on precisely this problem.
“Never in the history of humanity,” write Iyad Rahwan and his colleagues in Nature, “have we allowed a machine to autonomously decide who should live and who should die, in a fraction of a second, without real-time supervision. We are going to cross that bridge any time now, and it will not happen in a distant theatre of military operations; it will happen in that most mundane aspect of our lives, everyday transportation.”
“For driverless cars, the Trolley Problem is no longer hypothetical”Rahwan affirms that the Trolley Problem comes into its own even for regular driving. “Some people like to dismiss these situations as extremely rare,” he says. “But they are missing the point. The Trolley Problem is an abstraction to capture the presence of trade-offs. Where you position a car in a lane or how aggressively you overtake can alter risks for cyclists versus passengers versus pedestrians.”
We make those calls instinctively according to our temperament. But if we’re programming such behaviour into an artificial-intelligence device, we need to make an explicit, conscious choice. The Moral Machine explores the range of opinion on that.
The principle is simple. The researchers created an online platform that posed Trolley-Problem-style accident scenarios and asked participants to indicate their preferences. For example: an autonomous vehicle has a brake failure as it approaches a road crossing. If it continues straight ahead, it will collide with and kill two elderly men and an elderly woman. If it swerves into a barrier, it will kill all three passengers: a man, woman and young boy. Should it swerve or not?
Already you can see the problem: the number of permutations is endless. In different scenarios the Moral Machine included young versus elderly, fit versus not fit, males versus females, pregnant mothers, criminals and doctors. The aim, though, was not to arrive at majority views on every scenario imaginable but to look for common priorities. The researchers wanted also to see if and how the choices depended on participants’ own demographics: age, gender, education, religion, and culture. The experiment attracted around 2.3m users, distributed worldwide but mostly in Europe and the United States, giving a total of 40m decisions from 233 countries and territories.
One can certainly imagine a survey of this kind being used to find real-world, implementable solutions that are likely to be deemed ethically acceptable by the greatest number of people. For example, there were clear preferences for sparing humans over animals, the young over the elderly, and the lawful over the criminal. No surprise, you might think—except that already those outcomes conflict with the one set of official ethical guidelines so far drawn up at the national level for autonomous vehicles, in Germany. These state that distinctions of preference based on personal features such as age should not be permitted.
Does this mean that such guidelines should respect public opinion? Not necessarily, say Rahwan and colleagues—but it does mean that when there is conflict, policymakers may find themselves called on to justify unpopular choices. But “this isn’t a new problem,” says Rahwan—“policymakers often have to negotiate conflicts of opinion between different stakeholders, and provide protections of basic rights even if the majority object.” (That’s one reason not to risk putting the death penalty to a referendum in the UK.)
Regardless of whether the Moral Machine might ever become a tool for guiding policy, it already supplies a way to explore cultural differences in ethical reasoning. The researchers say they could identify three broad clusters of similarity between countries, loosely categorised as western (North America and Europe), eastern (Japan, Taiwan and some Islamic countries) and southern (Latin America and some French territories). “Southern” participants were the most likely to spare younger characters, “eastern” the least likely. But “southerners” had much less inclination to spare humans in preference to (pet) animals.
Does this mean that autonomous vehicles might follow different ethical rules in different parts of the world? Is it, indeed, right to answer ethical questions at all by a show of hands? It’s not just moral philosophers but also, in the light of current political trends, many Prospect readers who may shudder at the thought.
One option might be to take Harari’s suggestion seriously: to provide vehicles programmed for different behaviours in these scenarios and let the customer decide. But that seems likely just to bring us up against the inconsistencies of human behaviour. In previous work, Rahwan conducted much smaller online surveys about rules for whether autonomous vehicles should preferentially sacrifice passengers or pedestrians. They found that, while participants were broadly in favour of sacrificing one or a few passengers to save several pedestrians, they wouldn’t be willing to ride in those vehicles, and neither would they approve of regulations that made such driver self-sacrifice mandatory.
Rahwan and colleagues are optimistic about the outcome of their massive online experiment. “We might not reach universal agreement,” they say, “but the fact that broad regions of the world displayed relative agreement suggests that our journey to consensual machine ethics is not doomed from the start.” Maybe so, and the experiment highlights the urgency as well as the complexity of the challenge.
But it could also be seen as an extension of the nascent discipline of “experimental philosophy.” Rather than regarding these results as instructions on how we ought to design our artificial intelligence systems, we might see them as data about human moral reasoning itself. Can we understand the cognitive factors that guide us to these judgments, and how they are influenced by culture and other demographics? Can we, in other words, reverse-engineer the answers so as to better understand how we frame and navigate the questions? Such efforts have already begun. Some years ago, for example, a team at Princeton University used MRI brain scanning to study people’s “emotional engagement” as they made judgments about moral dilemmas—based, as it happens, on the Trolley Problem.
