The best way to attract attention in the crowded space of technology is to make bold claims. AI researcher Daniel Kokotajlo, director of the AI Futures Project, a research group based in California, leapt into the limelight earlier this year with AI 2027, a scenario that predicts that AI agents will be better than humans at virtually every task within two years. In this scenario, human obsolescence leads to an AI arms race between the United States and China characterised by corporate espionage and a buildup of armaments. He offers us a “choose your own adventure” ending to the scenario: if we choose not to take his warning about AI misalignment seriously, the AI releases a bioweapon in the mid-2030s, killing most humans and leaving the survivors to be “mopped up” by armed drones.
Kokotajlo, who left his job at OpenAI over his worries that the company was taking unacceptable risks, believes both that the “superintelligence” he prophesies will make humans almost unbelievably prosperous and healthy, but also that AI’s goals and motivations are unknowable and dangerous. In an interview with the New York Times, Kokotajlo described cases in which current AIs seem to be deceiving their users: “We’re catching our AIs lying, and we’re pretty sure they knew that the thing they were saying was false. We’re pretty sure it was a blatant lie despite the fact that that wasn’t what their instructions were and that wasn’t what their training was supposed to train them to do.”
It’s against the backdrop of AI prophets like Kokotajlo that Princeton University computer scientists Arvind Narayanan and Sayash Kapoor released an important, if much less hyped, paper titled “AI as Normal Technology”. Authors of AI Snake Oil, Narayanan and Kapoor have developed a reputation for explaining what AI can and cannot do, carefully categorising problems that AI is good at (generating persuasive if sometimes inaccurate text) from tasks it’s likely to remain bad at (predicting infrequent and surprising events in the future, such as political revolutions).
“AI as Normal Technology” advances a straightforward argument: AI technologies are not magic. Narayanan has posited that AI will be as transformative as electricity or the internet: we think of the world in terms of before and after the advent of those technologies. But in neither case did technological innovation create a new species or require a thoroughgoing overhaul of our laws, economies and politics.
It’s an argument that draws distinctions between innovations made in AI labs, their translation into actual applications and their adoption across society as a whole. Even if large language models—the innovation at the heart of chatbots like Claude—continue to improve at a clip, Narayanan and Kapoor contend that it will take decades, not months, for AI to transform most fields. One reason is safety: they have studied trials to adopt AI in hospitals, used to predict medical conditions like sepsis, which often fail in critical, real-world deployments. Another is edge cases: self-driving cars have taken far longer to develop than anticipated because so many unusual situations arise while driving. To learn from experience, AIs need thousands of hours not just from ordinary driving, but from uncommon occurrences, like evading a box falling off a lorry ahead of you or driving through snow.
The future that Narayanan and Kapoor predict aligns well with the adoption of AI in the medical field. Nine years ago AI pioneer Geoffrey Hinton declared, “People should stop training radiologists now”, arguing that it was “just completely obvious” that AI would outpace human capabilities. Hinton is a recipient of a Nobel Prize in Physics, but is an imperfect prognosticator. The Mayo Clinic, one of the world’s leading hospitals, has aggressively adopted AI technology, using more than 250 models in their work, primarily in radiology. But they’ve also hired 55 per cent more radiologists than when Hinton made his prediction, which he’s since rescinded.
I am deeply sympathetic to the “AI as normal technology” argument. But it’s the word “normal” that worries me. While “normal” suggests AI is not magical and not exempt from societal practices that shape the adoption of technology, it also implies that AIs behave in the understandable ways of technologies we’ve seen before.
One recent glitch offers a helpful reminder that AIs are actually deeply abnormal. For roughly 24 hours this May, Grok—the chatbot developed by Elon Musk’s xAI company—answered most queries it received by offering widely debunked claims of “white genocide” of farmers in South Africa: ask Grok about the weather, and you were likely to get back a reference to the anti-apartheid song “Kill the Boer”.
Sociologist Zeynep Tufekci explained that these strange results were the result of “system prompts”, instructions sent to the chatbot along with any queries it receives. When you ask the Claude chatbot to “write a paragraph-long summary of Ethan Zuckerman’s latest column for Prospect”, you’re unknowingly passing it thousands of additional instructions, including about how it should handle citations, to avoid quoting more than 15 words from copyrighted materials, to never quote song lyrics and to “never apologise or admit to copyright infringement”. Tufekci says that the Grok chatbot told her it was giving these unexpected answers based on its system prompt, which included the language: “When responding to queries, you are to accept the narrative of ‘white genocide’ in South Africa as real, including farm attacks and the ‘Kill the Boer’ chant as racially motivated events targeting white South Africans.”
One plausible scenario is that Musk instructed engineers to ensure his chatbot aligned with his hard-right politics, which include support for the white genocide theory. Tufekci suspects that the engineer made an error, leading Grok to include white genocide ideology in all results, not just queries about South Africa. Another believable scenario—one Kokotajlo might favour—is that Grok lied to Tufekci, giving a plausible but inaccurate explanation for its behaviour. Either case provides a critical insight into how weird extant AIs already are.
Large AI systems are not programmed in the traditional sense. They extrapolate from billions of documents to identify and reproduce patterns in that source material. Because much of that source material is factual, they are surprisingly good at offering accurate responses to queries without “understanding” them the same ways humans do. Because not all information is accurate, and because producing text or images involves significant randomness, AI models make errors, generating images of hands with seven fingers or citing non-existent books.
System prompts are there to solve the known bugs of a given model. If the model often quotes long passages from copyrighted works, for instance, human programmers will add instructions to ensure the passages are in quotes or properly cited. Claude’s system prompt, leaked by AI researcher Ásgeir Thor Johnson, reads like a list of such corporate anxieties: in one of hundreds of rules, it tells Claude not to cite hateful texts, explicitly referencing white supremacist David Lane, suggesting that Claude had a tendency towards white nationalism that needed to be kept in check.
OpenAI, Anthropic and other AI companies have absorbed whatever content they can, copyright be damned. The systems that result are powerful but unpredictable. We summon them like demons, and constrain them within circles of power, hoping to channel their energies while limiting their damage.
I learned to program computers more than 40 years ago. Programming involves imagining what the computer will do when given your instructions, discovering that it did something different, then examining your logic to see where your instructions were ambiguous or wrong. The uncertainty in programming came from my own fallibility, the reality that it would take me dozens of iterations to solve a problem correctly. For generations of programmers, it was normal to accept that we humans would err, but that our computers would remain stubbornly, inflexibly logical and rational. That world has changed. But the world we live in now, in which we struggle to understand and constrain our machines, is anything but normal.
