Armageddon is trending. Jacob Coxon, an artificial intelligence researcher, has resigned from the AI company Anthropic in a spectacular blaze of whistleblowing. He has told the Wall Street Journal that the leading AI labs (including OpenAI, where Coxon also previously worked) are careering towards ASI: artificial superintelligence. In a post on X, he warned that ‘the people building AI earnestly believe that it could kill us all by the end of the decade’.
Amid the panic and the accusations of panic-mongering, perhaps the biggest surprise has been how few people were already aware of this increasingly grim situation. AI experts have been sounding the alarm for years. Geoffrey Hinton, known as one of the ‘godfathers of AI’, resigned from Google in 2023 and later said that he believed the odds of AI causing human extinction were roughly 10 to 20 per cent within the next thirty years.
He’s not alone. Jan Leike, OpenAI’s head of alignment, resigned in 2024 claiming that ‘safety culture and processes have taken a backseat to shiny products’. Mrinank Sharma, formerly in charge of Anthropic’s Safeguards Research Team, quit in February of this year; he declared that ‘the world is in peril’ and announced his intention to study for a degree in poetry.
It is all too easy to disregard such statements as ‘doom-mongering’, on a par with climate alarmists such as Guy McPherson, the ecology professor who predicted that ‘there will be no humans on Earth by 2026’. We are so jaded with the Greta Thunbergs of this world that there will be a natural instinct to dismiss these concerns about AI as hyperbole. But these are not crazed activists with an irrational fear of technology. Many of the warnings are coming from people with formidable credentials, including researchers who have actually built frontier AI systems.
Coxon, for instance, is no keyboard warrior with a conspiratorial mindset. He has been working at the coalface of AI development and has been directly involved in training models both at OpenAI and Anthropic. It is worth remembering that Anthropic was founded in 2021 by former OpenAI researchers who left amid disagreements about the company’s direction and concerns over safety. No doubt there will be many who will be comparing the likes of Coxon to the Luddites, those who smashed up the new machinery that they feared would undercut their wages during the industrial revolution. But the comparison is flawed.
For one thing, Coxon is almost certainly correct about the extent of the concerns within the industry. Responding to his post on X, Evan Hubinger, alignment science lead at Anthropic, said that ‘we really do earnestly believe AI could kill all humans!’ He estimates that the probability is over 10% in the next ten years, and that there is, as yet, no plan ‘to solve alignment for superintelligence’. (‘Alignment’ is the problem of ensuring that increasingly powerful AI systems continue to behave in ways that humans can control).
For most of us, the idea that all of these AI experts are fully aware that they are creating the conditions for Armageddon but are continuing anyway makes little sense. If AI achieves superintelligence, it will be far more capable than us in virtually every area that requires thought. It will be able to produce its own superior models, effectively entering into a loop of intelligence generation. It will be so sophisticated that no human being could come close to understanding its abilities or decisions. In intellectual terms, we are talking about the creation of gods.
What these gods choose to do with us mere mortals is the key question…
— To continue reading this article, please consider becoming a paid subscriber —
