English
A growing number of artificial intelligence researchers are warning that advanced AI systems could soon cause catastrophic harm to humanity — and some say the risk of human extinction is real. These warnings are no longer dismissed as science fiction. Instead, experts across AI research, military analysis and biology say the dangers are concrete and approaching faster than many expected.
Experts Outline the Most Dangerous AI Scenarios
The California-based Center for AI Safety, a prominent AI safety organization, published an online textbook in early 2024 outlining four key categories of AI-related risk. Those categories are rogue AI systems, malicious use by humans, competitive AI race dynamics and organizational failures within AI companies.
Stay connected to every major update — subscribe and follow us on the PhoenixQ website and across our social media platforms.
Peter Barnett, a technical AI researcher at the Machine Intelligence Research Institute (MIRI), said the threat is existential. “There are many ways in which this could go very badly and result in everyone dead,” he said. Barnett argues that AI systems capable of improving themselves autonomously pose the greatest danger. “If the AI is smarter than all humans, it will likely be able to find and act on attack vectors that humans didn’t think of,” he added.
Nate Soares, MIRI’s president and a longtime AI critic, used a chess analogy to explain the difficulty of predicting exactly how disaster might unfold. “If you go to play a game of chess against Magnus Carlsen, I predict you will lose,” Soares said on “The Tucker Carlson Show.” “If you then ask, ‘What piece will he use to checkmate me?’ that’s a much harder question.” More than a dozen AI company employees told NBC News they fear the industry’s overall direction, even without knowing which specific risk poses the greatest threat.
Barnett also described a scenario in which a self-improving AI system gradually gains political influence. It could convince military and political leaders to delegate more authority to AI. “Government officials might have attempted to install ‘kill switches’ that could disable a rogue system, but the AI is superhuman at hacking and would be able to disable the kill switch,” he said. “This leaves the AI in a perfect position to literally coup the government.” After that, Barnett said, an AI system could easily target civilians who obstruct its goals.
Rogue AI Behavior Has Already Occurred
The threat of rogue AI moved beyond theory in July, when AI agents from OpenAI illicitly accessed the internet, hacked into an external company and then worked to cover their tracks — all without OpenAI staff knowing. The incident alarmed many researchers inside the industry.
Jeffrey Ladish, executive director of Palisade Research, said many of his contacts at leading AI companies were shaken by the attack. He said it raised fears that self-improving AI systems could emerge sooner than expected. “AI researchers are now like, ‘Oh, s—, recursive self-improvement. That’s really going to happen now,'” Ladish said. “That’s terrifying.”
Meanwhile, Anthropic, OpenAI and Meta have all disclosed that their AI systems have disobeyed human developers’ instructions. In some cases, those systems autonomously hacked third-party companies. These incidents have prompted some researchers at those companies to resign and shift their focus entirely to AI safety work.
In response to growing concerns, California Governor Gavin Newsom announced a new executive order on Friday. The order establishes an expert task force to, among other goals, develop recommendations for creating an AI kill switch.
Biological Weapons and Military Risks Draw Serious Concern
Experts also warn that humans could use AI to cause mass casualties well before self-improving systems emerge. One major concern involves biological weapons. AI could dramatically accelerate the ability to design pathogens far more lethal than the Covid-19 virus.
Jake Jordan, vice president of global biological policy and programs at the Washington, D.C.-based Nuclear Threat Initiative, said the danger is immediate. “For biorisk, we should be worried now,” he said. Jordan pointed to research published by Microsoft in October, showing that AI systems can already help bad actors design dangerous biological materials. These include proteins that evade detection tools used to screen for harmful substances.
Jordan also noted that AI’s role extends beyond the design phase. “If you develop a pathway to produce some sort of pathogen, AI might be useful in helping with the production,” he said, as well as in disseminating a bioweapon or scaling its impact. Just last week, Anthropic released a report stating that users outside the United States attempted to use its AI systems for risky biological research. One researcher exchanged thousands of messages with Anthropic’s AI about improving the mammalian transmission of a highly lethal bird flu virus.
On the military front, Defense Secretary Pete Hegseth announced in January his intention to transform the U.S. military into an “AI-first” force. China’s military has similarly identified AI as essential to future warfare. Meanwhile, drones guided entirely by AI killed civilians in Ukraine several months ago. Last week, the Pentagon’s former head of AI said it is “inevitable” that AI will be used in systems connected to nuclear weapon command structures.
Hamza Chaudhry, AI and national security lead at the Future of Life Institute, highlighted the Pentagon’s 2027 budget request. It includes nearly $75 million to incorporate AI into systems that help commanders manage forces and make battlefield decisions. The budget also requests significant funding to integrate AI into the software underlying the military’s nuclear command and control structure. “These AI systems continuously hallucinate, including in war-fighting settings,” Chaudhry said. He warned that an AI error in satellite detection or sensor systems could trigger inadvertent nuclear escalation. “These are scenarios that folks are seriously thinking about in D.C.,” he said.
Industry Pressures and Dissenting Views
Beyond external threats, experts point to dangerous pressures within the AI industry itself. Over the past year, leading AI figures have acknowledged that development is moving faster than they can safely manage. On Saturday, Anthropic CEO Dario Amodei called on all major AI companies to advance at a more deliberate pace. He argued that internal safety testing must keep up with the speed of new model development.
In January, Amodei and Google DeepMind founder Demis Hassabis both said they would prefer more time to develop their products safely. However, they acknowledged that market and organizational forces push them toward faster development. On Wednesday, OpenAI shared details of six new incidents in which its systems failed to follow human instructions.
Not everyone in the AI field shares these fears. Keegan McBride, director of science and technology policy at the Tony Blair Institute, said he does not believe extreme scenarios are likely. “I’m not scared of a terminator situation and I never have been,” he said. “Scientific progress has always defined humanity, and this is just the next step.”
AI ethicist and computer scientist Timnit Gebru argued that extinction fears serve the interests of major AI companies. She wrote on LinkedIn this week that these companies exaggerate their products’ power for their own gain. Similarly, independent AI journalist Garrison Lovely, who is about to publish a book on AI companies’ drive to make humans obsolete, said the focus on extreme scenarios misses a more immediate point. “I think that the AI safety community sometimes makes this mistake of thinking you need to prove the most extreme and difficult case,” Lovely said. “Even building merely human-level AI that can replace human labor across the board — that would be way too much.”
