When asked privately what AI risk worries them most, many AI architects and leaders give the same answer: a lethal pathogen that spreads too silently, widely and quickly to stop.
What the research shows
A joint study by MIT FutureTech and the University of Queensland surveyed 272 researchers and asked them to rank 24 top AI risks. The respondents judged there to be about a 12% chance that dangerous AI capabilities will produce a catastrophic outcome by 2030. They also estimated a separate 12% chance that AI will enable weapons or mass-harm capabilities.
The study defines "catastrophic" as more than 1 million deaths or at least $100 billion in damage. Helping to create chemical or biological weapons was placed clearly among the highest-ranked risks.
Those probability estimates already assume mitigation efforts to reduce risk; the researchers warn that without mitigation the chances rise above 20%.
Why this matters
The relative consensus among experts is notable because such agreement is uncommon in the field. The core worry is that advanced models could map biological vulnerabilities and make it easier for an actor with malicious intent to identify genetic changes that increase transmissibility, evade detection, or defeat existing treatments.
Not every expert views the worst-case scenarios as likely, and some prominent optimists reject doomsday narratives. Still, as AI’s ability to understand biological systems improves—similar to how it has learned about large-scale cyber systems—the scenario becomes plausible enough to merit serious attention.
Leading voices have warned publicly
The concern has been voiced openly by industry leaders: last month Sam Altman (OpenAI), Dario Amodei (Anthropic), Demis Hassabis (Google DeepMind), Mustafa Suleyman (Microsoft) and Alexandr Wang (Meta) co-signed an open letter warning about the risks of AI-derived bioweapons and calling for stronger safeguards.
How experts imagine the threat could unfold
Today, creating a genuinely novel pathogen typically demands rare expertise, specialized laboratory equipment and years of failed experiments. The fear is that AI could accelerate or lower those barriers:
- A bad actor—whether state-sponsored or a well-resourced individual—could use a powerful model to map human biological vulnerabilities and identify which genetic tweaks make an existing pathogen more transmissible, harder to detect, or resistant to treatments.
- A model trained on extensive biological and genomic datasets could generate viable candidate modifications. The computation might resemble a swarm of experts working nonstop.
- Those candidates could be synthesized using increasingly affordable and accessible gene-editing tools, a separate technology progressing rapidly.
- If the resulting pathogen is novel enough to evade current biosurveillance systems, it could spread before public health authorities recognize what they are confronting, making containment far more difficult than with COVID.
Implications for regulation and the open-source debate
Two policy tensions matter here. Frontier model developers can, in principle, bake in guardrails and protective capabilities. Open-source models, however, can be downloaded, adapted and run in private settings outside regulatory oversight, and could be fine-tuned by actors operating beyond any governance regime.
At the same time, many critics note that frontier models with massive compute are more likely to produce genuinely novel capabilities—so improving model power increases both beneficial and harmful possibilities.
Bottom line
The issue is serious and deserves attention: ignoring the risk will not make it go away. The findings from researchers and warnings from industry leaders argue for continued, concrete efforts—technical, regulatory and surveillance-focused—to reduce the chance of misuse while developing tools that can detect and mitigate emerging biological threats.
