AI vs. Humanity: The Disturbing Rise of AI-Designed Bioweapons and How Concerned We Should Be

It might seem unbelievable that a snail could be lethal, yet certain highly venomous sea snails—known as cone snails—can indeed be deadly to humans. Their venom consists of a mix of tiny proteins called conotoxins, some of which can disrupt nerve signaling by blocking ion channels. Currently, there is no antivenom available.

There are hundreds of thousands of known conotoxin structures, and many pose no threat to humans—some are even medically beneficial: one has already been developed into an approved treatment for chronic pain. However, research on particularly dangerous conotoxins is tightly controlled in certain countries.

So when Chinese researchers announced in 2024 that they had created an AI tool capable of designing new conotoxins¹, it sparked concern in some circles. In an email to a private AI and biotechnology forum reviewed by Nature, a senior U.S. government official raised the study as a potential biosecurity risk. The official, who requested anonymity due to job-related concerns, noted it was especially troubling that the AI tool was built on an open-source protein language model originally developed by American scientists.

One of the study’s authors dismissed the alarm as unnecessary. Weiwei Xue, a computational chemist at Chongqing University and co-author of the paper, told Nature that the research was purely focused on drug discovery. His team has already identified promising therapeutic candidates after lab testing, he said. While acknowledging the importance of considering misuse potential, he emphasized the tool was never intended to create harmful substances, and turning digital designs into real molecules demands substantial expertise and specialized equipment. Other experts also told Nature that the actual risks appear minimal.

Still, this incident highlights a rising worry about new AI tools in biology: while they’re being built to accelerate drug development and deliver societal benefits, they could also lower the barrier to creating biological threats. Breakthroughs like AlphaFold have made it possible for scientists to design custom proteins and viruses at the click of a button—even general-purpose chatbots can help people learn how to build these in a lab. Could the latest AI systems also make it easier to engineer more potent toxins, viruses, or even bioweapons?

Interviews with over 20 scientists and policy experts suggest the biosecurity risk is real. “Theoretically—and this is what keeps me awake at night—someone could now engineer toxins as deadly as ricin or worse, and they’d be nearly impossible to detect,” says Martin Pacesa, a structural biologist at the University of Zurich.

But there’s disagreement on how to respond. Some advocate for restricting access to biological AI tools; others warn this could stifle vital research. “We’ve always believed the global benefits far outweigh the risks,” says David Baker, a computational biophysicist at the University of Washington in Seattle and 2024 Nobel laureate for his work in protein design. “But as these tools grow more powerful, that balance will need constant reevaluation.”

Some argue the priority should be detecting and responding to AI-enabled bioweapon attacks, rather than trying to block them through software controls. “That ship has sailed,” says Timothy Jenkins, a protein designer at the Technical University of Denmark.

What’s the worst that could happen?

James Black, an AI biosecurity researcher and visiting scholar at Johns Hopkins University, outlines two main concerns regarding AI and bioweapons.

First, individuals in makeshift labs might use chatbots to learn how to produce or deploy known threats like anthrax. Second, more advanced actors—such as governments or well-funded terrorist groups—could pair chatbots with specialized biological software to engineer novel bioweapons.

Researchers say the gravest danger could be AI-designed pandemic viruses. The most likely path would involve tweaking existing viruses—like SARS-CoV-2 or flu—to worsen traits such as immune evasion. AI tools built for viral surveillance and vaccine design could be repurposed this way, notes Doni Bloomfield, a biosecurity law professor at Fordham University.

Alternatively, AI might create entirely new pathogens that are hard to detect or counter. A 2025 preprint used AI to design viral genomes, with about 5% proving functional when synthesized in the lab². However, those viruses targeted bacteria, not humans.

Still alarming? A 2025 report³ from the U.S. National Academies of Sciences, Engineering, and Medicine (NASEM) tempers fears. It found major obstacles to using AI to meaningfully enhance pandemic pathogens or build them from scratch. Key challenges include scarce data linking traits like virulence or transmissibility to genetic sequences, making reliable predictions difficult. Another hurdle is the complexity of producing and testing pathogens in the lab—a step AI hasn’t simplified.

Some scientists also question whether malicious actors would even need AI, given nature’s abundance of threats. Traditional methods like random mutagenesis can enhance dangerous traits without AI, says Brian Hie, a computational biologist at Stanford.

“If you want to cause massive harm, you don’t need protein design,” adds Baker.

Still, the NASEM report acknowledges that existing AI tools could potentially design novel toxins—though producing and delivering them remains a major challenge. Such a toxin might be undetectable if previously unknown and more suited to targeted attacks like assassinations, says Jenkins. “I think the tools already published give bad actors a solid starting point,” he warns.

Seth Donoughe, AI director at the nonprofit SecureBio in Cambridge, Massachusetts, says his biggest concern is AI’s ability to amplify the capabilities of malicious users—whether through general-purpose chatbots or specialized biological models.

In a February preprint⁴, Donoughe and colleagues showed that access to advanced large language models (LLMs) allowed people with little biology training to match or surpass PhD scientists in tasks like troubleshooting virology protocols or coding lab robots. Related early findings suggest some AI agents—systems that can autonomously run code or execute tasks—could even guide biological AIs to enhance the harmful properties of viral proteins.

However, another February preprint⁵ from scientists at Active Site, a Cambridge-based research nonprofit, found that novices using LLMs didn’t perform significantly better at tasks like DNA manipulation or virus production than those using only standard internet resources (see ‘Does AI raise novices’ biology-lab skills?’). So it’s possible the current “AI boost” isn’t yet enough to close the expertise gap.

Creating a bioweapon might seem like a remote possibility today — but that perception could soon change.

AI is rapidly advancing across every domain we challenge it with, Donoughe notes, and this dual-use progress means it’s becoming just as easy to do harm as it is to do good.

Screening safeguards

Many experts argue the most effective defense against AI-driven biological threats lies in catching malicious actors at the stage where actual viruses or toxins are being made. According to Pacesa, what happens inside a computer is often secondary — the real danger emerges when digital plans are turned into physical proteins or chemical compounds.

Scientists wanting AI-generated proteins or synthetic DNA typically outsource the building process to specialized firms that chemically construct strands of DNA or RNA. Some of these companies belong to the International Gene Synthesis Consortium, an industry alliance requiring its members to check customer orders for sequences encoding dangerous elements like toxins, disease-causing proteins, or other hazardous molecules. However, a 2025 Microsoft study⁶ demonstrated that AI can bypass these protective measures.

Eric Horvitz, Microsoft’s chief scientific officer, along with Bruce Wittmann in Redmond, Washington, led a team using publicly accessible protein-design software to reengineer 72 molecules posing potential biosecurity risks, including various toxins and viral proteins. Their team created 76,000 modified versions — called ‘synthetic homologues’ — designed to maintain the harmful properties of the originals while having sufficiently different genetic signatures to escape detection by screening systems at four participating companies.

Results varied. The detection software caught some but not all modified designs, with roughly 25% of the highest-risk variants — those most closely mimicking actual threats — going unnoticed. After three companies updated their screening algorithms, only about 3% of top-tier designs remained undetected. A subsequent March preprint by Horvitz, Wittmann, and colleagues⁷

Horvitz’s team has kept their designs, target identities, and proprietary details confidential, though researchers can request access.

In contrast, Xue and coworkers openly published the conotoxin sequences created by their AI tool. Unlike Horvitz’s project, their goal wasn’t to produce dangerous compounds¹ (see ‘AI-designed toxins’). Nonetheless, some of these published sequences might escape detection if submitted to a DNA synthesis provider. When Nature ran the paper’s 45 designs through BLAST — a standard bioinformatics tool that searches genetic databases for matches — only five were flagged as cone snail toxins. An open-source screening tool identified additional hits.

An important consideration is that redesigned toxins evading detection may differ so significantly from natural versions that they become nonfunctional. This possibility was supported by a follow-up experiment. A collaborative study led by Wittmann and Elizabeth Strychalski from the National Institute of Standards and Technology in Gaithersburg, Maryland, lab-tested whether screening-evading molecules retained their original biological activity⁸. While basic proteins sometimes remained functional, enzymes consistently failed. “It really demonstrated how complex and unreliable this approach is,” says Horvitz, a co-author on the study.

James Diggans, vice-president for biosecurity and policy at Twist Bioscience in South San Francisco, California, and a contributor to the experimental work, found the synthetic homologue studies encouraging. “Current screening practices remain a highly effective defense against misuse, even when adversaries use AI tools to evade detection,” he states.

Still, Diggans acknowledges the field’s rapid evolution. Newer AI models may surpass the 2023 protein-design tools used by Horvitz’s team in their ability to create undetectable threats.

Researchers are developing advanced screening methods that analyze the 3D structure and potential biological function of ordered molecules, not just their genetic sequence. Baker has long advocated for a confidential global registry of all synthesis orders, but this proposal hasn’t gained industry-wide support.

Currently, basic sequence-based screening is voluntary in most cases. However, aligned with a 2025 US executive order, American research funders may soon mandate that grantees use screeners. The United Kingdom, European Union, and countries like New Zealand are exploring similar requirements. Yet most nations lack any mandatory rules. China, responsible for over 30% of global DNA synthesis orders, has encouraged — but not yet required — screening compliance, according to Weiwen Zhang, a synthetic biologist at Tianjin University. Foreign orders, typically under export controls, often receive closer scrutiny than domestic ones, he notes.

Tessa Alexanian, who develops screening standards at the International Biosecurity and Biosafety Initiative for Science in Geneva, Switzerland, warns that numerous providers worldwide would flag a toxin sequence without triggering any alarms.

Further concern arises from emerging ‘tabletop’ DNA synthesizers. Though currently limited to producing very short sequences, experts anticipate these machines will soon handle longer fragments.

AI model safeguards

Another strategy focuses on restricting access to AI tools themselves — especially those tailored for biological research. Baker explains that his team routinely assesses potential risks before releasing any protein-alignment tool. This approach aligns with responsible AI and biodesign guidelines issued in March 2024 (see go.nature.com/4cjbu6t), endorsed by nearly 200 scientists. However, enforcement relies on self-regulation within the scientific community. To date, Baker says he hasn’t found it necessary to restrict tools developed in his lab for fundamental protein and biomolecule design.

Major AI companies like OpenAI in San Francisco, California, already train their general-purpose chatbots to decline or carefully respond to harmful requests, including those related to biosecurity. Specifically, an OpenAI safety policy states the models shouldn’t offer “detailed, actionable instructions” for large-scale harmful actions involving chemical or biological weapons.

Some researchers suggest applying similar protective measures to specialized AI for biology. Developers are beginning to incorporate safeguards during model training. For example, the Evo 2 ‘genomic language model’ was trained on 128,000 genomes spanning diverse life forms but deliberately excluded viruses that infect humans and animals. Consequently, Evo 2 performs poorly at designing sequences from these viruses or predicting mutation effects⁹.

Yet these protections aren’t foolproof. Donoughe’s team⁴ found that nearly 90% of study participants could extract high-risk biological data from general-purpose large language models. Other researchers consistently report that chatbots still disclose sensitive information. An April New York Times article detailed how an Indian man arrested last year for allegedly planning to produce ricin for terrorism reportedly consulted ChatGPT and AI-enhanced Google searches. (An OpenAI spokesperson told the Times that public reports suggested the information sought was already available online.)

Regarding specialized AI, Stanford bioengineer Le Cong and colleagues demonstrated that a general-purpose AI agent could manipulate Evo 2 into generating novel SARS-CoV-2 and HIV-1 proteins¹⁰. Another study showed that ‘fine-tuning’ Evo 2 with publicly accessible viral genome data restored its threat-design capabilities¹¹.

Hie, a co-developer of Evo 2, isn’t surprised by these bypasses, given the abundance of publicly available training data. He also wouldn’t be alarmed if such models could design human-infecting viruses. Despite these risks, he favors open access: “Transparent model development actually enhances safety, because researchers worldwide can freely evaluate and improve defenses,” he explains.