A teachable moment for dual-use

To the Editor –

Dual-Use Research of Concern (DURC) can be defined as research, mainly in the life sciences, that has the potential to be misapplied for harmful purposes¹. Key examples are the synthesis of mousepox ², synthesis of polio virus ³, generation of the 1918 flu virus ⁴, gain of function studies with H5N1 in ferrets ⁵ and synthesis of horsepox, the viral cousin of smallpox ⁶. These seminal events involved the physical synthesis of a biological agent. However, the time may have come to also consider dual use risk of the development of toxic agents in silico: We recently reported the alarming results of a computational experiment performed for a biennial arms control conference where we used a generative artificial intelligence (AI) approach previously developed for drug discovery applications, and found it could easily design a range of nerve agents including VX ^7,8. The experiment demonstrated the alarming speed and ease with which software - based on open-source tools and datasets from the public domain - could be used for bad purposes. Our experiment was subsequently covered widely in the media, reaching a network of scientists, experts, and lay people alike⁸, and its implications were recognized at the highest levels of governments within a matter of days. The level of interest was likely amplified due to the enfolding war in Ukraine, with Russia’s invasion and the threat of biochemical weapons use.

While we are a small team, the perspectives we provide are nationally diverse, span the private, academic, and government sectors, and draw on expertise from the natural and social sciences as well as more technical fields like computing and drug discovery. We believe our experience holds several important lessons which we wish to communicate to the scientific, ethics and security communities. First, the experiment is a clear and powerful example of a concrete dual-use risk concern arising from converging technologies and this should be harnessed to raise awareness of the security dimension of life science research. Second, our experience as a whole, obtained from this publication from reviewer and editorial feedback through to the many groups we have since presented to or been interviewed by taught us the importance of increasing awareness in a responsible, non-alarmist way. Third, we need to consider what these dual use findings mean for responsible science in drug discovery, and what action the community should be taking.

Responses to our article ⁷ have run the gamut. Some academics and government employees have requested the compound structures (this was denied); some suggested we should only use the technology for good (yes). Others asked whether the software could help identify treatments for diseases of interest to them (yes, potentially). Some felt our thought experiment was obvious; while several experts on chemical weapons accepted that they had not considered it and saw novelty. Many were concerned about the security of the data generated. There have also been questions on why we published and whether the details of the experiment should have even been published at all—in line with responses to prior biological dual-use examples. In response to this point, we believe that this new example highlights an important message, that dual use risk potential in the life sciences goes beyond the synthesis of biological agents. For governments, our thought experiment highlights the challenge of how and when to limit access to generative and machine learning software, including through export controls. For the drug design community, it will now be necessary to agree on ways to share data and models securely.

Our thought experiment has already become a ‘teachable moment for dual-use’ – a positive unintended consequence of our study. It can be drawn on as a test case for considering the risks of research involving converging technologies, in contrast to prior dual-use examples that focus on physical biological agents. It can also be used to provide dual-use risk training for those applying AI in drug discovery in the context of nerve agents and chemical weapons. Dual-use concerns in AI is already an urgent topic on the agenda for policymakers, but our results point to the need for further action in developing regulation. Our preemptive publication may lead to increased diligence around AI technologies, datasets, models, and related software for designing new molecules and the subsequent consideration of ethics and societal consequences ⁹. Dual-use potential of AI is of concern to all scientists, not just those in the field of drug discovery. We hope our thought experiment puts dual-use risk on the radar for a wider area without raising undue alarm and that it stimulates the search for potential solutions ⁸.