Toward a framework for risk mitigation of potential misuse of artificial intelligence in biomedical research

Abstract

The rapid advancement of artificial intelligence (AI) in biomedical research presents considerable potential for misuse, including authoritarian surveillance, data misuse, bioweapon development, increase in inequity and abuse of privacy. We propose a multi-pronged framework for researchers to mitigate these risks, looking first to existing ethical frameworks and regulatory measures researchers can adapt to their own work, next to off-the-shelf AI solutions, then to design-specific solutions researchers can build into their AI to mitigate misuse. When researchers remain unable to address the potential for harmful misuse, and the risks outweigh potential benefits, we recommend researchers consider a different approach to answering their research question, or a new research question if the risks remain too great. We apply this framework to three different domains of AI research where misuse is likely to be problematic: (1) AI for drug and chemical discovery; (2) generative models for synthetic data; (3) ambient intelligence.

The notion that scientific advances can be misused for harm is not a new one. From the Dr Frankenstein’s monster of literature to the real-world atomic bomb and lab-created pathogens, the scientific community has long been intertwined with risk of misuse. As AI proliferates through academia and industry, scientific research can be further misused for harm, and researchers have an obligation to prevent it.

Individual research teams are discovering the potential for misuse that their work brings in drug and chemical discovery. Urbina et al.¹ showed how a drug-discovery algorithm could be repurposed for the de novo design of chemical weapons. By inverting the logic of their machine-learning models, originally designed to discover therapeutic molecules, they created a system capable of generating potentially deadly compounds. In just 6 h, their model produced 40,000 molecules within a specified toxicity threshold, including both known chemical warfare agents and novel molecules predicted to be even more toxic. Shankar and Zare² highlight how AI-driven material design, although promising for developing improved medicines, could be maliciously used to create compounds capable of poisoning community water supplies. The authors took seriously the misuse potential of their work, but were unable to find the guidance and expertise they needed to address it, even after consulting governmental agencies and industry.

Since then, governmental organizations have started to weigh in on the AI misuse problem. For example, the EU AI Act prohibits certain uses of AI and places the responsibility for assessing reasonably foreseeable misuse with the creators of AI systems³. US Executive Order 14110 also calls upon federal agencies to require testing and evaluation of AI tools to prevent misuse and also to ensure they are resilient against misuse⁴.

Though a helpful starting point, such guidance is often written for different actors and scopes of misuse. Given the emerging attention to AI misuse, the recommendations may not directly translate to individual research teams and their projects. Accordingly, it can be difficult for researchers to navigate when, where and how to mitigate such misuse risks within their work.

We provide a framework for researchers to mitigate misuse risks of AI within their biomedical research. This framework (Fig. 1) recommends that researchers look to three different areas for navigating misuse risks in their work: (1) existing guidance and regulation; (2) existing mitigation strategies to address other AI-related harms (off-the-shelf strategies); (3) design-specific solutions they can instantiate in their AI model development. We present three use cases to guide implementing our framework along different misuse risks.

Fig. 1 |. A framework for developing mitigation strategies to address misuse risks of AI in biomedicine.

The framework starts when researchers ‘begin developing technology’, and proceeds through the workflow following the arrows. When researchers’ mapping of the risk/benefit landscape shows greater than low risk of misuse, the grey pop-out box provides three types of mitigation strategy and guiding questions to aid researchers in reducing misuse risks. The iterative nature of research development and risk/benefit calculus are represented by dotted arrows that lead from the green ‘proceed with research’ boxes back to risk/benefit assessment. If the remaining risks cannot be sufficiently mitigated, the red box highlights alternative options for researchers.

To mitigate misuse risks, researchers must first be able to identify them. Although the framework itself is not intended to aid in risk identification, we group misuse risks in each case along key dimensions to show how different decisions throughout the research process can exacerbate or attenuate such risks. These dimensions include the types of data collected, how those data are collected, how data are stored and processed, the access provided to the data, model and model output, the types of prediction being made with the model, and the potential inferences and predictions the model could enable.

A framework for mitigating foreseeable, harmful misuse

We developed our framework using a National Academies-style fact-finding process, followed by consensus-building, with a large group of experts from Stanford University and the drug-development industry. Group members brought expertise in areas ranging from bioengineering and medicine to social science and philosophy, including experience tackling dual-use issues in bioengineering along with ethical and regulatory issues in AI.

We began by first defining ‘misuse’ of AI in biomedical research for the purposes of our framework—motivated actors’ deliberate use of AI tools to cause harm or engage in unethical activities^4,5. Next, we used casuistry to identify AI-specific misuse risks in biomedical research. Unlike top–down methods that start with general ethical principles to apply to certain cases, casuistry offers a flexible and practical approach to moral reasoning because it provides a method for navigating complex moral landscapes, drawing on precedent and analogy, while accounting for the individual circumstances of each case⁶. We held group meetings and communicated with members individually to produce the use cases to guide our casuistical reasoning. We reduced the use cases from an initial five to the three we present in this Perspective, as their misuse risks seem uniquely shaped by the incorporation of AI into biomedical advances: ambient intelligence in healthcare, generative AI for synthetic data and AI for chemical and drug discovery.

Following agreement on these use cases, we turned to group meetings and individual discussions to identify strategies for mitigating the misuse risks presented. Engaging experts in both formats ensured all input was equally considered. Themes began to emerge regarding the sources of mitigation strategies our experts were identifying. Some were drawn from existing guidance, regulations and policies regarding misuse, whether related to AI or non-computational areas, like biosecurity, dual-use research of concern (DURC) and the life sciences. Others were adapted from best practices to protect against other forms of AI-related harms, and the remainder focused on design-specific decisions researchers could make during AI development and deployment to prevent misuse. These three sources of mitigation strategies then formed the basis of our framework.

Figure 1 shows our mitigation strategy framework and where it fits within the research development and ethical risk assessment cycle. We recommend that researchers developing AI for biomedicine, whenever possible, engage ethicists, affected groups and societal impact review processes to help scope the risk and benefit landscape of their project. For the purposes of this framework, ‘risk’ refers to the potential magnitude and likelihood of the AI tool’s misuse, and ‘benefit’ refers to the potential magnitude and likelihood of anticipated positive outcomes following from the design and use of the AI tool. In weighing the risks and benefits of the research, researchers should take care where the benefits do not outweigh the misuse risks, even after mitigation attempts. In these cases, research teams must consider whether to proceed with the work as originally planned. They may consider whether related lines of enquiry could be pursued at lower risk or if certain misuse risks must be better understood before proceeding. Finally, if no viable path remains that keeps risks outweighed by anticipated benefits, the research team may decide not to pursue the project.

Table 1 includes solutions from ethical frameworks and regulatory measures of potential design-specific mitigation strategies, Table 2 provides off-the-shelf solutions, and Table 3 a list of potential design-specific mitigation strategies that can all be applied to cases of misuse. We intend these categories to be helpful heuristics for researchers, rather than an empirical taxonomy. Although we reference many strategies within the text, the tables provide further details on each. Inclusion within a table does not mean a strategy prevents misuse entirely or cannot be further improved, but should instead serve as a starting point for researchers and clinicians committed to responsible AI development.

Table 1 |.

Ethical frameworks and regulatory measures for mitigating misuse risks

Adapted mitigations	Purpose and the specific misuse risk it addresses	When to implement
Establish training and education for participating researchers and users of AI tool3,49	Improve the identification of misuse risks and development of mitigation strategies Specific misuse risk: enhances awareness and capability to identify and mitigate misuse risks among researchers and stakeholders49,50	During project ideation and before proceeding to AI tool or research development
Allow individuals to opt out of AI systems and disclosure their use20	Respect individuals’ autonomy and right to know and choose when to engage with an AI tool or system Specific misuse risk: prevents unauthorized or non-consensual data collection	During tool development and prior to deploying the AI tool on a user population
Assess whether and how actors could bypass safeguards and misuse an AI model38	Improve identification of misuse risks and locate where mitigations could be implemented Specific misuse risk: prevents unauthorized access to AI models	During project ideation, AI system development and prior to deployment in sensitive areas such as drug discovery or healthcare settings
Develop system to collect, monitor and respond to instances of misuse38	Enable researchers to identify and react to the misuse of their AI tool Specific misuse risk: enhances researchers’ awareness of tool or system misuse and their potential responsiveness to problems as they arise	During AI system development and prior to implementation
Develop screening for users of an AI model and/or its corresponding data37	Enable researchers to track who has access and how the AI model and data are being used Specific misuse risk: keeps researchers apprised of uses of the AI tool and data and enhances their potential responsiveness to problems as they arise	During AI system development and prior to implementation
Incorporate model safeguards38	Establish conditions that prevent undesirable or harmful output from AI tools Specific misuse risk: prevents misuse of tool for harm	When developing AI models for sensitive tasks, like drug discovery or patient symptom identification, that could potentially be misused to create harmful substances or otherwise disclose or misuse patient information

Table 2 |.

Off-the-shelf strategies for mitigating risks of misuse

Example of mitigation strategy	Purpose and the specific misuse risk it addresses	When to implement
Fairness-aware machine-learning techniques	Allow researchers to assess whether AI model prioritizes one group over another39 Specific misuse risk: prevents biased outcomes that favour one group over another, ensuring equity in AI applications	When developing AI models for healthcare diagnostics, treatment recommendations or resource allocation to ensure equitable outcomes across diverse populations
Adversarial testing	Reveal biases within data or model by examining model outputs to a range of unanticipated inputs39,51,52; compare algorithmic objective function to its intended use53,54 Specific misuse risk: identifies and mitigates hidden biases that could lead to unfair or unethical outcomes	During the development and testing phases of AI models, especially those used in critical decision-making processes such as drug discovery or clinical diagnostics
Red-teaming	Experts interact with an AI in attempts to reveal undesirable outcomes55 Specific misuse risk: exposes potential vulnerabilities and malicious uses of AI models	Before deploying AI systems in sensitive areas such as drug discovery or chemical synthesis, to identify potential misuse scenarios
Blue-teaming	Experts interact with an AI in attempts to resolve undesirable outcomes Specific misuse risk: enhances AI model robustness by addressing vulnerabilities and improving safety measures	After red-teaming exercises, to address identified vulnerabilities and improve the overall safety and reliability of AI systems in biomedical applications
AI transparency/explainability methods	Enable human intellectual oversight of AI decisions and predictions to provide users with appropriate information on data22,56,57 Specific misuse risk: ensures accountability and understanding of AI decisions, preventing opaque or unexplainable outcomes	When implementing AI systems for clinical decision support or automated diagnostics, to ensure healthcare providers can understand and validate AI recommendations
System of qualified data storage and tracking	Provide infrastructure for oversight of access and use of data (for example, structured access to data from the All of Us Research Program) Specific misuse risk: prevents unauthorized access and ensures proper data governance and tracking	When establishing large-scale biomedical research databases or collaborative AI projects involving sensitive health data from multiple sources

Table 3 |.

Design-specific strategies for mitigating misuse risks

Example of mitigation strategy	Purpose and the specific misuse risk it addresses	When to implement
Data fiduciary	Prioritize user autonomy by allowing users to customize which data are collected, how it is used, who can access it, and so on28 Specific misuse risk: prevent unauthorized use or misuse of personal data by ensuring users have control over their data	When developing AI systems that handle sensitive personal data, especially in healthcare or ambient intelligence applications
Digital watermarking	Enable researchers to track unauthorized data use or sharing40 Specific misuse risk: detect and prevent unauthorized distribution or misuse of sensitive data	When sharing datasets or AI models, particularly those containing synthetic or potentially sensitive biomedical data
Face-blurring in images and video data	Ensure individuals cannot be easily identified in collected data25 Specific misuse risk: protect individual privacy and prevent unauthorized surveillance or identification	When collecting or using visual data in ambient intelligence or healthcare monitoring systems
Differential privacy	Allow data analysis while ensuring results do not reveal specific information about individual observations in data58 Specific misuse risk: prevent re-identification of individuals from aggregated data analysis	When working with large datasets for population health studies or when generating synthetic data
Homomorphic encryption	Allow for computations on encrypted data without need for decryption59 Specific misuse risk: protect data confidentiality during processing and prevent data breaches	When collaborating on AI projects involving sensitive data across multiple institutions or when using cloud computing resources
Federated learning	Enable model training across multiple devices/servers while keeping data localized60 Specific misuse risk: prevent centralized data storage, reducing the risk of large-scale data breaches	When developing AI models that require data from multiple sources or institutions, especially in multi-centre clinical trials or distributed healthcare systems
Execute model on safeguards	Enable researchers to limit certain types of model use and flag inappropriate use patterns Specific misuse risk: detect and prevent misuse of AI models, such as generating harmful or unethical outputs	When developing AI models for drug discovery or chemical synthesis that could potentially be misused to create harmful substances
Tailored evaluations	Examine how the raw or synthetic data corresponds to target populations and other metrics30,61,62 Specific misuse risk: ensure the reliability and ethical use of data by verifying its accuracy and representativeness	When using AI models for clinical decision support or population health analysis to ensure fair and accurate outcomes across diverse groups

Ambient intelligence

Ambient intelligence (AMI) refers to electronic environments that are sensitive and responsive to the presence of people^7,8. In biomedical research, AMI encompasses systems that seamlessly integrate with users’ environments to provide continuous health monitoring, early detection of anomalies and personalized interventions^9,10. For instance, in care of older people¹¹, AMI can monitor daily activities, detect falls and provide medication reminders, thereby enhancing safety and well-being^9–14.

Misuse risks and ethical challenges

The types of data collected for AMI, the predictions being made and the inferences it enables can make it particularly attractive for misuse. Data collected for biomedical AMI can be especially sensitive, including information on a person’s face, voice and gait¹⁵. Health-related inferences are also made on AMI data, ranging from detecting pain within individuals to assessing mobility patterns and analysing patients’ sleep¹⁵. If motivated actors were to gain access to such information, they could misuse it to identify and/or expose individuals’ health information. Similarly, AMI developed to track and analyse individuals’ health information, such as smart toilets, could enable additional misuses, such as disease profiling by insurance companies or unauthorized drug monitoring by employers.

As AMI technology advances, so too do the capabilities for identity and behaviour recognition. These new capabilities are inherently linked to the potential for misuse, as evidenced by cases of state surveillance. For example, AMI has been used to monitor Uighurs in China¹⁶ and track Black Lives Matter protesters in the USA¹⁷, illustrating how such technology can be used for broader state surveillance activities¹⁸.

Furthermore, the manner in which AMI data are collected can shape misuse risks. As AMI continues to be deployed in semi-public spaces such as hospital waiting rooms or doctors’ offices and in private spaces such as patients’ homes, these environments can lead to unintentional or non-consensual data collection¹⁹. Designed for passive and continuous monitoring, AMI deployed in areas frequented by people beyond the individual patient or user of interest will necessarily collect data from others, further contributing to risks of misuse.

Applying our framework

Addressing the misuse risks described above requires preventing inappropriate access and use of the data and AMI models as well as ensuring proper data-collection practices. Following our framework, we can look first to the available regulatory guidance and legal requirements that apply or could be adapted to the case of AMI and misuse (Table 1).

Guidance from four regulatory bodies can help chart an initial path toward mitigation. Following the high-risk classification of biometric systems in the EU AI Act, researchers could begin with plans for continuous risk assessment against the reasonably foreseeable misuse of their AI tool and develop training for users on the appropriate uses of the system. These activities could help researchers spot risks as they emerge and prevent authorized users from incidental, though still harmful, misuse. The US Blueprint for an AI Bill of Rights touches on giving individuals the ability to opt out of inclusion in an AI system and providing disclosure on the use of one²⁰, which serves as a starting point for data-collection mitigations. Finally, the US AI Safety Institute (AISI) Managing Misuse Risk for Dual-Use Foundation Models document provides detailed guidance on red-teaming, a practice from cybersecurity, to assess whether actors could bypass safeguards and misuse an AI model, which reflects a similar call in US Executive Order 14110.

With the patchwork of mitigation strategies thus far, we turn to the next step in our framework: existing strategies for mitigating AI risks that can be adapted to issues of misuse (off-the-shelf strategies; Table 2). Research teams can conduct adversarial testing of their AI models to help spot unfair and unintended model output, preventing the misuse of data. Researchers can also employ blue-teaming practices to enhance an AI model’s robustness to vulnerabilities as a complement to red-teaming²¹. Finally, to better supplement user training, researchers can use AI transparency and explainability methods, such as SHAP (Shapley Additive Explanations)²² values, to ensure that users of the AI tool understand its output and resulting decision-making.

Finally, researchers should consider the design-specific mitigation strategies, outlined in Table 3, they can instantiate within their work to prevent misuse. These strategies can involve limiting the sensitivity of the data collected for AI tools, lowering its appeal for misuse and the severity of the consequences should unauthorized data or model access occur. For example, implementing differential privacy techniques can allow useful insights to be extracted from AMI data while mathematically protecting individual privacy. This could involve adding calibrated noise to collected data or query results, ensuring that the presence or absence of any individual cannot be inferred from the output²³. Researchers could also implement privacy-preserving camera lenses²⁴, face-blurring with ambient sensors²⁵ and body-masking techniques²⁶ to further reduce the data sensitivity in AMI applications.

Although obtaining meaningful consent in AMI environments is challenging, it is paramount for the proper use of collected data²⁷. Researchers could build on the consent procedures identified at the start of the framework to further implement granular, context-aware consent mechanisms, such as allowing residents in a smart-home setting to specify privacy preferences for different rooms or times of day through user-friendly interfaces. A similar principle could be implemented more broadly, creating a tiered system with stricter controls for AMI in private versus public areas. Implementing data fiduciary systems²⁸ can also help manage user preferences securely, acting as trusted intermediaries to ensure that data are used only in ways explicitly approved by the user.

Generative AI for synthetic data

Generative models in biomedical research offer a powerful tool for creating synthetic population data. These models, drawing on existing population data, can generate datasets that mimic real-world characteristics while addressing privacy concerns associated with sharing sensitive patient information^29–31. This approach not only reduces barriers to accessing restricted real data³⁰, but also retains the statistical properties of the underlying population³². For instance, in rare disease research, synthetic data can provide a larger sample size for analysis without compromising patient privacy.

Misuse risks and ethical challenges

The sensitivity of raw data, the data-generation process and the representativeness of the underlying data all contribute to the potential misuse of generative AI for synthetic data. Given the sensitive nature of the data used to generate synthetic datasets, a critical risk lies in the potential for re-identification, particularly in datasets representing rare disease populations. For example, in a synthetic dataset of a rare genetic disorder, specific combinations of genetic markers and sociodemographic characteristics could allow adversaries to triangulate information and identify individuals, even in supposedly anonymized data.

Data bias presents another substantial challenge. The quality and representativeness of synthetic data are inherently limited by the generative model and the underlying real data. If these sources contain biases, the synthetic data will reflect and potentially amplify these biases. This can lead to serious consequences in healthcare applications, such as underdiagnosing conditions in underrepresented populations or perpetuating existing health disparities^33–36.

The risk of data fabrication is particularly concerning in scenarios where real data are inaccessible and summary statistics are unavailable. For instance, in emerging infectious-disease research, there might be a temptation to generate synthetic data that fit preconceived notions rather than reflecting the limited real-world data available, potentially leading to misleading conclusions that could affect public health policies.

Applying our framework

We look again to existing guidance to begin mapping out mitigation strategies. Following the US Office of Science Technology and Policy’s Framework for Nucleic Acid Synthesis Screening, researchers can devise a system for screening users³⁷ of the synthetic datasets they create. Borrowing from AISI’s guidance for managing the misuse of foundation models, researchers can also implement systems to collect, monitor and respond to instances of misuse of synthetic datasets³⁸. Both strategies can aid researchers in identifying misuse.

Off-the-shelf solutions can help researchers further address data bias. Researchers can incorporate fairness-aware machine-learning techniques into the generation of synthetic data to explicitly optimize for both accuracy and fairness during model training³⁹. For example, in a synthetic dataset for heart disease risk prediction, the model could be trained to ensure equal predictive accuracy across different demographic groups.

To facilitate user screening and tracking recommendations adapted from existing regulations, researchers can use design-specific techniques such as digital watermarking⁴⁰ and blockchain-based systems to create immutable records of data origin and transformations⁴¹. This could be implemented by embedding a unique, encrypted identifier in each synthetic dataset that can be traced back to its generation parameters and source data characteristics. Furthermore, as with the AMI cases, researchers can implement differential privacy methods to prevent the identification of any individuals in the underlying data.

AI for chemical and drug discovery

AI has emerged as a transformative force in the field of drug and chemical discovery^42–44. The combination of reduced barriers to access for AI models and extensive computational power can accelerate the drug-development process and reduce costs compared to traditional methods⁴². For instance, AI algorithms can rapidly screen vast libraries of chemical compounds, predicting their potential efficacy and toxicity, thus substantially shortening the initial stages of drug discovery.

Misuse risks and ethical challenges

The very features that make AI-driven chemical discovery efficient and cost-effective also introduce new pathways for potential misuse. As more actors gain access to powerful AI models capable of processing massive amounts of chemical, clinical and biological data, the potential for misuse increases dramatically. In particular, misuse concerns arise regarding the predictions and output enabled by such AI tools, including classes of chemical weapons^1,45 and toxic compounds².

Applying our framework

Drawing first on existing guidance and regulation, we can begin to sketch a profile of mitigation strategies to address these misuse risks. Guidelines for reducing DURC in gain-of-function research highlights the importance of containment measures⁴⁶. Researchers could adapt this to the use of AI prediction tools for chemical and material development and establish conditions under which users can access or implement them. Building on AISI’s guidance, researchers could also collect, monitor and respond to instances of misuse. Furthermore, they could adapt AISI’s guidance for safeguarding against undesirable output of foundation models³⁸ to incorporate model safeguards against predicting known classes of chemical weapons and toxic compounds.

Next, researchers can consult off-the-shelf to fill out their mitigation plans. For example, red-teaming, blue-teaming and adversarial testing can all aid researchers in determining whether their AI model produces undesirable output, where model security remains vulnerable, and areas to improve safety measures within the model. These exercises could even involve ethical hackers attempting to misuse the AI systems to generate harmful compounds, thereby revealing potential loopholes that need to be addressed. To fulfil guidelines for collecting, monitoring and responding to instances of misuse, researchers could develop a system of qualified data storage and tracking, similar, for example, to the structured data access from the All of Us Research Program⁴¹. This could involve implementing a tiered access system where researchers’ credentials and project goals determine their level of access to AI tools and chemical databases. Continuous monitoring of AI usage patterns could help detect potential misuse, such as attempts to generate sequences of highly toxic compounds.

Design-specific safeguards within the AI models are also key. For example, researchers could implement ‘ethical constraints’ in their AI systems, programming them to avoid generating compounds with characteristics like known weapons or highly toxic substances. This could be achieved through a combination of rule-based filters and machine-learning models trained to recognize potentially dangerous molecular structures.

Discussion

The potential for misuse looms large in biomedical AI research. Our framework provides researchers with a path to follow for mitigating misuse risks that arise in their research. To begin, we recommend that researchers refer to the current landscape of regulation and guidance. Researchers should look to guidance for preventing the misuse of different scientific activities and policies regarding AI to piece together broad recommendations. Next, they can turn to off-the-shelf solutions for strategies developed to address other risks of AI that could also help resolve misuse concerns. Finally, researchers can look for design-specific mitigations that can be instantiated within their research to forestall risks of misuse. If misuse risks continue to outweigh anticipated benefits, even after developing mitigations, researchers should consider whether they could pursue a similar project that lessens the intractable misuse risks accompanying the work; otherwise, they can choose not to pursue the initial project.

It is also important to recognize that AI system building is fundamentally an exercise in applied engineering. Although this may lead to the discovery of new knowledge, it presents substantial challenges in terms of building capacities that could be misused. This reality underscores the importance of our framework, which aims to guide researchers towards responsible AI use.

Although this framework has focused on the responsibilities and capabilities of individual research teams to address misuse risks, we recognize that many more actors comprise the science ecosystem and therefore play a role in mitigating these concerns. Academic researchers can call on their universities and research institutions to help promote self-regulation in this area. For example, they can request training and education for identifying, addressing and preventing AI misuse risks. Additional requests could include ongoing ethical review processes for AI-related research, similar to the Ethics and Society Review model that requires researchers consider and mitigate downstream consequences of their AI research prior to accessing grant funding⁴⁷. Researchers can also urge their institutions to provide reporting guidance for documenting and discussing attempts and mitigating AI misuse risks. Finally, to prevent power disparities and promote responsible AI use, measures should be taken to ensure that AI tools and knowledge are accessible to a wide range of responsible researchers.

Furthermore, these challenges extend beyond technology development and encompass the need for a supportive institutional culture that emphasizes ethical practices, diligence and addresses privacy concerns, such as data persistence, repurposing and spillovers, ensuring informed consent and the right to data deletion. Such an approach is vital for preventing misuse and maintaining public trust in AI applications, particularly in sensitive areas like biomedical research and AMI development.

This framework is particularly useful for researchers once they have already identified potential risks of harmful misuse of their AI tool, and it presumes the AI tool functions properly⁴⁸. Similarly, researchers should pursue engaging relevant community members and stakeholders in the AI design and development process to amplify potentially harmful use cases that researchers did not consider due to differing lived experiences. When considering risks in AI, it is crucial to link the setting of intentional misuse with the associated risks. These components should be viewed as interconnected, not separate entities, in our risk assessment framework.

Limitations and future directions

Our framework, while providing a valuable starting point for addressing AI misuse risks in biomedical research, has several limitations. We used a National Academies-style fact-finding process complemented by individual interviews and group discussions, rather than the Delphi method widely regarded as the gold standard for consensus-building. This approach, while flexible, may lack structured iterative feedback and anonymity, potentially introducing biases. Additionally, our expert panel was predominantly based in the USA, potentially limiting the framework’s global applicability due to varying international regulatory and cultural contexts. Future research should address these limitations by incorporating the Delphi method and expanding the framework’s international relevance. Given the rapidly evolving nature of AI in biomedicine, this work should be viewed as an initial proposal requiring regular updates to keep pace with technological advancements and emerging ethical considerations. We must collectively work towards a future where AI enhances healthcare, advances scientific knowledge and ensures equity, while simultaneously safeguarding against potential misuse and protecting the rights and privacy of individuals and enhancing the trustworthiness of AI researchers.