The ‘Hippocratic Oath’ for AI-based clinical decision support systems

Abstract

Background

The implementation of Artificial Intelligence assisted Clinical Decision Support Systems (AI-CDSS) shows significant potential to improve healthcare. However, implementing AI-CDSS has many associated challenges. This article introduces the ‘Hippocratic Oath’ for AI which promotes safe and effective AI-CDSS development and implementation.

Methods

This paper summarises discussions which took place during the Turing-Roche Clinical AI Interest Group Joint Workshop. The workshop began with scoping lectures from AI experts, leading into focus group discussions of key themes surrounding AI-CDSS implementation. These include the ethics, trust, evaluation, regulation, human factors and challenges involved with implementing AI-CDSS into healthcare settings. Focus group outcomes, alongside insight from lectures, were used to formulate the arguments in this paper.

Results

This article presents a consensus definition of AI-CDSS and outlines a comprehensive table of implementation challenges alongside mitigation measures. It introduces the ‘Hippocratic Oath for AI’ and discusses its potential to promote safe and effective AI-CDSS implementation through addressing human factors and explainability.

Conclusions

The ‘Hippocratic Oath for AI’, can be used by AI-CDSS implementers and developers as a framework to mitigate challenges involved with implementing AI-CDSS into healthcare settings. This framework is likely to promote safe and effective implementation and maximise HCP uptake of the AI-CDSS. Through facilitating AI-CDSS use, this oath can transform health care practice via reducing medical errors, healthcare costs and improving patient outcomes.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-026-03474-5.

Introduction

In this era of expanding development and deployment of Data Science and Artificial Intelligence (AI), a key focus is how these new technologies can be adopted safely to benefit health and care.

Coinciding with this is an increased burden on healthcare professionals (HCPs) due to factors such as increased administrative demands, patient volume and complexity and public health emergencies such as the COVID-19 pandemic. HCPs felt the brunt of COVID-19 with many reporting heightened levels of acute stress, anxiety and feelings of burnout due to increased workload with more strenuous shifts [1]. There are strong associations between medical errors and physician burnout [2]. Moreover, research has suggested that stressful hospital environments, defined by high levels of perceived patient stress, yield poorer patient outcomes [3].

Clinical decision support systems (CDSS) are tools which aid users in making evidenced based decisions surrounding patient care. They contribute towards better clinical practice through reducing error, enhancing patient safety and ultimately improving patient outcomes [4]. Implementing and adopting progressive systems such as AI-assisted CDSS (AI-CDSS), may become especially important to address the increasing demands on current healthcare systems.

Despite advances in healthcare, medical errors are still prevalent, and they can prove fatal. Within the NHS alone, there are an estimated 237 million medical errors each year, which costs the NHS nearly £100,000,000 and some patients their lives [5]. Artificial intelligence has great potential to improve CDSS. AI-CDSS may be able to analyse a greater variety of data at significantly faster speeds and construct patterns from data which is otherwise inaccessible to humans. They have the potential to improve decision making, provide intelligent and personalised health suggestions, mitigate errors and minimise adverse drug effects [6].

There is potential for AI-CDSS not only to reduce cognitive load but also streamline workload and save HCPs’ time [6]. AI-CDSS may offer a powerful solution for reducing HCPs’ stress and fatigue, ultimately enhancing patient care and outcomes [7]. Despite this potential, the implementation and application of AI-CDSS in healthcare settings remains limited and poses many challenges [8]. These include difficulties surrounding data interoperability, quality, and availability, regulatory considerations, limited evidence supporting the safety and efficacy of particular systems, and the practicalities of integrating these systems effectively into existing organisations and work practices.

This paper will explore the challenges associated with implementing AI-CDSS and introduce the ‘Hippocratic Oath for AI,’ a set of guidelines to promote ethical and effective AI-CDSS development and implementation into healthcare settings.

Methods

On October 31st, 2023, the Turing-Roche Partnership and the Alan Turing Institute’s Clinical AI Interest group jointly brought together a multi-disciplinary group, including data scientists, clinicians, regulatory experts, behavioural scientists and patients. The workshop began with a selection of scoping talks which reviewed the latest developments and literature in the field, leading into focus group discussions. These talks helped address knowledge imbalances, promoting inclusive dialogue.

There were 6 different focus groups which the attendees cycled through to explore each of the three different themes. These were ‘Ethics, trust and accountability,’ ‘Regulation, evaluation and monitoring,’ and ‘Human factors: usability and maintaining the human in the loop.’

Each focus group featured a chair who led the session ensuring the talking points were addressed. The chair also guided the conversation to ensure it remained accessible, allowing all participants to contribute meaningfully. This included preventing the discourse from becoming too technical or specialised.

During the workshop, attendees were also given a table of risks and consequences of AI-CDSS implementation based on the work of Wyatt (Personal Communication, 2018). They were then given the opportunity to comment and reflect on the complex challenges associated with AI-CDSS, within their respective fields. Their feedback, alongside remarks during focus groups, were collated and used to revise and expand the existing list to produce ‘Table 1: 20 Risks and consequences of AI-CDSS implementation and adoption.’ Overall, the event facilitated evidence synthesis and the collection of opinions from a broad range of stakeholders to produce key outcomes. This paper summarises these discussions and conclusions.

Table 1.

20 Risks and consequences of AI-CDSS implementation and adoption

Source of AI-CDSS risk	Consequences	Mitigation Measures
1. AI-CDSS connected to incomplete or incorrect patient records; wrong clinical codes used	AI-CDSS gives inconsistent advice leading to over / under investigation and over / under treatment	Follow rigorous data management and programming practices during the development and use of AI-CDSS. Data lineage can be used to trace the origins of data so its accuracy can be assessed
2. Poor standard knowledge bases	AI-CDSS give inaccurate, potentially dangerous advice. Leads to worse patient care and erosion of HCP, AI-CDSS trust	Apply robust evaluation before deployment and monitoring post-deployment to ensure the AI-CDSS is performing at an acceptable level
3. Structured patterns (non-random) of missing data in training datasets.	Can reduce the accuracy of AI-CDSS predictions, especially in underrepresented sub-populations. This can lead to disparities in AI-CDSS supported decision making and unreliable advice	Consider the effects of structured missing data during design and development of AI-CDSS to minimise its impact. Computerised models can reduce the impact of unavoidable structured missingness in large multimodal health datasets [52]
4. Algorithmic drift, where “case mix”, intervention effectiveness, and disease definitions evolve without updates to AI-CDSS	AI-CDSS becomes inaccurate, providing outdated clinical recommendations. Leads to worse patient care and erosion of HCP, AI-CDSS trust	Developers must monitor and update the AI-CDSS, so it uses current data and guidelines. Frequent HCP, developer meetings can help ensure AI-CDSS are appropriately responding to clinical needs
5. Algorithm adapts to incoming data in an uncontrolled manner	AI-CDSS advice can become unreliable and inaccurate without warning	Adopt a robust monitoring process for the AI-CDSS to highlight any fall in performance in a timely manner
6. Poor regulatory standards	AI-CDSS may give inaccurate, potentially dangerous advice. Poor regulatory standards could lead to disparities in AI-CDSS quality resulting in loss of HCP, AI-CDSS trust	Apply robust evaluation before deployment of AI-CDSS. Monitoring and regular audits post-deployment is critical to confirm the system is functioning at an acceptable level
7. Narrow or unclear scope/capabilities of AI-CDSS where users may not know the range of tasks that a specific AI-CDSS can be used for	Users unsure how to act at boundaries of AI-CDSS scope and unsure where these boundaries are. This can lead to lack of trust even when the AI-CDSS is correct	AI-CDSS developers must define the scope and use case for their decision support system. AI-CDSS capabilities and functionality must be explained during training
8. Black box algorithm inhibits user understanding of how AI-CDSS formed its decision (low explainability)	Lack of user trust in AI-CDSS leads to lost opportunity for benefits and inconsistency throughout workforce	Provide training on how the AI-CDSS works, what data it uses and how it forms its advice. Ensure that explanations regarding the specific AI-CDSS functionality is easily available to users whenever they use it. Conformal uncertainty predictions can complement explainability and increase transparency
9. Algorithm bias, including racial and gender disparities	Advice and clinical actions based on decisions from AI-CDSS are biased and potentially inaccurate	Formulate AI-CDSS from research which is representative of those to whom the tool will be applied. Identify and allow for any imbalances in the training data during the model development process. Test performance for bias on relevant subsets of the population.
10. Unclear accountability for AI-CDSS supported decisions	Ambiguous responsibility for errors or poor outcomes when using the AI-CDSS could lead to poor HCP uptake and a lack of trust	AI-CDSS implementers should ensure that during training it is made clear where accountability lies according to regulators such as the Medicines and Healthcare products Regulatory Agency for HCPs using AI-CDSS.
11. Insufficient AI-CDSS training for staff	Misinterpretation or improper use of the AI-CDSS. Could compromise patient outcomes and safety as well as minimise potential benefits	Ensure staff are given sufficient training so that they are confident with the AI-CDSS capabilities and how to use it. A ‘local champion’ can provide continuing staff support
12. Over reliance on AI-CDSS leads to automation bias	Users allow incorrect advice to overrule their correct judgement – automation bias. This could lead to an inability to act without AI-CDSS support.	Use of a certainty index as well as further research into how user biases and decision making context affect AI-CDSS use
13. Poorly designed AI-CDSS interface with limited accessibility	Inconsistent and reduced use with the potential for medical errors	AI-CDSS must be designed with a personalisable interface, where users have given feedback to ensure usability. User centred design and testing
14. AI-CDSS advice delivered at inappropriate frequencies or times	Disruption of clinical flow – could lead to alert fatigue where AI-CDSS advice is ignored	AI-CDSS developers must prioritise the most essential alerts as well as produce customisable AI-CDSS design so that users can choose in which situations they receive alerts. Adopting two stream models in AI-CDSS, which provide clinical guidance alongside considering the method and timing of presentation, can enhance advice delivery
15. Alert Fatigue where users become desensitised and show decreased responsiveness to an alert due to repeated exposure	Users may ignore critical alerts and therefore risk patient safety. It may lead to user frustration and decreased engagement with AI-CDSS and therefore a loss of potential benefits	Users should receive training on the value of alerts and how to manage and customise their settings, so they can choose which situations require notifications. Alert systems must be regularly assessed so they prioritise significant alerts and minimise low importance notifications
16. Lack of integration of AI-CDSS into existing healthcare infrastructures	Loss of the potential benefits of hybrid intelligence	Developers must ensure the AI-CDSS meets the specific needs of the infrastructure and are clear regarding its scope of functionality. A multidisciplinary team including software developers, clinicians, patients and administrative staff should be involved with creating and implementing a detailed plan
17. Failure to successfully integrate AI-CDSS into existing practices	User frustration and rejection of AI-CDSS and therefore loss of the potential benefits of hybrid intelligence. Poor integration could lead to disrupted workflows, increased confusion and errors potentially compromising patient care	Impose detailed implementation plans with sufficient training and education regarding the benefits, drawback and scope of functionality of the AI-CDSS. They should have customisable, accessible interfaces so users can personalise how they receive assistance. They must seamlessly integrate into workflows to maximise uptake
18. Data breach and cyber-attacks	Exposure of personal confidential information, denial of access to AI-CDSS	AI-CDSS should be installed on secure systems with appropriate levels of data security
19. Reduced HCP, patient interaction	Decreased patient satisfaction and drug adherence.	HCPs must still discuss results and decision making with patients. The time spared via AI-CDSS use may mean HCPs can spend more time providing support and developing a rapport with their patients
20. Unreliable internet connection or slow software performance	Delayed decision making could lead to frustration and rejection of AI-CDSS	Invest in reliable, high speed networks including backup systems. Offline functionality via secure downloading and data storage would also be beneficial

This table provides an overview of 20 risks associated with AI-CDSS implementation and adoption. It explores their potential consequences and offers practical mitigation strategies to minimise their effects

Results

The working group and focus groups were attended by 41 stakeholders. Of these, 19 were female and 22 were male. For further details, please refer to Fig. 1: Attendee Affiliations and Fig. 2: Attendee Expertise below.

Fig. 1.

Attendee background representation. This graph shows the background representation of the 41 individuals who attended the workshop

Fig. 2.

Attendee expertise. This graph shows the expertise distribution of the 41 participants who attended the workshop

The resultant outcomes, from the working groups and focus groups, are thematically summarised below.

What is an AI-CDSS?

The first major outcome from the workshop involved participants collating opinions and defining what a CDSS is. It was felt that in its simplest form, a CDSS is a type of decision tool that will aid medical professionals in forming a clinical decision and treating patients. Attendees emphasised that they are not necessarily technological and have been used by clinicians for decades. An example of such a CDSS given during the workshop was the Advanced Life Support, ‘ABCDE’ model, which clinicians use to form standardised approaches to assessing and treating critically unwell patients. However, the workshop’s consensus definition was that more commonly, CDSS can leverage technological advances to support decision making for many users (including carers, patients and clinicians) by making specific treatment, care and support options explicit. CDSS give evidence-based information about the associated benefits and harms, providing a decision aid for patients to help consider what matters most to them.

Attendees agreed that it is important to acknowledge the incomplete overlap between ‘AI as a medical device (AIaMD)’ and AI-CDSS. AIaMD describes a subset of software products which qualify as ‘Medical Devices’ and hence are required to comply with medical device regulations. Precise definitions of AIaMD vary across jurisdictions, but in general these tools are characterised as having a ‘Medical Purpose’ such as diagnosis or treatment of disease, or the investigation or modification of a physiological process, etc [9]. Many AI-CDSS will have a medical purpose and hence qualify as AIaMD. Whereas some AIaMD are able to operate autonomously without input from clinicians, AI-CDSS must be used in conjunction with HCP expertise, where human decision making remains the final authority. Accordingly, some tools will be AIaMD but not AI-CDSS, some will be AI-CDSS but not AIaMD, and some will be both AIaMD and AI-CDSS.

While using AI-CDSS, it was felt that HCPs must apply their clinical judgement to each situation, as they alone have the necessary scientific, ethical and moral understanding and are ultimately legally accountable for their decisions [10]. This complementary collaboration between human and artificial intelligence is known as Hybrid Intelligence. Research suggests that this partnership, which combines AI-CDSS data processing and pattern recognition power with human strengths such as empathy and creativity, can outperform human or artificial intelligence alone [11]. Studies have found that patients strongly prefer the use of Clinical AI when it’s paired with HCP oversight. Moreover, public acceptance of AI in healthcare relies on collaboration between HCPs and AI, rather than HCP replacement [12]. Reflecting these findings, there are now legal requirements such as the EU AI Act which mandates the involvement of HCPs in the oversight of AI-CDSS use, further supporting the importance of AI-CDSS, HCP collaboration [13].

The use of AI assisted decision aids spans many industries, whilst healthcare lags behind [14]. For instance, during the workshop, a maritime engineer and data scientist discussed how AI is being used to optimise routes for trans-ocean crossings, saving fuel, time and reducing CO2 emissions. Software integrates data on the vessel with data on tides, waves, wind and weather to provide a route which reportedly reduces fuel consumption and gas emissions by between 10 and 15% [15]. Although AI use within healthcare is currently limited, there are already promising signs. For instance, within radiology, studies have demonstrated that AIaMD tools can diagnose X-rays and mammograms quicker and with greater accuracy than human experts in certain circumstances [16, 17]. Moreover, AI-CDSS use can outperform human predictions alone when classifying dermal lesions [18] and improve accuracy in emergency department triage [19].

Challenges associated with AI-CDSS implementation

As previously mentioned, healthcare lags behind other industries in terms of AI adoption, with many unique challenges such as data quality and the complexity and diversity of healthcare organisations. During the workshop attendees had the opportunity to comment and reflect on the challenges of using AI-CDSS within their respective industries. This was used to construct ‘Table 1: 20 Risks and consequences of AI-CDSS implementation and adoption.’

The ‘Hippocratic Oath for AI.’

It is clear that implementation of AI-CDSS into current healthcare systems is complex with challenges ranging from practical considerations such as accessibility and usability, through to psychological factors such as motivation and trust, as well as technical concerns regarding data bias quality. The workshop introduced the novel idea of a ‘Hippocratic Oath for AI’ which could help address these challenges. This concept was raised across the focus groups and then developed in post workshop analysis, using comments and discussions recorded by the scribe.

The multi-disciplinary group consisting of a variety of experts who were present at the workshop enabled a wider appreciation of the challenges underlying AI-CDSS implementation, achieving a more holistic set of guidelines which underpin the ‘Hippocratic Oath’ for AI.

The Hippocratic Oath, one of the earliest forms of medical ethics, has long served as an ethical standard that physicians are expected to uphold [20]. Its purpose is to ensure that patients are treated equitably with instructions which highlight the importance of confidentiality, prioritising patient care and teaching within medicine [21]. There have been many alterations to the original oath, including changes such as the need for patient autonomy and the removal of instructions against abortion and euthanasia [20, 21].

While the fundamentals of the oath remain relevant, modern medicine and digital health has led to calls for it to evolve [20, 22]. Suggested updates emphasise the need to preserve the humanity in healthcare whilst recognising the growing influence of data, algorithms and Software Engineers [20]. Moreover, the Hippocratic Oath may have utility beyond just physicians. Research has suggested that adapting the oath for AI scientists could help instil an ethical responsibility and accountability in developers. This should enable us to reap the societal benefits of AI whilst keeping it safe [23].

Our proposed oath, outlined in box 2, combined the opinions of AI experts, ethicists and clinicians, providing a guiding framework for safe and effective AI-CDSS development and implementation into clinical settings. AI-CDSS developers and implementers may wish to adopt the oath, so it influences their strategies for designing and introducing AI-CDSS. If followed, we believe this has the capability to minimise challenges such as algorithm aversion and ineffective AI-CDSS use whilst maximising user trust and system accuracy and impact. This oath could therefore help unlock AI-CDSS potential, addressing critical issues in modern medicine such as staff shortages, long waiting lists and physician stress. Through supporting ethical development and safe integration, this oath paves the way for enhanced healthcare with AI, saving both physician time and improving patient care.

Table 2.

‘The Hippocratic Oath for AI’

i. AI-CDSS will be developed from evidence-based research using datasets which are representative of the patients in which the tool will be applied.

ii. AI-CDSS will be co-developed with clinician and patient input.

iii. Developers will be clear and specific about the proposed purpose and scope of their AI-CDSS.

iv. AI-CDSS developers will be transparent about what data they have used and their bias mitigation strategies.

v. Patient data will be stored safely and securely.

vi. Users will receive training addressing: explainability, usability, alerts and the strengths and weaknesses of the AI-CDSS. They will also be informed of the legal aspects of using AI-CDSS, including where accountability lies and potential risks.

vii. Each AI-CDSS will have ‘local champions’ who receive additional training, advocate for the software and can provide continuing help to staff members.

viii. Explanations of the AI-CDSS will be available to users if they desire it.

ix. Users will be given options on how they want to integrate AI into their practice.

x. AI-CDSS output will be continually monitored with multi- disciplinary oversight, and users can report on usability, accuracy, errors and impact to developers.

xi. AI-CDSS will be updated in response to changes in guidelines, research and clinical recommendations. Users must be notified of these changes and significant alterations to functionality will require new evaluation studies to be carried out.

xii. AI-CDSS will have clear, measurable standards and evaluation metrics which are continuously monitored to assess performance. Independent stakeholders should be involved in auditing and ensuring that these standards are met.

xiii. AI-CDSS will be evaluated against both human performance and ideal outcomes.

xiv. AI-CDSS will comply with all relevant regulatory guidelines.

Through the discussions and outcomes of the workshop, it became evident that successful implementation must consider human factors. This became a central theme as attendees highlighted how technical effectiveness alone is insufficient for successful AI-CDSS implementation. Implementors must consider how to engage users so that they trust the tool and want to adopt it into their practice.

Discussion

The reinvention and increased useability of AI in medicine, from early systems such as MYCIN and INTERNIST-1 [24], promise huge potential for improving patient care, alleviating physician workload and reducing healthcare costs [6].

The implementation of AI into CDSS can enhance their capabilities through advanced computational techniques such as natural language processing and other forms of deep learning. These have revolutionised the scale and detail in which data can be analysed [6]. Some AI-CDSS use patient data alongside algorithms capable of interpreting large amounts of relevant past medical information to amalgamate evidence-based diagnoses and deliver personalised treatment plans [7, 25]. The rise of ‘Big Data’ amplifies the potential of AI-CDSS by providing vast and detailed data sets for development and evaluation. These range from clinical information such as electronic medical records, investigation results and omics data, to current scientific research and practice guidelines, as well as financial data and personal information such as self-monitored sleep and exercise. Using this wealth of data, AI-CDSS has the potential to work beyond diagnosis and enhance healthcare in ways such as optimising staff and hospital procedures, predicting disease spread and enhancing preventative medicine [26].

The following discussion addresses factors influencing AI-CDSS adoption and uptake. However, while this is extremely important, the primary purpose of the ‘Hippocratic Oath for AI’ is to establish a framework for safe, ethical and useable AI-CDSS implementation. Successful implementation requires the establishment of both user trust and effective uptake, alongside ensuring that AI-CDSS are used safely and appropriately in line with ethical medical principles. These include preserving autonomy, ensuring transparency, minimising harm and maintaining accountability. However, these considerations are closely linked as AI-CDSS cannot be safely and ethically implemented without effective uptake and appropriate use. The subsequent sections therefore explore measures that can be taken to support adoption in ways that enable ethical and safe AI-CDSS implementation.

Human factors in AI-CDSS implementation

Despite the exciting prospects of AI-CDSS, research has shown that integrating them into practice can be challenging. Design issues such as the interface between the AI-CDSS and the user, along with a lack of integration with the clinical workflow, can contribute to these difficulties [27, 28]. One significant concern is alert fatigue, where users become desensitised and show decreasing responsiveness to alerts. Research suggests that between 40 and 96% of alerts are overridden and that clinicians are less likely to accept alerts the more of them they receive, especially with repetition [29, 30]. Alert nonadherence mitigates the benefits the CDSS provides and can become dangerous, resulting in unintentional patient harm. These considerations are reflected in the ‘Hippocratic Oath for AI’ through lines ‘vi, vii and ix.’ These lines highlight the need for training on the value of alerts and emphasise the importance of user autonomy so they can customise their settings and choose which situations require notifications. Moreover, line ‘x’ calls for continual monitoring so that alert systems can be regularly assessed so they prioritise significant alerts and minimise low importance notifications to reduce the impact of alert fatigue.

‘Algorithm aversion’, where users show reluctance to utilise algorithmic/statistical models despite their demonstrated superiority over human judgement, is another potential implementation barrier. There is a lack of research on whether clinicians in healthcare settings show algorithm aversion, as studies which report low AI-CDSS uptake, often do not provide sufficient information to attribute low adoption to psychological factors [28]. However, ‘algorithm aversion’ has been demonstrated in experimental settings with medical and non-medical forecasting tasks [31, 32]. Algorithm aversion can undermine the benefits of AI-CDSS implementation and reduce the potential for hybrid intelligence [11]. Research indicates that through addressing psychological factors such as perceived lack of transparency, ability to adapt outcomes (autonomy), and social norms, it is possible to improve algorithm acceptability and adoption [27, 33, 34]. These factors are considered through lines ‘iii, iv, vii and ix,’ of the ‘Hippocratic Oath for AI,’ and therefore can support AI-CDSS implementors in reducing algorithm aversion.

However, while reducing algorithm aversion, attendees felt that AI-CDSS implementers must also take care to promote education which prevents overtrust in AI. For instance, research has found that dermatologists using AI to assist with diagnosis of skin lesions, changed an initially correct clinical diagnosis to an incorrect AI suggested diagnosis in 39% of cases [35]. HCP overtrust In AI can similarly neglect the potential benefits of hybrid intelligence, but also disregards the essential aspects of the HCP in decision making such as intuition, experience and contextual understanding.

Automation bias occurs when users become overly trusting or reliant on decision support systems, leading to reduced vigilance and a potential for errors [36]. Research supports the existence of automation bias, and it is more common in the case of less experienced users [37, 38]. This is particularly relevant to future AI-CDSS implementation, as advancements in technology and AI-CDSS capabilities could increase the tendency for HCPs to rely on them [39]. To mitigate the effects of automation bias, studies have highlighted the need for further research to investigate how specific user biases and context surrounding decisions influence AI-CDSS use [39]. Moreover, the use of a certainty index, a measurement of the certainty/confidence the AI-CDSS has in its recommendation, has been shown to reduce the impact of automation bias [39, 40].

Considering and understanding human factors such as medical and technological expertise, personality, cognitive biases, and trust [41] and how they interact with the AI-CDSS will be instrumental for successful implementation and was critical when constructing the ‘Hippocratic Oath for AI.’ Anticipating which users are likely to benefit will enable developers to tailor the AI-CDSS in terms of accessibility and training. Research has shown that there is no one-size-fits-all approach and has highlighted the need for personalised strategies to support diverse user needs [42]. These considerations are reflected in the oath’s emphasis on adaptability, transparency and autonomy (lines ii, iv, ix and x.) For long term success, attendees of the workshop emphasised the importance of continuous monitoring, including user feedback post-integration. It was suggested that regular user-developer meetings would enable refinement of AI-CDSS, so that it can adapt to meet evolving user expectations and clinical needs, whilst staying usable and practical for everyday use. This is aligned with the oath’s call for continually monitoring with sustained multi- disciplinary oversight (line x.)

Explainability, transparency and trust

For widespread AI-CDSS implementation, attendees agreed that users must trust the algorithms and technology underpinning it [43]. Trustworthy AI requires transparency, with clear processes which are open to scrutiny by regulatory bodies such as the Medicines and Healthcare products Regulatory Agency (MHRA). This includes using well-documented and appropriate data sources and algorithms that can be investigated and have their biases mitigated or alternatively made explicit so that users can compensate [44]. The ‘Hippocratic Oath for AI,’ supports this principle through lines ‘i’ ‘iii’ and ‘iv.’ These call for the use of representative datasets and for developers to be transparent with the scope of their AI-CDSS, the data sources used, and any potential biases present in the system.

Explainable AI (XAI) takes transparency further by making the AI-CDSS theoretically understandable to users. This can be achieved through making the logic and algorithms interpretable, comprehensive and by providing insights into how different input data items influence the algorithm’s advice [45]. Alongside developing trust, XAI may yield other benefits including improving how HCPs communicate decisions to patients, reduced automation bias and enhanced error identification [45]. Unfortunately, XAI also poses challenges such as the limitations of approaches which attempt to produce explanations from complex, often opaque deep learning models [45, 46]. Critics of XAI are concerned that explainability will come at the price of reducing predictive power in favour of simplicity. For instance, developers may prioritise simpler algorithms without the use of complex deep learning systems which, while having greater explainability, may have inferior predictive accuracy and efficacy [47].

During the workshop, it was suggested that the importance of explainability may be overstated. It was felt that we cannot expect total explainability from an AI-CDSS when we do not have total explainability for many of the mechanisms of drugs it may be recommending [48]. For example, research estimates that 6300 tonnes of paracetamol are sold in the UK alone each year [49]. Despite this huge quantity, doctors prescribe it, and patients take it, with neither fully understanding its mechanism of action [50]. Current decision making by HCPs is also not fully explainable. Decisions can be affected by subtleties such as mood, intuition and experience. In cases where decision making and competence are assessed and scrutinised, HCPs must rely on their potentially fallible memory and notes which will likely not provide complete explainability. There is, of course, a question as to the extent to which AI CDSS would need to be more explainable than our current gold standard, provided it is safe and effective.

Furthermore, there is research which undermines the importance of XAI in AI-CDSS adoption. When GPs were given information regarding the derivation, validation and accuracy of a cancer risk algorithm, it did not significantly improve integration into their judgements. Interestingly, ‘social proof’ (where information concerning algorithm usefulness was transferred between peers) did significantly improve algorithm adoption [34]. This suggests other factors such as comprehensive training on how and when to use the AI-CDSS and guidance on its strengths and weaknesses [51], may be more impactful in increasing uptake than explainability.

After detailed discussion, it was felt that a balanced implementation of XAI would be the most appropriate, as explainability should not be viewed as a universal and standardised requirement for all users. Instead, explainability should be tailored to the specific function of the AI-CDSS and the needs of its users. For instance, patients may not need a deep understanding of the mechanisms underpinning AI-CDSS involved in their care anymore than they would need a detailed understanding of pharmacology when deciding whether to start a new medication. However, as with prescribed medications, they should be adequately informed so they can balance the risks and benefits of an intervention being used in their care.

This approach is reflected in the ‘Hippocratic Oath for AI.’ For example, while full explainability is not a universal requirement, line ‘viii’ calls for explanations of the AI-CDSS be available to users if they desire them. This knowledge can help HCPs trust the AI-CDSS, effectively communicate its insights to patients and confidently integrate it as one of many sources of evidence into their decision making. Alongside explainability, the oath includes other factors which may improve algorithm adoption. For instance, comprehensive training on explainability, usability and the strengths and weaknesses of the AI-CDSS (line vi) or the presence of ‘local champions’ who can advocate for the tools, providing social proof (line vii.)

Limitations and future directions

Overall, the oath provides a comprehensive ethical framework that addresses a broad range of factors essential for successful AI-CDSS implementation. These include human factors and explainability but also critical ideas for maintaining AI-CDSS safety and efficacy as reflected in lines ‘xi’, ‘xii’ and ‘xiii.’ Although the o ath provides a strong basis that implementors can use to guide successful implementation, it has limitations. Firstly, while the oath provides a generic framework, it may lack details needed to support effective implementation in some specific clinical contexts. Moreover, the rapidly evolving nature of AI may lead to the oath becoming outdated, requiring updates to maintain relevance. Finally, some of the criteria of the oath such as continuous monitoring with multi-disciplinary oversight, and the comprehensive training may be financially or logistically unfeasible, especially in resource limited settings.

Moving forward, further iterations of the oath would become more specific, with adaptations tailored to different clinical areas of AI-CDSS implementation. Moreover, adaptations would include versions with practical alterations which may make the oath’s criteria more economically feasible. Continuous monitoring of the oath, alongside implementor feedback, can help make updated revisions technologically relevant while also being practically effective in clinical contexts.

Conclusion

This paper summarises discussions and outcomes of the Turing-Roche and Clinical AI Interest Group Joint Workshop. At the workshop, experts explored themes such as: defining a CDSS, human factors, explainability and challenges involved with AI-CDSS implementation and adoption. Debate on these themes led to the idea of a ‘Hippocratic Oath of AI,’ which could facilitate safe and effective AI-CDSS development and implementation.

As AI advances and healthcare data increases in quantity and availability, we expect to see an exponential growth in the development and adoption of AI-CDSS. To match this, there is a pressing need to establish transparency and safety frameworks as mandated by law through the recent EU AI act [44]. Our ‘Hippocratic Oath of AI’ suggests a concrete pathway to promote the development and implementation of AI-CDSS are in a safe and effective manner. The clauses of our oath should be considered as a guiding checklist for developers and those responsible for procuring and implementing AI-CDSS. The oath calls for essential requirements, such as prolonged user-developer feedback, transparency in data usage, the need for explainability, and a requirement for AI-CDSS to allow personalisation. Among others, these requirements will aid us all in creating, implementing and developing safe, usable, trustworthy and effective AI-CDSS. Our oath has the capability to help unlock AI-CDSS potential, facilitating hybrid intelligence where AI-CDSS can complement HCP expertise in a personalised manner. This collaboration can lead to many benefits including improved patient outcomes and reduced healthcare costs, ultimately revolutionising healthcare practice.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

AI

Artificial intelligence

CDSS

Clinical Decision Support System

HCP

Health Care Professional

AIaMD

Artificial intelligence as a Medical Device

XAI

Explainable AI

EU

European Union

Author contributions

SB and NF drafted the initial manuscript. JA, CB, TC, VH, CH, OK and EK substantially altered the manuscript. SB, BA, JA, CB, TC, KC, AD, VH, CH, JN, OK, EK, JW, BM, NF edited, refined the manuscript and reviewed the final draft.

Funding

This work was supported by The Alan Turing Institute through Turing Clinical Interest Group funding and the Turing-Roche Strategic Partnership.

Data availability

Not applicable.

Declarations

Ethics approval and consent to participate

This study involved a multidisciplinary stakeholder workshop with expert discussions rather than a research study involving human participants where personal data is recorded. In line with guidance from the UK Health Research Authority, formal research ethics approval was therefore not required. Participation in the workshop and related discussions was entirely voluntary and was not a requirement of membership in the Clinical AI Interest Group. All contributors provided consent for their contributions to be used in the preparation of this manuscript and for their acknowledgement as contributors.

Consent for publication

All authors have consented to publication.

Competing interests

NF reports consultancy for DocHQ. CH is affiliated with Roche Pharmaceuticals. Authors affiliated with these organisations were involved with the design of the event and drafting of the manuscript.