Using Artificial Intelligence in Patient Care—Some Considerations for Doctors and Medical Regulators

Abstract

This paper discusses the key role medical regulators have in setting standards for doctors who use artificial intelligence (AI) in patient care. Given their mandate to protect public health and safety, it is incumbent on regulators to guide the profession on emerging and vexed areas of practice such as AI. However, formulating effective and robust guidance in a novel field is challenging particularly as regulators are navigating unfamiliar territory. As such, regulators themselves will need to understand what AI is and to grapple with its ethical and practical challenges when doctors use AI in their care of patients. This paper will also argue that effective regulation of AI extends beyond devising guidance for the profession. It includes keeping abreast of developments in AI-based technology and considering the implications for regulation and the practice of medicine. On that note, medical regulators should encourage the profession to evaluate how AI may exacerbate existing issues in medicine and create unintended consequences so that doctors (and patients) are realistic about AI’s potential and pitfalls when it is used in health care delivery.

Introduction

Artificial Intelligence (AI) is broadly defined as a computer programme with the capability to perform tasks and to reason—processes that are usually associated with human intelligence. It encompasses a wide range of technologies including facial recognition, machine learning, natural language processing and robotics—each with its own distinct ethical and legal implications (Lupton 2018; Rigby 2019).1,2 Because AI excels in image-recognition tasks and providing quantitative assessment in an automated fashion, it is particularly suited for specialties that interpret images such as radiology, pathology, dermatology and ophthalmology (Hosny et al. 2018). In addition, with algorithms increasing in their capabilities, there is scope for AI technologies to transform health care by tackling variation, optimising treatment planning and forecasting outcomes of care (Truog 2019; Hashiguchi et al. 2022).

Despite its potential to improve the technical aspects of care, what is less clear are AI’s implications on the clinician-patient relationship, and health care more broadly. That in turn raises ethical challenges for health professionals who use AI in patient care and the professional bodies who regulate them (Rigby 2019; Nagy and Sisk 2020; Australian Medical Association 2023). Those challenges are numerous and varied ranging from how to protect patient privacy and ensure that clinical data is stored securely and used appropriately, to ascertaining what measures should be taken when an AI system generates unexpected results. Other challenges include possible disruptions to current health care roles, the need to reconfigure core clinical skills and the de-personalisation of care if greater emphasis is given to machines (Zhang and Zhang 2023; Cabitza et al. 2017). All that highlight the need for safeguards so that the quality of care and safety remain paramount when AI is used with patients.

This paper discusses the key role medical regulators (health registration authorities) have in setting standards for clinicians who use AI in patient care. Given their mandate to protect public health and safety, it is incumbent on regulators to guide the profession on emerging and vexed areas of practice. This is necessary particularly as technological advancements have outpaced current policy and professional guidance, and the medical profession remains largely unaware of the ethical and practical challenges that accompany AI technology3 (Rigby 2019; TerKonda and Fish 2023). While this paper focuses on doctors and medical regulators, the issues discussed will also be relevant to other health professions as a range of disciplines are often involved in the patient’s care (McKinlay et al. 2021).

The paper contains three main sections: starting with the premise that effective regulation of AI involves understanding what AI is and the challenges using it raises, followed by what medical regulators should consider when devising professional guidance on AI. As effective regulation of AI extends beyond setting standards, the third section discusses the broader role of regulators. The overall aim of this paper is therefore to help medical regulators consider what effective regulation of AI might mean in their context, and to take steps to facilitate that.

Effective Regulation of AI involves Understanding What AI is and the Ethical and Practical Issues Related to its Use

Medical regulators are the only professional body with a statutory mandate to regulate the practice of medicine (TerKonda and Fish 2023; Wolf 2020). One way they do that is by setting standards for the profession on what good medical practice entails. These standards typically cover basic fundamental topics such as record-keeping, prescribing medication and maintaining professional boundaries (Medical Council of New Zealand 2021).

However, few medical regulators have specific professional standards on AI. Given that AI is likely to introduce new risks and amplify existing ones when used in patient care, it is vital that regulation keeps pace (Smith et al. 2023). That said, formulating guidance in a novel area of practice is a challenging exercise and one that is likely to involve considerable time and effort. Essentially, regulators are navigating unfamiliar territory as their board members and staff will have to be educated on what AI is and the ethical and practical issues related to its use in order to guide the profession effectively (TerKonda and Fish 2023). This section discusses some pertinent and overlapping issues that regulators should be aware of but those issues are by no means exhaustive.

First, is the issue of agency. To date, professional standards have tended to focus on the doctor-patient relationship and intra-professional relationships between health professionals. As AI could change the dynamic between patients and health professionals (particularly doctors), medical regulators will need to devise standards that take into account the presence of a non-human agent and its involvement in clinical care. This requires thinking through the implications of a triadic relationship between patients, doctors and technology, and the accompanying lines of responsibility and accountability especially when errors or near-misses occur (Smith et al. 2023).

Several professional bodies (such as the Australian Medical Association and the Royal Australian College of General Practitioners) state that ultimately, decisions on patient care should always be made by a human (usually a doctor). However, holding an individual responsible for the actions of a system could be problematic especially if that system is uninterpretable (the black box conundrum), or takes on more autonomous decision-making functions (Boniface 2020). Although there have been attempts to classify AI as assistive, augmented or autonomous4 in part to ascertain the appropriate level of oversight and applicable regulatory framework, it can still be difficult to identify the persons or processes driving or controlling the AI tool even with these classifications.

Secondly, AI could add to the complexity in clinical decision-making. Unlike other software used in health care, AI systems are able to auto-learn from the large datasets they are fed and to improve in their performance. This means that an AI system could potentially alter itself based on user interactions and the clinical situations in which it is applied such that it could be a fundamentally different machine at a later time. This is a feature in deep learning which relies on a “family” or network of algorithms and uses many layers of artificial neurons to solve more complex problems, although reliability and predictability issues have also been identified in machine learning tools (Macrae 2019). All that creates novel regulatory challenges if doctors are not able to fully comprehend and explain to their patients, the basis of a result or recommendation generated by an AI system. Even where a doctor is able to offer some form of explanation to the patient, they would need to be sure that their advice remains robust given the possibility that the AI system could continue to evolve. This has implications particularly if the patient queries the recommendation at a later point in time or experiences an adverse event or a near-miss (Boniface 2020). Where that mishap is major, it could undermine public and health professional confidence in AI (Reddy et al. 2020).

Thirdly, AI systems can often be opaque to end users and even to its designers which raises issues of trust and transparency. An example is a deep-learning algorithm that may have a superior diagnosis rate including the ability to generate results in a shorter timeframe.5 However, there may not be any way to determine how and why it reached a particular recommendation given that deep learning algorithms continuously fine-tune their parameters (Reddy et al. 2020). This lack of transparency makes it difficult to troubleshoot and ascertain the logic of an AI system when seeking to apply it to a different patient population (Hosny et al. 2018). Consequently, it would be near impossible for doctors to learn from their decisions and to explain their reasoning to their patients. Even if they manage to do so, translating AI’s recommendations in lay terms remains a challenge. This is because AI generates statistical correlations that may be difficult to convert into a meaningful medical explanation especially if the doctor is not well-versed on data science (Zhang and Zhang 2023). That in turn affects patients’ trust in AI-generated recommendations which could be problematic when obtaining consent for treatment (Eglinton et al. 2023).

Trust is foundational for good care, and medicine is primarily a communal knowledge system that relies on doctors communicating with their patients and colleagues about diagnoses, situations and treatments. Trust can also improve clinical outcomes when patients engage with and adhere to their doctor’s advice (Nagy and Sisk 2020). Given that doctor-patient interactions are integral to medical information gathering, patients may be less accepting of a technological substitute if it adds complexity and alienates them from their doctor (Royal Australian College of General Practitioners 2021; Grouse 1983). Furthermore, patients may also not be receptive to AI’s involvement if they perceive that they are receiving lower quality care as a result (Boniface 2020; King 1987).

Fourthly, medical regulators will need to ensure that their regulatory framework is sufficiently robust such that it can accommodate safety and medicolegal risks arising from doctors’ use of AI (Royal Australian and New Zealand College of Radiologists 2022). For example, the Medical Council of New Zealand (2022a) currently defines the practice of medicine as “assessing, diagnosing, treating, reporting or giving advice in a medical capacity, using the skills, attitudes and competence initially attained for the MBChB degree (or equivalent) and built upon in postgraduate and continuing medical education, wherever there could be an issue of public safety.” That definition assumes that a human is involved in diagnosing and providing medical advice to the patient. Going forward, it may require revision especially if AI assumes a greater role in clinical decision-making.

Furthermore, regulators may also need to review existing mechanisms such as how different scopes of practice are defined if more doctors use AI in patient care or take on AI-specific clinical roles. In particular, it involves assessing whether certain scopes of practice6 should be expanded to reflect the increasing involvement of AI and/or whether to create a new AI-specific scope of practice. Such decisions entail re-visiting the required qualifications, training and continuing professional development programme for doctors registered under each scope and evaluating the impact a new scope will have on the medical profession as well as other health professions (Smith et al. 2023; Scott et al. 2023). Ascertaining that involves considerable time and effort not just on the part of the medical regulator but also other stakeholders such as AI developers, medical schools, specialist medical colleges, clinical supervisors and clinical facilities (particularly where they have teaching and training responsibilities).

One important consideration is whether creating another clinical specialty will ultimately be beneficial (particularly for patients). Modern medicine is already extremely complex with a wide range of specialties,7 and with treatments available for thousands of diagnoses and conditions. While specialisation has led to significant medical and surgical advances over the years, there is also a risk that introducing another specialty into the mix could make medicine even more complex and result in further fragmentation within the health system (Gawande 2011; Chong 2014).

Fifthly, AI technology is primarily driven by large commercial interests whose motives may not align with the medical profession or the public. For example, professional bodies such as the Australian Medical Association and Royal Australian College of General Practitioners state that AI must only be used in patient care with appropriate ethics oversight, and where there is rigour in assessing the safety and efficacy of that AI tool. In contrast, venture capitalists tend to promote “breakthroughs” based on theoretical evidence even though that may not result in cost-effective care or improved outcomes for patients (Francis 2023; Scott et al. 2023). While it is beyond the scope of this paper to discuss the regulation of private companies who develop AI systems, medical regulators will need to be aware of the lucrative landscape fuelling AI’s growth and influence, and to understand its implications for medical practice.

In summary, these are complex issues. However, they are issues that medical regulators will need to grapple with if they are to guide doctors effectively in a dynamic and evolving area of medicine.

What should Medical Regulators Consider when Devising Professional Guidance on AI?

Devising guidance on a novel area of practice involves incorporating the professional values, knowledge, skills and behaviours expected of doctors and practical guidance that they can apply to their setting. While this section highlights some broad considerations for regulators, what is discussed is not exhaustive.

Draw on Existing Standards as a Starting Point

As end users of AI systems, doctors need professional guidance that is clear, practical and legally and ethically defensible. Where regulators have existing standards on key aspects of practice (for example, obtaining informed consent), they should draw on that as a starting point. This is because the principles of good medical practice (such as effective communication and professionalism) are still applicable even when using AI (Federation of Medical Regulatory Authorities of Canada 2022). If anything, as health care becomes more high-tech, the rudimentary aspects of practice such as good bedside manner are likely to matter even more to patients (Truog 2019).

However, merely transposing existing regulatory standards is unlikely to be adequate. Given the ethical and practical challenges that using AI raises, medical regulators will need to tailor their guidance by considering the implications of involving a non-human agent in a clinical setting (Smith et al. 2023; Ganapathy 2021). For example, when discussing informed consent, professional guidance on AI should highlight the importance of patients being aware that their treating doctor is drawing support from AI applications. Patients should also be informed about the capabilities and limitations of those applications so that they are empowered to accept or decline treatment involving AI (Victor et al. 2023; Zhang and Zhang 2023). While that may involve more time and effort during a clinical encounter, professional guidance should remind doctors to identify what matters to the patient they are caring for and to consider what would be reasonable to convey to that patient. This is because the quality of the doctor-patient relationship and the clinical decisions made as a result depend crucially on how they discuss information and communicate with each other (Loftus et al. 2020; Quinn et al. 2022). In other words, for professional guidance on AI to be instructive, it should help doctors adhere to good medical practice and to navigate situational challenges that arise when they use AI in their respective setting.

Recognise which Aspects of AI are Within the Medical Regulator’s Remit

Globally, AI technologies remain under-regulated as many countries have yet to formalise standards for assessing AI’s impact and safety. That can create barriers for clinical facilities and doctors who are looking to deploy AI in their practice setting, and perpetuate unsafe practices in clinical settings where AI is already being used (Reddy et al. 2020). Even so, a medical regulator’s remit is confined to doctors, and not how AI systems or its developers should be regulated even if those impact on doctors as end users.

This does not mean that doctors and medical regulators should have no involvement in offering a values-based perspective, or advocating for safe and fit-for-purpose AI technologies in health care. On the contrary, encouraging doctors to provide input on the design, testing, implementation and post-market surveillance of AI products increases the likelihood that these technologies will be safe and appropriate for use with patients (Pierson and Tsai 2023; Australian Medical Association 2023). By recognising which aspects of AI are within the regulator’s purview, it ensures that regulators remain focused in their approach and in the guidance they provide.

Emphasise the Need to Use AI Responsibly

How algorithms operate are affected by their design and the assumptions and values they are programmed with. Where an algorithm is flawed, the implications are greater than a single misdiagnosis by a doctor because of the risks that an automated system will replicate more errors (Zhang and Zhang 2023). On that note, medical regulators should emphasise the need to use AI responsibly so that it does not worsen existing health and social inequalities (Federspiel et al. 2023; World Health Organisation 2023a). In line with the principles of beneficence and non-maleficence, this entails using AI systems only for clinically justified tasks that enhance patient care and in areas of practice where the doctor is skilled and trained in. For example, before performing a robotic-assisted procedure, surgeons and surgical teams will need to undergo simulation and training to develop their psychomotor skills, to be competent in operating the robotic console used in theatre and to find ways to communicate effectively between team members given the inevitable reduction in non-verbal cues and exchanges during surgery. All of that is crucial so that the robotic-assisted procedure does not compromise safe, efficient surgical care for the patient (Clanahan and Awad 2023).

Using AI responsibly also includes doctors evaluating the context in which an AI system was trained and tested when assessing its suitability for the local patient population (Royal Australian and New Zealand College of Radiologists 2022; Royal Australian College of General Practitioners 2021). However, given each patient’s unique medical history, condition, views and preferences, doctors would need to ensure that the AI-assisted technology they use facilitates patient-centred and individualised care even for automated tasks. In reality, this can be difficult to achieve particularly if the data driving the AI system is biased or contains inaccuracies or incorrect assumptions which could then generate erroneous recommendations or results for a patient and affect their treatment options and/or access to care (Chappell and Teven 2023).

That said, the problem of bias is difficult to avoid given that most training data are imperfect and may not contain sufficiently representative, diverse and accurately labelled data that one would like to have. Even if an AI tool is trained on very diverse data, doctors should still be mindful of preconceived notions and possible bias on their part when they interpret the results generated by AI (Haug and Drazen 2023). While data governance is outside a medical regulator’s remit, they have a role nevertheless in cautioning the profession that data sets powering AI systems are often imperfect and incomplete which can have implications for individual patients. On that note, regulators should remind the profession that an AI tool is not (and should not be) a substitute for exercising clinical judgement as a doctor, and that deferring to AI as the decision-maker could impair a doctor’s relationships with his/her patients (Quinn et al. 2022).

However, because AI is rapidly evolving, medical regulators will need to balance between a risk-based approach, and encouraging innovation and improvement within the profession. Specifically, medical regulators should avoid introducing uncertain and impractical requirements such that compliance becomes burdensome and costly (Australian Medical Association 2023). In other words, the regulator’s overall objective should be that doctors practise safely when they use AI, and to minimise bureaucracy so that doctors do not spend a disproportionate amount of time on administrative tasks.

Start Somewhere given that AI Is a Fast-Changing Field

Opinions vary among professional bodies on what guidance they should provide to doctors who use AI in patient care. For example, the Federation of Medical Regulatory Authorities of Canada (FMRAC)’s Working Group on Artificial Intelligence and the Practice of Medicine consider it premature for medical regulators to recommend minimum standards expected of doctors. In FMRAC’s view, it would be more beneficial to gather additional evidence and real-world experiences on how AI is integrated into patient care before establishing specific standards for doctors (Federation of Medical Regulatory Authorities of Canada 2022). In contrast, the Royal Australian and New Zealand College of Radiologists has taken a pro-active approach by acknowledging AI’s impact and potential in the fields of radiology and radiation oncology and developed a set of ethical principles, practice standards and a position paper on regulating AI in medicine.

Professional bodies will invariably differ in their views and approaches about what (if any) guidance they should have on the use of AI in patient care. In part, this reflects the extent in which AI has penetrated their country’s health care.8 Irrespective of what position medical regulators adopt, that will not hamper AI’s progress nor deter doctors from exploring its uptake (for example, using ChatGPT or Nabla Copilot to transcribe patient notes). For that reason, a proactive approach9 is advisable given that regulators have a mandate to promote good medical practice and protect public health and safety. Conversely, leaving it to industry to fill that gap is risky particularly as they have vested interests and have at times exaggerated AI’s potential (Pierson and Tsai 2023). For medical regulators that do devise professional guidance on AI, they should cover areas such as:

• The skills and knowledge doctors require to use AI safely with patients.

• What effective communication with patients and other clinicians means when a non-human agent is involved, and how to facilitate informed decisions in that setting.

• When is it appropriate to use AI applications in the diagnosis and treatment process? Conversely, in what situations should a doctor revert to their own assessment/clinical judgement?

• Expectations for reporting concerns when using AI (for example, errors or unusual recommendations, inaccurate or unrepresentative data, privacy breaches and systems malfunction) and guidance on handling adverse events and near-misses.

(Smith et al. 2023; Medical Council of New Zealand 2020; Royal Australian College of General Practitioners 2021).

Even so, professional guidance has its limitations as it is only one aspect of regulating medical practice involving AI. While it outlines what is expected of doctors who use AI in patient care, the quality of care provided is also contingent on other factors beyond a medical regulator’s control such as how easily AI can fit within existing clinical workflows and how rigorously it will be assessed before deployment (Macrae 2019; Kovoor et al. 2023). Nevertheless, it is important that medical regulators start somewhere so that doctors are aware of the regulatory and liability implications, and the standards they should adhere to when they use AI in patient care.

Broader Role of Medical Regulators

Effective regulation of AI extends beyond devising guidance for the profession. It includes keeping abreast of developments in AI technology and considering its implications for the practice of medicine. Because AI technology is constantly changing, it is vital that regulators are agile and responsive in their approach (World Health Organisation 2023b).

At present, there is considerable hype and hope that AI could transform medicine even though its use is still nascent in real-world clinical settings (Royal Australian College of General Practitioners 2021; Hashiguchi et al. 2022). That is because AI is unlike human intelligence despite the intention to simulate intelligent human behaviour. For example, current AI applications are “narrow” in that they are programmed to accomplish very specific tasks but are rarely able to generalise outside the boundaries of their training and to make associations the way a human brain does.

Because the field of AI is still in its infancy, it is important that expectations are tempered (Hosny et al. 2018). On that note, a responsible approach on the part of medical regulators includes cautioning doctors against novelty bias (the assumption that the newest invention or breakthrough will improve health outcomes) and an overly optimistic faith in technology (Francis 2023). Related to that, the medical profession would do well to understand that the use of AI could potentially exacerbate rather than mitigate existing issues in medicine and create unintended consequences for patients, doctors, health systems and society more broadly (Nabi 2018; Cabitza et al. 2017).

The issues discussed below are not exhaustive. While they are difficult to reflect succinctly in a set of professional guidelines, they touch on the business of medicine and the institutional and economic contexts in which medicine is practised today (Sparrow and Hatherley 2020). As such, they are pertinent issues that medical regulators should encourage doctors to reflect on particularly as AI continues to grow in its use and influence. Although each issue warrants more in-depth discussion than the scope of this paper allows, they have been included to raise awareness and facilitate dialogue within and beyond the medical profession.

The Use of AI could Exacerbate Overdiagnosis and Overtreatment

Advances in medicine and technological developments have fuelled demand for care and pushed up the cost of providing it. There is also a growing recognition that overdiagnosis and overtreatment occur in the course of providing care which are potentially harmful for the patient and wasteful for the wider health system (Hashiguchi et al. 2022).

Overdiagnosis and overtreatment affect accessibility and affordability which are growing concerns in health care. While some patients stand to benefit from technological advancements, the availability of high-tech care could also make health care less accessible and affordable for others. For example, the demand for medical appointments could increase if patients self-refer based on recommendations from an AI tool (Royal Australian College of General Practitioners 2021). This requires asking questions about what it means to use AI responsibly so that it does not encourage or exacerbate overtreatment, or widen existing disparities. On that note, medical regulators could take the lead by encouraging doctors to reflect on whether using AI could:

• expand the definition of diseases, and what the flow-on effects of that would be on patients, doctors, treatment facilities, health systems and societies more broadly?

• make it harder to manage patients’ expectations especially if the results generated by an AI tool was unexpected, unusual and/or led to an adverse outcome?

• encourage more people to undergo investigations and screening even when there is no clinical basis or advantage in doing so?

• increase the demand and use of health care services (especially when the health system is already stretched)?

• divert health resources and funding from where health needs are higher to invest in developing more AI applications?

Conflicts of Interests may arise from Engaging and Interacting with the AI Industry

Many AI applications are developed in research labs and environments that require significant capital outlay. In the last few years, AI applications in health care have created enormous attention and generated immense public and private sector investment. In particular, billions of dollars are being poured into companies whose aims are to develop artificial general intelligence, that is machine intelligence that surpasses human abilities (Pierson and Tsai 2023). Consequently, the health AI market is estimated to grow tenfold from 2020 to 2026, and to attract technology giants not traditionally associated with health such as Google, IBM and Microsoft (Zhang and Zhang 2023; Banja 2020).

In the previous section, a point was made that doctors should be involved in guiding and overseeing the development and deployment of AI systems particularly as they will be the end users of these technologies. However, literature on AI is often silent about the influence industry could have when doctors engage with them whether that is through undertaking advisory/consultancy work for an AI developer, receiving funding to conduct research on AI, attending an educational event or hosting an industry representative on site to learn more about an AI system and its potential to enhance different areas of clinical practice.

Irrespective of the form of engagement, interactions with AI commercial enterprises could potentially influence a doctor’s views about AI and his/her clinical decisions to use it particularly if the doctor receives some form of benefit from that commercial enterprise. Those benefits have a subliminal effect in that social science research has found that gifts and incentives engender a certain sense of obligation and reciprocation in the recipients (Chong 2017). In particular, doctors tend to prefer or favour a company’s products or services, and to report findings from their research in a more positive manner when they receive some form of benefit from that company (Royal Australasian College of Physicians 2018; Medical Council of New Zealand 2023).

This is not to say that doctors’ interactions with industry is unethical per se. Indeed, it can be beneficial to some extent as doctors and industry do share some common goals in the prevention, control and management of diseases, and in undertaking research to improve health outcomes. What is at stake is recognising and managing potential conflict of interests because doctors have an obligation to maintain their professional integrity and to prioritise their patients’ interests ahead of their own. On that point, professional guidance on AI should ideally caution doctors about the significant commercial interests tied up with AI systems which could compromise a doctor’s professionalism and influence their judgement particularly when they have some form of engagement with the AI industry.

Managing Uncertainty even though AI Increases the Amount of Clinical Information Available to Doctors

The outset of this paper noted that AI has the potential to increase and improve precision in medicine, and how information is organised and managed. AI also has the potential to generate more information about an individual patient including recommendations about their lifestyle and how certain conditions should be monitored and addressed. However, there is a downside to the growing volume of information as there can be unintended consequences such as information fatigue and privacy breaches (Federspiel et al. 2023). It can also compound the pressure of modern medical practice as more time will be spent discussing clinical information with patients and coming to a shared decision, along with keeping abreast and sifting through a greater volume of information especially if those relate to novel treatments. A related problem is how best to manage an exponential increase in the volume of clinical information available (Francis 2023; Truog 2019).

While access to more clinical information can guide treatment decisions, it will not eliminate or reduce the patient’s risk factors, nor the risks of undergoing any recommended course of treatment or procedure. More clinical information will also not eliminate uncertainty which is an inherent and unavoidable part of medicine irrespective of how skilled or experienced the doctor (Chen and Asch 2017; Hatch 2017). Put simply, no medical treatment is risk-free even if it incorporates cutting-edge technology such as AI.

For many patients, continuity of care and connecting with their doctor matter more than getting answers to specific clinical questions (Wellbery 2012). On that note, if the focus of care becomes more about precision and arriving at the correct diagnosis, there is a risk that using AI could reduce the quality of relationships between doctors and patients and contribute to an impersonal, technology-dominated health system (Nagy and Sisk 2020; Grouse 1983). For that reason, medical regulators should emphasise that good bedside manner remains fundamentally important when AI is used in the patient’s care, and that includes recognising and accepting that ambiguity and uncertainty will always be part of every clinical setting (Helman 2007, 167–169).

Understanding the Limits of AI Technology and Why that Matters

AI is capable of assessing complex data and generating recommendations at impressive speed and volume. However, it is not a panacea because technology is only one aspect of the health care ecosystem. Good health care also requires clear and effective communication with patients and other members of the clinical team, co-ordination of care, teamwork and working collaboratively with colleagues. Often, it is these core functions done consistently and continually over a period of time that go a long(er) way in improving health outcomes for the patient (Gawande 2017). So, although AI could open up new possibilities and solutions, not all clinical problems can be solved by creating algorithms. Rather, for an AI tool to be useful in a clinical setting, it will need to address real-world problems (Coiera 2019; Reddy et al. 2020).

Doctors who use AI in their care of patients would do well to recognise AI’s capabilities and limitations including which tasks and areas of practice AI is better suited for. Understanding that helps doctors to use their time and limited health resources wisely, and to be realistic about what AI can and cannot achieve for them and their patients (Rigby 2019). Given the explosive growth of AI powered applications in all aspects of daily lives, it is even more important to keep AI in perspective—that AI is a tool that could support and potentially enhance medical practice but it is not the ultimate solution in and of itself (Richards 2023; King 1987).

Conclusion

AI is predicted to transform how health care will be delivered in the future. Even though AI’s use is still nascent in real-world clinical settings, there are clearly risks along with clinical and social implications when it is deployed and integrated in patient care. To ensure that doctors use AI safely and appropriately, it is incumbent on medical regulators to guide the profession by devising standards. To do that, medical regulators will need to familiarise themselves with what AI is and the myriad of ethical and practical issues using it raises. While many of those issues are complex and professional guidance is only one aspect of regulating medical practice involving AI, it is important that regulators start somewhere and take the lead by outlining expectations for the profession in a vexed and rapidly evolving field.

That said, effective regulation of AI extends beyond devising guidance for the profession. It includes keeping abreast of developments in AI technology so that medical regulators can be more agile and responsive. In addition, medical regulators have a broader role in encouraging dialogue within the profession (and beyond) on how the use of AI could potentially exacerbate existing issues in medicine and create unintended consequences for doctors, patients, health systems and the wider public. In particular, regulators should encourage the profession to reflect on contemporary issues such as overdiagnosis and overtreatment, potential conflicts of interests especially as the health AI market grows and the challenges of managing uncertainty even though AI increases the amount of clinical information available.

Irrespective of what steps regulators take, AI is only one aspect of the health ecosystem. Clear and effective communication, co-ordination of care and teamwork are also crucial aspects of improving health outcomes for patients. Put simply, patients should still be central no matter what technology is employed. In the quest to improve health care delivery by using AI, it would be regrettable if the focus shifts from patients to machines, and doctors spend the vast majority of their time caring for patients away from the bedside rather than with the patient.

Declarations

Conflict of Interest

I have been a Senior Policy Adviser and Researcher with the Medical Council of New Zealand (MCNZ) since December 2014. MCNZ is responsible for registering doctors who practise in New Zealand and setting standards for the way our doctors practise. A large part of my role involves researching and writing those standards which are used to assess a doctor’s conduct and competence.

I researched and wrote the following MCNZ resources mentioned in my paper:

• • Discussion paper on When Artificial Intelligence is involved in the care of patients10

• • Good medical practice

• • Doctors and health-related commercial organisations.

I was a member of the Editorial Board of Cole’s medical practice in New Zealand when it was updated in 2021. (Cole’s is a handbook published by MCNZ for new doctors and doctors new to medical practice in New Zealand.) A chapter from Cole’s is referenced in my paper.

Although my paper references several MCNZ resources and sections from our website, the views expressed in my paper are my own, and are not to be attributed to MCNZ.