Relational Ethics in the Administration of Healthcare Technology: AI, Automation and Proper Distance

ABSTRACT

Automation and AI‐driven decision support systems are increasingly reshaping healthcare, particularly in diagnostic and clinical management contexts. Although their potential to enhance access, efficiency and personalisation is widely recognised, there remain ethical concerns especially around the shifting dynamics of healthcare relationships. This article proposes a conceptual framework for understanding the relational ethics of healthcare automation, drawing on the work of Levinas and Silverstone to interrogate the ethical implications embedded in regulatory processes. Focusing on the Australian Therapeutic Goods Administration (TGA) database, we analyse clinical decision support system (CDSS) approvals to examine how healthcare relationships are discursively constructed within regulatory documentation. Through close reading of these technical and administrative texts, we investigate how ethical concerns such as patient autonomy, informed consent and trust are acknowledged or elided. Our findings reveal a limited framing of relational dimensions in regulatory discourse, raising important questions about how ethics are operationalised in the oversight of automated systems. By making visible the administrative practices shaping healthcare automation, this study contributes to emerging debates on AI governance and the ethical integration of automation into clinical practice.

1. Introduction

Automation and data‐driven decision support systems are changing how diagnosis, treatment and client management take place in healthcare (Boone et al. 2022; Kumar et al. 2023; Prictor 2022). These developments have been met with enthusiasm for their innovative potential, but also with concerns about risks. Healthcare automation aims to improve resource allocation, access to healthcare, prediction and diagnosis and enables personalised treatment planning (Jiang et al. 2017; Tekkeşin 2019). Automation is particularly prominent in fields such as oncology, radiology, neurology and cardiology (e.g., Castillo et al. 2020; Ilhan et al. 2021; Jiang et al. 2017; Lopez‐Jimenez et al. 2020; Pedersen et al. 2020). Technologies servicing these areas of medicine are therefore well represented in the Australian Register of Therapeutic Goods (ARTG), administered by the Therapeutic Goods Administration (TGA), the regulatory body tasked with assessing medical or therapeutic products, including software, for safety and quality (Prictor 2022). Other emerging automated decision‐making systems, such as generative AI or large language models (LLMs), are being developed with less ethical scrutiny to date (Zhang and Kamel Boulos 2023). For all types of automation in healthcare, there is a need to develop an associated ‘relational ethics’ (Birhane 2021) that can help to assess their impact on healthcare relationships. This article establishes a conceptual framework for considering the relational ethics of systems designed for healthcare automation, building on the ideas of Levinas (2002 [1961]), Silverstone (2013) and others through an analysis of TGA‐assessed items.

In this article, we explore the extent that healthcare relationships are considered or discursively present in the TGA database entries on clinical decision support system (CDSS). It contributes to emerging debates about how to govern artificial intelligence (AI) and algorithmically supported healthcare technologies by opening a critical space for debate regarding the relational ethics of these technologies (Birhane 2021). The TGA is an Australian regulatory agency comparable to the FDA (Food and Drug Administration) in the United States or the MHRA (Medicines and Healthcare products Regulatory Agency) in the United Kingdom. In 2021, the TGA approvals process was altered to better regulate medical and healthcare software (Therapeutic Goods Administration 2021b). The impact of such increased regulation on outcomes and safety is unclear (see, e.g., Moshi et al. 2019), but the inclusion of CDSS arguably provides a mechanism for validation that product developers may appreciate (Ceross and Bergmann 2021), aiding companies in the marketing of products while embedding new rules of engagement in the product marketplace (Karnik 2014, 202).

To better understand how relationships are shifting in healthcare with emerging automation and AI technologies, we examine the way relational aspects of technology are framed and imagined within the documentation and approvals for CDSS provided through the TGA. CDSS is defined by the Therapeutic Goods Administration (2021a) as ‘software that can perform a broad range of functions that facilitate, support and enable clinical practice’ but is only subject to regulation if it meets the definition of a medical device. This means that it is used for diagnosis, monitoring, prediction, prevention, prognosis, treatment, compensation or alleviation of disease, injury or disability, or is an accessory to such a medical device. Our relational ethics approach draws on the tradition of Silverstone’s work to consider the relational ethics of media.

Drawing on a dataset of approved software tools and systems, including the supporting documentation provided by developers, we explore the discursive framing of the healthcare relationship in the administrative imaginary of automation. Additionally, in the materials provided by manufacturers and CDSS developers, how are ethics present or absent? How is the healthcare relationship framed and understood within the documentation? Is informed consent and patient autonomy considered and, if so, how? The approach and findings inform further research on the effects of automation on the healthcare relationship and, therefore, how relational ethics should be considered in the administration of such technologies.

There is a growing urgency for more nuanced approaches to the ethics relevant to healthcare technologies and automation. Further development is expected in these technologies (Coiera et al. 2023), particularly with the addition of generative AI via large language models, which will have unpredictable impacts on the way healthcare is conducted and how it is mediated. We acknowledge that much of the expanding field of automated decision‐making in healthcare happens outside such approval processes (Attwooll 2023; Kannelønning 2023), but the analysis of these texts allows us to understand how administrative regimes understand ethics and to advocate for the expansion of these frames. Below, we provide an explanation of a conceptual and ethical framework grounded in relational ethics, followed by an explanation of our methods and presentation of findings.

2. Background

Our approach to relational ethics is informed by the study of Emmanuel Levinas (2002 [1961], 197) and others who have built out his ethical frameworks into medical and technology ethics (Benaroyo 2022; Clifton‐Soderstrom 2003; Silverstone 2013) grounded in his theorisation of ethics as being grounded in the relationship between self and other, and Silverstone’s (2013) expansion of this into media and technology. This Levinasian tradition of relational ethics has been further exemplified by the later work of Butler (2005) and Thiem (2008) with their focus on accountability and responsibility. However, for the purposes of this article, we limit ourselves to Silverstone’s (2013, 47, 119) concept of ‘proper distance’, in particular, for how it helps us understand how technology mediates ethics through the mediation of the relationship. Proper distance refers to ‘the importance of understanding the more or less precise degree of proximity required in our mediated relationships if we are to create and sustain a sense of the other sufficient not just for reciprocity but for a duty of care, obligation and responsibility, as well as understanding’ (Silverstone 2013, 133), and that distance itself is a moral category that is changed by technology (172). This understanding of the other and proximity is derived from Levinas’ work on proximity and distance (109) and the face‐to‐face relationship (43, 1.4–219) but significantly extends it into technology and media ethics. This concept has relevance to the mediating impact of medical and healthcare technologies and their effects on the healthcare relationship. These impacts have been explored previously in the context of broadcast media by Silverstone (2013, 119) himself, as well as in the context of telehealth and mental health applications (Shaw and McCosker 2019; Shaw et al. 2023).

Levinas’ key concepts for our understanding of relational ethics are his theorisation of the relationship between the self and other and his concept of the face‐to‐face relationship, which has relevance for practitioner–patient relationships (Levinas 2002 [1961]). Although we do not conflate the healthcare relationship with relational ethics, we argue that the mediation of the healthcare relationship has an impact on the ethical. Levinas argues that the other is irreducible to the I, or the self—the other person’s alterity is absolute (43), and therefore, a face‐to‐face interaction is necessary to establish an ethical relation. This has relevance in the context of a philosophy of technology in which patients are easily knowable through image and data analysis, without an explicitly intersubjective relation. Although we may object on the basis that technology allows for other forms of contact and closeness, we can ask what forms these proximities take and what ethical relationship is established in light of this ‘[f]or how could intermediaries reduce the intervals between terms infinitely distant?’ (Levinas 2002 [1961]).

For Levinas (2002 [1961]), the face‐to‐face relationship (as opposed to the mediated one) calls us to responsibility (203). ‘I cannot disentangle myself from society with the Other’, he says, even as ‘my comprehension of Being in general’ grows (Levinas 2002 [1961], 47). Silverstone’s (2013) concept of proper distance adds to this understanding of the relationship by allowing for a set of norms about what is appropriate in a mediated relationship where one is necessary. Silverstone (2013) asks how close anyone should be to one another if we are not face‐to‐face. In either case (face‐to‐face or through mediation), the healthcare practitioner does not and should not have a close relationship with a patient, but they have a relationship nonetheless. Some distance is necessary and some proximity, and a different degree of distance is appropriate (proper) in different kinds of relationships (Silverstone 2013). Each practitioner‐patient pair may be different in this respect. As Levinas (2002 [1961], 250) puts it, ‘the relation between me and the other commences in the inequality of terms’ or the specificity of the relationship.

Technology introduces new forms of distance (and proximity) into relationships through representation and datafication and altered forms of contact between healthcare professionals and patients (e.g., via representation through images or scans as opposed to face‐to‐face narrative self‐disclosure or physical examination). It also changes the mediation of patient histories through structured and often proprietary ways of creating, processing, storing and using patient data (Cahill and Makadon 2014; Rentmeester 2018). This is not new in healthcare, with a long history of healthcare devices, tools and equipment intervening in and altering the healthcare relationship and its mediation. However, automation may be understood as having a more profound effect on proper distance, as it intervenes in the decision‐making process, therefore impacting responsibility, accountability and consent. Use of CDSS, for example, has been facilitated by the integration of digital systems and electronic medical records into all parts of healthcare practice. These systems have traditionally helped to augment clinicians' knowledge at the point of care but increasingly draw on data and include algorithmic or machine learning interpretation processes that are inaccessible or ‘black boxed’ even for the clinicians using them (Sutton et al. 2020).

In the Australian context, ethical considerations in the governance of healthcare automation have mainly considered ethics in terms of justice, equity, trust, bias, risk and safety (Coiera et al. 2023; Megaro 2023; Prictor 2023). The literature on technology ethics is more likely to consider healthcare relationships and patient autonomy (e.g., Dalton‐Brown 2020; Rogers et al. 2021), as does the World Health Organisation guidance, which also emphasises the need to avoid ‘diffusion of responsibility’ (WHO 2021, 28). Although, of course, a risk‐based approach to regulating technology is necessary to prevent harms (Weatherall et al. 2023), considerations of ethics in automated healthcare should also consider relational ethics. A risk‐based approach to ethics leaves out the impact of technology on the healthcare relationship and the establishment of proper distance. We argue that technology can be understood to potentially intervene in the ability of both patient and practitioner to develop an ethical relationship in the Levinasian sense or to change the quality of that relationship. Here, we explore these aspects of technology ethics through a discursive analysis of CDSS documentation.

3. Methodology

The research team came from the disciplinary backgrounds of media studies with a focus on technology in nonprofit and public good settings, as well as a background in political theory, technology ethics and the social implications of healthcare technologies. Both authors have a background in discursive analysis and qualitative research. These backgrounds and the team’s research area on the impact of automation in healthcare informed our approach. We were focused on the question of relational ethics in technology implementation. We undertook a thematic and discursive analysis of the existing CDSS documentation in the TGA database to assess how the ethical relationship is conceptualised within it, with a focus on decision‐making and responsibility, as well as shared decision‐making, consent and patient autonomy (Sandman and Munthe 2009). This review included the creation of categories of CDSS included in the database. Initially, entries were coded within an Excel spreadsheet by category and then analysed within each category using analytical notes prompted by the sensitising concepts to the point of saturation.

3.1. Selection

There is no clear way to neatly separate out software from non‐software‐based medical devices in the TGA’s database, so CDSS was identified in the database using the search term ‘software’. This may mean that some relevant digital devices and systems where this terminology is not used may not be represented in the data, but the search strategy nonetheless captures a substantial sample of technologies described in this way and allows for a discursive analysis. We did not limit this search temporally to cover all the CDSS currently regulated. The search was undertaken on the 26th of April 2023 and resulted in 998 entries ranging from 2003 (3 entries) to 2023. Necessarily, this included CDSS that were regulated before the required registration—submission of such medical devices was optional at that time (Therapeutic Goods Administration 2021a).

3.2. Categorisation

Category codes were based on the device or system’s purpose in healthcare with a focus on its relevance to aspects of healthcare surrounding the relationship between practitioners and patients, including diagnosis, the analysis of health information, and monitoring and management of conditions. The category codes used included diagnosis and/or identification (of illness), analysis and/or interpretation of data, data processing, monitoring and/or management (of illnesses or conditions), procedure or planning, self‐care, treatment devices and uncategorisable or other. The coding hierarchy is outlined below. For the most part, coding in this category was prompted by words directly stated in the documentation, for example, ‘the tool is used for diagnosis’; however, sometimes interpretation was necessary. For example, ARTG 338258 states that the CDSS ‘detects suspected abnormalities in the Chest X‐ray and […] provides clinically relevant tags’. This is coded as a diagnosis or identification tool.

These categories are hierarchically organised, meaning that if the description provides multiple purposes for CDSS, it will be coded under the higher‐level category. Codes are listed below with lower‐level categories listed first (Table 1).

TABLE 1.

Coding frame, categories of CDSS.

Code	Definition	Count
Data processing	Data processing software for the collection, management, secure storage and visualisation of data relating to healthcare. Related to administrative data and systems management; does not include any evaluative or treatment application.	303
Analysis and interpretation	Information provision to practitioners that includes analysis or interpretation of information related to a patient’s care, including pathology, imaging and other data. May include keywords such as ‘analysis’, ‘interpretation’ or ‘evaluation’, ‘review’ or ‘assessment’ but does not assist with diagnosis or risk assessment.	207
Procedure or treatment planning	Decision support for treatment, including surgery planning and visualisation, the fitting or design of prostheses, fitting and design of hearing aids etc., or dosage calculation for medication or radiology treatment purposes.	187
Diagnosis and identification	Decision support for diagnosis or automated identification of suspected conditions. Explicitly mentions or implies diagnosis. Risk calculation and the automated identification of risk also included.	128
Monitoring and management	Monitoring systems for existing patients, whether remotely or in a hospital setting.	120
Treatment device	Devices for treatment running on the software in question.	40
Uncategorisable	Do not fit into any of the above categories and another clear category cannot be identified or the information provided is unclear.	6
Self‐care	Includes apps that are provided for self‐management of conditions and are to be used without the involvement of a practitioner.	4
	Total	998

3.3. Analysis

The sensitising concepts (listed below) were selected by both authors. The documentation entries were then coded according to category as listed above. Category selection was completed according to a hierarchical approach. The entries within each of these categories and their accompanying text descriptions were then analysed by the lead author according to the consideration of relational ethics within the documentation and the treatment of accompanying concepts listed below, such as relationality, responsibility, patient agency and consent. The categorisation process enabled us to identify documentation most relevant to our research questions about the impact of CDSS on relational aspects of healthcare. Data processing (303 entries) was excluded from analysis; however, all other entries were analysed.

To analyse coded documents, we used an abductive analytical approach (Earl Rinehart 2021; Timmermans and Tavory 2012), drawing on existing concepts and theories in conversation with the text being analysed. Because of the high volume of entries, we refined our analysis through an iterative process, moving between the documentation and concepts relevant to relational ethics to identify themes for a more focused analysis. This focused analysis within the categories above revolved around aspects of healthcare relationships: autonomy, responsibility and distance were among the related concepts we considered. Finally, notes were then taken within the category texts to document patterns and repeated phrasing and note any unusual or standout entries that treated ethics in a different manner to the norm, given that language was often standardised and similar across multiple entries. Because there was considerable repetition within and across categories, we noted repeated language but did not do any numerical analysis of the codes.

4. List of Sensitising Concepts for Analysis

4.1. Relationality

This refers to how practitioners and patients do not exist in isolation but in relationship to one another (Kazimierczak 2020); how narratives and stories assist caregiving (Charon 2008; Ostherr 2022) and how two or more people or things are connected (including technology). This concept sensitises the analysis to the facilitation of history‐taking and patient–practitioner relationships in general.

4.2. Responsibility

This concept sensitises the analysis for themes relating to the professional responsibility of the practitioner and their ability to respond ethically to the patient. In more conventional terms, healthcare responsibility is about ensuring patient safety, providing high‐quality care and being accountable for medical decisions and advice. Responsibility is related to the practitioner’s agency or ability to act.

4.3. Patient‐Centred Care and Shared Decision‐Making

Patient‐centred care (Kerasidou 2020; Malamateniou et al. 2021) refers to the prioritisation of the patient’s needs in healthcare, ensuring respect and empathy for the patient and personalising care to their needs. It also has a focus on the patient’s experience of care. As a sensitising concept, this enabled us to consider where the patient’s experiences and involvement in care are facilitated or considered in terms of the regulated technologies.

4.4. Informed Consent and Patient Agency/Autonomy

The concept of informed consent is related both to the responsibility of the practitioner to provide adequate information about what is involved in treatments, their potential risks and benefits, and possible alternatives. This relates to how the patient is then able to freely and in an informed way agree to or decline a particular treatment or procedure.

4.5. Power Dynamics (Between Patients and Practitioners, Between Practitioners and Technologies)

Related to consent, this sensitising concept enables us to consider power in the use of technology for healthcare. Is power considered in the documentation? Does technology change power relationships? See, for example, Forbat et al. 2009 for an analysis of this.

5. Findings and Analysis

As described above, the categorisation of entries enabled a prioritisation of particular documentation for analysis. Although the most common category was data processing, the main categories of interest in this review are systems that perform ‘analysis and interpretation’ (e.g., of medical images and other test results), ‘identification and diagnosis’ (of health issues or areas of concern), ‘monitoring and management’ (of a diagnosed condition or health issue) and procedure planning, which includes surgeries and other treatment planning. These are the categories most likely to mediate healthcare relationships, consent and both practitioner and patient agency and responsibility. They also have knowledge effects on decision‐making, with healthcare practitioners' expertise potentially altered or overridden (Aquino et al. 2023; Benaroyo 2022; Irvine 2005). Our key analytical findings include the absent presence of the healthcare relationship in the technology documentation, the disavowal of responsibility from the CDSS developers as a stand‐in for technology ethics and the implications of such regulatory discourses for healthcare technology overall.

5.1. The Diagnostic Relationship in TGA Documentation

This section considers the way the patient–practitioner relationship is included or hidden in the documentation for diagnostic tools. From a liability perspective, responsibility and expertise are strongly present as a theme. Responsibility is included as a caveat to say that CDSS are not to be used without the final clinical judgement of a doctor. This is to ensure that devices are not given sole responsibility for diagnosis or for accuracy of analysis. Such caveats may be prompted by documentation requirements: Standardised or boilerplate phrasing is often used with little to acknowledge the potential for deferral to the system in clinical judgement or how such concerns have been addressed by the software itself. Such caveats are often repeated across multiple entries with the same or very similar phrasing. Because of this repetition, it is not meaningful to count instances of particular phrases or themes in the documentation. Examples of qualifying phrasing include the following:

As an adjunctive tool, the device is intended to be viewed by interpreting physicians and it is not intended as a replacement for a complete clinician’s review.

Patient management decisions should not be made solely on the basis of analysis by [the software].

The results of [software] are intended to be used in conjunction with other patient information and based on the user’s professional judgement.

[The software] should only be used as an adjunctive tool.

The information provided by [software] should only be used in conjunction with other clinically accepted information to guide medical decision‐making at the discretion of the clinician.

These phrases reduce the direct responsibility of automated decision‐making or decision support tools for healthcare decisions (e.g., Bjerring and Busch 2021). This is a necessary intervention from a legal liability perspective. However, it tends to elide how such technologies may in practice alter the role of clinical judgement and medical expertise.

The TGA’s assessment of CDSS therefore generates a contradiction. The importance of the consideration of responsibility to the assessment means that the actual impact of software on the responsibility of practitioners in decision‐making is paradoxically not considered because of the use of generic text directing healthcare practitioners to take responsibility for decision‐making.

Some documentation uses more involved language to maintain the responsibility of the practitioner in detecting possible conditions (ARTG ID: 316044). In this example, the software’s function is described as follows:

[Using] an artificial intelligence algorithm to analyse images and highlight cases with detected findings on a standalone desktop application in parallel to the ongoing standard of care image interpretation. The user is presented with notifications for cases with suspected findings. […] The results of [the software’s analysis] are intended to be used in conjunction with other patient information and based on the user’s professional judgement.

However, this simple statement that final responsibility rests with the practitioner contrasts with an example of documentation that explicitly names the patient–practitioner relationship in the diagnostic process and affirms practitioner autonomy while also describing mechanisms for this to be maintained:

[The software] is designed to compliment [sic] and enhance (not replace) the patient / clinician relationship by assisting them to efficiently navigate the diagnostic assessment and management process.

(ARTG ID: 365811)

Notably, the documentation for this software promises mechanisms for patient involvement in the diagnostic process:

All data inputs and outputs require validation by the patient and the clinician, and the clinician retains clinical autonomy with the ability to override and/or modify any clinical recommendations.

(ARTG ID: 365811)

Other software that mentions the importance of professional judgement does not explain ways that the CDSS might facilitate such judgement. As we will see in the next section, most descriptions of CDSS do not mention the patient’s role in decision‐making at all. This documentation is exceptional in describing mechanisms for patient and practitioner involvement. In the analysis, we will argue based on these observations that responsibility is not framed as relational in the majority of this documentation.

5.2. The Patient as Data

As described above, the responsibility of the practitioner is affirmed in the documentation, mainly without reference to the healthcare relationship. Patients are mentioned frequently in the documentation, with 464 mentions, primarily in relation to the data associated with them, including their DNA in the case of DNA screening tools (8 mentions), their tissue or other samples (12 mentions) in pathology tools or ‘scans’ (23 mentions) or ‘images’ (223 mentions) in the context of medical imaging analysis or diagnosis.

These mentions situate the patient as data or in relation to the forms of data that are processed, analysed and interpreted via the CDSS. One typical CDSS, for example, isolates and automates patient ECG data analysis to ‘assist the physician’:

…intended for analysing, editing, reviewing, reporting, and storing of pre‐recorded ambulatory EGG and accelerometer data on [name of device]. The results of the automated analysis are intended to assist the physician in the interpretation of the recorded data. This information is not intended to serve as a substitute for the physician overread of the recorded ECG data.

(ARTG ID: 387791)

The software is described in the documentation as analysing ‘patient data’, ‘patient information’ or ‘patient images’. As noted above, the patient–practitioner relationship is rarely mentioned in this discussion of what software does or factored into its functionality, nor is the decision‐making involvement of the patient in their own healthcare. The word ‘decision’ occurs 50 times in the documentation, referring to ‘treatment decisions’ or ‘patient management decisions’, but the CDSS is framed as interfacing with the practitioner to facilitate their decision‐making. Any instance of automation in the processing or analysis and interpretation of data is qualified by a requirement for clinician or physician oversight, but never with some reciprocal requirement to engage or involve patients in the decision‐making process.

‘Patient history’ (relating to the previous analysis of responsibility) is something that should be considered in clinical judgement; however, this caveat is only present in four entries. In one of these exceptional instances:

The software performs computer simulation to predict device‐anatomy interaction to support the evaluation for device size and placement. The results are intended to be used by qualified clinicians in conjunction with the patient’s clinical history, symptoms, and other preprocedural evaluations, as well as the clinician’s professional judgement.

(ARTG ID: 351533)

Documentation for CDSS generally has a ‘distancing’ effect on the physician–patient relationship.

One exception to this pattern is self‐care technologies and self‐monitoring software. Such tools make up a tiny proportion of the total number of entries in the database (n = 4/998), likely because these tools generally do not require TGA approval as they sit outside the scope of TGA regulation or are not defined as CDSS. The following example is one of the few such CDSS entries that situates usage in a broader set of relationships:

[the software is] for use by patients, caregivers and healthcare professionals to assist people with diabetes and their healthcare professionals in the review, analysis and evaluation of historical glucosemeter data to support effective diabetes management.

(ARTG ID: 325903)

Ultimately, the patient is left out of the documentation except as data, images or rarely as having a history that could be considered alongside CDSS, and is also largely absent from the description of what the tool does. In this way, the patient exists in these texts only as data or images rather than as a full, rational subject capable of building an ethical relationship with a healthcare practitioner. This coincides with a view of automated technologies as having the capacity to know a patient more effectively or efficiently through their data in such a way as to be able to assist in the process of diagnosis, analysis and treatment planning. However, shared decision‐making appears not to be part of this imaginary.

5.3. The Absence of Consent

Informed consent as a concept is largely absent from the ARTG documentation for digital devices that we reviewed in this study. We acknowledge that informed consent may be considered as given, sitting outside the practitioner’s decision to use or not use CDSS during a patient’s care and course of treatment. Consent may therefore not be considered within the primary product description and other documentation provided for the purposes of regulation. However, when reference is made to treatment decisions exclusively without reference to consent or to patient‐centred care or collaborative decision‐making, this has clear impacts on healthcare expectations and the diagnostic relationship (Dalton‐Brown 2020), and further research is necessary to understand how.

Whether the purpose of the software or device is for procedure planning, analysis and interpretation, or diagnosis, the choice for its use is placed with healthcare professionals. This is more pronounced with diagnostic systems. For example, a typical automated diagnostic system such as for ADHD:

provides healthcare professionals with objective measurements of attention and inhibitory control. The visual component aids in the assessment of, and evaluation of treatment … Results should only be interpreted by qualified professionals.

(ARTG ID: 345436)

However, there is a more pronounced, but still only implied, requirement for consent for CDSS designed for some forms of monitoring and management, where patients are required to take on their own data collection responsibility. As the standout example and exception when it comes to informed consent, the collaborative mental health diagnosis application described in the previous section (ARTG ID: 365811) refers to requiring ‘patient validation’. Although the documentation does not directly address consent in the automated decision‐making processing of the CDSS, it is a rare case of seeking to involve patients in analysis and interpretation.

The word consent does not appear at all among the downloaded entries analysed. Because of this, we have used discussion of treatment planning in the data as a proxy to understand how the patient’s role and consent are framed. Treatment planning is the area of automation most relevant to the issue of consent.

a computer‐generated treatment plan is created using a commercial Treatment Planning System (TPS) is generated for the patient’s anatomy

(ARTG ID: 388754)

Some systems are described as determinist: Although the final decision is ‘validated’ by a practitioner, the CDSS recommends a course of action or treatment plan, whereas other systems allow for the evaluation of multiple ‘treatment options’ or evaluate/assess treatments that the practitioner suggests, allowing practitioner‐users to ‘simulate/evaluate implant placement and surgical treatment options’ (e.g., ARTG ID: 197644). There are many tools within this category designed to match prostheses or other treatment devices (e.g., dental and orthodontic accessories or joint replacements) to the anatomy of patients. Such software likely already bypasses the question of consent if the patient has consented to the fitting of such devices. However, devices that suggest treatment based on a set of inputs (e.g., tools for ‘evaluating surgical treatment options’ [ARTG ID: 146193]) could potentially alter the deliberation of treatment options in the clinical setting in ways the patient may not be aware of. Across all cases, it is clear that new and emergent forms of automated clinical decision support systems and devices are easily incorporated into treatment patterns without additional effort to provide explanation or transparency as to how decision‐making—and hence care relationships—are altered.

In our findings in this section, we acknowledge that informed consent may be considered a separate issue outside the practitioner’s decision with a patient to use or not use a device or treatment during their care. Consent may therefore not be considered a relevant consideration for the primary product description and other documentation provided for the purposes of regulation. However, when reference is made to treatment decisions exclusively without reference to consent or to patient‐centred care or collaborative decision‐making, this has clear discursive impacts on healthcare expectations and expectations for the diagnostic relationship (Dalton‐Brown 2020), and further research is necessary to understand how.

6. Discussion

These findings and our analysis present a starting point for applying a relational ethics framework to automation in healthcare settings. Although further empirical work is needed, this section draws out some of the central tenets of a relational ethics approach that can apply to an increasingly wide range of innovations in healthcare automation. We have shown above that the documentation for CDSS provides a range of assurances (whether mechanisms are documented or not) that automated analysis is conducted in the ongoing standard of care and used in conjunction with professional judgement, which are necessary to maintain practitioner responsibility. The use of caveats such as those described above places responsibility in the practitioner’s expertise while simultaneously allowing CDSS to mediate the practitioner–patient interface.

This absence of the patient in the documentation is not necessarily surprising. The TGA’s documentation format is focused on the regulation of healthcare technologies as devices, tools or medications that may only be in contact with the metric or visual analysis of the blood, DNA or biochemistry of a person, and the CDSS is assessed upon the accuracy of its assessment against outcomes. However, diagnostic and analytical tools feed into decision‐making about treatment for which informed consent and shared decision‐making are required according to healthcare norms, therefore potentially changing and disrupting such norms. This leaves an area of uncertainty for the impact of healthcare technologies on ethical relationships and considerations of care ethics in the evaluation of technologies. The healthcare relationship is absent from the description of what the tools do, in their focus on calculation and planning and the absence of the patient as a partner in decision‐making. The healthcare relationship may be fundamentally altered by automated systems through the treatment of patients as data (Ruckenstein and Schüll 2017).

As Benaroyo (2022, 330) puts it:

In clinical medicine, the unfolding of the responsibility‐for‐the‐other entails being aware of the radical otherness of the other, as well as the same time being aware of the common vulnerability that links physician and patient, namely to the inescapable responsibility one owes to the other, more particularly for the physician, to the responsibility to care for the other in [their] radical otherness.

We saw in the documentation a focus on CDSS effectiveness and responsibility framed as avoidance of risk. An approach that evaluates technology predominantly in terms of risks and effectiveness is an approach that is focused on outcomes and healthcare rationalisation. Cain (2019) documents how healthcare workers need to navigate multiple logics and epistemologies in the practice of medicine. Medical philosophy and medical ethics approaches emphasise the shared vulnerability of the healthcare relationship (e.g., Delgado 2021; Delgado et al. 2021) in a way that echoes relational ethics approaches. An ethical relationship necessitates having responsibility for the other in a way that makes a practitioner also vulnerable to and accountable for the decisions they make together (Nortvedt 2003). In the context of healthcare technology, this is ‘proper distance’ (Silverstone 2013). Benaroyo discusses the related concept of hospitality. This should not be taken literally as making the patient feel welcome but rather an intersubjective kind of hospitality in which the practitioner ‘takes responsibility‐for‐the‐suffering‐other’ (Benaroyo 2022, 331). This is an investment not necessarily based on feeling/empathy but rather a personal, subjective investment in which the practitioner recognises their own responsibility in the encounter.

These perspectives ground the practice of medicine in knowledge, embodiment, speech and response (Aquino et al. 2023; Benaroyo 2022; Burns 2017; Clifton‐Soderstrom 2003; Falque 2019). They are grounded in the assumption of the uniqueness and unknowability of the other (Nortvedt 2003, 25), which contrasts with the ontology of machine learning and automated approaches that assumes that outcomes and diagnoses are discoverable through predictive technologies trained on past data (e.g., Malik et al. 2018). Precision medicine, personalisation, decision‐support and diagnostic systems may contribute to the so‐called ‘crisis of expertise’ (Eyal 2019) that many argue has already shifted the onus of care away from healthcare professionals towards data‐driven machine learning systems (Egher 2023).

On the other hand, as Nortvedt (2003, 26), drawing on Scheffler (2001, 65), argues, the reality is that resources are increasingly stretched in healthcare and automated tools can be a way to increase distributive justice, so there is a need to find ‘responsibility we can live with’, with an ethics grounded in the particular. Vallès‐Peris et al. (2021) show that patients see the introduction of automation in healthcare as inseparable from the reality of unequal and inadequate resourcing of the healthcare sector, therefore showing that the introduction of automation is both a result of injustice and a potential solution towards justice if used in ways that benefit all. However, for Levinasian ethicists (e.g., Cohen 1986), distributive justice is, if not opposed to an ethics grounded in the particularity of the other, then at least partially working at cross‐purposes to it:

Ethics, as the extreme exposure and sensitivity of one subject to another, becomes morality and hardens its skin as soon as we move into the political world of the ‘impersonal third’—the world of governments, institutions, tribunals, prisons, schools, committees, and so on (Cohen 1986, quoted in Nortvedt 2003, 31).

Justice is therefore conceptualised as the multitude outside the dyadic patient–practitioner relationship, to which practitioners also have a responsibility and must also be part of how they consider how to act (Buddeberg 2018). Levinasian ethicists see justice therefore both as necessary and as a danger because justice ‘creates distance and puts into question the proximity experienced in the encounter with the other’ (Buddeberg 2018, 151). In so doing, it ‘runs the risk of becoming […] calculable’ (Buddeberg 2018, 151). Therefore, ‘this non‐reducible anarchical responsibility for the Other must be protected and must hold in check the world of institutions and justice’ (Nortvedt 2003, 31). The relevance of such a conception of justice and responsibility to the particular other is clear in the balance between the calculable—automated decision‐making and distributive justice (the number of patients seen and cured and the resources they receive)—and an ethics of medicine focused on patient autonomy, shared decision‐making, informed consent and proper distance. To strip this concept back to concepts with more contemporary currency, automated decision‐making has its basis in the neoliberal distribution of resources in ways that may be efficient and ‘just’ in aggregate but unjust and irresponsible in the particular.

Informed consent (which may be understood as accountability in ethical terms) is a relational process in which practitioners provide information to patients and make decisions collaboratively. Although this may still be the case when using CDSS, there is a difference in what the consent is based on when the information comes from the practitioner's own knowledge and when the recommendation comes through a process of automation. Oxholm et al. (2022) argue that the algorithm becomes a ‘third agency’ in the relationship. The conceptual link between informed consent and shared decision‐making in the literature (e.g., Whitney et al. 2004) demonstrates that automated decision‐making has an impact on informed consent that goes beyond the question of how to inform a patient about what an algorithmic or machine learning‐based health tool might involve (explicability/explainability) and extends also to the way that healthcare decisions are normally made. Although there can be automated tools designed to facilitate shared decision‐making (e.g., Stacey et al. 2016), these are largely not apparent in the TGA database under review.

7. Conclusion

We have argued that decision‐based technologies have implications for relational ethics in that they potentially intervene in the mediation of the patient–practitioner relationship, the moral responsibility of the practitioner and the autonomy of the patient. Much of the discourse around responsibility and AI has necessarily focused on risk, benefit, transparency and the removal of bias (e.g., Megaro 2023). Although slow, careful and evidence‐based implementation is indeed important, it leaves out the dimension of responsibility grounded in relationships. The exclusion of the patient from the documentation also demonstrates that there is an impact on the patient–practitioner relationship through technologies, with technologies increasing the distance (or placing an additional barrier) between patients and practitioners. Although a degree of distance is appropriate in all relationships (‘proper’ distance) and barriers are used to necessary effect in healthcare, this may also need to be renegotiated with the advent of automation that intervenes to some degree in patient consent, practitioner responsibility and expertise, and shared decision‐making. The inclusion of mechanisms for collaboration and considerations of consent, such as ‘dynamic consent’ (Lee et al. 2023), could go some way to redressing these changes.

One question for the regulation of CDSS is whether a product‐based evaluation (on the same basis as the evaluation of medical devices or pharmaceuticals) is appropriate for such software. A product‐based approach traditionally relies on measures of both efficacy and safety, whereas CDSS also mediates healthcare relationships and intervenes in assumptions of practitioner knowledge and responsibility, as argued above. How can practitioner responsibility, shared decision‐making, and patient autonomy and consent be built into product design beyond caveats that the final decision must be made by the practitioner?

The TGA database is not necessarily representative of the kinds of technology that are currently being developed and implemented in healthcare‐related fields. The inclusion of technologies is based on what the TGA has framed as in need of regulation and what they explicitly or by omission have left out of regulatory regimes (Therapeutic Goods Administration 2021a). For example, large language models (LLMs) are currently not regulated but are increasingly used in healthcare (e.g., discussions including Duffourc and Gerke 2023; Meskó and Topol 2023; Zhang and Kamel Boulos 2023). This is also likely to have a considerable impact on healthcare relationships and ethics and should therefore be considered in any further work.

The study of Vallès‐Peris et al. (2021) reviews patients' views on automation in healthcare and finds that patients are often uncomfortable with the reality of increased automation. Further work could include interviews with patients, practitioners and patient advocates to understand their views on healthcare automation in terms of its effect on relationships, equity, decision‐making processes, responsibility and the impact of regulation of the implementation of such technologies. An understanding, through interviews with practitioners, of how decision‐making software is used within practitioner‐patient decision‐making processes would assist in understanding how they mediate the relationship or adjust their practice to compensate for what automated tools leave out.

Author Contributions

Frances Shaw: conceptualization (lead), data curation (lead), formal analysis (lead), methodology (equal), writing – original draft (lead), writing – review and editing (equal). Anthony McCosker: conceptualization (supporting), funding acquisition (lead), methodology (equal), project administration (lead), supervision (lead), visualization (supporting), writing – original draft (supporting), writing – review and editing (equal).

Data Availability Statement

The data that support the findings of this study are available in the Australian Register of Therapeutic Goods. These data were derived from the following resources available in the public domain: https://www.tga.gov.au/resources/artg.