Trust in AI: why we should be designing for APPROPRIATE reliance

Abstract

Use of artificial intelligence in healthcare, such as machine learning-based predictive algorithms, holds promise for advancing outcomes, but few systems are used in routine clinical practice. Trust has been cited as an important challenge to meaningful use of artificial intelligence in clinical practice. Artificial intelligence systems often involve automating cognitively challenging tasks. Therefore, previous literature on trust in automation may hold important lessons for artificial intelligence applications in healthcare. In this perspective, we argue that informatics should take lessons from literature on trust in automation such that the goal should be to foster appropriate trust in artificial intelligence based on the purpose of the tool, its process for making recommendations, and its performance in the given context. We adapt a conceptual model to support this argument and present recommendations for future work.

INTRODUCTION

Artificial Intelligence (AI) has been defined as a “machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions.”¹ Big data and increased computing power have allowed for more predictive yet complex AI approaches, such as machine learning.^2,3 AI algorithms show promise in improving healthcare, but few of the algorithms developed are actively used in clinical practice.^2–8

Trust in AI is an important but challenging problem.⁹ AI in healthcare often automates cognitively challenging tasks, such as predicting patients at risk for poor outcomes. Therefore, previous literature regarding automation holds lessons for AI in healthcare. Lee and See’s 2004 review, specifically, describes why trust is increasingly important as automated systems become more complex, “Trust guides reliance when complexity…make[s] a complete understanding of the automation impractical…By guiding reliance, trust helps to overcome the cognitive complexity people face in managing increasingly sophisticated automation.”¹⁰ A 2015 update to Lee and See’s review included 127 articles stemming from other settings (eg, military, transportation), suggesting that important literature regarding trust in automation has not been applied in healthcare.¹¹

We argue informaticists should conceptualize trust in AI as a complex, dynamic entity with the goal of fostering appropriate trust based on the purpose of the AI, its process for making recommendations, and its performance in the given context. This perspective aims to contextualize Lee and See’s model for an informatics audience and advocate for systems supporting appropriate reliance on AI in combination with clinical expertise, with the ultimate goal of improving health outcomes.

A MODEL FOR APPROPRIATE RELIANCE IN AI

Figure  1 presents a conceptual model for developing appropriate reliance on AI. The model focuses on trust by the end user, or the person(s) who are intended to take some action based on the AI (typically, the health professional). In this model, response to AI involves 4 stages. First, people learn about the system and form beliefs. Next, they develop trust, which is a necessary step toward intention to use the AI. Finally, intention precedes actions (ie, using the system or not). Trust evolution is also affected by the individual, organizational, cultural, and environmental context. We can then evaluate trust appropriateness based on how well users’ reliance actions on the AI match the AI’s capabilities.

Figure 1.

Lee and See’s model for fostering appropriate trust on AI in medicine.

THE AI SYSTEM ATTRIBUTES

Purpose, process, and performance are key attributes for forming appropriate trust in AI.^12,13  Purpose explains why the automation was developed. Conveying the purpose of an AI system can help prevent misuse, which could occur, for example, if a health professional uses an AI system intended to predict sepsis in adults to predict sepsis in children.

Process describes how the AI operates. Process-related information may include data inputs, analysis procedures, and data outputs. Providing this information to the user is important to the initial establishment of trust.^9,14 The World Health Organization and others have advocated for improved explainability, whereby the AI describes how algorithms produced their predictions.^15–17 Previous studies have indicated that users tend to trust automation if its process can be understood.¹⁸ The concept of explainability poses a challenge with machine learning and deep learning methods, which do not produce explanations for their findings without post hoc interpretation.¹⁹ However, qualitative research by authors of this perspective has indicated that the issue is not making deep learning itself more understandable, but instead providing enough information to facilitate appropriate trust and decision-making. Specifically, users reported wanting information on the high-level drivers of algorithms, such as whether they included both clinical and social risk factors, and which factors were most important in driving a patient’s risk score. This information was considered critical to determining what resources or interventions might be most appropriate for the patient.⁹

Performance refers to what the AI does, and how well the system supports users’ goals. AI performance is commonly measured by statistics such as area under the curve (AUC). It is unclear, however, if these performance statistics are conveyed to users. In addition, the AUC describes the system’s overall performance, and thus only indirectly addresses whether a health professional should trust its recommendation for a specific patient. For the health professional, it would be more helpful to show how often a prediction was correct for a specific type of patient (eg, the PPV [positive predictive value]). Yet health professionals struggle to understand concepts such as PPV and NPV (negative predictive value), and patients have even further difficulties.^20,21 A more intuitive measure, such as a clearly described record of true positives versus false positives for a specific patient group (ie, a likelihood ratio), might be more meaningful.⁹

CONTEXT AND ITS IMPACT ON TRUST FORMATION AND EVOLUTION

Trust development is affected by individual, organizational, cultural, and environmental factors as described in Table  1.

Table 1.

Description of factors impacting trust formation

Factor affecting trust	Description	Example(s)
Individual	Individuals have varied predispositions to trust a system based on experiential, psychological, and knowledge-based differences.	An attending physician who had negative experiences with a drug-drug interaction system during residency may be more skeptical than another attending who had positive past experiences. Those with higher technology literacy and more experience with AI tend to trust AI at higher rates.22
Organizational	Interpersonal communication, organizational culture, leadership, and the trust in the person or group who developed the system can all impact trust	Gossip and observed use (or disuse) by trusted colleagues may impact trust.23 Trust may vary based on whether the system was developed internally (eg, by the local clinical informatics team) or externally (eg, EHR vendor or third party).
Cultural	Individual cultural differences may impact trust based on country of origin, role/profession (eg, physician, nurse, patient), age, race, or ethnicity.	Older adults have indicated lower trust in AI.24–26 Significant differences were found in initial trust of AI systems based on country of origin in the same region (Nordic nations).27
Environmental (physical and external)28	Physical and external environmental factors impact a user’s ability to utilize AI, understand system performance, and ultimately develop trust. The physical environment (eg, hospital layout, location of devices, resource access) may be less relevant to healthcare-related AI systems than physical automated systems in other domains (eg, military, transportation, industrial settings) or other patient safety-related problems (eg, patient falls, medication administration errors). Other external drivers (eg, governmental policies, activities of third-party organizations), however, may have important influences on trust.	A patient may not have sufficient internet connectivity to access an AI-based tool. Trust may be impacted by who supports or sponsors the AI system (eg, National Institutes of Health, professional medical organizations, insurance companies).

Trust is a dynamic process involving information assimilation, trust formation, intention formation, and actual reliance actions (Figure  1), so different contextual factors have greater impacts at different points. We mention these ever-evolving contextual factors to demonstrate that trust development is not a one-size-fits-all approach.²⁹

MEASURING TRUST APPROPRIATENESS

The appropriateness of trust in AI can be assessed through 2 distinct yet related constructs—calibration and resolution (Table  2).

Table 2.

Description of constructs related to measuring trust appropriateness

Measurement construct	Description	Example
Calibration	The extent to which a user’s trust matches the performance of the system	Trust should be higher for better performing systems. For example, if an AI system has a high AUC, people should trust it frequently. If the system has a poor AUC, people should trust it less frequently.
Resolution	The users’ ability to adapt their trust based on changing functions and goals, or how the AI performance changes over time30	Resolution can be measured through:
		1) “functional specificity” or how the user adapts their trust based on the problem context. For example, a natural disaster, such as a pandemic or weather event, may change the data so dramatically that an algorithm’s recommendations may not be relevant for future forecasts. In other scenarios, AI may not perform as well for certain populations for which there is biased or insufficient data, and the user trust should shift accordingly.
		2) “temporal specificity” or how the users’ trust adapts over time as the decision support becomes more or less accurate. For AI that becomes more accurate over time as it gains more information about the patient, user trust should also adapt accordingly. Alternately, if the system becomes outdated, trust should decline with time.

RECOMMENDATIONS AND FUTURE DIRECTIONS

Integration of the concepts presented here into AI systems will require a collaborative effort from informaticists, computer scientists, application developers, health organizations, and patient/family advisory groups. We provide recommendations for conceptualizing trust as a complex, dynamic concept where the goal is to achieve appropriate reliance on AI in conjunction with clinical expertise to facilitate treatment decisions that improve health outcomes.

Allow the user to view the purpose, process, and performance of the AI system

Provision of this information should be dynamic and flexible as different users may want different levels of detail, or need different levels of explanation (eg, patients vs. health professionals). Typically, this information should be provided within the AI system interface. Provision of this information should also adhere to appropriate principles regarding user-centered and inclusive design, ensuring the information can be easily found and understood by the various users.³¹ Barda et al provides a framework for user-centered design of machine learning predictions in healthcare.²⁹ Patient-facing systems, for example, should use plain language, and provide options for having the text read aloud for less literate or visually impaired users. It is also important to ensure accessibility across different device types (eg, mobile device, tablet, laptop, or desktop).

Purpose descriptions may be most critical in the early implementation of an AI system and may be conveyed for first time users of the system. Safety-critical scenarios (ie, where misuse may result in patient harm) may require more engagement from users such as answering questions to demonstrate understanding of the system’s purpose. At minimum, the purpose should briefly describe why the system was created, by whom, for whom, using what training data (ie, from the local site or another), and appropriate and inappropriate scenarios for using the system. Ideally, the AI support systems would not display for inappropriate use cases. Following AI introduction, purpose information should remain available for new users and review by existing users. If developers update the purpose post-deployment, display windows regarding these updates upon the user's next entry to the system may be appropriate.

Process display guidance can be broken into system inputs (data) and outputs (recommendations). Data inputs should be reviewed early and iteratively throughout development. Review by end users (health professionals or patient advisory groups) and clinical informaticists may detect data inputs that are prone to missingness or quality issues, as well as information that may cause bias. AI that rely on historical data can be systematically biased based on age, race, or gender, and recent guidance from the WHO stresses the need for inclusive, equitable AI design.^15,32 Displaying process-based inputs to stakeholders early to detect problematic inputs can mitigate potential biases in AI systems. One study, for example, gained feedback from health professionals to determine if model outputs at certain points were plausible or relevant.³³ Reviewing inputs with experts prior to implementation may also foster trust with the larger group of end users once the system is deployed. Further, working with end users may help determine which inputs users find surprising, so that descriptions of these inputs may be provided and used as a means for demonstrating the utility of the system above-and-beyond clinical gestalt. Post-deployment, it may also be helpful to provide a means for all users to view a list of data inputs.

Displaying process outputs (decision support) involves providing explanation as to how the system arrived at its recommendation. One group, for example, developed an explainable AI system which doubled the correct identification of hypoxemia during surgery. The interface overlaid predictions with which features of the model have increased or decreased risk and provides related values (eg blood pressure, body mass index, etc.).³⁴ Developments in computational methods, such as SHapley Additive exPlanations (SHAP values) which allow for post hoc quantification of each predictor in a model, have made the process of displaying the most important values more feasible.³⁵

Performance metrics are essential to allowing users to trust systems appropriately. At minimum, displaying variables such as PPV and NPV may be beneficial. As described, health professionals and patients can struggle to understand these concepts,^20,21 so future research should continue to assess creative, understandable ways for conveying system performance. To the extent possible, performance information should be dynamic and tailored to the context at various levels, for example, the institution, department/unit, and patient variables such as demographics.

Work with local sites to adapt purpose-, process-, and performance-related information display appropriately

Often, AI systems may start with baseline models that need to be adapted to different institutions.³³ The same is true for designing an AI interface that facilitates appropriate trust formation. Updating the AI interface design may also be done in parallel with the aforementioned activities related to determining the display for process inputs. Working with end users at local sites will also allow developers to understand and account for contextual differences.²⁹ As described, organizational, cultural, and environmental factors impact the formation and evolution of trust. Incorporating diverse end users from local sites will help understand these contextual differences and how they may be accounted for in the AI interface. Utilizing information design concepts such as overview, filter, details on demand, will also allow different individuals to adaptively gain the information needed to develop appropriate trust.³⁶

Redefine how we measure success in AI

Measures of success of AI have hinged on model parameters such as PPV, NPV, AUC, and Akaike information criterion (AIC) that account only for model performance under controlled conditions and not how people utilize the models in practice. Calculating and monitoring statistics such as calibration and temporal and functional specificity (to assess resolution) will help developers understand if end users appropriately trust AI systems. Temporal specificity can help monitor AI systems over time to detect if performance improves or degrades, also presenting an improvement to current practice where performance statistics are only calculated pre-deployment. It may also be helpful to investigate how these metrics may be productively displayed to end users to help them improve the appropriateness of their trust as has been done with systems in other domains.³⁷ Similar examples of showing performance information can also be seen in other aspects of healthcare, such as ordering of expensive imaging studies.³⁸ Others have also proposed decision-based statistics such as the “net reclassification improvement,” which measures if utilization of a certain model improves the users’ ability to identify true positive cases.³⁹

CONCLUSION

AI approaches, such as the predictive power of machine learning, hold immense promise in advancing the future of healthcare. However, AI must work collaboratively and seamlessly with the people inherently embedded in our health systems. This requires that the means by which AI are developed and interfaces through which AI are delivered foster appropriate trust by the people working in our health systems (both professionals and patients). Conceptualizing trust such that sophisticated human minds should appropriately trust and collaboratively work with AI will help us realize the potential of human-AI teams in advancing health outcomes.

AUTHOR CONTRIBUTIONS

NCB proposed the initial concept, which was further articulated with LLN, CR, and JSA. NCB drafted the manuscript with substantial input from all other authors.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

No new data were generated or analysed in support of this research.