Medical Artificial Intelligence and Human Values

Recent progress in generative artificial intelligence (AI) has given rise to large language models (LLMs) that can be prompted to craft persuasive essays,¹ pass professional competency exams,^2–4 and write patient-friendly empathetic messages.⁵ Amid growing recognition of the capabilities of LLMs, many have expressed concerns about their use in medicine and healthcare, citing known risks of confabulation, fragility, and factual inaccuracy.⁶ As these risks are measured and mitigated, what is coming into focus are unresolved questions about the human values that will remain embedded in AI, both in their creation and in their use, and how the “values of an LLM” may misalign with human values even if AI models no longer confabulate and have been scrubbed of obvious toxic output. Such “human values” pertain broadly to the principles, standards, and preferences that reflect human goals and guide human behaviors (Glossary). As we review here, LLMs and new foundation models, as technically impressive as they are, are only the latest incarnation of a long line of probabilistic models integrated into medical decision-making, all of which have required their creators and implementers to make value judgments.

Many of the challenges we address here were evident to the pioneers of medical decision analysis of the 1950s⁷ and to scholars in subsequent decades^8–11 who conducted careful and creative studies of both human and algorithmic decision-making to disentangle probability, the chance of an event occurring, from utilities, the quantified value judgments that are often only indirectly articulated in much of medical decision-making (Glossary). The nuanced understanding of individual values and risks is what makes the thoughtful clinician so indispensable. These considerations have renewed relevance with unprecedented and ubiquitously available AI models like LLMs. In this article, we first describe how value judgments enter predictive models, using familiar clinical settings and new AI language models. We then connect early work in reasoning about probabilities and utilities to the emerging issues of newer AI models, identifying unresolved challenges and future opportunities in designing high-performance and safe AI models.

HOW HUMAN VALUES SHAPE AI MODELS

Myriad examples spanning automated chest X-ray interpretation,¹² skin disease classification,¹³ and algorithmic decisions about healthcare resource allocation¹⁴ have illustrated how the data used in training AI models encode individual and societal values that may become cemented in the model. As reviewed recently in the Journal,¹⁵ biased training data may both amplify and reveal the values and biases present in society. Conversely, studies have also shown that AI can be used to reduce bias. For example, researchers applied deep learning models to knee X-ray images and identified factors within the knee that were missed by standard severity measures graded by radiologists, reducing unexplained pain disparities between Black and White patients.¹⁶

Despite growing recognition of bias in AI models, particularly with respect to training data, less appreciated are the many additional entry points for human values along the development and deployment journey of an AI model. Explicit considerations and modeling of human values, and how they interact with risk assessment and probabilistic reasoning have been largely absent amidst the otherwise impressive recent successes of medical AI.

A MOTIVATING CLINICAL EXAMPLE

To make these abstract concepts concrete, imagine that you are an endocrinologist who has been asked to prescribe recombinant human growth hormone (hGH) to a 8-year old male whose height is falling below the 3rd percentile for his age and who has a post-stimulation hGH level of less than 2 ng/L (normal > 10 ng/L and > 7 ng/L in many countries outside the US), and is found to have a rare loss-of-function mutation in the gene encoding hGH. We posit that in this setting proceeding with hGH treatment is straightforward and uncontroversial. Much more controversial would be the administration of supraphysiologic doses of hGH to a 14-year-old male, whose height has consistently been at the 10th percentile for age, whose has a post-stimulation hGH peak of 8 ng/L, who harbors no known functional mutations that impact height, has no other known cause for his short stature and whose bone age is 15 years (i.e., not delayed). Only part of the controversy is due to divergence in the hGH threshold that experts, informed by dozens of studies, used in making the diagnosis of isolated growth hormone deficiency (IGHD).¹⁷ At least as much of the controversy stems from the risk-benefit tradeoff as seen from the perspective of the patient, his parents, the health professional, the pharmaceutical company, and the payor. The pediatric endocrinologist might weigh the rare adverse effects of 2 years of daily GH injections against the likelihood of no or minimal gain in adult stature. The boy might feel that even the possibility of a 2 cm gain is worth the effort. The payor and the pharmaceutical company may further disagree.

In 2024, the second case would have a default recommendation from an LLM such as the Generative Pre-trained Transformer 4 (GPT-4) model, which reflects not only its training data but also the process by which the model was trained, including methods such as supervised fine-tuning and reinforcement learning from human feedback (described below). Beyond this, each of the people or parties involved in the second case—the patient, the parents, the physician, the drug maker, and the payor—could instruct GPT-4 to inject “custom values” to reflect their viewpoints on the matter (Figure 1). This “tunability” of the model’s output is a desirable feature of these models, but it raises several questions.

Figure 1. How contemporary AI models may be “steered” to capture different human values.

Large language models (LLMs) such as GPT-4 encode human values based both on their training data and how they subsequently “tuned.” As this example illustrates, they can further be powerfully “steered” to adopt different roles. The human prompts in this example are about an identical case of a 14-year old short-statured boy; GPT-4 is instructed to “adopt” three different perspectives: A the treating physician, B the insurance company, C the boy’s parents. GPT-4 prompts and output are abridged to fit.

Whose values do a given AI model reflect? How should we proceed in adapting generalist AI models¹⁸ for medical decision-making? Will AI models facilitate rational decision-making that reflects the values of the patient or those of other parties? How will financial forces shape the creation and use of these models in medicine? How steerable should an AI model be when used by a physician for an evaluation and treatment plan?

At every stage of model creation and model use, human values enter (Figure 2). We illustrate this first with a simple statistical model familiar to clinicians (the estimated glomerular filtration rate, eGFR) and then, using LLMs, we show that beyond the data underlying an AI model, the model design, training and use cases will encode “human values”. These examples are not intended to be exhaustive but only to illustrate how human values enter across the spectrum of model complexity.

Figure 2. Entry points and choices for human values in traditional clinical equations and new AI models.

In both traditional clinical equations (e.g. eGFR) and new AI models (e.g. LLMs), human values enter at every stage including in choices about training data, model development, and model use. While the examples are highly varied, often the same questions can be used to elucidate human values in both traditional clinical equations and newer AI models.

IMPLICIT AND EXPLICIT VALUES IN FAMILIAR CLINICAL EQUATIONS

Consider a familiar clinical equation, the creatinine-based estimated glomerular filtration rate (eGFR), a widespread index of kidney function used to diagnose and stage chronic kidney disease, including setting thresholds to determine if an individual is eligible for kidney transplantation or donation, and to determine dose reductions and contraindications for many prescription drugs.¹⁹ The eGFR is a simple regression equation developed to estimate the measured glomerular filtration rate (mGFR), which is the gold standard but more onerous to assess.²⁰ This regression equation could hardly be considered an AI model but nonetheless illustrates many principles about human values and probabilistic reasoning.

Human values first enter eGFR in the data used to fit the equation. Most of the original cohorts involved Black and White participants;^21,22 generalizability to many other race groups was unknown. Human values further enter into this equation in the choice of accuracy (with respect to mGFR) as the primary target to optimize in estimating kidney function, what constitutes an acceptable accuracy level, how accuracy should be measured, and in the use of eGFR as a threshold to trigger clinical decisions (such as eligibility for transplant or prescription drugs). Finally, values enter in the choice of inputs to the model and their impact on its outputs. For example, until 2021, guidelines²³ recommended adjusting creatinine levels in the eGFR equation based on a patient’s age, sex, and race (categorized as Black or non-Black). Race adjustment was introduced to improve the equation’s accuracy (with respect to mGFR), but in 2020, major hospitals started challenging race-based eGFR, citing concerns including delayed transplant eligibility and the reification of race as biology.^24,25 Studies showed that how the eGFR model is formulated with regard to race can have profound and varying effects on both accuracy and clinical outcomes,^26–28 and thus selectively focusing on accuracy or on a subset of outcomes reflects value judgments that could obscure transparent decision-making.^29,30 Ultimately a national task force recommended³¹ a new equation³² refit without race to balance both performance and equity concerns.³³ eGFR illustrates that even a simple clinical equation has many entry points for human values.

THE VALUES EMBEDDED IN LARGE LANGUAGE MODELS

In contrast to clinical equations with few predictor variables, large language models (LLMs) may be composed of an inscrutable combination of tens to hundreds of billions of parameters or more. We say “inscrutable” because the exact way that a query leads to a response in most LLM is not mappable. GPT-4’s parameter count is undisclosed; its predecessor GPT-3 has 175 billion parameters.³⁴ More parameters do not necessarily equate with more capability, as smaller models subject to more compute cycles like the LLaMA³⁵ model or models that are carefully fine-tuned with human feedback can outperform their larger counterparts. For example, the InstructGPT³⁶ model (a 1.3 billion parameter model) outperformed GPT-3 as assessed by human raters in preferred model outputs.

The exact training details of GPT-4 are not publicly available but details for predecessor models including GPT-3, InstructGPT, and many other open-source LLMs have been published. Many artificial intelligence models now come with “model cards”³⁷ (Glossary); evaluation and safety data for GPT-4 have been released in an analogous “system card”³⁸ provided by the model’s creator, OpenAI. The creation of LLMs can be broadly divided into two phases, an initial “pre-training” phase followed by a “fine-tuning” phase to refine the model’s output.³⁹ In the pre-training phase, large corpora including raw internet text are provided to the model, which is trained to predict the next word. This seemingly simple “autocomplete” process yields a powerful base model but one that may also exhibit harmful behavior. Values entered here in the choice of the pre-training dataset(s) for GPT-4, and furthermore the decision to scrub inappropriate content (e.g., erotic content) from the pre-training data.³⁸ Despite these efforts, the base model may be neither useful nor free of harmful output.³⁸ It is the next phase of “fine-tuning” where much of the useful and non-toxic behavior emerges.

In the fine-tuning phase, “supervised fine tuning” (SFT) and “reinforcement learning from human feedback” (RLHF) are used to change, often profoundly, the language model’s behavior. In the SFT phase, hired human contractors write example responses to prompts which directly train the model. In RLHF, human raters rank model outputs for example inputs. These comparisons are then used to learn a “reward model” that further optimizes the model using reinforcement learning.³⁶ A surprisingly modest level of human participation can fine-tune these large models. For example, the InstructGPT model used a team of only about 40 human contractors, recruited from crowdsourcing websites, who passed a screening test that was used to “select a group of labelers who were sensitive to the preferences of different demographic groups.”³⁶ With LLMs like GPT-4, further complexity emerges from the infinite ways in which the model can be “steered” (Figure 1) to encode values long after the model is first trained.⁶ Many of these same considerations of how human values shape general-purpose LLMs apply not only to GPT-4 but also to the ecosystem of competing LLMs⁴⁰ produced by other organizations. There is also a growing cadre of medical large language models, for example Google’s Med-PaLM models.⁴¹ Finally, we note that LLMs will often not be used in a standalone manner, but rather after they have been customized and embedded in a larger system, creating further entry points for values.

As illustrated by these two extreme examples, i.e., simple clinical equation (eGFR) and in the powerful LLM (GPT-4), human decisions and therefore human values play an indispensable role in shaping model outputs. Do these AI models capture patient and physician values which themselves may be quite varied? How can we openly guide AI implementations in medicine? As described below, a principled approach to these questions may arise from revisiting medical decision analysis.

MEASURING HUMAN VALUES: MEDICAL DECISION ANALYSIS AND UTILITY ELICITATION

MEDICAL DECISION ANALYSIS

Although unfamiliar to many practicing clinicians, medical decision analysis provides a systematic approach to complex medical decisions by disentangling probabilistic reasoning about uncertain outcomes related to a decision (e.g. whether to administer hGH in the controversial case in Figure 1) from considerations of the subjective values attached to those outcomes, quantified as “utilities’’ (e.g. the value to the boy of an additional 2cm of height). Decision analysis requires one to first identify all potential decisions, outcomes, and the probabilities associated with each outcome, and then incorporates patient (or other party) utilities attached to these outcomes to select the optimal choice. As a result, the validity of a decision analysis depends on how comprehensively the outcomes are specified as well as how well the utilities are measured and probabilities are estimated. Ideally, this method can help ensure decisions are evidence-based and aligned with patient preferences, bridging the gap between objective data and personal values. This approach was introduced to medicine decades ago^7,10 and has been applied to both individual patient decisions ⁴² and population health evaluations such as recommendations for colorectal cancer screening in the general population.⁴³

While we do not foresee physicians dramatically altering diagnostic practice using decision analysis in the era of LLMs, the core principle of utility elicitation offers lessons on aligning AI models for medicine. These lessons include the fundamental incompatibility of utilities from competing parties,⁴⁴ the importance of how information is presented,⁴⁵ and the benefits of enumerating and measuring both probabilities and utilities even when uncertainty remains in both.¹⁰

UTILITY ELICITATION

Many methods have been developed in medical decision analysis to obtain utilities. Most conventional ways of doing so involve direct elicitation of the value from an individual. The simplest approach is to use a rating scale, where an individual scores their preferences for an outcome on a numeric scale, such as a linear rating scale (e.g. 1–10), with the most extreme health outcome (e.g., perfect health and death) on either end.⁴⁶ While these are intuitive, a major challenge is incorporating uncertainties in healthcare decisions. Time-tradeoff is another commonly used method. Here, individuals are asked to make decisions about how much time in “good health” they would trade for a quantity of time in a lesser health state.⁴⁷ The standard gamble is another popular approach for determining utilities. Here, an individual is asked their preferences of two options: either they live t years in a normal health state with a given probability, p, and risk dying at a probability of 1-p, or they live t years in a lesser health state with certainty. An individual is asked this multiple times at different values of p until they do not show any preference toward either option, allowing the calculation of a utility based on the response.⁴⁶

In addition to eliciting the preferences of individual patients, methods for obtaining the utilities of a group of patients have also been developed. In particular, focus group discussions, where patients are brought together to discuss a specific experience, can be useful in understanding their perspectives.^48,49 To effectively aggregate the utilities from a group, many structured group discussion techniques have been proposed. For example, the nominal group technique allows participants to write down their thoughts and preferences independently, followed by idea sharing and group discussion. Finally, the preferences of the group are aggregated by a voting process.⁵⁰ Although these structured discussion techniques can overcome issues of groupthink, there are inherent limitations in the voting procedures for obtaining group preferences.⁴⁴ In addition, as is true of all such exercises, the aggregated decision is not necessarily reflective of individual preferences.⁵¹ These findings highlight the difficulty in defining optimal decisions when more than one stakeholder is involved.

In practice, the elicitation of utilities directly from each individual is time-consuming. As a solution, population-level utility scores are commonly obtained using questionnaires sent to a randomly selected portion of the population. Some examples of these are the EQ-5D,⁵² the SF-6D utility weights,⁵³ the Health Utilities Index,⁵⁴ and the cancer-specific QLQ-C30 instrument.⁵⁵ From large surveys, population-level utilities can often be generated from survey data using methods such as the time tradeoff method and the standard gamble. The discrete choice experiment is another survey-based method for understanding preferences. Here, individuals are given a series of choices to choose between, from which quantitative health utilities can be calculated.⁵⁶ In each of these approaches, an individual’s utility may differ from the group utility, which raises the issue of individual autonomy when group derived utilities are applied to individuals.

UNRESOLVED CHALLENGES AND FUTURE DIRECTIONS

The examples from medical decision analysis, current methods for utility elicitation, and their limitations point to several unresolved issues and key questions for contemporary AI models in medicine.

WHOSE VALUES SHOULD BE INJECTED?

As discussed above, human values can profoundly shape the inputs and outputs of both simple clinical regression models and advanced AI models. For example, with LLMs, fine-tuning methods, including SFT and RLHF, refine LLM outputs based on human input from crowdsourced workers, hired and instructed by the model developers. This transmutes the question of which values are encoded in models to whose values are encoded. The values that should govern the range of model behavior in clinical care and the healthcare system remain unresolved, but efforts to develop principles for responsible medical AI are underway.^57,58 The potential biases from crowdsourced inputs and the variability in values across cultures further compound this challenge. Studies that develop and evaluate AI in areas where resources may be limited, including low and middle-income countries, are needed.^59,60 Emerging work characterizing the “psychology” of LLMs is promising.⁶¹ Future studies of AI in realistic clinical settings that rigorously evaluate how AI affects human decision-making and skill development are urgently needed.^62,63 Undoubtedly, such studies will both rediscover and exploit many lessons from the psychology and medical decision-making literature about human cognitive biases and heuristics that both enhance decision-making and lead it astray.⁸

DATASET SHIFT

Dataset shift⁶⁴ refers to changes in the data characteristics that can undermine the accuracy and reliability of AI models. Such shifts can arise due to evolving medical practices, demographic changes in the population, or the emergence of novel diseases. When incorporating human values into AI systems, shifts in societal values and differences in values among subpopulations can lead to inappropriate treatment recommendations, poor alignment with common societal expectations, and a potential loss of trust in AI-driven tools among both clinicians and patients.⁶⁵ Ensuring models are periodically retrained and that model outputs are regularly monitored can help foster the safe and effective application of AI in medicine,^66–68 as with non-AI diagnostic tests and procedures.^69,70 AI governance teams can also help provide oversight,^71,72 and agencies worldwide are grappling with how to regulate AI models, a challenge that will become more complex with foundation models^73,74 and models that can reason over multiple data types.^75–77 Finally, considerations of the values of individual patients may cause physicians to ignore or override AI recommendations; the liability implications remain an active focus by legal scholars.⁷⁸ As medical AI becomes more integrated into care, recognizing and mitigating the risks associated with dataset shift will be paramount in aligning AI outputs with human values.

ALTERNATIVES TO DIRECT UTILITY MEASUREMENT

Although the utility elicitation methods described above can obtain human values, they are often limited to well-controlled study settings and miss the nuances of decision-making as individuals grapple with real-world healthcare scenarios. They can also be sensitive to framing and context,⁴⁵ biased,^79,80 as well as difficult to scale. Decision curve analysis^81,82 is an alternative paradigm to evaluate diagnostic tests and predictive models without requiring explicit utility elicitation. Another emerging line of research employs data-driven methods to extract human values and integrate them as long term objectives in order to support continual learning that may adapt to shifting data and values.

The subfield of reinforcement learning focuses on having a computer “agent” learn what actions to take in a given state and environment in order to maximize a specified “reward.” RLHF, seen above with LLMs, is one example of reinforcement learning. A key component is the reward function, which quantifies the desirability of each state. Given the myriad clinical scenarios and patient-specific utility variations, crafting this function is challenging but remains an active frontier.

Conclusion

At every stage of training and deploying an AI model, human values enter. Echoing the insights of decision analysts from decades ago, we recognize that AI models are far from immune to the shifts and discrepancies of values across individuals and societies. Past utilities may no longer be relevant or even reflect pernicious societal biases. Our shared responsibility is to ensure that the AI models we deploy accurately and explicitly reflect patient values and goals. As noted by Pauker and Kassirer in the Journal more than three decades ago in reviewing progress in medical decision analysis,¹⁰ “the threat to physicians of a mathematical approach to medical decision making simply has not materialized.” Similarly, rather than replacing physicians, AI has made the consideration of values reflected in the guidance of a thoughtful physician more essential than ever.

Glossary

Alignment

The degree to which an artificial intelligence system’s behaviors and actions are congruent with human values.

Generative Artificial Intelligence

A form of artificial intelligence designed to produce new and original data outputs, including those that resemble human-made content (including text, code, images, audio, video etc).

Human Values

A broad term for the principles, standards, and preferences that reflect human goals and guide human behaviors.

Large Language Model (LLM)

A type of artificial intelligence model that interprets and generates text. LLMs are often “pre-trained” with large text corpora and then fine-tuned with human feedback via supervised fine-tuning and reinforcement learning from human feedback.

Model Card

Comprehensive overview and performance characteristics of a machine learning model (e.g. training and evaluation data, training procedure), existing evaluations (e.g. observed safety or bias challenges and existing remediation strategies), intended use cases, and model performance across populations (e.g. key demographic or clinical groups). Similar to “System Card” for GPT-4.³⁸

Reinforcement Learning from Human Feedback (RLHF)

A method of fine-tuning LLMs where humans rank responses to prompts; reinforcement learning is then used to optimize the output to align with human preferences.

Supervised Fine-tuning (SFT)

A method of fine-tuning LLMs that uses human-written responses to example prompts.

Utility

The quantitative measure used in decision analysis to assess the value of a health state or outcome. Utilities may be elicited directly from individual patients or groups, or they can be learned from data. Utilities may be applied to individuals, groups, or populations.