“Black Box” Artificial Intelligence for Mortality Prediction: A Mixed-Methods Study of Palliative Care Team, Patient, and Caregiver Perspectives

Abstract

Background:

New artificial intelligence (AI)-based mortality prediction algorithms could support both patients’ prognostic awareness and person-centered palliative care. Although they promise accuracy, their outputs can be hard to explain – potentially affecting whether patients and care teams use them.

Methods:

To investigate perspectives on the explainability of AI algorithms in palliative care, we conducted a sequential mixed-methods study. We interviewed 30 palliative care physicians and nurses; 15 social workers, spiritual care providers, psychologists, and others; and 35 patients and caregivers at four U.S. academic centers (total n=80). The 53 interviews containing data on explainability were analyzed thematically to understand reasons for concern or unconcern. We randomly sampled and surveyed n=2500 palliative care physicians (overall adjusted response rate, 32.6%). The 537 surveys with complete responses on explainability items were analyzed descriptively; a multivariable model examined predictors of concern.

Results:

Among 53 interviewees, 18 expressed only concern about black box AI-based prognostication, 17 expressed only unconcern, and 18 interviewees expressed mixed sentiments. Reasons for concern related to: data transparency, mistrust of machines or their creators, patient-clinician communication, bias, and accuracy. Reasons for unconcern related to: inexplicability not unique to AI, greater accuracy, not using AI in isolation, trust in science, and being evidence-based. Notably, “accuracy” and “trust” appeared in both. Overall, 75% of physicians (n=396/528) reported being at least “moderately concerned” about unexplainable AI algorithms. Male physicians were less likely to be strongly concerned (aOR 0.57; 95% CI: 0.36, 0.89; P=0.01) about explainability. Those who perceived AI mortality prediction to be inaccurate were more likely to be concerned (aOR 2.06; 95% CI: 1.27, 3.41; P=0.003).

Conclusions:

Our findings suggest that if a black box model is perceived as accurate, there may be less demand for explainability. Nevertheless, in palliative care – where communication is key - explainability may still be central. Future efforts should seek to create models that are both accurate and explainable at the point-of-care.

1. INTRODUCTION

1.1. Background:

The use of artificial intelligence (AI) in healthcare has become increasingly common (1). Physician use of AI doubled between 2023 and 2024, with most clinicians feeling optimistic about AI use (2). Due to its ability to detect patterns in images, AI shows great potential in radiology, with nearly 1 in 3 radiologists already using AI (3). A 2025 survey from a network of 67 non-profit hospitals found that a majority are also applying AI to administrative tasks (e.g., note writing), image interpretation, or clinical decision-making (such as predictive algorithms for clinical deterioration or sepsis) (4). Indeed sepsis predictive algorithms and inpatient deterioration predictive models have shown positive results, including earlier intervention and reduced mortality (5,6).

The benefits of AI-based predictive analytics thus extend to all areas of medicine (7–11). For palliative care in particular, AI could support more goal concordant, person-centered care decisions. AI tools to identify patients at high risk of mortality are being rapidly developed and adopted in many different care settings (12), largely to predict life expectancy or prognosis and prompt serious illness conversations or palliative care consultations (13–17). Some think it could even predict incapacitated patients’ end-of-life goals (18). AI is appealing in part because prognosticating life expectancy is challenging: a 2022 review concluded that physicians’ estimates of their patients’ prognoses are often incorrect. Various factors impacted accurate prognostication, including ethical values, attitudes, opinions, and ego bias (19). Yet accurate knowledge of life expectancy – i.e., prognostic awareness – is one key component of care planning and helps patients make health care decisions that align with their wishes and values (20). AI could provide more accurate information to patients, families, and care partners (21).

AI-based prognostication also raises a significant number of important ethical issues, including around choice, unintended harms, fairness, and more. One concern relates to the explainability of AI models – i.e., the ability to understand and explain why a model came to a particular output or prediction (as distinct from the technical details of how a model operates). The more complex an AI model is, the more unintelligible its processes can become to the user (22). Some models, such as deep neural networks, even approach absolute inexplicability – “black box” models that, because of the complexity of their decision-making pathways and network of calculations, cannot be explained by the end user (23). These models raise concerns about uncaught errors and moral or legal responsibility, patient autonomy, and inappropriate influence over health care decision making (24,25). On the one hand, explainability is sometimes thought prerequisite to widespread public acceptance and uptake of AI in health care (26). On the other, if AI predictions are more accurate, others claim that there is an ethical obligation to use them, explainable or not (26,27).

1.2. Rationale:

The relative importance of explainability likely relates to the nature of the prediction or clinical context. For instance, we might demand more in the way of explainability for higher risk decisions, compared to lower risk ones. Additionally, the type of explainability that is desired may differ by context (e.g., patients and clinicians may desire different levels or forms of explainability). Yet little is known about palliative care physicians’ and patients’ perspectives on the importance of explainability in AI-based mortality prediction or prognostication. Previous interview (22,28,29) and survey (30–35) research of physicians from various specialties identified transparency and explainability as central concerns. However, these were not tailored to palliative care and were largely conducted outside the U.S. Only one study evaluated oncologists’ perspectives on AI mortality prediction algorithms (36). From a patient perspective, systematic (37,38) and scoping (39) reviews suggest patients are both optimistic about AI (37) but still concerned about explainability or black box algorithms.

1.3. Objective:

To address gaps in our knowledge related to perspectives on explainability in AI-based prognostication, we conducted a sequential mixed-methods study.

2.1. Study Setting:

We used an exploratory sequential mixed-methods approach (40) to assess perspectives on AI-based prognostication – specifically, mortality prediction – in palliative care. Given the exploratory nature and lack of prior evidence related to our research question, this design allowed us to gain rich insights from qualitative interviews that we were then able to use to inform the design of a national survey to achieve more generalizable findings. Part 1 involved 80 semi-structured interviews with palliative care physicians and nurses, other care team members (e.g., social workers, spiritual care, psychologist and others), and palliative care patients and caregivers at four geographically diverse U.S. medical centers (West, Mountain West, Southeast and Northeast). All four sites had implemented AI-based mortality prediction to varying extents. Part 2 involved a national cross-sectional survey of palliative care physicians. The full study investigated a range of ethical issues in AI-based prognostication; here we present mixed-methods findings related to explainability.

2.2. Interview Sample and Procedures:

For interviews, we recruited palliative care team members at the four institutions as previously described (41). We sampled purposively based on age, race, ethnicity, discipline, practice setting, and practice site to achieve maximal variation and elicit a wide range of views. We recruited patients and family caregivers via chain referral sampling from the care team group, sampling purposively based on age, race, ethnicity, and disease condition (dementia, advanced cancer, and heart failure – recognizing the different illness trajectories of these conditions).

We created an interview guide based on our review of the literature and the research team’s expertise. The guide included domains around communication, the patient/caregiver experience, and shared decision-making, as well as specific probes around certain ethical issues (such as black box explainability). Interviews were conducted over Zoom or phone between August 2022 and August 2024 and lasted up to 85 minutes. After oral consent, audio was recorded and then transcribed by a professional transcription service.

2.3. Interview Analysis:

Transcripts were analyzed by employing constructivist grounded theory methodology (42). Analysis began after each interview, with memos and notes discussed weekly amongst the team. While shaped by questions in the guide, coding began by open coding – allowing for important themes to emerge from the data. Codes and the codebook were iteratively refined during weekly team meetings. Three research team members were involved in coding, with 16 transcripts (20%) double coded to ensure inter-coder agreement. Data were managed using ATLAS.ti (version 23, Windows; Scientific Software Development GmbH, Berlin, Germany). Throughout, reflexivity (i.e., disclosure and management of potential biases) was used to ensure reliability and validity (43).

For this analysis, we retrieved codes related to black box explainability, occurring in 53 of the 80 interviews, and engaged in additional qualitative thematic analysis. Our analytic aim was to understand, first, whether the interviewee’s sentiment expressed concern or unconcern about explainability, and second, the reasons why.

To do this, we first blinded the identity of the interviewee, anticipating the desire to compare and contrast care team members’ to patients’ and caregivers’ perspectives, we wanted to avoid potential biases. Then, two team members independently coded all content for concern or unconcern, resolving disagreements through discussion. To categorize reasons behind concern/unconcern, we generated a list of potential subcodes based on existing research and what we had heard in interviews. For instance, we considered three initial reasons for concern (data transparency, effect on patient-clinician communication, and bias) and two for unconcern (evidence-base supporting the AI and not using AI in isolation). One team member read quotes several times to identify and group subcodes for each, eventually arriving at ten total subcodes. These ten were used by a second coder independently to code all quotations with differences again resolved by discussion between coders.

2.4. Survey Design and Sample:

We conducted a cross-sectional survey of U.S.-based physicians who are board certified in Hospice and Palliative Medicine. We randomly sampled 2,500 physicians who met this inclusion criterion from IQVIA’s OneKey healthcare provider database. Survey administration occurred in two waves (n=1250 each) between January 2024 and March 2025, with sampled physicians randomly assigned to either Wave 1 or Wave 2. Sampled physicians were mailed a survey packet containing a paper questionnaire with an individualized secure link to complete the questionnaire electronically if preferred and an upfront $50 check as an incentive to participate. Three additional mailings of the survey packet were sent to non-responders at monthly intervals. At the end of the fielding period, remaining non-responders for whom we had a valid email address (n=907) were invited by email to complete the questionnaire.

2.5. Statistical Analysis:

Survey respondents were excluded from analyses if they were missing responses to age (estimated from year of birth), gender, or race. We determined a pre-specified list of sociodemographic and attitudinal variables from our survey which we hypothesized would be related to the primary outcome, concern over black box algorithms. To create the primary outcome, we dichotomized responses to the primary survey item as “Strong concern” (“very concerned” and “moderately concerned” on the survey) and “Weak or No concern” (“a little concerned” and “not at all concerned” on the survey). Age was approximated by taking the difference of the survey year (2024 or 2025) and the birth year of each respondent, and then categorized into three groups, <40, 40–59, 60 and older. Race was defined into two categories, Person of Color (POC) which comprised of those who identified as Asian, Black, Middle Eastern, Native American, or multi-racial or Hispanic, and White which comprised of those who identified as White and not Hispanic. Early technology adopter was defined as those who answered “the first”, “Not the very first, but I am among the first”, and “Earlier than most of my peers” regarding adoption of new technology. Disease trajectory (i.e., specialty type) was created based on the largest proportion of patients cared for and categories included: rapid (cancer - all types and populations), intermittent (cardiovascular disease, pulmonary disease, liver failure), gradual (neurological disease, psychiatric illness, and kidney failure), and other (autoimmune, chronic pain, OBGYN, primary care, pediatrics, etc.). Response categories on the 4-item Likert scale for both general AI knowledge and AI prognostication knowledge were dichotomized such that “nothing at all” and “a little” were grouped into ‘low’ knowledge and “a moderate amount” and “a lot” were grouped into ‘high’ knowledge.

We performed descriptive statistics for variables of physician self-reported personal and clinical practice characteristics and AI-related attitudes. Categorical data were characterized using proportions (%) and frequencies were stratified by the outcome. To examine bivariate associations, simple logistic regressions were fit, and results are presented with odds ratios and their respective 95% confidence intervals, with two-sided P-values <0.05 considered significant. To evaluate the association between physician characteristics and concern over black box algorithms after adjusting for physician characteristics, a multivariable model was constructed by including all pre-specified sociodemographic and attitudinal covariates using a multivariable logistic regression framework. The adjusted odds ratios and confidence intervals are presented with the crude estimates. We employed complete case analysis. All analyses used R version 4.4.0 (2024–04-24).

3. RESULTS

We completed 80 qualitative interviews with palliative care team members, patients and caregivers, 53 of which referenced black box explainability. A total of 2500 physicians were invited to take the survey and 583 were returned. We used the AAPOR RR3 method—the most common AAPOR method used in reporting (44)—to calculate our response rate. Our known working address rate (71.5%) and known eligibility rate (99.9%) were used to estimate what proportion of our overall sample was possible to contact and eligible to participate, yielding an overall response rate of 32.6%. Among 583 respondents, 4 respondents were missing year of birth (and subsequently age), 12 were missing gender, 25 were missing race, and 5 were missing both age and race, which resulted in a respondent population of 537 for this analysis. Participant characteristics for respondents included in this analysis are in Table 1.

Table 1.

Participant Characteristics

Characteristic	Interview Participants n (%)	Survey Participants n (%)
Total	53 (100%)	537 (100%)
Age
Under 40	12 (23%)	148 (28%)
40–59	23 (43%)	256 (48%)
60+	18 (34%)	133 (25%)
Gender
Female	42 (79%)	284 (53%)
Male	11 (21%)	253 (47%)
Race and Ethnicity
White	33(62%)	393 (73%)
Asian	6 (11%)	86 (16%)
Black	4 (8%)	17 (3%)
Middle Eastern	0 (0%)	4 (1%)
Hispanic	1 (2%)	1 (<1%)
Multiracial and/or multiethnic	9 (17%)	36 (7%)
Asian and Black	1 (2%)	1 (<1%)
Asian and White	0 (0%)	4 (1%)
Black and Hispanic	2 (4%)	1 (<1%)
Black and White	1 (2%)	0 (0%)
Hispanic and White	2 (4%)	25 (5%)
Middle Eastern and White	0 (0%)	3 (<1%)
Native American and Black	1 (2%)	0 (0%)
Native American and White	1 (2%)	2 (<1%)
Unspecified	1 (2%)	─
Role
Physician	15 (19%)	537 (100%)
Nurse or Advance Practice Provider	9 (19%)
Other Care Team	13 (3%)
Patient	7 (24%)
Caregiver	9 (20%)
Religiosity
Some		261 (49%)
Not at all		175 (33%)
Very		99 (19%)
Unknown		2
General AI Knowledge
Low		426 (79%)
High		111 (21%)
AI Prognostication Knowledge
Low		491 (92%)
High		44 (8%)
Unknown		2
Early adopter of technology		369 (69%)
Unknown		2
Medical school appointment (yes)		282 (53%)
Disease specialty trajectory
Rapid		309 (59%)
Intermittent		91 (17%)
Gradual		86 (16%)
Other		39 (7%)
Unknown		12

3.1. Qualitative Findings:

Of the 53 interviewees who discussed black box explainability, 18 interviewees expressed only concern about black box AI-based prognostication, 17 interviewees expressed only unconcern, and 18 interviewees expressed mixed sentiments with varying reasons for both concern and unconcern. Of note, among interviewees expressing only concern or unconcern, care team members overall and physicians in particular were more likely to express only concern while patients and caregivers were more likely to express only unconcern. Of these interviewees, 8/10 (80%) physicians expressed only concern and 9/11 (82%) patients/caregivers expressed only unconcern.

3.1.1. Concern About Black Box:

Among 53 interviewees who discussed black box explainability, 36 expressed some concern about black box AI-based prognostication (29 care team members and 7 patients). We were able to group the reasons given for that concern into five types (Table 2).

Table 2.

Types of reasons given for concern about black box AI-based prognostication listed in order of descending frequency

Reason	Meaning	Example Quotations
Data Transparency	Describing wanting to know what data the algorithm used, or how it created the result	“I would like to know like what information they’re synthesizing to get to that end result, and what would those triggers look like... Like what are the overall factors they’re looking at?” (Other Care Team Member) “At least, for us, you need to understand how that...what does it factor in, and what does it not factor in?” (Caregiver)
Mistrust of Machines or Their Creators	Describing a mistrust of, or disbelief in machines’ capability to predict these outcomes or concerns about the motivations of the algorithms’ creators	“Somebody had to program it, and I personally don’t believe that...There’s no soul, and so where the computer is only as good as the people that programmed it... You’re dealing with people and so as long as we’re dealing with the medical field, we’re dealing with people. I don’t care what kind of machine it is.” (Caregiver)
Patient-Clinician Communication	Describing wanting the clinical team to be able to explain the algorithm, answer questions about its results, or add information when discussing it with the patient	“I’m a communicator, and I need to be able to tell people that ‘I have this tool, this is how it works, I believe that it works, and that’s why I’m bringing it to this conversation.’“ (Physician) “It’s somewhat unsatisfying to patients when they say, ‘Oh, well, the doctor told us three months, but how do they know?’ It’s like, ‘Well, he uses his personal and his professional experience, plus what the research tells him, plus dah, dah, dah.’ That’s not always satisfying to patients and families, and so I think it could get even worse if you say, ‘Oh, well, we don’t even know who came up with this or how....’” (Other Care Team Member)
Bias	Describing concerns that the algorithm performs differently for certain patient groups or is biased	“That’s at the end of the day, just very—it’s national. That doesn’t necessarily apply to me and the people around me in [my state]. It’s nice to know the national information, but it’s important to what I do on a daily basis to know our local information.” (Other Care Team Member)
Accuracy	Describing concerns about how accurate the algorithm is, or will continue to be	“I think the thing about if you had repeated validation in different ways that shows it’s consistently highly accurate, that that’s helpful to know. I think that the areas of discomfort are always, "But how is it getting that information? Is it gonna stay accurate?" Because everything in the chart changes over time, and how something’s documented or whatever it might be is gonna be different in the chart or in Epic over time. Is it still gonna remain accurate if we have no idea what it is that stuff’s coming from?” (Physician)

Data transparency was the most frequent reason for concern, appearing in 15/29 (52%) care team interviews and 3/7 (43%) patient/caregiver interviews. Most participants concerned about data transparency wanted to know which data points were included in the tool, particularly when the electronic health record (EHR) is involved and when algorithms are proprietary. As a physician said, “...because…it’s a proprietary thing, we only get a certain amount of information of what they’re utilizing…”

Some concerns over black box algorithms were closely tied to the need to communicate. From the palliative care team perspective, we observed this theme more commonly in interviews with other care team members, such as social workers, spiritual care providers, and psychologists (6/11, 55%), compared to physicians or nurses in our sample (5/18, 28%).

Yeah, it would make me much more hesitant to trust it if I can’t understand or can’t explain to patients...I feel like that just leaves a level of uncertainty that’s not helpful and can be harmful.

(Other Care Team Member)

Bias – understood as differential performance of AI-based prognostication between groups – arose in 11/29 (44%) care team interviews but only one patient/caregiver interview. Bias was not just about age, race, ethnicity, or other demographic factors. It also related to geographic location, given regional difference in patterns of health and disease.

Accuracy arose in 10/29 (34%) care team interviews and 2/7 (29%) patient/caregiver interviews. Some interviewees expressed a concern about accuracy changing over time:

...What’s gone from something very precise has now...things that we can’t even anticipate in being inaccurate. What if it decides on its own that it’s not gonna be accurate for select people?

(Physician)

Finally, a notable theme emerging from our analysis was that of mistrust, either of algorithms themselves or of the people who might have created them. This arose in 11/29 (39%) care team interviews and 3/7 (43%) patient/caregiver interviews. For some, mortality prediction was simply not the sort of task that should be left to a machine.

3.1.2. Unconcern About Black Box:

Among 53 interviewees who discussed black box explainability, 35 expressed some unconcern about black box AI-based prognostication (21 care team members and 14 patients and caregivers). We were able to group the reasons given for that concern into five types (Table 3).

Table 3.

Types of reasons given for unconcern about black box AI-based prognostication listed in order of descending frequency

Reason	Meaning	Example Quotations
Inexplicability Not Unique to AI	Describing being unconcerned with explainability because they use other things they do not understand or that can only be explained by experts.	“I don’t—I—that wouldn’t really bother me too much not knowing the nitty-gritty about exactly putting it together. Again, there’s so much care delivery that happens for my patients that I don’t know on the level of some of the specialists that refer to me if that makes any sense. There’s a lot of stuff that I feel like I’m a little bit in the dark about, but I still do what I do ’cause I know what I do and how it helps. I know my part really well. I guess I’m very good at suspending understanding and not what things are going on...” (Physician) “‘Cause [doctors] will not understand the algorithm. They’ll be able to describe how all kinds of data was taken in and it was crunched around. How many people can explain a computer program? Very few. Very few. Yeah. That’s the way it’s gonna be, but the doctor would have to be able to answer the question from the patient of ‘Why do you trust it?’ The doctor could have a very good answer. He might say, ‘I’ve seen this thing—I’ve seen this thing work before, and it’s been pretty accurate. You know what? I’ve been reading these papers from the other doctors, and they’ve been monitoring it also. It’s right most of the time, and that’s why I trust it. I trust the doctors who have reviewed the data in their place and all that.’ They would not be able to explain how it worked, no more than they can explain how a rocket engine works.” (Caregiver)
Greater Accuracy	Describing not being concerned with explainability because they believe the algorithm is likely more accurate or outperforms humans.	“Humans make errors, and if the AI works but nobody understands it that doesn’t mean that it’s not good anymore, it just means that the AI is farther advanced than the doctor.” (Patient)
Not Using AI in Isolation	Describing not being concerned with explainability because the AI tool will be used in conjunction with personal and clinical judgement	“Because I don’t care what the AI has to say, or even another clinical expert, I still by and large listen to my body and understand it. I understand how it works. I pay attention to nuances that occur, and my body. The fact that they forthright about it, I’d be comfortable of the fact that I would tend to trust it more than them to be secretive about it.” (Patient)
Trust in Science	Describing not being concerned with explainability because they believe in science and cutting-edge technology.	“I’m all about science and moving forward. I’m fortunate that in the healthcare system I work, they have a strong belief and support of palliative care. It kind of lends itself to how I would think anyway. We wanna be at the forefront of everything. That’s what makes us a great nation.” (Nurse)
Evidence-based	Describing being unconcerned with explainability, if the tool is evidence-based and supported by scientific literature.	“I think they would just wanna know that it was backed up by someone that they could trust. When you get the Good Housekeeping seal of approval, as long as there was some entity that could say, ‘We’ve done our research on this. We know that this is to be,’ then that’s who we would put our trust in.” (Caregiver)

The most common reason for lack of concern - appearing in 11/21 (52%) care team interviews and 7/14 (50%) patient/caregiver interviews – was that inexplicability was not unique to AI. Interviewees stated they could accept a black box algorithm because they already accept things -- like the EHR, basic lab tests, and computers in general – even when they did not fully understand them. While some participants hoped that someone, somewhere (e.g., a technology expert) would be able to explain the algorithm, they did not always expect this of the care team.

Four other reasons explained unconcern. First, several participants emphasized that, so long as a tool was evidence based, explainability was not a concern. Second was a distinct but related reason about the accuracy of the model. This appeared in 8/21 (38%) care team interviews and 6/14 (43%) patient/caregiver interviews. Here – perhaps reflecting known inaccuracies in human prognostication – interviewees seemed to think that AI might be more accurate than physicians, leading them overlook explainability. In some cases, participants dissociated AI being trustworthy with explainability; as one physician said related to the black box nature of some AI, “...That wouldn’t be a huge issue for me if I felt like it was accurate and trustworthy (emphasis added)...” suggesting that other features besides explainability might lead to trust in AI. Third, trust in science and a belief in technological progress led some to be unconcerned about black boxes. Lastly, while only appearing in 4/21 (19%) care team interviews and 3/14 (21%) patient/caregiver interviews, the assumption that AI would not be used by itself- that other inputs would be used – led some to be unconcerned about explainability.

3.1.3. Shared Reasons, But Different Assessment of Concern:

As evident in Tables 2 and 3, two types of reasons – related to accuracy and trust – appeared both in interviews among those who were concerned and among those who were not. Those concerned about explainability tended to believe that algorithms might be inaccurate or develop inaccuracies over time, or that algorithms or their creators could not be trusted. By contrast, those unconcerned about explainability appeared to hold a belief that algorithms would exceed current mortality prediction or that their trust in scientific progress supported trust in the algorithm.

3.1.4. Comparing Concern or Unconcern and Their Reasons:

Overall, palliative care team members appeared more concerned about black box algorithms than patients and caregivers. The ratio of care team members who expressed concern about black box to those who expressed unconcern was 1.4 (29 concerned, 21 not concerned) whereas that ratio for patients and care partners was 0.5 (7 concerned, 14 not concerned). We also found slight differences between palliative care team members and patients or caregivers in the types of reasons given, especially among those expressing concern. No specific reason was unique to any participant type, but the percentage of interviewees expressing each concern differed (Figure 1). Again, the nature of qualitative research means that frequencies and proportions reflect trends (not generalizations); frequencies are not meant as prevalence estimates. However, we observed that care team members cited issues related to patient communication and bias more often than patients or caregivers.

Figure 1. Comparison of care team members’ and patients/caregivers’ reasons for concern or unconcern about black box algorithms.

3.2. Quantitative Results:

Overall, 75% of respondents (n=396/528) reported being at least “moderately concerned” about using AI-generated prognostic information from unexplainable AI algorithms; nine respondents were missing an answer to this question. When asked about factors that could influence their decision to use AI-generated prognostic information in their clinical practice, 67% (n=352/527) of respondents reported that it was “very important” that they were able to explain to patients how the prognostic information was created and 80% (n=421/526) responded that it was “very important” to know what data were being used by the algorithm (Table 4).

Table 4.

Distribution of responses on physician survey items related to explainability

	Not concerned at all	A little concerned	Moderately concerned	Very concerned	Missing
Sometimes, it is impossible to know exactly how AI algorithms arrive at their predictions. How concerned are you about this when thinking about using AI-generated prognostic information?	22 (4.2%)	110 (21.0%)	222 (42.0%)	174 (33.0%)	9
	Not at all important	Slightly important	Somewhat important	Very important	Missing
When deciding whether to use AI generated prognostic information in your practice how important is it that you are able to…?
Know what data are being used by the algorithm?	6 (1.1%)	27 (5.1%)	72 (13.7%)	421 (80.0%)	11
Explain to patients how the prognostic information was created?	6 (1.1%)	43 (8.2%)	126 (23.9%)	352 (66.8%)	10

Participant characteristics stratified by concern about black box AI after dichotomizing responses to “Strong concern” versus “Weak or no concern” are presented in Table 5. Gender and perception of AI accuracy were both significantly associated with concern over “black box” algorithms, after adjusting for age, race, general AI knowledge, AI prognostication knowledge, religiosity, and perception of AI accuracy relative to physician (Table 6).

Table 5.

Distribution of physician characteristics by concern over black box algorithms (n=528, row percents displayed)

Characteristic	Strong concern N = 396†	Weak or no concern N = 132†
Age
< 40	106 (72%)	42 (28%)
40 – 59	192 (76%)	60 (24%)
60 & older	98 (77%)	30 (23%)
Gender
Female	218 (78%)	60 (22%)
Male	178 (71%)	72 (29%)
Race
POC	107 (74%)	37 (26%)
White	289 (75%)	95 (25%)
General AI Knowledge
High	85 (77%)	25 (23%)
Low	311 (74%)	107 (26%)
AI Prognostication Knowledge
High	31 (72%)	12 (28%)
Low	363 (75%)	120 (25%)
Unknown	2	0
Religiosity
Not at all	136 (79%)	37 (21%)
Some	194 (76%)	62 (24%)
Very	64 (66%)	33 (34%)
Unknown	2	0
Perception of Accuracy: AI generated mortality prediction
Mostly & Very accurate	224 (70%)	97 (30%)
None/A little accurate	157 (82%)	34 (18%)
Unknown	15	1
Perception of Accuracy: relative to physician
AI more accurate	142 (70%)	62 (30%)
Doctors more accurate	144 (83%)	29 (17%)
Equal	97 (75%)	33 (25%)
Unknown	13	8

^†n (%)

Table 6.

Multivariable logistic regression modeling concern about black box AI-based prognostication: crude and adjusted results

	Crude Results			Adjusted Results (n=496)
Characteristic	OR	95% CI	P value	aOR	95% CI	P value
Age			0.54			0.15
< 40		(—)			(—)
40 – 59	1.27	(0.80, 2.01)		1.30	(0.79, 2.12)
60 & older	1.29	(0.75, 2.24)		1.85	(0.99, 3.50)
Gender			0.06			0.01
Female		(—)			(—)
Male	0.68	(0.46, 1.01)		0.57	(0.36, 0.88)
Race			0.8			0.9
White		(—)			(—)
POC	0.95	(0.62, 1.49)		1.03	(0.64, 1.69)
General AI Knowledge			0.53			0.09
High		(—)			(—)
Low	0.85	(0.51, 1.39)		0.61	(0.34, 1.07)
AI Prognostication Knowledge			0.66			0.98
High		(—)			(—)
Low	1.17	(0.56, 2.30)		0.99	(0.41, 2.21)
Religiosity			0.07			0.08
Not at all		(—)			(—)
Some	0.85	(0.53, 1.35)		0.83	(0.50, 1.36)
Very	0.53	(0.30, 0.92)		0.50	(0.27, 0.92)
Perception of Accuracy: relative to physician			0.007			0.14
Equal		(—)			(—)
AI more accurate	0.78	(0.47, 1.27)		0.91	(0.54, 1.52)
Doctors more accurate	1.69	(0.96, 2.97)		1.54	(0.85, 2.77)
Perception of Accuracy: AI generated mortality prediction			0.002			0.003
Mostly & Very accurate		(—)			(—)
None/A little accurate	2.00	(1.30, 3.14)		2.06	(1.27, 3.41)

(—) indicates reference category.

On average, the odds of strong concern about black box algorithms for male respondents is 0.57 times the odds of female respondents (i.e., 43% lower), after adjusting for covariates (95% CI: 0.36, 0.89; P=0.01). The odds of strong concern about black box algorithms for those who perceived AI-generated mortality predictions to be only a little accurate or not accurate at all was 2.06 times when compared to those who perceived AI-generated mortality predictions as mostly accurate or very accurate, after adjusting for covariates, and this was statistically significant (95% CI: 1.27, 3.41; P=0.003).

Our study found no significant differences in concern about black box algorithms based on age, race, general AI knowledge, specific knowledge of AI-based prognosis, religiosity, or perceived prognostic accuracy of AI relative to human physicians.

4. DISCUSSION

4.1. Key Findings:

Mortality prediction appears to be one of the more prevalent uses of AI in palliative care (45,46). Our study is the first to examine real-world perspectives of palliative care team members, patients, and caregivers on black box explainability in mortality prediction algorithms. Of note, themes surrounding patient communication and biases in performance were more prevalent in care team interviews, and certain themes (such as accuracy and trust) were cited as either cause for concern or unconcern. Results of our national survey suggest that palliative care physicians overall are concerned about explainability and the use of black box AI tools, and that concern is mostly associated with whether an individual believes AI is accurate.

4.2. Explanation of Findings and Comparison to Prior Research:

Our results add to the broader literature and the ongoing discussion regarding the importance of explainability for AI tools in medicine. A recent survey of UK physicians found that the overwhelming majority felt AI algorithms should be explainable (47). Other studies have documented that health professionals and the public value transparency and understanding of the workings of AI models (37,48–50). With a near even split among our interviewees and the vast majority of physician survey respondents expressing concern about explainability, this concern was evident in our study.

One reason for this may be the unique setting of palliative care. AI algorithms are being increasingly developed in this field, and a recent systematic review of explainable AI in palliative care found that models are becoming increasingly complex and sophisticated (51). Prior research has suggested that people value explainability more in higher stakes situations like emergencies and scarcity (48). Palliative care often involves tackling high stakes decisions (such as around life expectancy and serious illness) and placing high priority on communication (52). Thus, it is not surprising that accuracy and impact on patient-clinician communication emerged as major themes in our study. This concern was evident across the entire palliative care team, not just physicians or nurses, a previously understudied group that is likely to intersect with AI and that is critical to patient-centered care (53).

Our results also shed light on ongoing discussion regarding the relative importance between knowing that something works (“accuracy”) and knowing why it works (“explainability”) (27). Prior research suggests that accuracy is a key component for AI acceptance (37,50,54), perhaps preferred over explainability or interpretability by the general public (48). From the patient perspective, evidence is mixed. A survey found that the most patients were uncomfortable with a highly accurate but unexplained diagnosis (55), though others suggest that patients tend not to question AI but instead rely on trust in their clinician’s decision to use it (37,53,56,57). In interviews, participants acknowledged that much of medicine can be equally inexplicable (35,50). Our survey found that the more accurate a palliative care physician believes AI is, the less concern exists about explainability. Overall, our findings lend more support to the idea that accuracy matters more to physician users than explainability in this context.

Counter to prior work (37,47,53,58), attitudes toward black box explainability did not appear to associate with AI knowledge in our study. But we found that physicians who identified as female in our study were more concerned about explainability. We are cautious not to over-interpret this finding. However, other studies have found sex or gender-based differences in technology use. In general, men appear more positive about AI than women (59,60). A survey of physicians in Portugal found that men were significantly more likely than women to see advantages to AI use and believe that AI improves healthcare quality (58). Given prior research suggesting female physicians engage in more patient-centered communication than male physicians (61), our finding around the importance of communication in AI and palliative care offers a potential explanation for the gender differences we observed. More research is needed before drawing firm conclusions on this point.

4.3. Implications:

Our study has implications for the design and implementation of AI-based mortality prediction. First, although we observed that accuracy can be traded off against explainability, our findings overall suggest explainability is important, especially in palliative care where communication is paramount. Data transparency was still a common theme among interviews; moreover, our participants were generally less interested in understanding all the inner workings of an AI model; instead, they wanted to understand why the model might generate a specific prediction for them. Thus, when considering implementation, hospitals and health systems should prefer models that are capable of providing this insight. For example, models that incorporate feature importance techniques (62) can highlight which elements of a patient’s health record contributed to the prediction. Moving forward, developing models that are both interpretable (i.e., how a model results in an output, in general,) and explainable (i.e., why a model resulted in an output for this particular patient) are both important to support patient-centered care (63). The supposed inherent trade-off between model accuracy and explainability has been recently questioned (64).

Second, the fact that palliative care teams and patients/caregivers shared some reasons in common (e.g., data transparency, accuracy, AI not used in isolation) and some that were different (e.g., less emphasis on communication or biases among patients/caregivers) can help finding common ground between patients, families, and clinicians for managing these issues. For instance, clinicians may need to explain why bias is a concern even for individual patients, who need to know how a model may apply to them. Patients and families could also benefit from understanding how the explainability of a model affects a clinician’s willingness and responsibility in using it, or how to incorporate its use in a way that respects autonomy and choice.

4.4. Strengths and Limitations:

Our study has several strengths. First, we conducted a multi-site study, drawing our qualitative sample from geographically distinct settings and including participants from a broad range of palliative care stakeholder groups. Our qualitative results thus feature a greater diversity of perspectives than is often the case in qualitative research. Second, we conducted the first national survey of palliative care physicians related to their perspectives on black box AI-based prognostication. Additionally, by relying on IQVIA’s OneKey database, we were able to target physicians who are actively practicing for our sample. Our study also has limitations. Our interview sample included only four health systems, each with familiarity using AI, so our findings may not generalize to other health systems. Due to the nature of qualitative methods, we cannot draw any conclusions about the prevalence of concern about explainability in non-physician populations, nor the prevalence of the themes expressed on the topic, especially because we analyzed comments related to explainability from a larger study; a future study focusing only on explainability might be useful. Our survey yielded a low response rate and thus may exhibit response bias. For example, it is possible that respondents are more concerned about AI than non-respondents suggesting true population-level concern is lower than in our sample. In addition, we did not survey care team members other than physicians; nurses, social workers, psychologists, and spiritual care providers frequently engage in deep conversations with patients and families. These clinicians might be more concerned about explainability as a result – or less, because of differences in how specific patient care responsibilities intersect with AI outputs or the level of responsibility they feel for using them. However, we are limited in our ability to compare the different participant roles because we only completed a survey of physician stakeholders. Future work can quantitatively compare the groups and consider the direct impact of these themes on different aspects of explainability.

5. CONCLUSION

Understanding real-world perspectives toward AI-based prognostication or mortality prediction in palliative care is critical to ethical implementation. Although palliative care team members and patients or caregivers appeared willing to give up explainability for the sake of accuracy, given the importance of communication in palliative care, this should not be done lightly. Future research should examine, in more detail, the types of explanations patients, families, and care team members find most helpful for patient-centered care.

Key findings

• Interviews with patients, caregivers, and palliative care team members at four geographically diverse academic health systems in the United States found a diversity of views about black box artificial intelligence (AI) algorithms when used for mortality prediction. Overall, care team members expressed more concern about black box algorithms.

• In a national survey of palliative care physicians, the vast majority (75%) were at least moderately concerned with black box algorithms used in prognostication. Concern was greater among those who perceived AI to be less accurate, and among physicians who were female.

What is known and what is new?

• Prior research has suggested that physicians and the public may desire explainability in AI-based systems in order to support widespread use. However, this perception has not been fully examined in the setting of palliative care or AI-based mortality prediction.

• This manuscript adds rich contextual data from semi-structured interviews and national-level estimates from practicing physicians, providing insights to support practice and policy improvement and shape technology development.

What is the implication, and what should change now?

• Although palliative care team members and patients or caregivers may be willing to trade some level of explainability, in the unique setting of palliative care – where communication is especially valuable – explainability of AI models used may be particularly important. Technology developers should continue to pursue explainability in parallel with accuracy.

Abbreviation List

AI

artificial intelligence

EHR

electronic health record

POC

person of color

US

United States