US and Dutch Perspectives on the Use of COVID-19 Clinical Prediction Models: Findings from a Qualitative Analysis

Abstract

Introduction

Clinical prediction models (CPMs) for coronavirus disease 2019 (COVID-19) may support clinical decision making, treatment, and communication. However, attitudes about using CPMs for COVID-19 decision making are unknown.

Methods

Online focus groups and interviews were conducted among health care providers, survivors of COVID-19, and surrogates (i.e., loved ones/surrogate decision makers) in the United States and the Netherlands. Semistructured questions explored experiences about clinical decision making in COVID-19 care and facilitators and barriers for implementing CPMs.

Results

In the United States, we conducted 4 online focus groups with 1) providers and 2) surrogates and survivors of COVID-19 between January 2021 and July 2021. In the Netherlands, we conducted 3 focus groups and 4 individual interviews with 1) providers and 2) surrogates and survivors of COVID-19 between May 2021 and July 2021. Providers expressed concern about CPM validity and the belief that patients may interpret CPM predictions as absolute. They described CPMs as potentially useful for resource allocation, triaging, education, and research. Several surrogates and people who had COVID-19 were not given prognostic estimates but believed this information would have supported and influenced their decision making. A limited number of participants felt the data would not have applied to them and that they or their loved ones may not have survived, as poor prognosis may have suggested withdrawal of treatment.

Conclusions

Many providers had reservations about using CPMs for people with COVID-19 due to concerns about CPM validity and patient-level interpretation of the outcome predictions. However, several people who survived COVID-19 and their surrogates indicated that they would have found this information useful for decision making. Therefore, information provision may be needed to improve provider-level comfort and patient and surrogate understanding of CPMs.

Highlights

• While clinical prediction models (CPMs) may provide an objective means of assessing COVID-19 prognosis, provider concerns about CPM validity and the interpretation of CPM predictions may limit their clinical use.

• Providers felt that CPMs may be most useful for resource allocation, triage, research, or educational purposes for COVID-19.

• Several survivors of COVID-19 and their surrogates felt that CPMs would have been informative and may have aided them in making COVID-19 treatment decisions, while others felt the data would not have applied to them.

People hospitalized with coronavirus disease 2019 (COVID-19) may require admission to an intensive care unit (ICU), possibly escalating to the need for invasive mechanical ventilation (MV). Individual preferences for ICU admission and/or MV are often influenced by concerns about poor outcomes including prolonged MV and subsequent mortality.^1,2 However, the COVID-19 pandemic has been widely characterized by high degrees of clinical uncertainty in terms of severity of symptoms, disease trajectories, and mortality for those contracting the virus. ³ In addition, variation in governmental public health responses among countries and surges in COVID-19 cases (“waves”) have been significant, and overall outcomes have varied both by geographic region and temporally with each wave. ⁴ Over the course of the pandemic, these factors have therefore exacerbated clinical uncertainty among health care providers, people with COVID-19, and their loved ones/surrogate decision makers (surrogates). This has resulted in difficulty predicting outcomes and subsequent treatment decisions, particularly for those people admitted to the hospital with COVID-19.

Clinical prediction models (CPMs) have the potential to support providers and people with COVID-19 and their surrogates in medical treatment decision making and communication about prognosis. Furthermore, given the continuous pressure on health care systems, CPMs may also support decision making in triaging people with COVID-19 in the emergency department (ED) for hospital or ICU admission and discharge. Since the start of the pandemic, several prognostic models have been developed to predict outcomes in people with COVID-19. However, almost all published models were identified as high risk of bias, indicating that their reported predictive performance is most likely overly optimistic. ⁵ Two CPMs that have been developed to predict outcomes in people suspected of COVID-19 are the Northwell COVID-19 Survival (NOCOS) ⁶ and the Erasmus Medical Center COVID Outcome Prediction in the Emergency Department (COPE) models. ⁷ The NOCOS model was developed using data from 13 New York City area hospitals, and the COPE model was developed based on data from 4 hospitals located throughout the Netherlands (Rotterdam, Zwolle, Eindhoven, Heerlen). Data from the models were based on first-wave data from people who presented to the ED with suspected COVID-19. These models were validated on second-wave data at the same sites and further validated against each other to determine their temporal and geographic transportability. Both are based on readily available predictors in the electronic health record.^8,9Figure 1 depicts the timeline of the development and validation of the 2 models.

Figure 1.

Timeline of the development and validation of the Northwell COVID-19 Survival (NOCOS) and COVID Outcome Prediction in the Emergency Department (COPE) models.

The NOCOS and COPE models provide risk predictions (e.g., risk of mortality expressed as a percentage given clinical characteristics of an individual patient) for people with COVID-19 based on a combination of clinical predictors. Specifically, the clinical predictors used in the NOCOS model are patient’s age, oxygen saturation (%), absolute neutrophil count (k/uL), red cell distribution width (%), serum sodium (mmol/L), and serum blood urea nitrogen (mg/dL) and are used to calculate the probability of hospital survival. The clinical predictors used in the COPE model are patient’s age, respiratory rate (per minute), lactate dehydrogenase (U/L), C-reactive protein level (mg/L), serum albumin (g/L), and serum urea (mmol/L) and are used to calculate mortality and ICU admission within 28 d. The models do not explicitly define treatment decisions or make treatment recommendations; that is, they are not clinical practice guidelines for defining either 1) mortality risk thresholds below which a person with COVID-19 can be sent home from the ED or 2) thresholds above which a person should be admitted to the ICU. Rather, they provide evidence-based information to assist in decision making. These models have the potential to support providers, people with COVID-19, and surrogates about decisions concerning hospital and ICU admission for COVID-19 and the use of MV. They also allow providers and health systems to define their own risk thresholds to guide decision making based on each health system’s most up-to-date protocols and available resources as well as the patient’s own goals of care. Figure 2 depicts the COVID-19 care pathways and intended uses of the models, namely, to support decisions regarding admission to the hospital and ICU or discharge.

Figure 2.

Domains addressed and themes identified in focus group interviews with United States–and Netherlands-based providers.

Before prognostic tools such as NOCOS and COPE can be implemented in clinical practice, we should understand end-user perceptions of CPMs, including those of health care providers, people with COVID-19, and their surrogates. It is important to develop an understanding of how these individuals navigate clinical uncertainty and understand probabilistic data used in CPMs during decision making. Earlier work on CPMs has primarily focused on difficulties in understanding prognosis in clinical practice. Such work has shown that there may be both layperson- and provider-level misunderstanding when interpreting probabilistic data generated from the CPMs, which can occur when data are presented in relative, as opposed to absolute, terms. For example, both Bodemar et al. ¹⁰ and Gigerenzer ¹¹ have shown that while relative and absolute risks are based on the same data, both providers and laypeople may interpret data in a more favorable light when it is presented as relative as opposed to absolute risk, which may ultimately affect decision making. In addition, there may also be provider-level discomfort when communicating the prognosis to patients and surrogates due to the high degrees of prognostic uncertainty inherent in applying probabilistic data generated from population-level outcomes to the individual patient. ¹²

While earlier work focused primarily on probability, expansion of this earlier work has shown that clinical uncertainty is often multidimensional. Work by Han et al. ¹³ led to the development of a taxonomy for categorizing different types of clinical uncertainty. This taxonomy highlights that uncertainty may result from what are broadly categorized as 1) the source (probability of a specific outcome, ambiguities related to the reliability and credibility of the sources from which the CPMs were developed, and complexities linked to myriad social contextual factors such as prior functional status, postdischarge access to care, or health system resources), 2) the issue (i.e., the contexts in which the uncertainty occurs, including scientific uncertainty [cause of disease, diagnosis, treatment, and prognoses], practical uncertainty [i.e., linked to the health system structures and processes], and personal uncertainty, which focuses on the impact of the illness on future well-being, quality of life, and personal relationships), and 3) the locus of the uncertainty (which may reside with the provider, the patient, or both).^13,14

Finally, a separate focus of existing work has explored CPM objectivity and the extent to which users perceive CPM data to be objective and based on a patient’s clinical characteristics. This work has examined how specific clinical variables are selected by the model’s developers while leaving out other variables (e.g., using or not using race or ethnicity as variables in a model and the extent to which this may exacerbate racial and ethnic inequalities) and whether CPMs have successfully been able to incorporate wider sociocontextual disparities affecting outcomes.^15–18

The primary objective of our qualitative study, therefore, was to better understand how the NOCOS and COPE CPMs may be implemented to support providers, people with COVID-19, and surrogates in making critical, patient-centered decisions in COVID-19 care by situating our study within the aforementioned bodies of work on clinical uncertainty; the communication of prognostic data among providers, patients, and surrogates; and CPM development and variable selection. To that end, we convened participants to gather information about what and how risk information was integrated in their prior COVID-19 treatment decision making and to obtain pointed feedback on the NOCOS and COPE CPMs for future use. Because of cultural and health care system differences between the United States (US) and the NL,^19,20 we sought to understand the perspectives of providers, people who were hospitalized with COVID-19, and surrogates in both countries.

The aims of our study were 2-fold:

• to understand specific experiences with COVID-19 decision making among our stakeholder groups—including estimating prognosis and information communicated for decision making, and

• to identify facilitators and barriers to implementation of CPMs in COVID-19 care, including perceptions and attitudes about the usefulness of CPMs for COVID-19 decision making.

Methods

Study Design

For this qualitative study, we conducted online focus groups and a limited number of one-on-one interviews among Dutch stakeholders who were unavailable for the focus groups. In both the US and the NL, the online focus groups were held separately for 1) health care providers and 2) people who had COVID-19 and surrogates. The semistructured focus group guides (Supplemental Appendices A–B) for this study were developed to facilitate a flexible conversational approach with open-ended probes focused on assessing the following:

1. Stakeholder experiences with COVID-19:

   1. Provider’s estimating prognosis, communication about prognosis, and decision making
   2. Information communicated to people with COVID-19 and surrogates about prognosis and decision making related to COVID-19 treatment seeking

2. Stakeholder attitudes and beliefs about the NOCOS and COPE CPMs

3. Facilitators and barriers for using the NOCOS and COPE CPMs in COVID-19 care

The study was approved by Research Ethics Committee of Erasmus Medical Center, Northwell Health’s Institutional Review Board (IRB) in the Feinstein Institutes for Medical Research (IRB No. 20-1017) and the Tufts Health Sciences Institutional Review Board (IRB No. 00001044). We followed the Consolidated Criteria for Reporting Qualitative Research guidelines in reporting this study (Supplemental Appendix C). The interviews took place between January 2021 and July 2021.

Participants and Inclusion Criteria

Health care providers in both countries were invited to take part if they had experience caring for people with COVID-19. In the US, people who had been hospitalized for COVID-19 and surrogates of people who had COVID-19 were invited to take part if they were older than 18 y and had proficiency in the language in which the focus groups were conducted (i.e., English or Spanish). In the NL, people who had visited the ED or had been admitted to the hospital (COVID-19 ward or ICU) for COVID-19 and surrogates of people who had COVID-19 were invited to take part if they were older than 18 y of age and had proficiency in the language in which the focus group or interview was conducted (i.e., Dutch). Surrogates could include a partner, parent, child, or other significant other (e.g., roommate). Among surrogates, death of the patient had to be at least 3 mo prior to enrolling in the study. All participants needed access to a device (e.g., laptop, computer, tablet) with a working camera and microphone, and they needed to provide informed consent to participate. The US-based focus groups invited only those who were hospitalized with COVID-19 or were relatives of hospitalized patients, while the NL-based focus groups invited individuals who either visited the ED (i.e., they were not admitted during their ED visit) or were hospitalized for COVID-19 (either in a COVID-19 ward or ICU). All other eligibility criteria as described above were similar across countries.

Recruitment

Health care providers were recruited through (clinical) collaborators from the study team at Tufts University Medical Center, Northwell Health, and Erasmus University Medical Center using purposive sampling of COVID-19 health care providers to capture a diverse array of clinical specialties (e.g., critical care physicians, pulmonologists, acute intensivists, geriatricians, nurses, and members of palliative care support teams such as clinicians, hospital chaplains). People who had COVID-19 and surrogates were recruited through online advertisements and hard-copy fliers placed in our pulmonary and respiratory clinics. Surrogates were further identified via our COVID-19 participants already enrolled in the study. Members of the study team contacted by telephone people who had COVID-19 and surrogates who responded to the advertisements and were interested in participating to further explain the study. If they agreed to participate, the research team sent an information letter and informed consent form, which were returned to the researcher by e-mail (US and NL) or by post (NL only). Individual subjects could withdraw from the study at any time without consequences and without having to state their reasons for withdrawal. Following local norms and common practice for compensation for research study participation, US-based participants received $200 for each session in which they participated (providers had the option of attending 2 sessions). All Dutch participants received €25, which was not disclosed beforehand.

Data Collection

Focus groups and interviews used a semistructured interview guide (Supplemental Appendix). All focus groups and interviews were conducted via Microsoft Teams or Zoom. In preparation for the online interviews, a brief online practice session was held with participants to test the internet connection and familiarize participants with the software that was used. Each session took approximately 1 h and was led by an experienced moderator. For the US-based focus groups, M.J.B. (a PhD-level medical anthropologist) and N.H. (a critical care pulmonologist) conducted the 2 provider focus groups and English-speaking patient/surrogate focus group. J.G.P.(a research coordinator and bilingual English/Spanish speaker) conducted the Spanish-speaking patient/surrogate focus group. A bilingual Spanish-/English-speaking interpreter was present to translate the discussion to the non–Spanish-speaking members of the study team. Additional members of the US and NL research teams were present at each of the sessions. For the NL-based focus groups, I.R.A.R.H. (a PhD candidate in public health with a focus on CPMs) and J.R. (a PhD-level associate professor with expertise in qualitative research) conducted all focus groups with H.F.L. present. All focus group moderators were female. All focus groups and interviews were audio-recorded (following introductions and verbal consent for recording) and later transcribed by a professional company with all identifiers removed to maintain confidentiality. Spanish and Dutch focus groups and interviews were also translated to English. In addition, members of the study team took notes during the focus groups. At the start of each session, the moderators introduced the study team including their academic and or clinical positions, and participants introduced themselves to the group. The moderators gave an overview on CPMs in general and provided details about the development of NOCOS and COPE specifically (including which predictors were included in the model and the prediction [estimate] that the model would give). Participants were shown static PowerPoint images of the 2 CPMs including the specific clinical data points used to calculate prognosis in both models as described above, and we explained that the models were developed based on data from multiple hospitals.

Data Analysis

Qualitative analysis involved an iterative coding process using open, axial, and selective coding, allowing an inductive exploration of themes and constructs. ²¹ First, the open and axial coding was done separately for the US- and NL-based focus groups and interviews. The US-based transcripts were coded independently by 3 researchers and the NL-based interviews by 2 researchers. Second, the codebooks were compared and similarities and differences between codes for the US- and NL-based focus groups and interviews were examined. Third, the codebooks were merged into 1 final codebook to allow for the selective coding process (Supplemental Appendix D). Results were reported separately for providers and people who had COVID-19 and surrogates. In addition, differences between the US and the NL participants were examined. All transcripts were hand coded (i.e., coding software was not used).

Results

Study Population

In the US, we conducted 4 online focus groups: 2 for health care providers (n = 9), 1 for English speakers who had been hospitalized for COVID-19 and surrogates (n = 12), and 1 for Spanish speakers who had been hospitalized for COVID-19 and surrogates (n = 12) between January 2021 and July 2021. In the NL, we conducted 3 focus groups and 4 individual interviews with health care providers (n = 6) and people who had visited the ED and were subsequently either discharged or admitted to the hospital for COVID-19 and surrogates (n = 9) between May 2021 and July 2021 (3 Dutch provider-participants and 1 Dutch surrogate who were recruited but unable to participate in the scheduled focus groups took part in one-on-one interviews using the focus group guides). Participant demographics are presented in Table 1.

Table 1.

Participant Demographics

Characteristic	n (%)
	US	NL
People with COVID-19	n = 5	n = 5
Sex
Male	2 (40)	2 (40)
Female	3 (60)	3 (60)
Other	0	0
Race/ethnicity a
White	2 (40)	5 (100)
Black	2 (40)	0
Hispanic	2 (40)	0
Asian	0	0
Other	0	0
Language of focus group
English	3 (60)	0
Spanish	2 (40)	0
Dutch	0	5 (100)
Admitted to ICU for COVID-19	5 (100)	1 (20)
Experienced mechanical ventilation	3 (60)	1 (20)
Caregivers of people with COVID-19	n = 7	n = 4
Relationship to individual with COVID-19
Parent	1 (15)	0
Partner or spouse	0	3 (75)
Other relative	5 (72)	1 (25)
Other nonrelative	1 (15)	0
Sex
Male	2 (29)	2 (50)
Female	5 (75)	2 (50)
Other	0	0
Race/ethnicity a
White	3 (43)	N/A b
Black	1 (15)
Hispanic	2 (29)
Asian	1 (15)
Other	1 (15)
Declined response	1 (15)
Language of focus group
English	5 (72)	0
Spanish	2 (29)	0
Dutch	0	4 (100)
Cares for someone admitted to ICU for COVID-19	7 (100)	1 (25)
Cares for someone admitted who experienced MV for COVID-19	4 (57)	0
Providers by clinical specialty a	n = 9	n = 6
Geriatric hospitalists	2 (22)	0
ED physicians and surgical/critical care physicians/ICU intensivists	1 (11)	0
Internal medicine	1 (11)	0
Pulmonary and critical care physicians	6 (67)	1 (16.6)
Palliative care physicians	3 (34)	0
Internal medicine physician trainees	1 (11)	1 (17)
Emergency department physician in training	0	1 (17)
Physicians	0	1 (17)
Nursing home physicians	0	1 (17)
ICU nurse and senior researcher	0	1 (17)
Pastoral care/chaplains	1 (11)	0

ICU, intensive care unit; MV, mechanical ventilation; NL, Netherlands; US, United States.

^aSome US respondents selected/identified with multiple categories for race/ethnicity and clinical specialty.

^bData on race/ethnicity were not collected from the NL participants.

Summary of Results

US and Dutch providers

Overall, we found greater similarities than differences among providers in the 2 countries (Table 2 shows the results of the qualitative analysis from the provider responses). In the first wave, similarities noted among providers were the high degrees of clinical uncertainty for outcomes and a sense of “groping in the dark.” In the initial absence of COVID-19–specific outcomes data, providers prognosticated based on intuition (“gut feelings”), gestalt, “laying eyes,” objective real-time data such as vital signs and the course of illness of the patient (i.e., rapidity of deterioration, days of infection, and severity of illness), and the patients’ comorbidities and prior functional status. Providers used descriptions and general estimates to communicate prognosis to patients and families rather than numbers (e.g., “things look bad” or “the odds are not good”). As stated by one Dutch provider, “I never use numbers or whatever, because if I’m worried then, indeed, I’ll just say that ‘I am worried’ about the patient.” In subsequent waves, clinical uncertainty was reduced as patterns in outcomes began to emerge.

Table 2.

Domains Addressed and Themes Identified in Focus Group Interviews with US- and NL-Based Providers

Domain Addressed	Themes	Exemplary Quotes
Experiences with estimating prognosis and clinical decision-making for COVID-19	• Focus on clinical factors, gut feeling, and gestalt • No use of CPMs • Uncertainty	“We don’t use a decision model. But of course, we know what factors are favorable and unfavorable [for outcome in COVID-19 patients].” (Dutch provider) “I know that in that first wave we were really inventing the wheel” (Dutch provider)
Communicating about COVID-19 prognosis	• Prognosis in words instead of numbers • Short-term prognosis instead of long-term prognosis • Difficult as surrogates could not be present	“I would group their family member into a category and tell them that, you know, we’re doing our best and based on these vital signs and the lab data it’s looking like it’s going this direction.” (US provider) “Then we want to say we think the chances are very bad if you go to the ICU, we advise you not to go there. We speak more in terms of poor odds than percentages.”(Dutch provider)
Possible use of CPMs	Bedside use: • Communication with patients/surrogates • Decision to admit patient to ward or ICU • Consensus building among providers Nonbedside use: • Education/training • Risk stratification • Resource allocation • Triaging	Bedside use : “If there’s something more standardized, you know, then it helps everybody sort of speak that same language. Because I think that confusing messages between physicians can be very difficult for families.” (US provider) Nonbedside use: “The Department of Public Health, or whoever, if they would know ahead of the time that there are these 100 patients around in the hospitals who are not in the ICU, but they are at a higher risk, they could have looked at things differently, or moved patients around better.”(US provider) “For new doctors and the next generation it might be good.”(Dutch provider)
Attitudes toward CPMs	Positive: • Helpful • Supportive • Objective Negative: • Not relevant • Misleading • Not patient specific	Positive: “But precisely in that middle area where you may not be sure, then I think it’s very good to indeed have that based on a larger database of patient data, yes then it can actually be helpful in your own decision-making.” (Dutch provider) “I think that it would be an extra tool that we can get into our decision-making tree.” (US provider) “If I’m speaking to families, I think it does give a nice objective measure to use.” (US provider) Negative: “[Families may think] ‘Oh, this is a really valid piece of information.’ But what really does that information tell us?” (US provider) “It’s always hard for me to take a prognostic model from a big pool of people and apply it to the person I’m seeing.” (US provider) “Much of the work can’t just be captured in a score” (Dutch provider)
Facilitators for use of CPMs in practice	Acceptability: • External and temporal validation • Impact analysis Implementation: • Linked to treatment decision • Able to adapt daily based on changes in patient’s condition Application: • Accessible through website • Smartphone application • Electronic patient record • Information on how to interpret risk estimate	Acceptability: “It does give more confidence when you say it has been externally validated, including the second wave.” (Dutch provider) Application: “You could easily embed this into an EMR, or just have either a phone app, or just have an app on the computer, that you could plug the numbers in real quick.” (US provider)
Barriers for use of CPMs in practice	Provider level: • Limited knowledge about CPMs • Score fatigue • Difficulty interpreting risk estimates Model level: • Model incomplete (e.g., no comorbidities) • Outcomes are of less relevance (e.g., no ICU mortality)	Provider level: “The tricky thing is that many clinicians are not familiar with prediction models. That makes it difficult.” (Dutch provider) Model level: “Like I said the model is quite limited. . . . So in the model, maybe then it gets too complex, but I would put in risk factors like BMI. Does someone have an underlying comorbidity, recently had chemo, recent immune suppressive therapy, an organ transplant? I think you can do more with that.”(Dutch provider) “I always worry whether is this a case where something pops out statistically as correlating, but clinically is not relevant?”(US Provider)

BMI, body mass index; CPM, clinical prediction model; EMR, electronic medical record; ICU, intensive care unit; NL, Netherlands; US, United States.

Providers suggested that the NOCOS and COPE CPMs could be used at the bedside to build consensus among providers and in nonbedside contexts for triage and resource allocation. Positive feedback on the CPMs was that they appeared objective and that a tool to augment subjective decision making was needed. Negative feedback was that the CPMs were not patient specific and did not include variables such as patient baseline functional status or sociocultural contexts that might heavily influence outcomes. Perceived facilitators to using CPMs included enhancing provider acceptability by providing data on CPM validation and making the CPMs easily accessible within the health system software. Barriers to use were providers’ limited knowledge of prediction models (i.e., a provider-level barrier) and provider perceptions that the CPM outcomes used were not relevant to their decision making, for example, not including the likelihood of morbidity if a patient survived MV (i.e., model-level barrier).

The one key difference noted among providers regarded critical care decision points. Among Dutch providers, the most important decision point was in the ED and focused on whether to admit a person with COVID-19 to the ICU. Dutch providers explained that they had specific criteria for ICU eligibility and that, in some cases, a patient may be too sick to survive in the ICU. Here, decisions were typically made between providers and people with COVID-19 directly before an individual was admitted to the ICU and therefore before MV decision making. In the US, decisions about treatments such as MV would typically be decided among providers, people with COVID-19, and surrogates once the individual was already admitted to the ICU.

US and Dutch surrogates and people who had COVID-19

Participants in both countries described uncertainty regarding decisions about when to go to the hospital and when a high chance of relapse after discharge from the ED or hospital would be a concern (Table 3 shows the results of the qualitative analysis from the patient and surrogate responses). Respondents reported that prognosis was often not explicitly discussed, that providers tended to use words instead of numbers (supporting providers’ own descriptions of how they presented information about prognosis to their patients and surrogates), and that data were rarely referred to— possibly due to a lack of data at the start of the pandemic. References to media representation of COVID-19 came up in each of the focus groups. For example, participants in both countries described hearing about high mortality rates and poor outcomes for people with COVID-19 needing MV. This weighed heavily on their own perceptions of likely outcomes and decision making. For example, as one person who had COVID-19 requiring MV in the US/Spanish language focus group stated about his prior knowledge of MV, “The only thing I heard was that people who were intubated in New York were all dying, so my feelings about intubation were very bad and I refused intubation twice.”

Table 3.

Domains Addressed and Themes Identified in Focus Group Interviews with US- and NL-Based Surrogates and People Who Had COVID-19

Domain Addressed	Themes	Quotes
Experiences with clinical decision making in COVID-19	• Shared decision-making • Acute care decisions • Lay knowledge	Shared decision making: “So, when it came time to be intubated, we knew that my dad had already agreed that he wanted that. So, we did allow that, we had to allow that because that’s what his wishes were.” (US surrogate, English language) Acute care decisions: “I really had no choice myself, so I was brought to the [hospital] and there I was immediately put on the ventilator.” (Dutch patient) “[ICU admission] was not in consultation with us. It was immediate because it was necessary.” (Dutch surrogate) Lay knowledge: “At a certain point there was an intervention and it is called faith, and in my religion, that is called a miracle, and that is what happened.” (US surrogate, Spanish language)
Communication with providers about COVID-19 prognosis	• Communication with patient and/or caregivers • Prognosis often not explicitly discussed • In words instead of numbers	“There was never a discussion of probabilities or predictive variables. In my case, I certainly would have valued that.” (US patient, English speaking)
Attitudes toward CPMs	Positive: • Supportive • Preference for risk estimates is individual Negative: • Induce fear and anxiety • Incorrect/not patient specific • No added value	Positive: “I think it’s nice to be as transparent as possible. And especially if someone is intubated or ventilated and the patient themselves cannot communicate, you really only have the doctor and the nursing staff. So then you prefer to have all available information.”(Dutch surrogate) “I do think that [preference for risk estimation] is different from person to person. And as such a doctor will also have to sense that, I think, to whom they can say that and to whom not.”(Dutch patient) Negative: “Had I known any prediction models, they might have been a little scary and anxiety-provoking.”(US patient, English speaking) “I think that the problem is that I am the exception, they gave me very less chances of survival, then I think, according to me, if they had used with me that model, I would not be here.”(US patient, Spanish speaking) “I think at the time I was admitted, [risk estimations] wouldn’t have mattered to me at all. It would have added absolutely no value.” (Dutch patient)
Possible use of CPMs	• Guideline for conversations • To support prognosis in “words” • Explain and support treatment decisions	Support prognosis in words: “Yes, it is serious, but how serious, on a scale of one to ten. For me [providing a number], that is kind of pleasant, then I know how to look at it.” (Dutch patient) Explain and support treatment decisions: “That it [CPM] is kind of part of the decision of whether or not to admit someone. Particularly if there is also a ‘code black’ or emergency situation or that there are indeed too few beds. Or in the case of transfer for example that in that case it should be part of the decision.” (Dutch surrogate) “Sometimes I regret and I think wow, maybe if he [dad] had been with the mask for some more days, maybe he could have recovered, so yes, data and also some assurance with numbers, or any other thing, probabilities, a doctor saying what it is best in order to make a decision.” (US surrogate, Spanish language)
Facilitators for use of CPMs in practice	• Understandable language Real-world interpretation by clinician (i.e., pertaining to real life)	“Indeed, in normal understandable [language]. I always tell doctors that. I like to hear things I can understand. All that medical language and numbers, I didn’t study [medicine].” (Dutch patient)
Barriers for use of CPMs in practice	• Not being able to understand the risk estimates • Dangerous misunderstanding • Use of difficult language No added value	“I don’t know what would have happened If I had not been able to understand him [the doctor], I don’t know . . . a bad decision or a misunderstanding. I think that for these medical situations, language is extremely important. A language with less words but that has to be accurate. In a translation you cannot add information, misunderstand information.”(US surrogate, Spanish language)

CPM, clinical prediction model; ICU, intensive care unit; NL, Netherlands; US, United States.

Positive attitudes toward the use of CPMs included 1) that they could provide greater transparency to how the providers were making their decisions and recommendations (i.e., a numeric value to lend support for a provider’s recommendation) and 2) that it allowed people with COVID-19 to view their chances compared with others on a visual scale. Negative attitudes included concern that the CPMs may induce fear in the person with COVID-19 or surrogate and that they therefore need to be carefully communicated to select patients. In fact, one participant suggested that the provider should decide whether or not to share this information with the person depending on whether they felt that he or she could handle the information and associated fears: “I think it differs from person to person. I think a doctor will have to have a feel for who he can say that to and who he can’t.” Participants suggested using CPMs to explain decision recommendations and to help them visualize the information being communicated. Perceived facilitators to CPM use included ensuring that the content was easily understandable to reduce the chance for misunderstanding. Barriers included the user perception that the predictions may not be accurate, that is, that some people with COVID-19 may have a different outcome from what the model might estimate.

A key difference among our Dutch and US participants was that most Dutch people who had COVID-19 were not admitted to the ICU. Therefore, the US participants overall had experienced more severe COVID-19. However, among those US and Dutch participants admitted to the ICU, both had experiences in which providers needed to make immediate decisions about MV and did not involve surrogates or COVID-19 patients in decision making. For example, one Dutch participant who had COVID-19 stated, “I was brought to the hospital and immediately put on a ventilator. I was so sick, they didn’t even ask.” However, most Dutch participants described consulting their general practitioners (GPs) at the onset of symptoms, and GPs often made house calls to assess them. Typically, people with COVID-19 and their GPs would decide together that the person should then go to the ED where triage would take place. Individuals would be assessed and then either sent home (with oxygen if needed), to a rehabilitation center, a COVID-19 ward, or the ICU. Some Dutch surrogates discussed losing contact with family members once an ICU admission had occurred. While all participants mentioned hearing about COVID-19 in the media, among US participants, references to lay or folk knowledge such as religion were more frequent compared with Dutch participants.

Discussion

The use of CPMs in critical care settings has precedent. Many centers have mortality prediction scores (e.g., the APACHE score)^22,23 incorporated into the electronic health records. Whether these scores are used by frontline providers in the direct care of their patients to inform their decision making or in their communication with patients and surrogates is unclear. The largest study measuring the effect of providing calculated prognostic estimates to ICU clinicians, conducted 25 y ago, was the Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments (SUPPORT) study. ²⁴ It did not find that the provision of objective (calculator-derived) prognostic estimates to ICU clinicians altered the use of life-supporting technologies or was associated with improved communication or care of critically ill patients—mainly due to provider-level lack of use. Therefore, when engaging our stakeholders in the US and the NL, we aimed to better understand how CPMs might be implemented to support health care providers, people hospitalized with COVID-19, and surrogates in making medical decisions about COVID-19 care in the US and the NL. This includes determining reasons for potential hesitancy of using the COVID-19 CPMs in the ED, ICU, or other clinical settings. We also sought to better understand end-user perceptions about CPMs, how prognostic uncertainly was communicated within the context of COVID-19, and how CPMs may support future communication in these settings.

Prior work on CPM implementation in clinical practice shows multilevel- and multistakeholder-based factors (e.g., provider, patient, CPM, and health system) potentially affecting the use and usefulness of CPMs. For example, the nonadoption, abandonment, scale up, spread, and sustainability (NASSS) framework ²⁵ explains that the implementation of technological health innovations is influenced by factors related to 7 domains: the characteristics of the condition or illness (i.e., is it well known or well understood?), the technology, the value proposition (i.e., the extent to which users find it valuable or “worth using”), adopter system (i.e., the professional staff, patients, and surrogates who will be adopting the technology), the organization, the wider institutional and societal context, and the interactions among all of these domains over time. Taken together, these domains allow consideration of COVID-19 CPM implementation in a wider context, namely, as an illness that was initially not well understood, with an uncertain trajectory, further complicated by the fact that many stakeholders are unfamiliar with CPMs in general and may not see the value of the CPMs in facilitating decision making. Moskowitz et al. ²⁶ also explored reasons for nonadaption of a well-known prediction model in traumatic brain injury (the IMPACT model) and found that mistrust in underlying data and the belief that presenting numbers derived from statistical methods may mislead patients and relatives were the main barriers to implementation. ²⁶ These are consistent with our findings. Additional prior work that we found to be significant for our analysis includes Han et al.’s taxonomy of clinical uncertainty, which provides a language for describing the myriad clinical uncertainties found among our study participants, work on communicating clinical uncertainty among providers, patients and surrogates,^12,27 and studies exploring the impact of variable selection in CPM development.^15–18

In our study, we found that among participants in both countries, first-wave experiences were characterized by various types of clinical uncertainty that fit within the taxonomy of uncertainty. Providers described an inability to draw on prior anecdotal experiences or clinical outcomes in assessing prognosis for people with COVID-19, which represents source uncertainty specifically ambiguity due to the absence of credible data available at the time. Therefore, providers from both countries frequently drew on intuition or gut feelings based on direct observations of patients at the bedside and their clinical trajectories over time. This uncertainty was reduced in subsequent waves as patterns in outcomes began to emerge, although here, too, CPMs were not used. Rather, published outcome studies and personal experience were extrapolated to make prognostic estimates. Providers in both countries felt that CPMs could be useful in aiding decision making since the CPMs could provide a standardized way of communicating prognosis based on an objective set of criteria. However, both groups were wary of relying solely on CPMs due to the fact that the models are based on the general population and may not reflect the individual characteristics of the people with COVID-19 they were treating, which represents probabilistic uncertainty.^13,14 Some providers also pointed out that factors such as baseline functional status should be considered, and quality of life should be predicted in addition to survival. Furthermore, providers stressed that access to resources postdischarge may affect outcomes, both of which represents clinical uncertainty rooted in complexity linked to patient-level personal and social contextual factors.

Providers felt that CPMs might be useful in communicating with surrogates and people with COVID-19, perhaps used as part of a larger “toolset” to estimate prognosis and support clinical decisions and communication, instead of as a standalone tool as is typically recommended in prediction model studies. However, some providers were hesitant to use models due to concerns that people who had COVID-19 may misunderstand the data or may not understand that the numbers were only estimates and not guaranteed outcomes. For example, several providers worried that patients and families would place too much value on the number given. Some providers were also concerned that surrogates and people who had COVID-19 would find the numbers dehumanizing. While some survivors of COVID-19 in our study pointed out that CPM estimates would likely not have accurately captured their own outcomes (many were survivors of the most severe COVID-19 illness), we found that most people who had COVID-19 or surrogates did not question the accuracy or validity of the data itself. Overall, a greater acceptance of using CPMs compared with providers was notable among surrogates and people who had COVID-19. Many stated that they would have wanted to see or hear about data and estimates in numbers and would have found this information helpful when making treatment decisions. This supports concerns of our providers about how patients and surrogates understand prognostic data and how they might rely on it for decision making. Therefore, an important caveat to the use of CPMs was that the communication of prognosis to surrogates and people with COVID-19 should be tailored to accommodate health literacy levels, and as prior work shows, should include discussions about prognostic uncertainty inherent in CPMs. ¹²

Although we identified fewer barriers to use among surrogates and people who had COVID-19, and some stated the information might be scary or “anxiety provoking,” the main priority for people who had COVID-19 and surrogates in our study was that data be presented in a way that is easy to understand—both in terms of health literacy levels—and accommodating of their cultural and linguistic needs. Subsequent revisions to the CPM platforms can accommodate these suggestions by making the information available in multiple languages as well as providing training to providers for integrating information from the CPMs into their conversations with people who had COVID-19. In addition, providers must remain aware that in some cases, sharing CPM data may do more harm than good in some individuals who misinterpret data or have their fears amplified when seeing prognostic estimates.

Our focus groups and interviews allowed us to assess facilitators and barriers that may occur when implementing the CPMs in clinical practice. Most providers felt that the CPMs could be easily embedded in the electronic health record since both NOCOS and COPE are already available through websites and can be made available on an app, tablet, or desktop workstation. Overall, the biggest barrier among providers was a general hesitancy toward using the CPMs in the first place, due to their attitudes and beliefs about CPMs. As mentioned, providers in our study saw as it as a limitation that the models did not take into account comorbidities, an individual’s prior functional status, or access to resources upon discharge when assessing prognosis. However, because the NOCOS and COPE models were explicitly developed based on quickly and objectively obtainable predictors at presentation to the ED, preexisting comorbidities or prior functional status were not considered. To address these concerns, providing detailed information on how the models were developed and validated, including the data and variables included and excluded in the models, is vital to obtaining provider-level buy-in for using the CPMs. However, this also underscores the need to balance objective data from the CPMs with wider patient-level sociocontextual factors including race, socioeconomic status, and disabilities, which may be left out of CPMs but that indeed impact outcomes^15–18 and which may result in some individuals being denied care if CPMs are used to determine which patients should be allocated scarce resources. Ensuring clarity on which variables are included, while at the same time offering alternative ways of “contextualizing” the patient during clinical conversations, may be a way of further tailoring communication about prognostic uncertainty when using CPMs. It is also vital that providers address lay knowledge (e.g., spiritual beliefs, media representations) and local contexts (e.g., resource shortages) in addition to goals of care when discussing prognosis as these factors may also be considered by patient and surrogate when making clinical decisions. Finally, CPMs must be continuously updated and maintained to ensure ongoing validity as new data become available or when conditions “on the ground” evolve rapidly, as has been seen with COVID-19 over time. This may further alleviate provider-level concerns about using the CPMs.

Finally, while providers had some concerns about using the CPMs for individual-level prognosis and communication, their suggestions for nonbedside uses are useful. Specifically, using the CMPs to predict community-level or hospital-level surges would allow for more efficient distribution of resources and triaging of individuals to locations with greater bed availability. Currently, the NOCOS and COPE models do not provide prescriptive risk thresholds, meaning that they do not make treatment recommendations based on a given risk estimate. These risk thresholds would be influenced by the availability of resources and social norms. For example, whether to admit a person with COVID-19 to the ICU may also depend on hospital bed capacity and patient preferences for care. Our study suggests that, with careful consideration of recommendations for implementation, the risk predictions provided by NOCOS and COPE can support providers, people with COVID-19, and surrogates when making decisions about hospital or ICU admission.

Strengths and Limitations

Our study is one of the few that provides insight on considerations of potential users of CPMs in 2 different countries. We obtained multiple perspectives from participants with different experiences and from different settings. Our findings support prior work on clinical uncertainty and may have practical implications that can guide the development, validation, and implementation of CPMs for COVID-19 in the future. Furthermore, we believe that some of the facilitators (e.g., external validation) and barriers (e.g., limited knowledge of CPMs among providers) that were identified apply to CPMs in other fields.

Several limitations of our study should be considered. First, in the NL, we did not include stakeholders from non–Dutch-speaking communities, some of whom may have different concerns about the use of CPMs. Second, in our study, context was limited to personal experiences; we do not present the wider sociocultural contexts that may shape those experiences. The fact that the NL has national-level, universal health care whereas the US is characterized by a private and more fragmented health system is well known. While relevant to decision making and prognosis—particularly regarding access to care and the use of life-extending treatment—these considerations were beyond the scope of our study. However, it must be emphasized that while race/ethnicity was not a variable used in either NOCOS or COPE, racial inequalities are known to affect health outcomes. Third, differences in disease severity, lived experiences of people who had COVID-19, and surrogate participants may affect the perceptions of the models. For instance, overall, the Dutch patients suffered less severe COVID-19 illness, which in turn likely influenced their feedback. Lastly, the participants represent health systems in Boston, New York City, and Rotterdam—all metropolitan areas that do not reflect the health care experiences of areas with more rural and isolated populations. Future work may explore how CPMs may be combined with lay knowledge and gestalt to tailor shared decision-making conversations, allowing a balance between CPMs on one hand and other epistemological frameworks on the other. We may also explore the use of CPMs during clinical conversations to obtain first-hand accounts of CPM use and to measure the impact of actual use on decision making and clinical outcomes.

Conclusions

Despite differences in health care systems and national-level public health programs, we saw more similarities than differences between our US and Dutch stakeholders. While providers had reservations about using CPMs for people who had COVID-19 due to concerns about CPM validity and patient-level interpretation of the data, surrogates and people who had COVID-19 indicated that they would have found this information useful for decision making provided that the information is carefully and possibly selectively communicated. Future studies must develop and test high-quality strategies for communicating prognostic uncertainty and to enhance shared decision making using CPMs, and they must continue to measure acceptability for surrogate decision makers and people with COVID-19 via ongoing stakeholder engagement. This may help increase both provider-level comfort and patient and surrogate decision maker understanding of CPMs.