Estimating ICU benefit: a randomized study of physicians

Abstract

Objective:

The distinction between overuse and appropriate use of the intensive care unit (ICU) hinges on whether a patient would benefit from ICU care. We sought to test (a) whether physicians agree about which types of patients benefit from ICU care and (b) whether estimates of ICU benefit are influenced by factors unrelated to severity of illness.

Design:

Randomized study

Setting:

Online vignettes

Subjects:

U.S. critical care physicians

Interventions:

Physicians were provided with eight vignettes of hypothetical patients. Each vignette had a single patient or hospital factor randomized across participants (four factors related and four unrelated to severity of illness).

Measurements and Main Results:

The primary outcome was the estimate of ICU benefit, assessed with a four-point Likert-type scale. In total, 1,223 of 8,792 physicians volunteered to participate (14% recruitment rate). Physician agreement of ICU benefit was poor (mean intraclass correlation coefficient for each vignette: 0.06, range: 0–0.18). There were no vignettes in which more than two-thirds of physicians agreed about the extent to which a patient would benefit from ICU care. Increasing severity of illness resulted in greater estimated benefit of ICU care. Among factors unrelated to severity of illness, physicians felt ICU care was more beneficial when told one ICU bed was available than if ICU bed availability was unmentioned. Physicians felt ICU care was less beneficial when family was present than when family presence was unmentioned. The patient’s age, but not race/ethnicity, also impacted estimates of ICU benefit.

Conclusions:

Estimates of ICU benefit are widely dissimilar and influenced by factors unrelated to severity of illness, potentially resulting in inconsistent allocation of ICU care.

Introduction

For many patients, care in the intensive care unit (ICU) can be life-saving.(1) For others, ICU care provides no added benefit and could result in harm.(2–4) For these patients, the abundance of intensive care beds in the U.S. is considered a major driver of ICU overuse.(5) Reducing the number of ICU beds nationally could force clinicians to more thoughtfully utilize intensive care.(6, 7) Yet, to safely succeed, clinicians would need to consistently identify patients who would benefit from ICU care. Otherwise, limiting access may unintentionally cause harm by depriving patients who would benefit.

The variation in ICU admission rates across hospitals raises the possibility there is little consensus regarding which patients should receive ICU care but is confounded by unmeasured heterogeneity among patients.(8) Observational data also suggest clinicians may be influenced in their use of the ICU by factors unrelated to severity of illness, such as ICU bed availability.(9) Since there is no objective evidence available to estimate ICU benefit,(10) we proposed critical, unanswered questions are (a) whether physicians agree about which patients benefit from ICU admission and (b) whether such decisions are influenced by information unrelated to severity of illness.

We chose to interrogate these questions using randomized clinical vignettes rather than observational data. Our study design with experimental manipulation has been shown to simulate clinical behavior(11) and permits causal interpretation of factors directly impacting the estimates of ICU benefit that cannot be made using observational data. While critical care guidelines recommend incorporating a patient’s potential to benefit into ICU admission decisions, there are currently no objective means to estimate ICU benefit.(10) As a result, we hypothesized there would be poor consensus and factors unrelated to severity of illness would affect estimates of ICU benefit.

Materials and Methods

Participants

We recruited volunteers from the membership roll of the Society of Critical Care Medicine (SCCM). We limited volunteers to U.S. physicians who were SCCM members in August 2017.

Instrument

Eight vignettes of hypothetical patients with pneumonia were developed (Appendix). Each vignette had a single patient or hospital factor randomized. Factors and their levels were selected based on a review of the literature, our clinical experience, and feedback from cognitive interviews of clinicians. Four patient factors related to severity of illness were selected (respiratory rate, oxygen requirement, blood pressure, mental status). Participants were randomized to receive four separate vignettes with a patient with: 1) a respiratory rate of 12, 18, 24, or 30; 2) an oxygen requirement of four liters per minute (LPM), six LPM, 50% via facemask, or 100% via facemask; 3) a blood pressure of 122/78, 105/74, 94/57, or 80/47; and 4) no confusion or confusion.

Four factors unrelated to severity of illness were selected (number of available ICU beds, presence of the patient’s family by the bedside, patient’s age, patient’s race/ethnicity). Participants were randomized to receive four separate vignettes with: 1) no mention of the number of available ICU beds, one available ICU bed, or five available ICU beds; 2) no mention of whether the patient’s family was present, the patient’s wife being present at the bedside, the patient’s wife being present at the bedside and crying, or the patient’s family being present at the bedside and crying; 3) a patient with an age of 25, 45, 65, or 85 years; and 4) no mention of the patient’s race/ethnicity or a patient with a race/ethnicity of White, Black, or Arab.

Vignettes included history of present illness, vital signs, physical examination, laboratory values, and chest x-ray image. Vignettes were designed to avoid patients with clear indications for ICU admission, such as receiving mechanical ventilation or vasopressor support.

Each vignette was followed by two questions: 1) “Would this patient receive the most benefit from admission to the general ward or the ICU?” (primary outcome); and 2) “How difficult was this decision for you?” (secondary outcome). Responses used four-point Likert-type scales ranging from “Definitely general ward” to “Definitely ICU” (for ICU benefit) and from “Not at all difficult” to “Very difficult” (for difficulty).

Vignettes were pilot tested with the University of Michigan Multidisciplinary Intensive Care Research Workgroup, a group of critical care scientists, and the University of Michigan Center for Bioethics and Social Sciences in Medicine Working Group, a multidisciplinary group of survey experts. The vignettes were cognitively tested with five critical care clinicians.

Vignette administration

A link to the vignettes was sent via e-mail from SCCM. The vignettes were administered online using Qualtrics (Qualtrics, Provo, UT), a survey development platform,(12) and fielded in August 2017. Participation was voluntary, and participants were offered a $5 gift card after completion. A single reminder e-mail was sent to all U.S. SCCM physicians two weeks after the first invitation.

Participants were randomized to vignettes after agreeing to take part in the study, similar to a clinical trial that requires consent prior to randomization. Vignette order, and the selected characteristics within each vignette, were independently randomized to prevent any systematic order-of-administration effects.

Analysis

SCCM provided the age, gender, and race/ethnicity of all individuals within the full membership roll. The specific characteristics of non-participants were not available. We compared the characteristics of participants to characteristics of the full membership roll using chi-square or t tests.

The consensus among physicians who received the same vignette was assessed using a one-way random effects intraclass correlation coefficient (ICC) model.(13, 14) The ICC represents the degree to which an individual physician’s estimate differs from the mean score for all physicians who received the same vignette, with agreement rated as poor (0.01–0.39), fair (0.40–0.59), good (0.60–0.74), or excellent (0.75–1.00).(15)

The primary outcome variable was the estimate of ICU benefit. The difficulty in assessing ICU benefit was evaluated as a secondary outcome. Responses were dichotomized for analysis. Each factor was analyzed separately. Logistic regression was used to evaluate the effect of the randomized factor on each outcome. Absolute rates for each outcome were estimated using predictive margins.

As a sensitivity analysis to assess whether any identified variation might be influenced by physician experience, we evaluated ICCs based on whether or not physicians reported they commonly cared for patients similar to those in the vignettes.

This research was deemed exempt from review by the Institutional Review Board for the University of Michigan (HUM00129113). Data management and analysis were performed using Stata 14.2 (StataCorp, College Station, TX). All tests were two-sided with P values less than 0.05 considered significant.

Results

Out of 8,792 U.S. SCCM physicians e-mailed, 1,223 physicians volunteered to participate (14% recruitment rate). All eight vignettes were completed by 913 physicians (75% completion rate). The median time to complete the vignettes was seven minutes (interquartile range 5–11 minutes). The average age of participants was 42 years, and most participants were male (65%) and White (61%) (Table 1). Participants were broadly representative of the full membership roll, though modest differences in age, gender, and race/ethnicity were noted (Appendix Table 1). After randomization, there were no significant differences among participants (Appendix Table 2).

Table 1:

Characteristics of participants^a

Characteristics	Participants
Number	859
Age, mean (SD)	42 (12)
25–35	37.8%
36–50	38.3%
51–80	23.9%
Gender
Male	64.6%
Female	35.2%
Other	0.2%
Race/ethnicity
White	61.0%
Black	2.6%
Other	36.4%
Years in practice
0–4	38.3%
5–10	15.1%
11+	46.6%
Practicing specialty
Pulmonary/critical care medicine	37.2%
Internal medicine	17.3%
Surgery	12.0%
Anesthesiology	9.4%
Other	24.1%
Clinical environment
Private practice	26.9%
Academic	73.1%
Hospital type
Community	33.3%
Academic	66.7%
ICU type
Mixed	39.0%
Medical	24.4%
Surgical	14.5%
Other	22.1%
ICU beds
0–20	28.1%
21–50	35.6%
51+	36.3%
Geographic region
Northeast	21.6%
Midwest	23.0%
South	37.0%
West	18.4%

^a859 of 1,223 participants (70%) completed the demographics section of the survey

Among physicians reviewing the same exact patient vignette, there was poor consensus. The mean ICC was 0.06 (standard deviation (SD) 0.08, range 0–0.18). For example, when considering an 80-year-old female with pneumonia and a respiratory rate of 30, who requires six LPM of supplemental oxygen, 18% of physicians felt the patient would definitely benefit from ICU care, whereas 17% of physicians with the same case felt the patient would definitely benefit from general ward care (Figure 1). At most, only 69% of physicians agreed about the extent to which a patient would benefit from ICU care (in this case, agreeing the patient with a blood pressure of 80/47 would definitely benefit from ICU care) (Appendix Table 3).

Figure 1: Agreement in estimated ICU benefit among physicians who received the same patient vignette.

Among the 231 physicians who received the same exact vignette describing an 80-year-old female with pneumonia, who has a respiratory rate of 30 and requires six liters per minute of supplemental oxygen, there was poor consensus about whether the patient would benefit from ICU or general ward admission. Appendix Table 3 displays the agreement between participants for each vignette.

Increasing respiratory rate or oxygen requirement, as well as decreasing blood pressure, resulted in greater estimated benefit of ICU care (Figure 2 or Appendix Table 4). Patients with confusion were felt to benefit from ICU care more than patients without confusion.

Figure 2: Effect of patient and hospital factors on estimates of ICU benefit.

For the four vignettes in which the factor was related to severity of illness (left column), increasing severity of illness resulted in greater estimated ICU benefit. The right column shows results for the four factors unrelated to severity of illness. ICU bed availability, family presence, and age each affected estimated ICU benefit. Race/ethnicity had no effect on estimated ICU benefit.

Physicians felt ICU care was more beneficial if they were told one ICU bed was available than if ICU bed availability was not mentioned (absolute increase in ICU benefit: 7.3%; 95% confidence interval (CI): 1.5, 13.1). There was no difference in estimated ICU benefit if physicians were told five ICU beds were available compared to if ICU bed availability was not mentioned (absolute increase: 5.3%; 95% CI: −0.3, 11.0), although the confidence intervals do not exclude the same effect as when one bed is mentioned.

Physicians felt ICU care was less beneficial if family was present than if family presence was not mentioned (absolute decrease in ICU benefit when the patient’s wife was present: 9.9%; 95% CI: −17.0, −2.8). There was no difference in estimated ICU benefit if the patient’s wife or family were crying at the bedside compared to if family presence was not mentioned, though the point estimates suggested decreased benefit to ICU care.

Older patients were felt to benefit from the ICU more than younger patients (absolute increase in ICU benefit for an 85-year-old patient compared to a 25-year-old patient: 8.6%; 95% CI: 0.3, 17.0). Race/ethnicity had no significant effect on estimates of ICU benefit, with point estimates all close to the null.

Physicians reported it was easier to estimate ICU benefit when the patient was hypotensive (absolute decrease in difficulty: 11.1%; 95% CI: −18.0, −4.1). However, it was more difficult for physicians to estimate ICU benefit when the patient was confused (absolute increase in difficulty: 8.2%; 95% CI: 1.4, 15.1) or when the patient’s wife was crying at the bedside (absolute increase in difficulty compared to when family presence was not mentioned: 7.0%; 95% CI: 0.2, 13.8) (Appendix Table 5).

In a sensitivity analysis, we assessed whether the identified variation was influenced by physician experience. Most physicians in our sample (63%) reported that they often cared for patients similar to those in the vignettes. ICCs, stratified by physician experience, both demonstrated poor consensus.

Discussion

One distinction between overuse and appropriate use of ICU care is whether a clinician reasonably believes the patient would benefit from ICU admission. Yet, in this study, we demonstrated (a) there is poor consensus among U.S. physicians about which types of patients benefit from ICU admission and (b) decisions to use the ICU may be affected by factors unrelated to a patient’s severity of illness.

The U.S. is unique, in that its number of ICU beds per hospital bed far outpaces other similar nations.(6) In fact, many have argued this oversupply of critical care in the U.S. leads to overuse and low value care.(5) As a result, one proposed strategy to reduce ICU overuse is to decrease the number of ICU beds nationally.(7) Tightening supply would work safely if clinicians consistently identified which patients benefitted from ICU care, thus allowing patients who would not receive added benefit from the ICU to be triaged to lower-intensity care. However, as recognized in over three decades of ICU admission guidelines,(10, 16, 17) no objective evidence exists to guide clinicians in establishing ICU benefit. Our study contributes to the literature by demonstrating there is also no professional consensus about ICU benefit—indicating that broadly reducing the availability of ICU care may successfully reduce overuse but may also unintentionally limit access to patients who would benefit from intensive care.

This study also suggests ICU admission decision-making may be influenced by factors unrelated to whether a patient may benefit from ICU care. We had hypothesized the presence of family members at the bedside would result in greater estimated ICU benefit. Surprisingly, we found physicians felt the ICU was less beneficial when the patient’s family was by the bedside. Physicians also had more difficulty estimating ICU benefit when family was present than when family presence was not mentioned. There may be two explanations for these findings. One, clinicians may believe having family at the bedside could provide an additional patient care resource—an extra layer of monitoring in case the patient’s condition deteriorates. Two, critical care physicians may be negatively influenced by the presence of family members at the bedside, resulting in a bias against ICU admission. The effect of family members at the bedside on clinical decision-making should be further evaluated.

Observational studies have previously demonstrated patients are less likely to be admitted to the ICU when fewer ICU beds are available.(9) Our study found, however, physicians felt the ICU was more beneficial when one ICU bed was available compared to when ICU bed availability was not mentioned. It is possible that, rather than prompting the scarcity of ICU beds, as intended, informing physicians an ICU bed was available may instead have served as a reminder of availability. Whether informing physicians of the number of available ICU beds could affect ICU use or patient outcomes, for better or worse, remains unclear.

Some may argue age is, at minimum, indirectly related to severity of illness. Yet, the role age should play on ICU admission is unknown. In most observational studies as well as in a similar vignette survey of Swiss physicians, age was negatively associated with ICU admission.(18, 19) However, physicians in our study felt increasing age was associated with greater likelihood of ICU benefit, suggesting clinicians recognize the ICU may be particularly beneficial to the elderly, despite the elderly being less likely to receive ICU care in clinical practice.

Our study found the patient’s race or ethnicity had no effect on estimated potential to benefit from ICU care. Most prior studies have similarly found no association between a patient’s race/ethnicity and likelihood of ICU admission,(20–22) despite the pervasive effects of race/ethnicity in other aspects of American medicine.(23)

Some may question whether hypothetical scenarios can mimic actual practice. While a noted limitation, physicians in this study responded as expected to factors related to severity of illness, which acted as “positive controls,” suggesting the vignette prompts were effective, the participants were attentive, and the participants were responding as they would in actual practice. Since patient characteristics cannot be readily randomized in real life, the randomized vignette approach provides high quality causal evidence relative to other approaches and has been shown to closely correspond with actual behavior.(11, 24)

Furthermore, we specifically asked participants, “Would this patient benefit from ICU admission?” rather than “Would you admit this patient to the ICU?” for two reasons. First, critical care guidelines recommend clinicians primarily use this concept of “ICU benefit” when making ICU admission decisions.(25) Second, we sought to minimize the impact of organizational constraints, such as ICU capacity, that might affect ICU admission practices but should not theoretically affect estimates of ICU benefit.

This study should be interpreted in the context of certain limitations. First, the recruitment rate was 14%, which is consistent with other surveys using the SCCM membership roll.(26, 27) The SCCM administration system could not conduct subsamples. Thus, the vignettes were sent to all 8,792 U.S. SCCM physicians, making unconditional, larger incentives impractical.(28) In addition, the SCCM system could send only one reminder e-mail, preventing targeted follow-up to reduce non-response.

Anticipating this recruitment rate, we took steps to mitigate response bias. The study design utilized randomization that occurred after physicians agreed to participate in the study, analogous to what is commonly seen in a randomized clinical trial. This, in combination with our high completion rate (75%), lessens the risk of response bias. Thus, while a recruitment rate of 14% is low on its face, randomization preserves internal validity in the face of non-response. To assess the threat to external validity from non-response, we compared participant characteristics to characteristics of the full sampling frame—U.S. SCCM physicians. Our participants were slightly more likely to be younger, female, or white, compared to the population of SCCM physicians. When considering the broader population of U.S. critical care physicians, our sample may have other differences, which may suggest participants had less experience caring for the patients described in our study. For example, one-third of our sample had been in practice for four years or less, and one-fifth practiced in surgery or anesthesia. These differences could increase the variation reflected in this study. In a sensitivity analysis assessing consensus based on experience caring for patients similar to those presented in our study, both experienced and inexperienced physicians demonstrated a similar lack of consensus.

Second, our vignettes did not include options for a neutral response or for intermediate care, where some clinicians may have chosen to admit these patients. However, we excluded intermediate care as an option because there is no uniform definition of intermediate care in the U.S.(29) Third, we asked participants to provide a qualitative estimate of ICU benefit on a Likert-type scale rather than a probability estimate. A probability estimate may have created the appearance of more precise responses; however, prior work has demonstrated clinicians’ inability to accurately prognosticate.(30) Finally, we intentionally selected cases which we believed represented common scenarios but which we also believed there might be disagreement. Further work is needed to identify the population incidence of cases for which there is consensus or disagreement among clinicians.

These findings have implications for patients, clinicians, and health system leaders. ICU admission can save lives when targeted properly but can also subject patients to unnecessary harms when administered inappropriately.(1, 31) This study demonstrates clinicians may allocate this important treatment—intensive care—inconsistently, with broad implications for both the U.S. and the global community. While many feel ICU care is overused in the U.S. and underused abroad, this study suggests a crucial problem may also be clinicians do not consistently identify potential to benefit from ICU admission. Thus, guidelines recommending ICU admission decisions be based primarily on the potential to benefit from ICU care,(10, 16, 17) while well-intentioned, are insufficiently precise to promote appropriate use. In the face of this uncertainty, critically ill patients may be harmed by inconsistent ICU admission decision-making. There is a critical need for an empirical base of evidence identifying patients who benefit from ICU admission.

Conclusion

Clinical estimates of ICU benefit are widely dissimilar and are influenced by factors unrelated to a patient’s severity of illness, potentially resulting in inappropriate ICU use.