Abstract
Rationale: Breakdowns in clinician–family communication in intensive care units (ICUs) are common, yet there are no easily scaled interventions to prevent this problem.
Objectives: To assess the feasibility, usability, acceptability, and perceived effectiveness of a communication intervention that pairs proactive family meetings with an interactive, web-based tool to help surrogates prepare for clinician–family meetings.
Methods: We conducted a two-arm, single-blind, patient-level randomized trial comparing the Family Support Tool with enhanced usual care in two ICUs in a tertiary-care hospital. Eligible participants included surrogates of incapacitated patients judged by their physicians to have ≥40% risk of death or severe long-term functional impairment. The intervention group received unlimited tool access, with prompts to complete specific content upon enrollment and before two scheduled family meetings. Before family meetings, research staff shared with clinicians a one-page summary of surrogates’ main questions, prognostic expectations, beliefs about the patient’s values, and attitudes about goals of care. The comparator group received usual care enhanced with scheduled family meetings. Feasibility outcomes included the proportion of participants who accessed the tool before the first family meeting, mean number of logins, and average tool engagement time. We assessed tool usability with the System Usability Scale, assessed tool acceptability and perceived effectiveness with internally developed questionnaires, and assessed quality of communication and shared decision-making using the Quality of Communication questionnaire.
Results: Of 182 screened patients, 77 were eligible. We enrolled 52 (67.5%) patients and their primary surrogate. Ninety-six percent of intervention surrogates (24/25) accessed the tool before the first family meeting (mean engagement time, 62 min ± 27.7) and logged in 4.2 times (±2.1) on average throughout the hospitalization. Surrogates reported that the tool was highly usable (mean, 82.4/100), acceptable (mean, 4.5/5 ± 0.9), and effective (mean, 4.4/5 ± 0.2). Compared with the control group, surrogates who used the tool reported higher overall quality of communication (mean, 8.9/10 ± 1.6 vs. 8.0/10 ± 2.4) and higher quality in shared decision-making (mean, 8.7/10 ± 1.5 vs. 8.0/10 ± 2.4), but the difference did not reach statistical significance.
Conclusions: It is feasible to deploy an interactive web-based tool to support communication and shared decision-making for surrogates in ICUs. Surrogates and clinicians rated the tool as highly usable, acceptable, and effective.
Keywords: critical care, communication, decision-making, internet-based intervention, clinical trials
Surrogate decision makers are routinely asked to make major treatment decisions for their incapacitated loved one in the intensive care unit (ICU), including end-of-life decisions (1, 2). However, surrogates often struggle in this role and report feeling emotionally and psychologically overwhelmed (3–5). In addition, a large body of literature documents frequent communication breakdowns between clinicians and surrogates regarding patients’ prognoses, values, preferences, and treatment options (2–4, 6–12).
To our knowledge, no easily scalable, efficacious tool exists to help surrogates manage the emotional and cognitive stresses of surrogate decision-making in ICUs. A variety of clinician-delivered interventions have shown promise, including adding staff to the ICU team focused on supporting surrogates (3, 13), deploying a support intervention delivered by the interprofessional ICU team (14), and involving palliative care (15, 16) or ethics consultants (17, 18). However, each of these interventions may be challenging to implement broadly because of financial costs and operational complexity, including workforce shortages (19–22).
On the basis of input from key stakeholders (23, 24), we developed and refined the Family Support Tool, a scalable, interactive, web-based tool to support surrogate decision makers. Here, we describe the pilot randomized trial we conducted to assess the feasibility, usability, acceptability, and perceived effectiveness of the tool in an ICU setting.
Methods
Between December 2017 and May 2019, we conducted a two-arm, single-blind, patient-level, pilot randomized trial comparing the Family Support Tool with enhanced usual care (the standard amount of care in the enrolling ICU enhanced with scheduled family meetings). The study was conducted in two high-acuity ICUs (a 10-bed neuroscience ICU and a 20-bed medical ICU, both staffed by team-based intensivist coverage) at a tertiary-care hospital in Pittsburgh, PA. Research staff obtained informed consent from each patient’s legally authorized representative and the participating surrogate for study enrollment. Because patients were incapacitated at the time of enrollment, research staff monitored each enrolled patient’s decision-making capacity during their ICU admission using a combination of daily chart review and communication with the ICU team. If the patient regained decision-making capacity during their ICU admission, research staff obtained direct consent from the patient. All patients approached for direct consent agreed to continued participation. The Institutional Review Board of the University of Pittsburgh approved the project (PRO16050247). The study was registered on ClinicalTrials.gov (NCT02955563).
Research staff screened patients within 48 hours of study ICU admission by reviewing electronic health records and confirming patient eligibility with the ICU team during morning rounds. Eligible participants included surrogates and physicians of incapacitated patients aged 18 years and older who were judged by their physicians to have ≥40% risk of death or severe long-term functional impairment, which was elicited by asking the attending physician or their designee, “Does this patient have at least a 40% chance of in-hospital mortality or severe long-term functional impairment?” Severe long-term functional impairment was defined as needing assistance with at least two activities of daily living. Patients were excluded if they were on the bone marrow transplant service because the physician group refused to allow their patients to be approached for research participation.
Patients were randomized to a control or intervention arm by an Excel randomization module programmed by the study statistician. All study participants received a refurbished iPad (valued at $200) upon enrollment as compensation for study participation. Surrogates in the intervention arm received the iPad preloaded with the Family Support Tool, and control surrogates received the iPad without the tool. Intervention surrogates received individualized username and password-protected access to the Family Support Tool, and research staff-oriented surrogates to the tool.
Description of the Intervention
The intervention consists of the following three components: 1) surrogates’ completion of Family Support Tool modules before the first two family meetings, 2) provision to the ICU team of a one-page summary sheet containing surrogates’ responses to questions presented in the tool, and 3) scheduled family meetings, detailed below.
The Family Support Tool is an interactive website designed to be used on computers or tablets. As described in a previous publication (24), the tool was developed in collaboration with a committee of previous surrogates through a user-centered design process (23). The tool consists of brief family-centered videos, animations, and interactive questions designed to address the emotions surrogates in the ICU often experience and to prepare them for family meetings (screenshots of select content are available in the previous publication) (24). The content of the Family Support Tool (Figure 1) is conceptually grounded in the cognitive emotional decision-making framework (25), the Ottawa Decision Support Framework (26, 27, 28), and empirical research reporting the difficulties family members experience as surrogates in ICUs (29–35).
Figure 1.
Description of the Family Support Tool. ICU = intensive care unit.
We asked surrogates to complete specific modules of the Family Support Tool at the following three time points: 1) at study enrollment, 2) before the first family meeting, which was scheduled within 48 hours of study enrollment, and 3) before the second family meeting, which was scheduled within 5–9 days of study enrollment. Research staff monitored real-time reports to assess completion of assigned modules. If modules were not completed within 24 hours of enrollment, research staff reminded the surrogate to complete the modules and assessed for technical or external barriers. If modules were still not completed by the morning of the family meeting (family meetings were held in the afternoon), research staff reminded the surrogate to complete the modules and offered technical assistance. We used the same monitoring and support process for module completion before the second family meeting. Surrogates were not required to complete additional modules if they participated in subsequent weekly family meetings, but they were encouraged to review the tool.
Figure 1 illustrates the Family Support Tool content. The first section of the tool, viewed upon study enrollment, is designed to orient surrogates to the ICU, provide support, and provide self-care tips. The second section, viewed before each family meeting, contains content to help the surrogate prepare for family meetings. Surrogates were given unlimited access to the tool throughout the hospitalization and, because the tool is web-based, had the ability to access the tool from a device of their choosing (tablet, laptop, or desktop computer).
Before each family meeting, research staff shared with clinicians a one-page summary of the surrogate’s main responses to the Family Support Tool. The one-page summary sheet (Figure E1 in the online supplement) includes the surrogate’s understanding of the patient’s health status before the hospitalization, the patient’s interests, and the patient’s values and preferences, as well as the surrogate’s prognostic expectations, treatment leaning, and questions. Study staff printed the summary sheet for both surrogates and clinicians and quickly briefed the clinicians on the surrogate’s responses, taking care to highlight the surrogates participating in the family meeting, their questions, and any differences between the surrogates’ and the clinician’s prognostic expectations, which were provided by the clinician in a premeeting survey administered immediately before the summary sheet briefing.
Control Group
Surrogates in the comparator arm received usual care enhanced to receive family meetings within the same time frame as occurred in the intervention arm (i.e., the first within 48 h of study enrollment and the second within 5–9 d of study enrollment). Surrogates were encouraged to use their iPads as they wished.
Data Collection
Our main and secondary outcome measures are summarized in Table E1. We collected baseline demographic information about the surrogate and the patient upon study enrollment. Immediately before each family meeting, intervention surrogates completed questionnaires about tool usability and acceptability. Physicians completed a brief questionnaire about the patient’s medical condition and their prognostic expectations before each family meeting. After each family meeting, all surrogates completed a questionnaire about their experience communicating with the ICU team during the meeting as well as their understanding of the patient’s medical condition and their feelings of decision-making preparedness (33). Intervention surrogates completed additional questions about their perceived effectiveness of the tool in helping them prepare for the family meeting. A subset of intervention surrogates participated in a semistructured interview after completing the postmeeting questionnaire, which elicited feedback to inform future tool improvement. At the 3-month follow-up, blinded research staff conducted telephone interviews with surrogates to assess the quality of communication and shared decision-making.
Baseline Demographic Data
We collected patient-level information, including age, sex, race and ethnicity, religion, Acute Physiology and Chronic Health Evaluation II score, whether written advanced directives were completed before the hospitalization, and code status. Surrogate-level demographic information included age, sex, race and ethnicity, education level, religion, and relationship to the patient.
Feasibility
To assess the feasibility of tool use among intervention surrogates, we measured user engagement with tracking analytics built into the tool using Pendo. Measurements included the proportion of surrogates who accessed the tool before the first family meeting, the mean number of tool logins throughout the hospitalization, and the amount of tool engagement time. Among both intervention and control surrogates, we assessed the feasibility of study procedures using a multipronged measurement including the number of eligible participants during the screening period, the proportion who agreed to participate, and the proportion who completed the 3-month follow-up interview. We monitored for adverse events, which we defined as the experience of severe psychological distress from exposure to the tool. Study staff were trained to detect severe psychological distress in surrogates; no formalized tool was used because of concern for potential interference with the intervention. If surrogates demonstrated severe psychological distress, they were referred for counseling and treatment in accordance with the institutional review board study protocol.
Tool Usability and Acceptability
We obtained usability and acceptability measures before the first family meeting (Figure 2). We measured intervention surrogates’ perceptions of tool usability with the validated 10-item System Usability Scale on a five-point Likert scale (1 = “strongly disagree” and 5 = “strongly agree”) (34). The mean summary score is calculated into a percentile score, which can be used to compare with other similar devices. We measured acceptability with an internally generated seven-item questionnaire using five-point Likert scales (increasing score indicates increased favorability). Specifically, we used the global question, “Overall, how acceptable did you find the Family Support Tool?”
Figure 2.
Timeline of study procedures. DCF = data collection form; FM1 = family meeting 1; FM2 = family meeting 2; ICU = intensive care unit; MD = medical doctor.
Perceived Effectiveness, Quality of Communication, and Shared Decision-making
We assessed intervention surrogates’ perceptions of tool effectiveness immediately after the first and second family meetings to capture any effect the tool may have had on the family meeting. We used an internally generated 11-item questionnaire using five-point Likert scales (in which increasing score indicates increased effectiveness) in which questions pertained to how well the Family Support Tool oriented them to the ICU environment; prepared them for the family meeting, helped them talk to the ICU team about their loved one’s values and preferences and their concerns, aided them in thinking about and understanding the pathways of care, and improved the degree to which their needs and concerns were met and the degree to which the care accounted for their loved one’s values and preferences. We calculated a mean summary score of the 11 items to produce a perceived effectiveness score.
During the 3-month follow-up call with blinded research staff, we measured quality of communication and shared decision-making among both intervention and control surrogates using items from the validated 19-item Quality of Communication (QOC) questionnaire on a 10-point Likert scale (1 = “the very worst I could imagine” and 10 = “the very best I could imagine”). To calculate QOC scores we used the global question, “Overall, how would you rate your physician’s communication with you?” Then, we abstracted from the QOC questionnaire six questions pertaining to shared decision-making (giving information, listening, discussing prognosis, eliciting patient preferences, including the surrogate in decision-making, and assisting family with decisions) (35).
Statistical Analyses
Usability, acceptability, and perceived effectiveness of the tool were summarized with means and standard deviations. To determine whether the tool results in better QOC and shared decision-making, we compared the mean ratings between the control and intervention groups using two-sample t tests. All analyses were executed using Stata version 16.0 (StataCorp LLC).
Results
Tables 1–3 summarize the sociodemographic data of enrolled patients and surrogates. Overall, surrogates were predominantly female (62.5%), middle-aged (mean 56.4 yr), and white (77.1%). Patients tended to be older (mean, 67 yr) and white (82%) and had a high likelihood of ICU mortality (54% of patients had an Acute Physiology and Chronic Health Evaluation II score of ≥25). Seventy-two percent of patients had no advance directive at the time of hospital admission. There were no significant differences between intervention and control arms.
Table 1.
Patient demographic data (n = 50) as reported by surrogates on the baseline questionnaire
| Characteristic | Intervention (N = 25)* | Control (N = 25) |
|---|---|---|
| Age, yr | ||
| Mean (SD) | 65.6 (17.7) | 69 (13.2) |
| Median (IQR) | 70 (56–77) | 68 (61–74) |
| Range | 25–93 | 39–95 |
| Sex, n (%) | ||
| M | 13 (52) | 11 (44) |
| F | 12 (48) | 14 (56) |
| Hispanic/Latinx, n (%) | ||
| Non-Hispanic | 23 (92) | 25 (100) |
| Hispanic | 1 (4) | 0 |
| Unknown | 1 (4) | 0 |
| Race, n (%) | ||
| White | 22 (88) | 19 (76) |
| Black | 3 (12) | 6 (24) |
Definition of abbreviations: IQR = interquartile range; SD = standard deviation.
Data exclude two withdrawn surrogates. One intervention surrogate did not complete the baseline questionnaire because of unexpected death shortly after enrollment; demographic data for this patient were abstracted from the medical record.
Table 3.
Surrogate demographic data
| Characteristic | Intervention (N = 23)* | Control (N = 25) |
|---|---|---|
| Age, yr | ||
| Mean (SD) | 58.65 (12.62) | 54.28 (13.15) |
| Median (IQR) | 61 (53–69) | 54 (44–66) |
| Range | 25–79 | 31–74 |
| Sex, n (%) | ||
| M | 6 (26.09) | 12 (48) |
| F | 17 (73.91) | 13 (52) |
| Hispanic/Latinx, n (%) | ||
| Non-Hispanic | 22 (95.65) | 23 (92) |
| Hispanic | 1 (4.35) | 1 (4) |
| Not answered | 0 | 1 (4) |
| Race, n (%) | ||
| White | 20 (86.96) | 17 (68) |
| Black | 3 (13.04) | 7 (28) |
| Not answered | 0 | 1 (4) |
| Education, n (%) | ||
| Some high school | 1 (4.35) | 1 (4) |
| High school diploma/GED | 7 (30.43) | 5 (20) |
| Some college | 5 (21.74) | 5 (20) |
| Completed college | 6 (26.09) | 6 (24) |
| 1+ graduate studies | 1 (4.35) | 2 (8) |
| Graduate/professional degree | 3 (13.04) | 6 (24) |
| Religion, n (%) | ||
| Roman Catholic | 5 (21.74) | 9 (36) |
| Other Christianity | 12 (52.17) | 14 (56) |
| Jewish | 2 (8.7) | 0 |
| None | 3 (13.04) | 2 (8) |
| Not answered | 1 (4.35) | 0 |
| Relationship to patient, n (%) | ||
| Spouse | 7 (30.43) | 9 (36) |
| Child | 6 (26.09) | 10 (40) |
| Parent | 4 (17.39) | 0 |
| Sibling | 2 (8.70) | 2 (8) |
| Other relative | 2 (8.70) | 3 (12) |
| Friend | 1 (4.35) | 0 |
| Other relationship (ex-wife) | 0 | 1 (4) |
| Not answered | 1 (4.35) | 0 |
Definition of abbreviations: GED = General Educational Development; IQR = interquartile range; SD = standard deviation.
Excludes two withdrawn surrogates. Two intervention surrogates did not complete the surrogate demographic form.
Table 2.
Patient demographic data based on chart abstraction
| Characteristic | Intervention (N = 25) | Control (N = 25) |
|---|---|---|
| Code status at enrollment, n (%) | 20 (80) | 20 (80) |
| Full code | 2 (8) | 2 (8) |
| DNR, intubation permitted | 2 (8) | 3 (12) |
| DNR/DNI | 1 (4) | 0 |
| Written advance directive on chart, n (%) | ||
| Yes | 9 (36) | 5 (20) |
| No | 16 (64) | 20 (80) |
| APACHE II score, n (%) | ||
| 10–14 | 2 (8) | 0 |
| 15–19 | 3 (12) | 3 (12) |
| 20–24 | 7 (28) | 8 (32) |
| 25–29 | 6 (24) | 7 (28) |
| 30–34 | 3 (12) | 2 (8) |
| >34 | 4 (16) | 5 (20) |
Definition of abbreviations: APACHE = Acute Physiology and Chronic Health Evaluation; DNI = do not intubate; DNR = do not resuscitate.
Data exclude two withdrawn surrogates.
Figure 3 shows the CONSORT diagram. Seventy-seven of the 182 patients (42%) who met inclusion criteria were eligible. Of those who met inclusion criteria, family unavailability (n = 28), an already-made decision to withdraw life-sustaining treatment (n = 14), and patient regaining of decision-making capacity (n = 13) were the most common reasons for ineligibility. Of eligible patients, 52/77 (67.5%) were enrolled. Surrogates feeling overwhelmed (n = 14) was the most common reason for declining enrollment. Twenty-seven and 25 surrogates were randomized to the intervention and control arms, respectively. Two surrogates in the intervention group withdrew after enrollment and did not complete any part of the intervention because of personal reasons; they were not included in the analysis.
Figure 3.
CONSORT flow diagram. BM = bone marrow; FL = Florida; FM1 = family meeting 1; FM2 = family meeting 2; FST=Family Support Tool; f/u = follow-up; ICU = intensive care unit; QNR = questionnaire.
Table 4 summarizes surrogates’ tool engagement metrics. Ninety-six percent of surrogates in the intervention arm (24/25) completed the assigned modules per study protocol. One surrogate did not access the tool because the patient died unexpectedly hours after enrollment, but they were included in our intention-to-treat analysis. Surrogates spent an average of 62 minutes with the tool before the first family meeting and logged into the tool an average of 4.2 times throughout the hospitalization. Surrogates selected an average of 5.1/13 questions from the question prompt list. Ninety-four percent (47/50) of enrolled surrogates completed follow-up 3 months after hospital discharge, and 6% were considered lost to follow-up. There were no adverse events detected by research staff, who maintained regular contact with surrogates.
Table 4.
Main outcome results: engagement metrics and follow-up rates
| Engagement Metrics in Intervention Arm* | ||||
|---|---|---|---|---|
| Measurement | Outcome |
Range |
||
| Tool completion rate, n (%) | 24/25 (96) |
|
||
| Mean time spent on tool | 62 min |
0 min–1 h, 48 min |
||
| Average logins, n | 4.2 |
0–8 |
||
| Mean questions selected from questions prompt list, † n | 5.1 |
0–13 |
||
| Adverse events, n | 0 |
|
||
| Withdrawal and Follow-Up Rate | ||||
| Measurement | Intervention | Control |
Total | |
| Withdrawal rate, n (%) | 2/27 (7.41) | 0 (0) |
2/52 (3.85) | |
| Loss to follow-up rate, n (%) | 1/25 (4) | 2/25 (8) |
3/50 (6) | |
| Follow-up completion rate, n (%) | 24/25 (96) | 23/25 (92) | 47/50 (94) | |
One surrogate did not access the tool because the patient died unexpectedly hours after enrollment, but they were included in our intention-to-treat analysis.
From the questions prompt list, which includes 13 questions.
Table 5 summarizes key usability and acceptability findings. Surrogates reported that the tool was highly usable (mean System Usability Scale summary score, 82.4, which translates to 94th percentile compared with other devices of its kind). Surrogates also rated the tool as acceptable (mean score, 4.5 ± 0.9, in which 1 = “very unacceptable” and 5 = “very acceptable”).
Table 5.
Secondary outcomes: surrogate perceptions on tool usability and acceptability before family meetings
| No. | Question about the Family Support Tool SUS | Family Meeting 1 |
Family Meeting 2 |
||||
|---|---|---|---|---|---|---|---|
| Mean* (SD) | Median (IQR) | Min–Max | Mean (SD) | Median (IQR) | Min–Max | ||
| Systems usability scale
†
|
N = 23 |
N = 7 |
|||||
| 1 | I think that I would like to use this program if I had a loved one in the ICU. | 1.5 (0.7) | 1 (1–2) | 1–3 | 1.4 (0.5) | 1 (1–2) | 1–2 |
| 2 | I thought the program was too complex. | 4.1 (0.6) | 4 (4–5) | 3–5 | 4.1 (0.4) | 4 (4–4) | 4–5 |
| 3 | I thought the program was easy to use. | 1.4 (0.7) | 1 (1–2) | 1–4 | 1.9 (0.4) | 2 (2–2) | 1–2 |
| 4 | I think that I would need the support of a technical person to be able to use the program. | 4.1 (0.9) | 4 (4–5) | 1–5 | 4.0 (1.0) | 4 (4–5) | 2–5 |
| 5 | I found the information and questions in the program were well integrated. | 1.7 (1.0) | 1 (1–2) | 1–5 | 1.9 (0.7) | 2 (1–2) | 1–3 |
| 6 | I thought there was too much inconsistency in the program. | 4.2 (0.7) | 4 (4–5) | 3–5 | 4.0 (0.6) | 4 (4–4) | 3–5 |
| 7 | I would imagine that most people would learn to use this program very quickly. | 1.6 (0.6) | 2 (1–2) | 1–3 | 2.0 (0.0) | 2 (2–2) | 2–2 |
| 8 | I found the program very cumbersome to use. | 4.2 (0.9) | 4 (4–5) | 2–5 | 3.9 (0.9) | 4 (4–4) | 2–5 |
| 9 | I felt very confident using the program. | 1.7 (0.8) | 2 (1–2) | 1–3 | 1.9 (0.7) | 2 (1–2) | 1–3 |
| 10 | I needed to learn a lot of things before I could get going with this program. | 4.2 (1.0) | 4 (4–5) | 1–5 | 4.3 (0.5) | 4 (4–5) | 4–5 |
| Overall, I would rate the user-friendliness of this product as... | 4.0 (0.8) | 4 (4–5) | 2–5 | 3.7 (0.8) | 4 (3–4) | 3–5 | |
| SUS summary score | |
||||||
| Overall SUS score ‡ | 82.4 (13.4) | 85 (72.5–95) | 45–100 | 78.2 (10.1) | 72.5 (70–82.5) | 70–97.5 | |
| Acceptability scale (“How acceptable was it...”)
§
|
N = 23 |
N = 7 |
|||||
| 1 | Overall, how acceptable did you find the Family Support Tool? | 4.5 (0.9) | 5 (4–5) | 1–5 | 4.4 (0.5) | 4 (4–5) | 4–5 |
| 2 | To enter questions you had for the doctor into the tool? | 4.5 (0.6) | 5 (4–5) | 3–5 | 4.3 (0.8) | 4 (4–5) | 3–5 |
| 3 | To ask you to record your thoughts about your loved one’s prognosis? | 3.9 (0.9) | 4 (3–5) | 1–5 | 4.0 (0.8) | 4 (3–5) | 3–5 |
| 4 | To hear tips from other families about how to talk to the doctors? | 4.4 (0.7) | 4 (4–5) | 3–5 | 4.0 (0.8) | 4 (3–5) | 3–5 |
| 5 | To hear about the different pathways of care that may be appropriate for your loved one? | 4.5 (0.7) | 5 (4–5) | 3–5 | 4.6 (0.5) | 5 (4–5) | 4–5 |
| 6 | To hear stories about how other families made decisions? | 4.6 (0.6) | 5 (4–5) | 3–5 | 4.4 (0.5) | 4 (4–5) | 4–5 |
| 7 | To answer questions to help you think about your loved one’s values? | 4.3 (0.6) | 4 (4–5) | 3–5 | 4.3 (0.5) | 4 (4–5) | 4–5 |
Definition of abbreviations: ICU = intensive care unit; IQR = interquartile range; Max = maximum; Min = minimum; SD = standard deviation; SUS = System Usability Scale.
SUS items 1–10 were scored from 1 to 5, in which 1 = “strongly agree” and 5 = “strongly disagree.” The overall user-friendliness item was scored from 0 to 5, in which 0 = “worst imaginable” and 5 = “best imaginable.”
Patients who died upon enrollment did not receive questionnaires; patients who were transferred before the family meeting did complete usability and accessibility questionnaires.
Overall SUS score was calculated using SUS items 1–10; the scale is 0–100.
Acceptability items 1–7 were scored from 1 to 5, in which 1 = “very unacceptable” and 5 = “very acceptable.”
Table 6 summarizes quantitative findings for overall effectiveness. Surrogates reported that the tool was effective (mean score, 4.4 ± 0.2, in which 1 = “not at all” and 5 = “extremely well”). Table 7 summarizes surrogate ratings of overall quality of communication and elements of shared shared decision-making at 3-month post discharge follow up. Compared with the control group at 3-month post–hospital discharge follow-up, surrogates who used the tool reported higher overall QOC (mean, 8.9 ± 1.6 vs. 8.0 ± 2.4, in which 0 = “very worst communication possible” and 10 = “very best communication possible”) and higher quality in shared decision-making (mean, 8.7 ± 1.5 vs. 8.0 ± 2.4 with the same QOC scale); however, the difference did not reach statistical significance.
Table 6.
Secondary outcome: surrogate perceptions on perceived effectiveness of the FST after the first family meeting*
| No. | Question about the FST “How Effective Was the Tool in…” | Family Meeting 1 |
||
|---|---|---|---|---|
| Mean † (SD) | Median (IQR)N = 19 ‡ | Min–Max | ||
| 1 | Introducing you to the ICU staff? | 4.5 (0.7) | 5 (4–5) | 3–5 |
| 2 | Introducing you to ICU rounds and routines? | 4.3 (0.7) | 4 (4–5) | 3–5 |
| 3 | Explaining ICU machines? | 4.3 (0.7) | 4 (4–5) | 3–5 |
| 4 | Explaining what happens in a meeting between the clinicians and family? | 4.5 (0.7) | 5 (4–5) | 3–5 |
| 5 | Assisting you in identifying your main questions to ask the clinicians in the family meeting? | 4.4 (0.8) | 5 (4–5) | 3–5 |
| 6 | Preparing you for questions the doctor may ask you? | 4.2 (0.9) | 4 (3–5) | 3–5 |
| 7 | Preparing you to speak up and ask your questions to the clinical team? | 4.6 (0.7) | 5 (4–5) | 3–5 |
| 8 | Helping you think about your loved one’s values and preferences? | 4.5 (0.8) | 5 (4–5) | 3–5 |
| 9 | Helping you understand the pathways of care that may be appropriate for your loved one? | 4.6 (0.7) | 5 (4–5) | 3–5 |
| 10 | Helping you think through what were the best goals for your loved one’s medical care? | 4.5 (0.7) | 5 (4–5) | 3–5 |
| 11 | Helping you communicate with the team of clinicians taking care of your loved one? | 4.4 (0.7) | 4 (4–5) | 3–5 |
| Summary score § | 4.4 (0.1) | 5 (4–5) | 3–5 | |
Definition of abbreviations: FST = Family Support Tool; ICU = intensive care unit; IQR = interquartile range; Max = maximum; Min = minimum; SD = standard deviation.
Scores not reported for second family meeting because of large fallout rate (patients died or were transferred) (N = 5).
Scale is 0–5, in which 1 = “not at all effective” and 5 = “extremely effective.”
After family meeting 1, the questionnaire was administered to 20 intervention surrogates, but one did not complete the FST effectiveness section.
Summary score was calculated as mean score of 11 questions.
Table 7.
Secondary outcomes: surrogate ratings of overall QOC and individual elements of shared decision-making at 3-month post discharge follow-up
| Shared Decision-making Questions (From QOC Questionnaire) | Intervention (N = 24) |
Control (N = 23) |
t Test P Value | ||
|---|---|---|---|---|---|
| Valid Responses | Mean (SD)* | Valid Responses | Mean (SD) | ||
| Giving information | 24 | 8.8 (1.8) | 23 | 8.3 (2.7) | 0.39 |
| Listening | 24 | 9.0 (1.4) | 23 | 8.3 (2.3) | 0.24 |
| Discussing prognosis | 22 † | 7.9 (2.2) | 23 | 7.6 (3.0) | 0.70 |
| Eliciting patient preferences | 22 † | 8.6 (1.8) | 22 † | 8.0 (2.7) | 0.32 |
| Including surrogate in decision-making | 24 | 8.9 (1.9) | 23 | 8.4 (2.6) | 0.50 |
| Assisting surrogate with decisions | 22 † | 8.9 (1.4) | 22 † | 7.9 (2.5) | 0.13 |
| Summary shared decision-making rating | 24 | 8.7 (1.5) | 23 | 8.0 (2.4) | 0.26 |
| Overall QOC rating | 24 | 8.9 (1.6) | 23 | 8.0 (2.7) | 0.19 |
Definition of abbreviations: QOC = quality of communication; SD = standard deviation.
Scale is 1–10, in which 0 = “the very worst I could imagine” and 10 = “the very best I could imagine.”
Missing scores were due to surrogates selecting “doctor didn’t do.”
Discussion
We found that it was feasible to incorporate the Family Support Tool into the care of critically ill patients and that surrogates and clinicians rated the tool as highly usable, acceptable, and effective.
This study provides important data on two important questions related to supporting surrogate decision makers in ICUs. First, before this study, it was unclear whether it would be acceptable to surrogates to use a technology-enhanced intervention as a longitudinal source of support and to enhance communication with clinicians. The concern was that, for many surrogates, the ICU is an overwhelming environment, in no small part because of the unfamiliar technology that is commonly used to treat and monitor critically ill patients, and that a technology-enhanced intervention may exacerbate surrogates’ sense of being overwhelmed. Our study suggests that this tool is indeed acceptable. We speculate that the codesign process and user-testing approaches may have been particularly helpful in creating an acceptable tool. The codesign process resulted in a tool that is both grounded in theory and attends to real-world needs identified by ICU surrogates, such as the intense distress they often feel in the early days of an ICU stay. The extensive usability testing with former and active ICU surrogates allowed us to preemptively identify and fix potential problems (24, 36).
Second, this study provides important data on the feasibility of integrating electronic decision support into the process of clinician–family communication in ICUs. Care processes around family communication are notoriously challenging to modify (37, 38). Our data suggest that it is indeed feasible for families to use the tool iteratively throughout the ICU stay and to complete the sections that are designed to be viewed before family meetings per protocol. We speculate that several aspects of the tool fostered feasibility. First, we used an extensive user-centered design process to originally develop the tool, which resulted in a tool that reflected the stated needs and preferences of surrogates and clinicians, and it was designed to fit into existing ICU workflow processes (24). Second, we designed each section of the tool to be relatively brief, lasting no more than 15–20 minutes. Third, we leveraged the concept of supportive accountability to increase surrogates’ motivation to complete the tool before family meetings (39). Supportive accountability involves designing process-oriented expectations that the user of a technology is accountable to a trusted person (i.e., the ICU team) for completing the tool before family meetings. We accomplished this by explicitly introducing the Family Support Tool to families as a step that the ICU team would like them to complete before the family meeting to maximize the usefulness of family meetings.
The Family Support Tool differs in important ways from other technology-enhanced support tools tested in ICUs. The web-based tool tested by Cox and colleagues (40) was a formal decision aid for patients undergoing prolonged mechanical ventilation that was intended to be used at a single time point late in the ICU stay. MyICUGuide (41), a decision support intervention designed to foster shared decision-making in the ICU, was deployed as either a paper-based guide or a web-based tool with additional video content. Similar to the Cox decision aid, it was also designed to be reviewed in a single, longer session; investigators noted the long duration to be a major barrier to study feasibility and tool use. The Family Support Tool is innovative in its design in several important ways, including the following: 1) it is the first web-based tool designed to attend to both the emotional and cognitive aspects of surrogate decision-making; 2) it is designed to be used early on and throughout the ICU stay; 3) rather than focusing on specific diseases, the tool can be broadly applied to a range of diseases that cause critical illness, offering the potential to use the tool among all critically ill patients and their families; and 4) surrogates can access the tool using their personal device and the internet, making it rapidly scalable to diverse ICUs.
We encountered two research challenges in this study that have implications for future studies. First, a substantial number of surrogates declined to participate in the study solely because they were too emotionally overwhelmed to think about research participation. This highlights the challenges of interventions designed to be deployed early in an ICU stay, when surrogates arguably most need support but when they may be too overwhelmed to be a study participant. This observation is consistent with previous studies of ICU surrogates and remains an ongoing challenge for studies involving ICU surrogate decision-making (13, 40).
Second, we observed that many patients either died or had improved substantially before the second protocol-driven family meeting could occur. This has implications for the intervention design in the sense that although it may be psychologically preferable to not broach the topic of goals of care in the first family meeting and instead focus on sharing information and building trust, this may be challenging given the rapid evolution of clinical cases. Future work should involve developing strategies to tailor the timing of the second family meeting based on the patient’s clinical condition rather than an arbitrary number of days into the ICU stay.
Our trial has important limitations. First, we used research staff, rather than clinical staff, to deliver the intervention to surrogates (i.e., to orient participants to the tool, to arrange family meetings, and to provide clinicians the summary sheet before the family meetings). This decision is grounded in National Institutes of Health recommendations for the development and testing of complex behavioral interventions, which recommend first testing interventions under “ideal” conditions, and then developing strategies to scale successful interventions. Therefore, if the intervention is efficacious in controlled conditions, future work will be needed to integrate these responsibilities into the workflow of ICU clinical staff (e.g., bedside ICU nurses). Second, we did not track whether surrogates needed to be reminded to complete tool content before family meetings. This should be assessed in future research involving web-based decision support tools in the ICU context. Third, the tool was not optimized for smartphone use, which may have limited the extent to which participants accessed the tool. Fourth, although this pilot randomized controlled trial established the feasibility and acceptability of the Family Support Tool, it was not powered or designed to assess efficacy regarding clinical outcomes. Future research should include assessments of the intervention’s impact on patients’ outcomes, surrogates’ outcomes, and healthcare utilization.
In conclusion, we successfully pilot-tested the Family Support Tool in high-acuity ICUs where surrogates were actively making decisions for their incapacitated loved one. Surrogates rated the Family Support Tool as highly usable, acceptable, and effective. These data support proceeding with an appropriately powered randomized trial to assess the efficacy of the Family Support Tool on patient and family outcomes.
Acknowledgments
Acknowledgment
The authors thank the patients, family members and friends, and clinicians who participated; the hospital administration, unit leaders, and bedside staff at the study sites; and the analytics support team at Pendo.
Footnotes
Supported by U.S. National Institutes of Health grant R21 AG050252 from the National Institute on Aging. H.O.W. was also supported by a Tier 2 Canada Research Chair in Human-Centred Digital Health. D.B.W. was also supported by National Institutes of Health grant (K24HL148314).
Author Contributions: A.O.S.: writing original draft, reviewing and editing writing, visualization, and formal analysis. R.A.B.: writing original draft, reviewing and editing writing, investigation, resources, visualization, data curation, formal analysis, and project administration. R.M.A.: conceptualization, methodology, reviewing and editing writing, and validation. B.M.: conceptualization, methodology, software, validation, reviewing and editing writing. H.O.W.: conceptualization, methodology, validation, and reviewing and editing writing. C.E.C.: conceptualization, methodology, and reviewing and editing writing. J.G.M.: investigation and reviewing and editing writing. P.B.: data curation, visualization, and formal analysis. A.-M.S.: investigation, resources, data curation, project administration, reviewing and editing writing. N.M.: investigation, formal analysis, and software. A.A.: investigation, project administration, and reviewing and editing writing. D.B.W.: conceptualization, supervision, funding acquisition, methodology, investigation, formal analysis, software, writing original draft, reviewing and editing writing, and validation.
This article has an online supplement, which is accessible from this issue’s table of contents at www.atsjournals.org.
Author disclosures are available with the text of this article at www.atsjournals.org.
References
- 1. Torke AM, Sachs GA, Helft PR, Montz K, Hui SL, Slaven JE, et al. Scope and outcomes of surrogate decision making among hospitalized older adults. JAMA Intern Med. 2014;174:370–377. doi: 10.1001/jamainternmed.2013.13315. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Raymont V, Bingley W, Buchanan A, David AS, Hayward P, Wessely S, et al. Prevalence of mental incapacity in medical inpatients and associated risk factors: cross-sectional study. Lancet. 2004;364:1421–1427. doi: 10.1016/S0140-6736(04)17224-3. [DOI] [PubMed] [Google Scholar]
- 3. McAdam JL, Dracup KA, White DB, Fontaine DK, Puntillo KA. Symptom experiences of family members of intensive care unit patients at high risk for dying. Crit Care Med. 2010;38:1078–1085. doi: 10.1097/CCM.0b013e3181cf6d94. [DOI] [PubMed] [Google Scholar]
- 4. Teno JM, Clarridge BR, Casey V, Welch LC, Wetle T, Shield R, et al. Family perspectives on end-of-life care at the last place of care. JAMA. 2004;291:88–93. doi: 10.1001/jama.291.1.88. [DOI] [PubMed] [Google Scholar]
- 5. Azoulay E, Pochard F, Kentish-Barnes N, Chevret S, Aboab J, Adrie C, et al. FAMIREA Study Group. Risk of post-traumatic stress symptoms in family members of intensive care unit patients. Am J Respir Crit Care Med. 2005;171:987–994. doi: 10.1164/rccm.200409-1295OC. [DOI] [PubMed] [Google Scholar]
- 6. Azoulay E, Chevret S, Leleu G, Pochard F, Barboteu M, Adrie C, et al. Half the families of intensive care unit patients experience inadequate communication with physicians. Crit Care Med. 2000;28:3044–3049. doi: 10.1097/00003246-200008000-00061. [DOI] [PubMed] [Google Scholar]
- 7. Anderson WG, Cimino JW, Ernecoff NC, Ungar A, Shotsberger KJ, Pollice LA, et al. A multicenter study of key stakeholders’ perspectives on communicating with surrogates about prognosis in intensive care units. Ann Am Thorac Soc. 2015;12:142–152. doi: 10.1513/AnnalsATS.201407-325OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Scheunemann LP, Cunningham TV, Arnold RM, Buddadhumaruk P, White DB. How clinicians discuss critically ill patients’ preferences and values with surrogates: an empirical analysis. Crit Care Med. 2015;43:757–764. doi: 10.1097/CCM.0000000000000772. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. White DB, Engelberg RA, Wenrich MD, Lo B, Curtis JR. Prognostication during physician-family discussions about limiting life support in intensive care units. Crit Care Med. 2007;35:442–448. doi: 10.1097/01.CCM.0000254723.28270.14. [DOI] [PubMed] [Google Scholar]
- 10. Chiarchiaro J, Buddadhumaruk P, Arnold RM, White DB. Quality of communication in the ICU and surrogate’s understanding of prognosis. Crit Care Med. 2015;43:542–548. doi: 10.1097/CCM.0000000000000719. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. You JJ, Downar J, Fowler RA, Lamontagne F, Ma IW, Jayaraman D, et al. Canadian Researchers at the End of Life Network. Barriers to goals of care discussions with seriously ill hospitalized patients and their families: a multicenter survey of clinicians. JAMA Intern Med. 2015;175:549–556. doi: 10.1001/jamainternmed.2014.7732. [DOI] [PubMed] [Google Scholar]
- 12. Curtis JR, Engelberg RA, Wenrich MD, Shannon SE, Treece PD, Rubenfeld GD. Missed opportunities during family conferences about end-of-life care in the intensive care unit. Am J Respir Crit Care Med. 2005;171:844–849. doi: 10.1164/rccm.200409-1267OC. [DOI] [PubMed] [Google Scholar]
- 13. Curtis JR, Treece PD, Nielsen EL, Gold J, Ciechanowski PS, Shannon SE, et al. Randomized trial of communication facilitators to reduce family distress and intensity of end-of-life care. Am J Respir Crit Care Med. 2016;193:154–162. doi: 10.1164/rccm.201505-0900OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. White DB, Angus DC, Shields A-M, Buddadhumaruk P, Pidro C, Paner C, et al. PARTNER Investigators. A randomized trial of a family-support intervention in intensive care units. N Engl J Med. 2018;378:2365–2375. doi: 10.1056/NEJMoa1802637. [DOI] [PubMed] [Google Scholar]
- 15. Campbell ML, Guzman JA. Impact of a proactive approach to improve end-of-life care in a medical ICU. Chest. 2003;123:266–271. doi: 10.1378/chest.123.1.266. [DOI] [PubMed] [Google Scholar]
- 16. Carson SS, Cox CE, Wallenstein S, Hanson LC, Danis M, Tulsky JA, et al. Effect of palliative care-led meetings for families of patients with chronic critical illness : a randomized clinical trial. JAMA. 2016;316:51–62. doi: 10.1001/jama.2016.8474. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Schneiderman LJ, Gilmer T, Teetzel HD, Dugan DO, Blustein J, Cranford R, et al. Effect of ethics consultations on nonbeneficial life-sustaining treatments in the intensive care setting: a randomized controlled trial. JAMA. 2003;290:1166–1172. doi: 10.1001/jama.290.9.1166. [DOI] [PubMed] [Google Scholar]
- 18. Dowdy MD, Robertson C, Bander JA. A study of proactive ethics consultation for critically and terminally ill patients with extended lengths of stay. Crit Care Med. 1998;26:252–259. doi: 10.1097/00003246-199802000-00020. [DOI] [PubMed] [Google Scholar]
- 19. Meier DE, Back AL, Berman A, Block SD, Corrigan JM, Morrison RS. A national strategy for palliative care. Health Aff (Millwood) 2017;36:1265–1273. doi: 10.1377/hlthaff.2017.0164. [DOI] [PubMed] [Google Scholar]
- 20. Spetz J, Dudley N, Trupin L, Rogers M, Meier DE, Dumanovsky T. Few hospital palliative care programs meet national staffing recommendations. Health Aff (Millwood) 2016;35:1690–1697. doi: 10.1377/hlthaff.2016.0113. [DOI] [PubMed] [Google Scholar]
- 21. Lupu D. American Academy of Hospice and Palliative Medicine Workforce Task Force. Estimate of current hospice and palliative medicine physician workforce shortage. J Pain Symptom Manage. 2010;40:899–911. doi: 10.1016/j.jpainsymman.2010.07.004. [DOI] [PubMed] [Google Scholar]
- 22. Hua MS, Li G, Blinderman CD, Wunsch H. Estimates of the need for palliative care consultation across united states intensive care units using a trigger-based model. Am J Respir Crit Care Med. 2014;189:428–436. doi: 10.1164/rccm.201307-1229OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Ernecoff NC, Witteman HO, Chon K, Chen YI, Buddadhumaruk P, Chiarchiaro J, et al. Key stakeholders’ perceptions of the acceptability and usefulness of a tablet-based tool to improve communication and shared decision making in ICUs. J Crit Care. 2016;33:19–25. doi: 10.1016/j.jcrc.2016.01.030. [DOI] [PubMed] [Google Scholar]
- 24. Suen AO, Butler RA, Arnold R, Myers B, Witteman HO, Cox CE, et al. Developing the family support tool: an interactive, web-based tool to help families navigate the complexities of surrogate decision making in ICUs. J Crit Care. 2020;56:132–139. doi: 10.1016/j.jcrc.2019.12.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Power TE, Swartzman LC, Robinson JW. Cognitive-emotional decision making (CEDM): a framework of patient medical decision making. Patient Educ Couns. 2011;83:163–169. doi: 10.1016/j.pec.2010.05.021. [DOI] [PubMed] [Google Scholar]
- 26. Murray MA, Miller T, Fiset V, O’Connor A, Jacobsen MJ. Decision support: helping patients and families to find a balance at the end of life. Int J Palliat Nurs. 2004;10:270–277. doi: 10.12968/ijpn.2004.10.6.13268. [DOI] [PubMed] [Google Scholar]
- 27. Légaré F, O’Connor AM, Graham ID, Wells GA, Tremblay S. Impact of the Ottawa Decision Support Framework on the agreement and the difference between patients’ and physicians’ decisional conflict. Med Decis Making. 2006;26:373–390. doi: 10.1177/0272989X06290492. [DOI] [PubMed] [Google Scholar]
- 28. Hanson LC, Danis M, Garrett J. What is wrong with end-of-life care? Opinions of bereaved family members. J Am Geriatr Soc. 1997;45:1339–1344. doi: 10.1111/j.1532-5415.1997.tb02933.x. [DOI] [PubMed] [Google Scholar]
- 29. Zier LS, Burack JH, Micco G, Chipman AK, Frank JA, Luce JM, et al. Doubt and belief in physicians’ ability to prognosticate during critical illness: the perspective of surrogate decision makers. Crit Care Med. 2008;36:2341–2347. doi: 10.1097/CCM.0b013e318180ddf9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Heyland DK, Dodek P, Rocker G, Groll D, Gafni A, Pichora D, et al. Canadian Researchers End-of-Life Network(CARENET) What matters most in end-of-life care: perceptions of seriously ill patients and their family members. CMAJ. 2006;174:627–633. doi: 10.1503/cmaj.050626. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Li L, Nelson JE, Hanson LC, Cox CE, Carson SS, Chai EJ, et al. How surrogate decision-makers for patients with chronic critical illness perceive and carry out their role. Crit Care Med. 2018;46:699–704. doi: 10.1097/CCM.0000000000003035. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Apatira L, Boyd EA, Malvar G, Evans LR, Luce JM, Lo B, et al. Hope, truth, and preparing for death: perspectives of surrogate decision makers. Ann Intern Med. 2008;149:861–868. doi: 10.7326/0003-4819-149-12-200812160-00005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.O’Connor AM. Ottawa, ON: Ottawa Hospital Research Institute; 1995. User manual - decision self-efficacy scale. [updated 2002; accessed 2019 Jun 1]. Available from: http://decisionaid.ohri.ca/docs/develop/User_Manuals/UM_Decision_SelfEfficacy.pdf. [Google Scholar]
- 34. Brooke J. SUS: a retrospective. J Usability Stud. 2013;8:29–40. [Google Scholar]
- 35. Charles C, Gafni A, Whelan T. Shared decision-making in the medical encounter: what does it mean? (or it takes at least two to tango) Soc Sci Med. 1997;44:681–692. doi: 10.1016/s0277-9536(96)00221-3. [DOI] [PubMed] [Google Scholar]
- 36. Argenas A, Myers B, Witteman H, Arnold RM, Shields AM, Buddadhumaruk P, et al. Developing a web-based tool to enhance communication and shared decision making for families of critically ill patients through user-centered methods. Am J Respir Crit Care Med. 2020;201:A4568. [Google Scholar]
- 37. Black MD, Vigorito MC, Curtis JR, Phillips GS, Martin EW, McNicoll L, et al. A multifaceted intervention to improve compliance with process measures for ICU clinician communication with ICU patients and families. Crit Care Med. 2013;41:2275–2283. doi: 10.1097/CCM.0b013e3182982671. [DOI] [PubMed] [Google Scholar]
- 38. Kodali S, Stametz R, Clarke D, Bengier A, Sun H, Layon AJ, et al. Implementing family communication pathway in neurosurgical patients in an intensive care unit. Palliat Support Care. 2015;13:961–967. doi: 10.1017/S1478951514000650. [DOI] [PubMed] [Google Scholar]
- 39. Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res. 2011;13:e30. doi: 10.2196/jmir.1602. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Cox CE, White DB, Hough CL, Jones DM, Kahn JM, Olsen MK, et al. Effects of a personalized web-based decision aid for surrogate decision makers of patients with prolonged mechanical ventilation: a randomized clinical trial. Ann Intern Med. 2019;170:285–297. doi: 10.7326/M18-2335. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41. Van Scoy LJ, Chiarolanzio PJ, Kim C, Heyland DK. Development and initial evaluation of an online decision support tool for families of patients with critical illness: a multicenter pilot study. J Crit Care. 2017;39:18–24. doi: 10.1016/j.jcrc.2016.12.022. [DOI] [PubMed] [Google Scholar]