Development and Pilot Testing of a Simulation to Study How Physicians Facilitate Surrogate Decision-Making Based on Critically Ill Patients’ Values and Preferences

Abstract

Context:

There are no evidence-based programs to train physicians to facilitate shared decision-making based on incapacitated intensive care unit (ICU) patients’ values and preferences.

Objectives:

To develop a high-fidelity simulation to fill this gap.

Methods:

Case development involved 6 steps: 1) drafting a case about an elderly patient receiving prolonged mechanical ventilation; 2) engaging an expert advisory board to optimize case content; 3) revising the case based on advisory board input; 4) training actors to portray the case patient’s daughter; 5) obtaining physician feedback on the simulation; 6) revising the case based on their feedback. We conducted a cross-sectional pilot study with 50 physicians to assess feasibility and acceptability, defined a priori as an enrollment rate > 40 physicians/year, study procedures <75 minutes/participant, >95% actor adherence to standardization rules, and high physician ratings of realism and acceptability.

Results:

Advisory panel feedback yielded two modifications: 1) refocusing the case on decision-making about tracheostomy and percutaneous gastrostomy; 2) making the patient’s values more authentic. Physician feedback yielded two additional modifications: 1) reducing how readily the actor divulged the patient’s values; 2) making her more emotional. All 50 physicians enrolled in the pilot study over 11 months completed study procedures in <75 minutes. Actor adherence to standardization rules was 95.8%. Physicians’ mean ratings of realism and acceptability were 8.4 and 9.1 respectively on a 10-point scale.

Conclusion:

Simulation is feasible, acceptable, and can be adequately standardized to study physicians’ skills for facilitating surrogate decision-making based on an incapacitated ICU patient’s values and preferences.

Introduction

Incapacitated, critically ill patients near the end of life often receive burdensome, expensive treatment that they would not want.(1–5) Such treatment violates the fundamental principle of person-centered care and harms patients, their families, and society.(6) Part of the problem is that clinicians and families struggle to communicate effectively about how to incorporate the patient’s values and preferences into a treatment plan.(7–9) Improving these communication skills is an important goal, but evidence-based interventions do not exist.

Before interventions can be tested, there must be a way of assessing change in clinicians’ communication skills. One strong possibility for doing so is simulation, which theoretically standardizes clinical cases to limit variability, permit intervention on specific skills, and test the impact on patient outcomes.(10–12) However, no methodology is available for testing whether medical simulations are adequately standardized for communication skills of interest.(13) For example, a simulation focusing on surrogate decision-making about an incapacitated patient’s values and preferences should require that actors discuss a defined set of patient values in a reliable way in response to clinicians’ questions. Such a methodology would allow researchers to study differences in clinicians’ communication skills that might influence decision outcomes, assess changes in clinicians’ communication skills in response to interventions, and demonstrate that the specific communication skills impact patient outcomes. Clearly, this is a critical gap.

Therefore, we developed and pilot tested a high fidelity simulation to study how physicians facilitate surrogate decision-making based on the values and preferences of an incapacitated, critically ill patient near the end of life. Our main aims were: 1) to develop a methodology for testing the standardization of the simulation; and 2) to apply it as part of an overall assessment of the simulation’s feasibility and acceptability.

Methods

We chose simulation methodology because it is safe, efficient, permits standardization, and can be applied across the range of observational and interventional communication research. The University of Pittsburgh IRB approved our study protocol, which was based on published guidelines for simulation development. The funding sources had no role in the study.

Development phase

We followed published frameworks for standardized case development(14,15) in six steps (Figure):

Figure.

Simulation Development

1) Drafting. The case focused on 78-year-old woman on day 12 of mechanical ventilation with a relatively poor prognosis receiving prolonged mechanical ventilation for ARDS.

2) Expert advisory panel. We shared the draft with an expert advisory panel consisting of clinicians and researchers from critical care medicine, critical care nursing, geriatrics, and palliative care. We conducted semi-structured interviews(16) to identify the strengths and weaknesses of the case for studying surrogate decision-making about patient values and preferences in the ICU.

3) Case revision. We made two changes based on the expert advisory panel’s feedback. First, we focused the case on whether the patient should undergo tracheostomy and percutaneous gastrostomy because these decisions involve trade-offs between deeply held values such as prolonging life, avoiding burdensome treatments, maintaining independence, and honoring spiritual beliefs.(17–19) Second, we made the patient’s values more authentic to the way families talk.

4) Actor training. We trained two experienced medical actors who were Caucasian women in their 30’s-40’s to portray the patient’s daughter. They studied written information about their character and learned two standardization rules: 1) do not volunteer information about the patient’s values and preferences unless the clinician asks; 2) give standardized information to specific types of questions. Although families do not operate under such rules, simulation always requires calculated trade-offs with realism. We considered this trade-off acceptable for two reasons. First, empiric evidence suggests that families do not talk much about patients’ values and preferences in ICU family conferences if not asked.(9) Second, we designed the rules to require that clinicians ask about patients’ values and preferences in order for them to be discussed and to reward them for asking, allowing them to move the conversation forward towards a patient-centered treatment decision. Thus, the rules support the overarching goal of developing a method of assessing clinicians’ communication skills and change in response to interventions.

We also trained the actors not to express too much emotion, so that clinicians did not spend the whole simulation on emotional support. They practiced in a series of 4 hours of role play with study investigators and 2 hours with volunteer physicians. The role plays included feedback about following the rules, as well as their authenticity to families encountered in clinical practice.

5) Clinician panel. Once the actors had learned the role, we recruited 16 fellows and attendings from critical care and palliative care to pretest it. They went through the simulation procedures described in the section below on the pilot phase and provided structured feedback about how to improve it.

6) Case revision. We made two modifications to the case based on the clinician panel’s feedback. First, we made her more emotional because she was initially so reserved that she was unrealistic compared to actual practice. Second, we reduced how much information the actors divulged in response to any single question about the patient’s values to 1-2 statements. This addressed both physicians’ sense that the actors shared more than most family members and our goal of studying how clinicians facilitate shared decision-making based on an incapacitated patient’s values and preferences; they would not have to use these skills if the actor provided information too freely.

Pilot Phase

Once we had completed these six steps, the simulation was ready for pilot testing. Although ICU family conferences typically include multiple family members and multiple interdisciplinary team members, we followed the precedent of prior research simulations about ICU decision-making which included 1-2 family members and one physician.(20–23) This choice was feasible for a pilot study, minimized participant scheduling burdens, permitted us to work on the methods for studying and measuring communication between case family member-clinician dyads before expanding to families an teams, and permitted us to study physicians’ communication skills in depth (analysis pending). We started with physicians because they strongly influence decision-making(24–27) and improving the quality of their communication skills is a high priority to improve patient outcomes. The intent of these choices was to build a foundation for future work incorporating non-physician team members, consistent with guidelines for shared decision-making near the end of life.(28,29)

We recruited a cross sectional convenience sample of attendings or fellows in palliative or critical care via email and advertising at local conferences, and offered a $25 gift card in appreciation for participation. We typically scheduled 3-4 of those who responded to participate in the same morning and ran simulations back-to-back in the family conference room of our medical ICU to avoid conflicts with real conferences which typically happen in the afternoons. In the week before participation, we emailed physicians a copy of the patient’s medical record including an off-service note from the prior intensivist as well as the morning’s vitals, labs, and chest x-ray report (Online Supplement 1). We obtained informed consent at the time of enrollment.

Enrolled physicians completed a demographic questionnaire in the physician workroom in our medical ICU and had a chance to review the patient’s medical record. We instructed them to conduct the family meeting as if this were their patient. They then went to the ICU family meeting room where the audio- and video-recorders were already running, met the simulated daughter, and led the family meeting. Simulations were capped at 30 minutes for feasibility, at which time the investigator knocked on the door and called the physician out. Afterwards, the physicians completed a questionnaire and a brief interview about the feasibility and acceptability of participation. The actors completed a brief questionnaire from the case daughter’s perspective. We noted whether physicians completed study procedures in <75 minutes.

Measurement

We considered feasibility to have three parts: 1) that participants were available, willing, and able to participate; 2) that the simulation was sufficiently standardized with regard to the target communication skills; and 3) that the simulation demonstrated some ability to discriminate between physicians’ skill for discussing patients’ values and preferences.

For the first goal, we measured physicians’ availability as the enrollment rate (number of physicians participating per year); their willingness to participate based on responses to previously used questionnaires(22) about realism and acceptability of participating in the simulation; and efficiency of participation as the time required to complete study procedures (to ensure study procedures would fit in physicians’ schedules and to keep actor payments within the study budget).

To test the second goal, we conducted a mixed methods analysis of simulation transcripts to test how well the actors adhered to the standardization rules. The qualitative analysis employed a previously developed coding scheme according to Crabtree and Miller’s template coding strategy,(30) focusing on two key aspects of shared decision-making: 1) discussing past expressions of the patient’s values and preferences, and 2) deliberating about how to apply them to treatment planning.(31) In order to test actors’ standardization, we added 3 subcodes indicating how actors’ statements related to physicians’ statements: 1). (which has correct content but does not require that physicians utilize the target communication skills); 2) answering incorrectly/not sharing when asked (which does not reward the communication skills physicians utilize); and 3) answering appropriately (responding to a physician prompt about the patients’ values and preferences with correct information from the case). Online Supplement 2 provides exemplars.

Two internal medicine residents who were masked to physician specialty trained in the coding scheme by coding a subset of 5 transcripts line-by-line and resolving any differences by discussion with the principal investigator. Next, they demonstrated excellent interrater reliability compared to each other and the principal investigator (average kappas ≥ 0.83) on 20-question tests constructed from a bank of exemplars. After this training, they each independently coded half of the transcripts. To ensure accurate coding, they conducted qualitative data cleaning with the principal investigator. This step involved reviewing all coding line by line to ensure appropriate inclusion of all eligible statements, exclusion of statements not fitting the code definitions, and categorization of all statements according to the codebook rules.

We approached the third goal, testing the simulation’s ability to discriminate between physicians’ skill in discussing the patient’s values and preferences, in two ways: (1) describing physicians’ communication skill use to assess the range and distribution for floor or ceiling effects; (2) assessing whether actors’ ratings of patient-centered communication on a modified Patient-Centeredness of Care-Surrogate(32) questionnaire were associated with physicians’ communication skill use. The Patient-Perceived Patient-Centeredness of Care scale has 14 items on a 4-point Likert scale; it has demonstrated construct validity and been adapted for surrogate decision-makers(33). We had to remove 2 of its items for use in the simulation because of its cross-sectional design. The rationale for these two strategies for testing discrimination was that they balanced a quantitative assessment of how frequently physicians used skills with a qualitative assessment of how it felt from the case daughter’s perspective.

Statistical analysis

We calculated descriptive statistics for all measures. We considered that feasibility required an enrollment rate of at least 40 physicians/year, retention of ≥95% of physicians for all study procedures, keeping the time burden of participation ≤75 minutes for all participants, median realism and acceptability scores ≥8/10, and <5% rate of errors for actors in following the standardization rules. To determine the error rate, we calculated the total number of opportunities for discussing the patient’s values and preferences by adding all the statements in which the actor volunteered, answered correctly, answered incorrectly, or failed to answer physicians’ questions about them. Any statement that was not answered correctly counted as an error, and the error rate was the number of errors divided by the total number of opportunities. To assess whether the simulation discriminated between physicians’ use of skills for discussing patients’ values and preferences, we first calculated descriptive statistics of their skill use. Because it was non-normally distributed, we used Spearman correlation to test its association with actor-rated Patient-Centeredness of Care.

Results

Study population

Table 1 shows the demographics of our study population. Overall, they were representative of palliative and critical care physicians at a large, urban academic tertiary care center in the United States. The young mean age with a wide standard deviation reflects the mix of fellows and attendings.

Table 1.

Participant characteristics

	Development Phase N=16 % or Mean (SD)	Pilot Phase N=50 % or Mean (SD)
Mean age (SD)	37.4 (9.0)	39.8 (9.8)
% Male	38%	52%
Race
% Asian	25%	12%
% Caucasian	75%	74%
% Other	0%	14%
% Latino or Hispanic	12%	6%
% Trainees	44%	32%
Mean years in practice (SD)	8.3 (9.3)	8.9 (8.8)
Area of practice
% Critical care	88%	72%
% Palliative care	12%	28%

Feasibility of study procedures

Study procedures demonstrated the feasibility of recruiting and retaining physicians to completion of data collection: the physician enrollment rate was 50 physicians/year, the study completion rate was 100% for enrolled physicians; and 100% of physicians completed the study in <75 minutes. Physician ratings of the simulation’s realism and acceptability of participation were also consistently high (Table 2). Semi-structured interviews with physicians suggested several reasons the simulation did not seem perfectly realistic, including that there was only one family member and only one clinician (when most family meetings have multiple family members and clinicians); the daughter character was not as emotional as a typical family member although they had experience with such reserved family members; and although the patient’s values and preferences seemed similar to real patients, the daughter expressed them more readily and clearly than most families.

Table 2.

Realism and Acceptability

Realism	To what extent were the following similar to what you encounter in your practice?	Pilot Phase Median Scale 1-10 (IQR)
	Family conference	8 (7-10)
	Actor portraying the family member	9 (7-10)
	Emotions expressed	9 (8-10)
	Discussion about the patient’s values & treatment preferences	8.5 (7-10)
	Conference room	10 (9-10)
	Clinical information	10 (8-10)
Acceptability	How acceptable did you find participation overall?	9.75 (8-10)

Feasibility of standardizing communication

Transcript analysis revealed the feasibility of standardizing communication of actors within medical simulations (Table 3). All of the simulations contained some discussion of the patient’s values and preferences. Overall, there were 1068 physician-actor statements about the patient’s values and preferences, which counted as opportunities for success or error. The overall error rate was 4.2%, with most errors reflecting problems volunteering or incorrectly responding (giving information that was not in the case, exemplar in Online Supplement 2), to physicians’ questions about past expressions of the patient’s values and preferences.

Table 3.

Frequency of Actor Errors During the Pilot Phase

Error Types	N Errors	% Errors
Values & preferences
Volunteered	23	2.1
Didn’t share/answered incorrectly	11	1.0
Treatment plans
Volunteered	1	<0.001
Didn’t share/answered incorrectly	10	0.9
Total (1068 Opportunities)	45	4.2

Discrimination among physicians’ use of communication skills

The simulation demonstrated the ability to discriminate differences in physicians’ communication skill use in both assessment strategies (Table 4). First, the lowest number of physician statements about the patient’s values and preferences in a conference was 1 and the maximum was 42, with a mean of 12.7, a standard deviation of 8.4, and a non-normal distribution with an interquartile range of 2-25. There was similar variation within the individual communication skills assessed; because deliberation is contingent on elicitation, the maximum number of statements was less, but the distribution was still non-normal. Second, physicians’ use of communication skills for discussing patients’ values and preferences showed significant association with Patient-Centeredness of Care-Surrogate (Spearman’s rho=0.36, p=0.01). Two questions on the Patient-Centeredness of Care-Surrogate are specifically related to communication about patients’ values and preferences: 1) to what extent did the doctor ask about your loved one’s goals/preferences for treatment? and 2) how much would you say that the healthcare team(s) care about your loved one as a person? In a sensitivity analysis testing the association of just these two statements with physicians’ use of communication skills for discussing patients’ values and preferences, the correlation remained significant (Spearman’s rho=0.43, p=0.01).

Table 4.

Physicians’ Use of Communication Skills within the Simulation

Communication Skills Used by Physicians	Simulation Characteristics (N=50)
Elicited the patient’s previously expressed values & preferences
Any question – N (%)	50 (100)
Number of questions – Mean (SD, total range)	9.8 (7.3, 1 - 36)
Deliberated about how to apply the patient’s values & preferences in the current situation
Discussed how the patient would think or feel about the current situation or possible futures
Any discussion – N (%)	40 (80)
Mean (SD, total range)	2.2 (1.8, 0 – 6)
Offered a treatment recommendation based on the patient’s values & preferences
Any recommendation – N (%)	23 (46)
Number of recommendations – Mean (SD, total range)	0.68 (0.94, 0 – 4)
Total communication about patients’ values and preferences
Any question – N (%)	50 (100)
Number of statements – Mean (SD, range)	12.7 (8.4, 1 – 42)
Association with Patient-Centeredness of Care – Spearman’s rho (p value)	0.36 (0.01)

Discussion

This study demonstrated that simulation is a feasible and acceptable methodology for studying physician-surrogate communication about an incapacitated patient’s values and preferences. Physicians were willing and able to participate and agreed that the simulation was realistic and an acceptable use of their time. The design of the simulation for targeting communication about patients’ values and preferences was successful, in that 100% of the simulations contained some discussion of them compared to about 70% in usual practice.(9) However, the simulation still discriminated both among physicians’ quantitative use of skills (i.e., how many times did they use each skill) and the qualitative impact of those skills (i.e., how patient-centered did it feel to the actor portraying the case daughter). The non-normal distribution of physicians’ skill use ranging from a minimum of 1 to a maximum of 42 statements for discussing patients’ values and preferences suggests low likelihood of important floor or ceiling effects. And despite the complexity of these conversations, the actors successfully followed standardization rules with <5% errors.

This last result deserves highlighting. Our novel analysis showed the feasibility of standardizing actor responses in medical communication simulations. As simulation moves from a medical education tool to a research tool, demonstrating its standardization takes on increased importance. The overall error rate of 4.2% has face validity for good standardization for such complex communication skills. Future research should refine methods for ensuring that simulations are standardized with regard to target communication skills. For example, if clinicians trained in facilitating communication based on the patients’ values and preferences demonstrated increased use of these skills in our simulation, it would support the simulation’s convergent validity as a tool for studying those skills. Another key refinement is increasing the efficiency of testing both simulation standardization and clinicians’ skill use, for example by a checklist or other tool that could be completed in real time during simulations. Finally, guidelines recommend that interdisciplinary teams facilitate shared decision-making with families.(28,29) Future studies should include non-physician team members and refine the methodology for assessing standardization when multiple participants are present.

Medical education research has long used simulation to train physicians in complex skills like communication based on the understanding that adult learners require practice and feedback to incorporate and use such skills.(34,35) Our study, a simulation studying physicians’ communication skills during critical care triage,(20,36) and a simulation studying how physicians manage conflict during family conferences(22,23) represent first steps in adapting this methodology for critical care outcomes research. The long-term goal of this work is translational(10): identify the “active ingredients” of communication that improve patient-and-family centered outcomes and develop, test, and disseminate interventions focused on those active ingredients to improve patient and family outcomes. Because the skills and aims of medical educators and intervention researchers are compatible but not overlapping,(37) this use of simulation methodology is an ideal model for interdisciplinary research collaboration.

This study’s most important limitation is that it was conducted with a convenience sample of physicians at a single center that has shown a strong commitment to communication training and research for critical care physicians.(35) Whether physicians with less strong institutional support would be as available to participate or would rate its acceptability as highly is unknown. It seemed important that we held the study in an easily accessible location, were committed to fitting physicians’ schedules, and committed to not running over 75 minutes. Second, while most of the communication simulations in critical care have focused on single-episode communication with one provider and one simulator, usual practice typically includes multiple family members and multiple interdisciplinary team members over time. This was a reason physicians downgraded the simulation’s realism. Future research should use this simulation to study interprofessional communication, and adapt it to study the process of communication over time. A third limitation is that we used a single case representing a “typical” patient, an “easy” family member, and easily elicited, clear values. These choices were reasonable to begin research in this area; we made them under the guidance of our expert advisory panel according to simulation guidelines despite the fact they decreased the case’s realism. Future work should investigate a range of ages, genders, ethnicities, clinical scenarios, family behaviors, and patient values. Finally, the standardization rules prevented the case daughter from volunteering information or changing the topic, which is not how families operate. However, physicians scored the realism well, indicating that they did not perceive a significant difference from practice. Again, the goal of the simulation is not to perfectly mirror actual practice, but to provide a good laboratory for learning and practicing new communication skills. The ultimate test of whether it is realistic enough will be whether clinicians demonstrate improved use of communication skills for discussing patients’ values and preferences in the simulation, transfer them to practice, and improve patient-centeredness of care – which are important questions for future research.

In conclusion, this study developed a feasible and reliable simulation to study physician communication with surrogate decision-makers about incapacitated, critically ill patients’ values and preferences. Its methods for demonstrating the simulation’s standardization are novel and may be useful to other researchers studying communication using simulation. The next step is to use the simulation to test the efficacy of a communication skills training intervention to improve how clinicians elicit and integrate patients’ values into goals of care decisions in ICUs.