Accuracy of a Decision Aid for Advance Care Planning: Simulated End-of-Life Decision Making

Abstract

Purpose

Advance directives have been criticized for failing to help physicians make decisions consistent with patients’ wishes. This pilot study sought to determine if an interactive, computer-based decision aid that generates an advance directive can help physicians accurately translate patients’ wishes into treatment decisions.

Methods

We recruited 19 patient-participants who had each previously created an advance directive using a computer-based decision aid, and 14 physicians who had no prior knowledge of the patient-participants. For each advance directive, three physicians were randomly assigned to review the advance directive and make five to six treatment decisions for each of six (potentially) end-of-life clinical scenarios.

From the three individual physicians’ responses, a “consensus physician response” was generated for each treatment decision (total decisions = 32). This consensus response was shared with the patient whose advance directive had been reviewed, and she/he was then asked to indicate how well the physician translated his/her wishes into clinical decisions.

Results

Patient-participants agreed with the consensus physician responses 84 percent (508/608) of the time, including 82 percent agreement on whether to provide mechanical ventilation, and 75 percent on decisions about cardiopulmonary resuscitation (CPR).

Across the six vignettes, patient-participants’ rating of how well physicians translated their advance directive into medical decisions was 8.4 (range = 6.5–10, where 1 = extremely poorly, and 10 = extremely well).

Physicians’ overall rating of their confidence at accurately translating patients’ wishes into clinical decisions was 7.8 (range = 6.1–9.3, 1 = not at all confident, 10 = extremely confident).

Conclusion

For simulated cases, a computer-based decision aid for advance care planning can help physicians more confidently make end-of-life decisions that patients will endorse.

INTRODUCTION

For situations in which doctors are unsure whether an incapacitated patient would want life-sustaining medical treatment, advance directives were initially heralded as a promising mechanism to help ensure that decisions about end-of-life medical care would be consistent with individuals’ actual wishes and goals.¹ However, the results from multiple investigations have challenged this assumption,² as have various theoretical critiques.³ While recent data suggest that completion of an advance directive increases the likelihood that a patient’s wishes for end-of-life care will be honored,⁴ concerns remain about the capability of advance directives to accurately represent a patient’s wishes and/or facilitate surrogate decision making that is consistent with the patient’s wishes for medical treatment.⁵ Here, we describe the results from a pilot study to evaluate the extent to which advance directives that were generated by an innovative, computer-based decision aid for advance care planning helped physicians to accurately translate a patient’s wishes into a treatment decision.

Detailed descriptions of our decision aid for advance care planning (ACP), Making Your Wishes Known: Planning Your Medical Future, have been published previously.⁶ Briefly, it is an interactive, self-directed computer program that uses a question-answer format involving audio, text, graphics, patient vignettes, and videotapes of “professional experts.” The computer program takes an educational approach, simulating the kind of discussion one might have with an experienced, reflective healthcare professional who is well-informed and not rushed for time. We recognize that technological solutions are imperfect and cannot replicate the kind of compassionate and personal interaction that occurs between people. Nevertheless, a practical solution is needed for providing effective ACP in the absence of trained professionals, and we have designed a private, self-paced process that attempts to meet this need. Specifically, its purposes are to:

1. Educate users about the various components of advance directives (including their uses and limitations);

2. Help individuals identify, clarify, and prioritize factors that influence their decision making about future medical conditions;

3. Explain several common medical conditions (stroke, coma, dementia, terminal illness) that often require life-sustaining medical treatment;

4. Describe and demonstrate several common forms of life-sustaining medical treatment (dialysis, mechanical ventilation, feeding tube, cardiopulmonary resuscitation) including indications, side-effects, and complications;

5. Help individuals reflect on the acceptability (to them) of various symptoms (for example, nausea, pain, shortness of breath) as well as specific disabilities (for example, inability to ambulate, communicate, live independently, or think clearly);

6. Help individuals consider and articulate the relation between a condition’s prognosis and their wishes for life-sustaining medical treatment;

7. Help individuals choose an appropriate spokesperson and substitute spokes-person(s);

8. Help individuals articulate a coherent set of medical wishes in the event they could not communicate in the future;

9. Create a tailored advance directive that accurately represents individuals’ views and wishes (with regard to future medical decisions), using a format that is readily and accurately interpretable by physicians; and

10. Prepare individuals to engage in discussions about their values and wishes with family, friends, and healthcare providers.

This computer program combines multiple strategies, building on the strengths of values histories,⁷ detailed preference instruction,⁸ and durable power of attorney designation.⁹ Unlike any other ACP tool of which we are aware, this program also includes a decision aid based on multi-attribute utility theory (MAUT),¹⁰ a method for helping people rank and prioritize aspects of complex decisions. In our program, we use MAUT to help translate an individual’s values and goals into a meaningful, usable advance directive that explicitly reflects that person’s healthcare wishes, and outlines a plan for how they wish to be treated. (See appendix 1 for an abridged version.) In ongoing studies, we are examining how well this decision aid promotes effective communication regarding ACP, as well as its real-life impact on end-of-life medical care. Here we report pilot results on how well the advance directive generated by this decision aid helped others accurately translate patients’ wishes into treatment decisions.

METHODS

This study used hypothetical vignettes to evaluate whether patient-participants agreed with end-of-life treatment decisions made by physicians whose only knowledge of the patients’ wishes was from an advance directive generated by a computer-based decision aid. Informed consent was elicited from each patient-participant. Due to an administrative oversight, this study (which represented an amendment to a previously approved protocol) was reviewed by the Penn State Hershey Medical Center Human Subjects Protection Office after data collection, at which point “the issue of noncompliance was considered to be non-serious and non-continuing,” and the investigators’ corrective action (which involved creating multiple, distinct institutional review board research applications in place of subsuming them under modifications to existing research protocols) was deemed appropriate.

Sample

The study included 19 patient-participants and 14 physicians. Patient-participants (hereafter, patients) were community volunteers who had used the computer program in a prior study that examined its ease of use for creating a personalized advance directive. After completing that study, individuals were recruited for the present project. Physicians consisted of internists, critical care physicians, and surgeons at Penn State Hershey Medical Center, chosen for their experience dealing with end-of-life issues. These physicians had no prior contact with, nor knowledge of, the patients, nor any access to their medical records.

Design

Physicians’ activity

Each patient’s advance directive was printed and rendered anonymous by blacking out the individual’s name, then photocopied and distributed to three physicians, using a randomization scheme that ensured that no two patients would have their advance directives evaluated by the same three physicians. No physician was assigned more than five advance directives. All advance directives used a standard format (see appendix 1) that contained the individual’s specific wishes and content.

For every advance directive, the physicians made a series of treatment decisions for each of six hypothetical clinical vignettes: (A) intraoperative hemorrhagic stroke, (B) trauma-induced cerebral hemorrhage in the setting of a patient with chronic pain, (C) trauma-induced spinal cord damage causing paralysis from the neck down, (D) trauma-induced closed head injury with significant long-term mental deficits, (E) renal failure and sepsis in the setting of a patient with metastatic colon cancer, (F) pneumonia and anorexia in the setting of a patient with Alzheimer’s dementia. (See figure 1 for full scenarios and treatment decisions.)

Figure 1.

Vignettes Given to Physicians (Version for Female Patients)

Additionally, for each vignette, physicians were asked: (1) their level of confidence in being able to appropriately translate the patient’s wishes into clinical decisions (1 = not at all confident, 10 = extremely confident), (2) who is the appropriate person to speak for the patient (fill in the blank), and (3) what additional information they would find helpful (open-ended response). Physicians reviewed patients’ advance directives and made decisions independently; they had no knowledge of the physicians they were teamed with for any given patient, or even which physicians were involved in the study.

For each patient, there were a total of 32 decisions to be made (across the six vignettes), including who should serve as the patient’s spokesperson. The three physicians’ responses (that is, their treatment decisions and answers to the open-ended questions) for each advance directive packet were examined by one of us (SH), and a “consensus physician response” was generated for each of the 32 decisions. The “consensus response” was determined by the predominant response—that is, agreement by two or more of the three physicians for each item.

Patients’ activity

This consensus response (framed as “the doctor’s response”) was shared with the patient whose advance directive had been reviewed, along with a “lay version” (that is, no medical jargon) for the six vignettes. The patient was then asked to indicate (1) whether the doctor made the decisions the patient would have wanted him/her to make for each of the clinical scenarios (yes/no), and (2) how well the physician translated the patient’s wishes into clinical decisions (1 = extremely poorly, 10 = extremely well).

Statistical Analysis

All analyses were carried out using SAS statistical software.¹¹ Patients’ overall satisfaction was calculated by averaging their satisfaction scores across the six vignettes. Physicians’ overall confidence scores were calculated by averaging all of the confidence scores for each vignette and then averaging them across the six vignettes. Patients’ agreement (yes/no) with the consensus of the doctor for each question and vignette was summarized with frequencies and percentages. Patients’ satisfaction for each vignette and physicians’ confidence in their decisions for each vignette were summarized using means and standard deviations.

RESULTS

Of the 26 patients contacted, six expressed interest in the study but declined due to scheduling conflicts, and 20 agreed to participate— although one did not complete the protocol due to subsequent scheduling conflicts. Because this was a pilot study not powered to detect demographic-based differences, only gender (12/19 female), age (mean = 62 years), physicians’ specialty, and race data were collected (see table 1).

Table 1.

Participants’ Demographic Characteristics

Characteristic	Physicians		Patients
	n	%	n	%
Gender
Female	6	43	12	63
Male	8	57	7	37
Race
White	13	93	18	95
Other	1	7	1	5
Physicians’ specialty
Surgeon	3	21	--	--
Critical care	3	21	--	--
Internist	8	57	--	--
	Physicians		Patients
	Mean	Range	Mean	Range
Age in years	48	35–61	62	28–87
Physicians’ years in practice	16	4–30	--	--
1–10	5	36	--	--
11–20	2	14	--	--
>20	7	50	--	--

Patients’ Agreement with Physicians’ Decisions

Patients rated the physicians’ decisions on the hypothetical clinical vignettes as highly accurate reflections of their (the patients’) wishes, via two measures. Measure 1 (a 10-point scale) asked, “How well do you think the doctor translated your wishes into actual clinical decisions?” (1 = extremely poorly, 10 = extremely well). For this, the combined mean score (that is, for all six vignettes) was 8.4 (S.D. = 1.1). (See table 2.)

Table 2.

Patients’ Overall Assessment of Physicians’ Decisions

Responses to query: “How well do you think the doctor translated your wishes into actual clinical decisions?”

	Vignette						Mean
	A	B	C	D	E	F	(A–F)
Average score	9.6	9.2	8.1	7.4	7.4	8.9	8.4
Standard deviation	0.9	1.5	2.8	3.2	3.3	2.6	1.1
Range of scores	7–9	5–10	4–10	2–10	1–10	1–10	6.5–10

Key: 1 = Extremely poorly, 10 = Extremely well

Vignettes:

1. Hemorrhagic stroke that will recover, leaving mild physical deficits
2. Traumatic brain hemorrhage with underlying poorly controlled chronic pain
3. Severed C-spine, resulting in lifetime paralysis and mechanical ventilation
4. Coma after head injury, anticipated to wake up, but with long-term mental deficits
5. Renal failure and sepsis with underlying metastatic colon cancer
6. Pneumonia, anorexia, and aggression with underlying Alzheimer’s dementia

Measure 2 asked for patients’ agreement (yes/no) with each of the 32 individual treatment decisions. Here, patients agreed with the consensus physician response 84 percent (508/608) of the time, including 99 percent agreement with physicians’ identification of the appropriate surrogate decision maker; 82 percent agreement with whether to provide mechanical ventilation; and 75 percent agreement with decisions about cardiopulmonary resuscitation (See table 3.)

Table 3.

Percentage of Patients Who Agreed with Consensus Physician Response

Question asked of physicians	Vignette						Mean
	A	B	C	D	E	F	(A–F)
Would you initiate mechanical ventilation?	*	*	*	*	74	84
Would you Withdraw mechanical ventilation?	100	89	74	74	*	*	82
Would you order a feeding tube?	100	89	79	79	58	95	83
Would you perform CPR?	100	79	79	68	47	79	75
Would you initiate renal dialysis?	100	84	79	73	68	84	81
Who is the appropriate surrogate?	100	100	100	100	95	100	99
Would you pursue surgery?	*	84	*	*	*	*	--
Would you initiate antibiotics?	*	*	*	*	63	*	--
Patients’ overall agreement w/physicians’ response	100	87	82	79	67	88	84

^*Question not asked for this clinical vignette

Vignettes:

1. Hemorrhagic stroke that will recover, leaving mild physical deficits
2. Traumatic brain hemorrhage with underlying poorly controlled chronic pain
3. Severed C-spine, resulting in lifetime paralysis and mechanical ventilation
4. Coma after head injury, anticipated to wake up, but with long-term mental deficits
5. Renal failure and sepsis with underlying metastatic colon cancer
6. Pneumonia, anorexia, and aggression with underlying Alzheimer’s dementia

In deriving the consensus physician response, two out of three physicians provided identical treatment recommendations in 100 percent of decisions (608/608), and three out of three physicians provided identical recommendations in 59 percent of decisions (358/608).

Physicians’ Confidence Scores

Responding to the question, “Given that this patient’s advance directive is the only information you have available before making the needed clinical decisions, how confident are you that you were able to appropriately translate the patient’s wishes into clinical decisions?” (1 = not at all confident, 10 = extremely confident), physicians’ combined mean confidence score (for all six vignettes) was 7.8 (range = 6.1 to 9.3, S.D. = 0.8). (See figure 2).

Figure 2.

Physicians’ Confidence Ratings. Physician’s confidence in being able to appropriately translate the patient’s wishes into clinical decisions—Vignettes (A)–(F)—as measured by the question, “Given that this patient’s advance directive is the only information you have available before making the needed clinical decisions, how confidentare you that you were able to appropriately translate the patient’s wishes into clinical decisions? (1 = not at all confident, 10 = extremely confident).”

Open-Ended Responses

In response to the question, “What additional information would you find helpful for making treatment decisions for this patient?” physicians most often cited the desire for input from family members. Physicians also had specific questions about the impact that chronic pain or various levels of mental function would have on the patient’s notion of “quality of life.” In several instances, physicians wished for more information about what “being a burden” meant for the patient, whether the patient would wish to be kept alive long enough for their family to say goodbye, and how long ago the advance directive was created.

Relatedly, physicians stated that they struggled at times with conflicting requests—an example being a patient’s expressed wish not to be a financial burden on his or her family, while at the same time requesting expensive long-term interventions. In the case of Vignette E (in which patients’ agreement with the consensus physicians’ response was lowest), many physicians wished for more information about whether the patient’s affairs were in order, whether the patient’s family would have the means and ability to care for the patient at home, and whether the patient would want hospice service in such a clinical setting.

DISCUSSION

This pilot study suggests that a computer-based decision aid has the potential to help physicians make treatment decisions that accurately reflect the wishes of patients who cannot speak for themselves. In comparison with other studies using hypothetical vignettes, in which patients agreed with only 60 to 70 percent of physicians’ decisions made on their behalf,¹² use of the advance directive generated by Making Your Wishes Known: Planning Your Medical Future resulted in 84 percent overall patients’ agreement with treatment decisions made by physicians. Even after removing all decisions about who is the appropriate surrogate decision maker (that is, one decision per vignette), patients’ agreement with the consensus physician response in this study remains quite high, at 80 percent (394/494).

Multiple studies have reported that, typically, neither family members nor physicians make accurate substituted judgment decisions on behalf of patients.¹³ For example, in one study (also using hypothetical clinical vignettes) researchers found that, in the absence of advance directives, physicians (many of whom had an existing relationship with the patient) accurately predicted patients’ wishes only ~65 percent of the time, and the use of a standard advance directive improved this by only ~5 percent.¹⁴ By contrast, the 84 percent concordance between patients’ wishes and physicians’ decisions in our study suggests that the advance directive generated by Making Your Wishes Known: Planning Your Medical Future was better at helping physicians make treatment decisions that are in alignment with patients’ wishes for lifesaving interventions. This is important because medical decisions at the end of life are not optional—they must be made one way or another. At such times, most patients will lack decision-making capacity¹⁵ and physicians who make these decisions often will not know the patient well enough to predict the patient’s goals and preferences.¹⁶

Clearly, an advance directive document is no substitute for meaningful conversations in which goals and values can be explored, misconceptions clarified, and preferences explained. Additionally, some individuals may not have formed preferences regarding specific treatment options. That said, for many people there is value to articulating their wishes, and the present data suggest that Making Your Wishes Known can help individuals create advance directives that physicians can readily interpret as intended by patients. This was particularly true for three of the vignettes in this study: (A) hemorrhagic stroke that will recover, leaving mild physical deficits (100 percent agreement); (B) traumatic brain hemorrhage with underlying, poorly controlled chronic pain (87 percent agreement); and (F) pneumonia, anorexia, and aggressive behavior in the setting of underlying Alzheimer’s dementia (88 percent agreement). (See table 3.) Additionally, physicians in our study were able to correctly identify the appropriate surrogate decision maker for patients in 99 percent of cases. Although this may seem a trivial task, the reality of end-of-life decision making is that it is often unclear who should speak for an incapacitated patient, and we know that as many as one-third of patients may not wish their next-of-kin to serve as their proxy.¹⁷ Hence, an accurate, readily interpretable mechanism for identifying the appropriate spokesperson is a valuable contribution.

For two of the vignettes that involved significant long-term debilities—(C) lifetime paralysis and mechanical ventilation and (D) prolonged coma with the prospect of serious lifelong mental deficits—patients’ agreement with physicians’ decisions was somewhat lower (~80 percent). Because this was a pilot study that did not include in-depth interviews with participants, the present data shed little light on why this was so. Still, the patient-physician concordance for these vignettes is substantially better than found in similar studies.¹⁸

In the context of these generally positive outcomes, the comparatively low level of patients’ agreement with physicians’ responses for Vignette (E) (renal failure and sepsis in the setting of a patient with metastatic colon cancer) warrants examination. If we look beneath the “percent agreement” to the actual views expressed regarding the five treatment questions in Vignette (E), we find that the patients in our study elected life-sustaining medical treatment only 24 percent of the time (23/95 decisions) whereas physicians decided in favor of life-sustaining medical treatment 54 percent of the time (51/95). If we omit CPR and consider only mechanical ventilation, feeding tube, and dialysis, the disparity becomes even sharper, with patients opting for life-sustaining medical treatment 24 percent of the time (18/76) versus 61 percent of the time (46/76) for physicians.

We can speculate that this may reflect medicine’s cultural tendency toward “erring on the side of life.” Alternatively, it may reflect the fact that those more familiar with death and dying have a greater appreciation for the value that patients often place on being able to say good-bye to family and friends or having time to “put their affairs in order.” But if either of these explanations were fitting, we would expect to see similar disparities with several of the other vignettes, which we do not. One thing that sets Vignette (E) apart is that it concerns a patient with end-stage cancer—specifically, renal failure, sepsis, and underlying aggressive, metastatic colon cancer. Many physicians are aware that, with end-stage cancer, an individual’s “will to live” often depends upon the level of family/community support that is available for caring for him/her,¹⁹ and that severe illness can cause patients to have greater uncertainty regarding their treatment preferences.²⁰ Hence, it is possible that physicians’ treatment decisions reflected a desire to keep options open until more information became available, but it is also a reminder that static documents should be interpreted with caution, and used as a tool (not a substitute) for meaningful conversation.

It is nonetheless promising that, across the six vignettes, patients agreed with 84 percent of the treatment decisions rendered by physicians whose only information about each patient was the advance directive generated by the computer-based decision aid. Given the design of this pilot study, we cannot be sure how much of this is due to the quality of the output—that is, that the advance directive was systematically and clearly articulated, making it easy for physicians to understand and apply—versus patients themselves having a clearer idea of what they wanted, due to having worked through the decision aid. This is because the program is designed to help users clarify and prioritize their values, and understand how their views map onto end-of-life decision making.

LIMITATIONS

There are several important limitations to the present study. First, it was conducted at a single site with only a small number of participants and little ethnic or racial diversity. Second, the physician-participants who volunteered time for this study may not be representative of the broad range of physicians who make end-of-life decisions. Third, there was no control group, thereby limiting the ability to compare the relative effectiveness of the decision aid. Fourth, the use of just six vignettes limits the generalizability of our findings, given the broad array of clinical scenarios that require decisions about life-sustaining treatment. Fifth, although written vignettes have been found to be an accurate means for assessing physicians’ performance,²¹ they tend to isolate decisions that in real life are influenced by multiple factors.

That said, the scenarios used for this study were chosen for being commonplace in end-of-life decision making. Additionally, the present data suggest that, in terms of identifying and communicating individuals’ wishes regarding end-of-life treatment decisions, a computer-based decision aid is an effective mechanism for doing so. As discussed elsewhere,²² we regard advance directive documents as tools for promoting discussion. In that light, the present findings are particularly encouraging insofar as they demonstrate that the advance directive generated by Making Your Wishes Known: Planning Your Medical Future accurately reflects individuals’ wishes, and thus can serve as a good starting point for conversations with healthcare providers, loved ones, and others. In ongoing studies, we are finding that decision aids can help individuals become more informed and better prepared for engaging in advance care planning discussions (and do so without increasing patients’ anxiety or diminishing their sense of hope). To the extent that such interactions generate meaningful dialogue, decision aids may not only promote respect for autonomy,²³ but help deepen people’s relationships.²⁴

CONCLUSION

The results of this pilot study suggest that, despite the many limitations of static documents, advance directives generated by Making Your Wishes Known: Planning Your Medical Future can help physicians make treatment decisions that patients would want, and do so with confidence. How well this intervention promotes conversations between patients and their loved ones or healthcare providers is the subject of ongoing study.

APPENDIX 1