Improving Medical Decisions for Incapacitated Persons: Does Focusing on “Accurate Predictions” Lead to an Inaccurate Picture?

Abstract

The Patient Preference Predictor (PPP) proposal places a high priority on the accuracy of predicting patients’ preferences and finds the performance of surrogates inadequate. However, the quest to develop a highly accurate, individualized statistical model has significant obstacles. First, it will be impossible to validate the PPP beyond the limit imposed by 60%–80% reliability of people’s preferences for future medical decisions—a figure no better than the known average accuracy of surrogates. Second, evidence supports the view that a sizable minority of persons may not even have preferences to predict. Third, many, perhaps most, people express their autonomy just as much by entrusting their loved ones to exercise their judgment than by desiring to specifically control future decisions. Surrogate decision making faces none of these issues and, in fact, it may be more efficient, accurate, and authoritative than is commonly assumed.

I. INTRODUCTION

The authors of the Patient Preference Predictor (hereafter PPP) proposal speculate that the PPP will be much more accurate in predicting people’s treatment preferences than their surrogates’ predictions (Rid and Wendler, 2014a, 2014b). This speculation about greater accuracy drives their proposal. Indeed, the mere fact that they consider as an option—although ultimately not endorsed—using PPP as a strong default that overrides surrogate decisions indicates their confidence in PPP’s empirical and normative potential.

The PPP proposal has been thought out in great detail, with integration of sophisticated theoretical and extensive empirical considerations. There are many aspects that deserve comment, but this brief piece will mostly sketch an alternative assessment of the surrogates’ role, arguing that, contrary to the premise of the PPP, surrogate decision making is in fact an impressively efficient, accurate, and authoritative mechanism for making decisions for the incapacitated.

II. LIMITS OF ELICITING PREFERENCES FOR FUTURE TREATMENT: A HYPOTHESIS

Imagine Mrs. Smith being interviewed by a researcher regarding her treatment preferences in case of future incapacity. Some of the decision scenarios she finds easy (e.g., cardiopulmonary resuscitation (CPR) in permanent comatose state, antibiotics in her current state of health). But some scenarios make her hesitate: she hasn’t seriously imagined or thought about, say, suffering a stroke with modest (but presumably real) chance of meaningful recovery. If she is inclined to self-reflection, she may think to herself that she cares deeply about living as long as possible. Her life is good, filled with grandchildren, friends, and travel. Yet she wants to avoid pain, suffering, and an undignified, prolonged death that burdens her family. Despite these convictions, it is not clear to her how she should decide in this stroke scenario. Is the “modest chance” of recovery worth the risk of a prolonged life with impairment? She wonders what her husband of 40 years would do (the nice young researcher is, in fact, asking her to answer that question too); she also wonders what he thinks she would decide.

Or, if she is not prone to self-reflection, she may not even feel much uncertainty as she simply chooses an answer on first impressions, without much thought—it’s not a fun thing to think about after all. She knows it’s only a survey; she is expected to give an answer, so she does.

If we returned a month, a year, or 2 years later, she may well give a different answer. She may not even realize that her views had changed; after all, it’s not as though she had a settled preference in the first place.

III. LIMITS OF ELICITING PREFERENCES FOR FUTURE TREATMENT: EVIDENCE

Does this hypothetical picture of Mrs. Smith have a basis in reality? In fact, there is considerable support for it, with important implications.

Unpredictable Instability of Preferences Inherently Limits Any Prediction Model

The data regarding stability of treatment preferences are clear. Back in 2004, a review reported that an unpublished meta-analysis of 11 studies showed an overall stability of 71% (range of 57%–89%) (Fagerlin and Schneider, 2004). For instance, in their study of 252 AIDS patients, Weissman et al. (1999) found “one-fourth of the respondents changed their minds about life-sustaining care” over a 4-month period. Another study of 495 outpatients and 102 members of the public found 77% stability over 12–24 months (Emanuel et al., 1994). Elderly persons in Israel showed 70% stability (Carmel and Mutran, 1999) and elderly persons in the United States were similar (Danis et al., 1994; Ditto et al., 2003). In one study, response stability was no better than chance in 16 of 35 scenarios, and among those who changed their preferences, only 20% were aware of the change in their own preferences (Gready et al., 2000).

Studies since 2004 have yielded similar results. Fried et al. (2007) interviewed elderly patients with serious, advanced diseases every 4 months over 2 years, finding that 35%–49% (depending on scenario) had inconsistent preference trajectories. Other studies have shown, for a variety of samples and scenarios: 13.8%–49.3% change (Pruchno et al., 2008), 38% change (Janssen et al., 2012), and 20%–59% likelihood of change (Wittink et al., 2008). Although some studies find factors that show associations with changes, the findings are inconsistent or even contradictory, and insufficiently robust for classificatory purposes; in other words, the changes in preferences are largely unpredictable (Danis et al., 1994; Pruchno et al., 2008; Wittink et al., 2008; Janssen et al., 2012). Indeed, even asking subjects whether they changed their minds is an unreliable method of identifying instability.

Thus, social science data provide an unusually high degree of confidence that end-of-life treatment preferences will change on the order of 20%–40% on average, with a greater range for individual scenarios (more stable for “easy” scenarios and less stable for “difficult” scenarios), even over short periods of time (as short as 4 months). What are the implications of these findings for the PPP? The main implication is that it will be impossible to validate the PPP beyond 60%–80% level of reliability. The inherent unpredictability of the main outcome variable limits the validation of any survey-based prediction tool beyond a certain level of accuracy. Of note, this achievable limit of accuracy is remarkably similar to the known accuracy level of surrogates’ predictions (Shalowitz, Garrett-Mayer, and Wendler, 2006).

Is the Prevalence of Actual Preferences Artificially Inflated by How We Measure?

When patients who complete the kind of surveys on which the above data are based (and presumably the kind of surveys needed for PPP) are given a chance to express their level of confidence regarding their preferences, one finds that there is considerable doubt. In one large study (Danis et al., 1994), 12% replied “don’t know” (when given “yes,” “no,” “don’t know” to a given treatment option—a response set not often used in these studies) to all questions and were removed from their stability analysis. These were not disenfranchised members of our society; they were more likely to be “married, male, white, and better educated.” Moreover, even after excluding these 12%, for any given treatment preference question, “don’t know” responses still ranged from 18% to 46%! It is also interesting that survey researchers who study preferences for future treatment will usually categorize “unsure” responses on Likert scales as “yes,” inflating the rate of people having actual preferences (Smucker et al., 2000). ¹

The prevalence of preferences therefore seems highly sensitive to the way we measure them, and we overestimate it. This hypothesis is consistent with the (in)stability data above and explains why only one in five persons who change their minds realizes that there is a change, and why people prefer to give broad instructions rather than specific ones (Hawkins et al., 2005; Miyata, Shiraishi, and Kai, 2006).

Just because people have strong—but broad—values regarding how they prefer to die (no pain, no burden on others, length of life, etc.) does not mean people always have, or can have (or that we can accurately infer), stable specific preferences. For a significant proportion of persons, such an inference seems unwarranted. The authors of PPP state that “the PPP primarily promotes . . . treatment consistent with patients’ considered preferences and values” (Rid and Wendler, 2014a, 5, 29). But what if people do not have “considered preferences and values” specific enough for decisions as PPP requires?

IV. “ACCURACY” IS NOT THE HIGHEST VALUE

The conclusion that the level of predictive “accuracy” even for the best mechanisms will, on average, be fairly good (60%–80%) but not much higher may be unwelcome news to the PPP project. But data suggest that this is really not a problem for most people because they are not invested in their surrogates’ perfect and strict adherence to their preferences.

Let’s imagine we first ask people about their preferences for certain treatment scenarios. Then, we ask: in case of future incapacity, would they be willing to let their surrogate and doctor make the decision as they see fit, or would they prefer that their stated preference be followed? Several such studies have been conducted.

In a study of 150 dialysis patients, asking whether they would want continued dialysis if diagnosed with Alzheimer’s disease, 39% would not give any leeway to their surrogates, but 19% would give a little leeway, 11% would give a lot of leeway, and nearly a third (31%) would give complete leeway (Sehgal et al., 1992). A study of elderly US outpatients regarding end-of-life treatment preferences showed that only 9% would allow no leeway, whereas 37% would allow a little, 28% a lot, and 24% complete leeway (Hawkins et al., 2005). The largest study to date on leeway (Puchalski et al., 2000) posed the following question (after first ascertaining patients’ CPR preferences) to older (N = 513) or seriously ill (N = 646) inpatients: “If you became too sick to tell your doctors what medical care you wanted, would you be willing to let [surrogate] and doctor make decisions or would you prefer that your current CPR choice . . . be followed?” Over 71% of older inpatients and 78% of seriously ill inpatients said they would want their surrogates to make decisions with their doctors, rather than follow their stated wishes about CPR.

One might think that terminally ill patients—patients already quite ill, with considerable experience being sick, and facing major decisions—might have more desire for their explicitly stated preferences to carry the day. However, even in such a group, only about 22% want only their own stated preferences to direct treatment (Sulmasy et al., 2007). Two studies from Japan yielded similar results regarding leeway preferences, leading the authors to conclude it is permissible for “family and physician to loosely interpret one’s directives” (Akabayashi, Slingsby, and Kai, 2003; Miyata, Shiraishi, and Kai, 2006).

I disagree with the authors of the PPP proposal who seem to demote the importance of these leeway data by, first, categorizing giving leeway as a process preference of patients, which promotes autonomy only “indirectly” and, second, dismissing it further by saying that most patients trust their surrogates because they mistakenly believe that they will be accurate in predicting their wishes (Rid and Wendler, 2014a). But the leeway data need to be reckoned with more directly.

The leeway data go directly against the view that subjects trust surrogates only for “accuracy” reasons (i.e., that their value is derivative of how well they follow or represent the patient’s preferences): a sizable minority to a solid majority are willing to say explicitly that their recently endorsed preference can be overridden by their surrogates. The authors of PPP might reply that this is because people who want leeway think their surrogates will actually be more accurate about what they would actually want; so again, the value of the surrogate reduces to an indirect way of enhancing accuracy. Here, I think there are two important points to make in response. First, the distinction between most recently stated wishes and the unknowable “what this incapacitated person would want in this current situation if he were capable now” is important. The patients are clearly saying they are OK with the first type of preference being overridden, which is the reference for accuracy in this discussion of the PPP: many care less about the sort of accuracy that PPP seems to assume they should care about.

Second, there is a better explanation for what leeway preference means. It is just as much an expression of autonomy to express that you want to give leeway to someone you trust as it is to express a specific preference. Clearly, we want someone to be more than an automaton when we entrust her with a serious task. Accuracy is an important consideration to be sure; but we want someone to incorporate that consideration in an exercise of judgment. Many people value the certainty of their trust in their loved ones over the uncertainty of applicability of their current preferences in the future. One might even see this as a more thoughtful and rational (and therefore more autonomous) choice.

V. SURROGATE “ACCURACY” IN PERSPECTIVE

The authors of the PPP proposal report that surrogate accuracy, averaged across various scenarios used in multiple studies, is 68% (Shalowitz, Garrett-Mayer, and Wendler, 2006). These data are said to “undermine the claim that reliance on surrogates is justified by their ability to predict incapacitated patients’ treatment preferences” (Shalowitz, Garrett-Mayer, and Wendler, 2006, 493). But this does not follow unless a much higher level of accuracy is achievable, available, and desired. And as we saw, this is a doubtful premise.

The authors of the PPP proposal cite two studies (Smucker et al., 2000; Shalowitz, Garrett-Mayer, and Wendler, 2007) to suggest that the PPP would be more accurate than surrogate predictions. Both studies showed that population-based predictions are similar—actually, identical—in accuracy to surrogate predictions. In one study (Smucker et al., 2000), the composite index of accuracy for both a community standard and surrogate predictions was 74%–75%, for nine health conditions (with multiple treatment options for each). In the second study, a population-based versus surrogate predictions gave identical level of accuracy for one scenario, at 78% (Shalowitz, Garrett-Mayer, and Wendler, 2007). The authors note that the surrogates’ accuracy is “inflated” relative to the overall 68% accuracy found in their review due to use of an “easier” scenario—as though the population-based predictions do not also benefit from the “easiness” of the scenarios. In fact, the two predictions will travel together, bumping up against the limit set by inherent instability of preferences that will vary depending on how “easy” the decision scenario is. Is it mere coincidence that in both studies the surrogates’ predictions and the statistical models give identical results?

This suggests something quite remarkable about surrogates’ medical decisions for the incapacitated. First, considering that there is no cost involved (as compared to the high cost and effort necessary to generate the PPP or any other large survey-based statistical model), and given the known level of stability of preferences, one must marvel at how accurate the surrogates actually are on average. Remarkably, little of surrogate “inaccuracy” seems to be due to the kind of biases mentioned by the PPP authors. If such biases were large, it is unlikely that surrogates and population-based predictions would match so closely. Second, although most people may believe that their surrogates know them best and can best represent their wishes (which turns out to be a true belief, as a matter of best alternative to one’s own contemporaneous decision making), most also seem to realize that their surrogates have to be given at least some degree of leeway in making their decisions. People do not value their own preferences as much as (some of) our normative models demand that they do. Their valuation of their own preferences seem to be more muted, more consistent with the inherent uncertainties of expressing a preference about an unknowable future.

VI. CONCLUDING THOUGHT: BEYOND “ACCURACY”

Mere “accurate prediction” provided by the PPP would be insufficient to authorize a decision made on behalf of a patient. We want the decision to be his in a meaningful way, with a thread of authorization that connects the prediction to the person. If we already had sufficiently detailed knowledge about his values and previously endorsed actions relevant to the current medical decision—sufficient for us to feel that the prediction given by the PPP is really his—then, paradoxically, we would have no need for the PPP. And those situations in which PPP is supposed to be most useful—that is, when we are stumped about what someone would have wanted—are precisely when PPP by itself will not be able to carry the weight of authorization. I suspect that it is the need for this thread of authorization that leads the authors of the PPP to suggest obtaining in advance a person’s preferences (specifically, an opt-out preference) regarding the use of PPP in order to justify its use. This is a puzzling suggestion because PPP’s main use is presumably for those unlikely to leave any advance clues, and because the futility of promoting advance directives is one of the reasons for proposing the PPP. But the authors’ intuition that something beyond accurate prediction is needed is correct. Surrogate decision making is not, as the authors of PPP put it, “an indirect way to respect patient autonomy” (Rid and Wendler, 2014a, 6). For expressing one’s values about an uncertain future, entrusting one’s loved ones with future decisions is a direct expression of one’s autonomy that also happens to be a pretty good way to get what one wants.