Physicians' perceptions of the value of prognostic models: the benefits and risks of prognostic confidence

Abstract

Background

The communication of prognosis in end‐of‐life (EOL) care is a challenging task that is limited by prognostic uncertainty and physicians' lack of confidence in their prognostic estimates. Clinical prediction models (CPMs) are increasingly common evidence‐based tools that may mitigate these problems and facilitate the communication and use of prognostic information in EOL care; however, little is known about physicians' perceptions of the value of these tools.

Objective

To explore physicians' perceptions of the value of CPMs in EOL care.

Design

Qualitative study using semi‐structured individual interviews which were analysed using a constant comparative method.

Setting and participants

Convenience sample of 17 attending physicians representing five different medical specialties at a single large tertiary care medical centre.

Results

Physicians perceived CPMs as having three main benefits in EOL care: (i) enhancing their prognostic confidence; (ii) increasing their prognostic authority; and (iii) enabling patient persuasion in circumstances of low prognostic and therapeutic uncertainty. However, physicians also perceived CPMs as having potential risks, which include producing emotional distress in patients and promoting prognostic overconfidence in EOL care.

Discussion and conclusions

Physicians perceive CPMs as a potentially valuable means of increasing their prognostic confidence, communication and explicit use of prognostic information in EOL care. However, physicians' perceptions of CPMs also indicate a need to establish broad and consistent implementation processes to engage patients in shared decision making in EOL care, to effectively communicate uncertainty in prognostic information and to help both patients and physicians manage uncertainty in EOL care decisions.

Introduction

Prognostication is a fundamental clinical activity and a common concern for patients and clinicians.1, 2, 3 In one study, nearly 90% of internists reported that a patient asked ‘how long do I have to live’ in the last year.4 Patients and clinicians need prognostic information to share in advance care planning and informed decision making about medical interventions at the end of life. Yet, physicians have been shown to be reluctant to engage in prognostication, in part because they lack confidence in the accuracy of their prognostic judgments.5, 6, 7, 8 This lack of prognostic confidence is likely justified, given that physicians' prognostic estimates have been shown to be inaccurate and optimistically biased.9

A potential solution to this problem is to provide clinicians with accurate, evidence‐based methods of estimating prognosis. To this end, there have been growing efforts to develop clinical prediction models (CPMs) – multivariate statistical algorithms that utilize characteristics of patients, diseases and treatments to estimate individualized probabilities of future health outcomes including survival and mortality.10 CPMs provide a needed means of modulating the known inaccuracies and biases in physicians' own prognostic estimates, which may increase their confidence in prognostic estimates.

Yet, lack of prognostic confidence is not the only factor limiting physician's participation in prognostication. Other barriers include lack of physician time or resources for discussing the complexities of end‐of‐life (EOL) care, and physicians' concerns that communicating a poor prognosis may upset patients and cause them to perceive physicians as abdicating their role as healer and taking away hope.11, 12 The ability of CPMs to overcome these barriers may be limited. This was illustrated by the landmark SUPPORT study, which showed that provision of evidence‐based prognostic information does not necessarily enhance the use or communication of prognostic information in clinical care.13

Thus, a number of unanswered questions remain regarding the clinical value of new evidence‐based CPMs in EOL care. Will CPMs increase physicians' prognostic confidence? If so, will this increased prognostic confidence alter physician engagement in prognostication – including the communication of prognostic information to patients and its use in decision making at the end of life?

The overall objective of this study was to use qualitative methods to explore the answers to these questions from the perspective of physicians. This study examined physicians' perceptions of the value of CPMs in EOL care, using a recently developed CPM, the Patient‐Reported Outcome Mortality Prediction Tool (PROMPT), as an exemplar.14 The study was undertaken both to inform further development and implementation of the PROMPT and to understand physicians' perceptions of the acceptability and appropriate use of CPMs more generally in EOL care.

Methods

Study design, participants and data collection

The study employed individual semi‐structured in‐depth qualitative interviews, conducted from June to August 2010, with a convenience sample of 17 physicians representing five different internal medicine specialties: Cardiology (3), General Internal Medicine/Geriatrics/Hospital Medicine (7), Pulmonary/Critical Care (3), Nephrology (1) and Haematology/Oncology (3). All participants were affiliated with Maine Medical Center (MMC), a large tertiary care medical centre in Portland, Maine, and were identified by local practice leaders and by study participants themselves using snowball recruitment methods. We excluded physicians who spent more than 50% of their time in non‐clinical activities (e.g. research, administration) and trainees (residents, fellows). The study was approved by the MMC Institutional Review Board.

Interviews lasted between 30 and 60 min and were conducted in‐person by the same experienced interviewer (PH) using an interview guide that began with open‐ended questions regarding participants' attitudes and practices regarding the communication of prognostic information to patients. Participants were then asked what sources of prognostic information they use and their opinions of the value of evidence‐based prognostic tools such as CPMs. Several participants reported use of prediction models in their practice. Examples included mortality prediction tools such as the Seattle Heart Failure Model,15 the Flacker nursing home mortality models,16 and the APACHE II.17 Clinical outcome prediction tools such as the FRAX^™,18 CHADS₂ 19 and the Framingham Risk Calculator20 were also used. Participants were also provided with a brief description of the PROMPT – a new CPM that utilizes patient‐reported outcome information to predict 6‐month mortality in general medical patients aged 65 and over – to use as another example.14 The description of the PROMPT included a list of its 11 predictor variables and basic information on its accuracy. Participants were then asked about their perceptions of the potential value, appropriate use and potential barriers to use of not only the PROMPT but other similar prognostic CPMs in clinical practice. During the course of the study, minor revisions were made to the interview guide to explore emergent and unanticipated themes.

Data analysis

All interviews were audio‐recorded and transcribed verbatim by a professional transcription service. Audiofiles and notes were stripped of personal identifying information; participants were subsequently referenced only by generic descriptors (e.g. ‘Cardiologist 1’, ‘Geriatrician 2,’ etc.). Three investigators (SH, NH and PH) then performed in‐depth analysis and line‐by‐line software‐assisted coding of all interview transcripts using the program NVivo® (Version 9; QSR International, Melbourne, Australia) using a ‘constant comparative’ method.21, 22 Participants' verbatim statements were categorized by thematic content, and emergent themes were organized hierarchically within an overall conceptual schema. The analysis of interview transcripts employed an inductive ‘grounded theory’ approach in which the study investigators strived to interpret the data with minimal pre‐conceptions, allowing important themes and insights to emerge and to build new theoretical insights from them.23, 24

Two investigators (NH and PH) conducted initial analysis of three transcripts and developed a preliminary interpretive codebook, which was iteratively reviewed and revised by the research team. Three investigators (SH, NH and PH) then reapplied the revised codebook to the interview text. Coding decisions were compared, new themes were identified, and areas of disagreement were resolved through further discussion among study team members.

Results

Characteristics of the 17 participating physicians are presented in Table 1. Sixteen of the 17 physicians interviewed reported that prognosis was central to their practice. Only one internist reported that he used prognostic information ‘zero percent of the time’ because ‘it didn't come up’. However, the analysis of the transcripts revealed otherwise high homogeneity and a general lack of dissent. Several major themes were identified, the first relate to perceived benefits of CPMs.

Table 1.

Physician characteristics

Participant characteristics	n	%
Subspecialty
Cardiology	3	18
General Internal Medicine/Geriatrics/Hospital Medicine	7	41
Pulmonary/Critical Care	3	18
Nephrology	1	5
Haematology/Oncology	3	18
Sex
Male	13	76
Female	4	24
Race
Caucasian	17	100
Years in practice
≤5	4	23.5
6–15	6	35
16–25	3	18
≥26	4	23.5
Average number of patients per week
≤25	2	13
26–50	6	35
51–75	4	24
76–100	3	18
101–125	1	5
≥126	1	5
% office or non‐hospital based		76
% time spent in direct patient care		64
% patients 65+ seen in practice		56

Benefits of CPMs: Increasing prognostic confidence

Similar to findings of past studies,4 physicians reported that they lacked confidence in estimating prognosis:

Geriatrician 1: I have learned first of all that I can't predict life expectancy. I can predict to some extent when I know that a person [is actively dying], but I can't even say how long that is going to last. I have seen many times that the medical establishment is wrong….

Nephrologist 1: But every now and then I'm dead wrong, just dead wrong, in the hospital and in the dialysis clinic and it's always humbling.

Lack of confidence in prognostic estimates, in turn, appeared to negatively influence the communication of prognostic information. Consistent with findings from past studies,8 several physicians reported that lack of prognostic confidence led them to be reluctant to discuss prognosis:

Cardiologist 3: People often ask me am I going to be here in 6 months or not? [I've said no and] then I walk into them on the street – bump into them! And they're still alive … so I've found that my own ability to make that prediction is not so good, so I try to avoid answering the question.

Geriatrician 1: We as doctors are only human and we can't predict the future. There are too many cases in life where we are wrong.

Prognostic uncertainty, furthermore, appeared to also affect the precision of physicians' prognostic communication. Physicians reported that even when they were inclined to communicate prognostic information, uncertainty caused them to avoid use of precise numeric terms. One internist reported that ‘I will often hedge because I don't honestly know’. Other physicians reiterated this theme:

Pulmonogist 1: … the predictions that I try to give are intentionally somewhat vague… I try to avoid specific numbers, and in the ICU, I try to be more qualitative than quantitative in the predictions. In that way, I hope to avoid gross inaccuracies.

Geriatrician 3: I had the experience of giving someone a timeframe once and having them call me back to let me know that I was wrong … I have learned that people will fixate on the number and that is not really what it is about. [It's about] knowing that the time is limited. I do not know if it is month or 6 months, but I know that it is getting shorter, and they will need to figure that out. So, I tend not to commit to a specific number.

Physicians perceived CPMs as a potential means of increasing their prognostic confidence by providing evidence‐based prognostic data that could assist in clinical decision making and communication with patients:

Geriatrician 4: I see it more as a tool to further enable the physicians… to have more confidence with doing it and saying, look, this is where your mom or your dad are or where you are.

Cardiologist 2: I think there is a need for developing a prognostic tool … to give the physician the confidence that that person should be in a hospice program. Much like we have confidence when we send a person for Ventricular Assist Device. Much like we have confidence when we send a person for transplant … It's a big black box right now when we're trying to figure out who to get into hospice and when to get them into hospice… It would be to try to prove to myself that making the hospice referral is the right one.

The prognostic confidence afforded by CPMs appeared to reflect a trust in the scientific evidence base underlying the tools themselves. Physicians viewed evidence‐based prognostic estimates produced by CPMs as superior to their own experience‐based estimates:

Pulmonologist 1: I am a firm believer in systematic type of population‐based tools to assist in practice. I mean, that's sort of what the whole of the ICU is designed to do and I guess another analogy that pops into my mind in this regard is the evidence on protocolized weaning assessment and extubation … when you actually assess everybody in a systematic, structured way every day… you have much better outcomes.

Even though prognostic estimates produced by CPMs were judged superior to physicians' own estimates, physicians felt CPMs had more utility as a confirmatory tool, validating or fine‐tuning rather than substantially changing their pre‐existing estimates:

Hospitalist 1: I think it would be very helpful [to have a tool] and I have to say off the top of my head I would [use it] mostly for either confirming or not confirming my clinical instinct.

Internist 2: to actually have numbers come out of this … [CPM] would improve what you're already doing or add to it. I guess it would fine tune it.

A few even felt that CPM evidence could cause them to correct their prognostic impressions or overcome a more general tendency to ignore or overestimate prognosis:

Geriatrician 4: I'm just looking for more ways to validate what I'm seeing with patients or give me some contradictory information, you know to have another look at the patient, because maybe I am missing something entirely or maybe I'm seeing too much.

Geriatrician 3: If I were in primary care practice and ran this tool on all my patients over 80, you know it would seem like maybe every now and then something surprising would pop up but then you might go, huh, I wonder why, and then …. it might make you look at their medications a little differently, it might make you talk with them a little bit differently… that sort of thing…

The overarching theme in all interviews, however, was the perception that CPMs were valuable scientific tools that increase prognostic confidence and thereby encourage communication of prognostic information and its use in clinical decision making.

Benefits of CPMs: enhancing prognostic authority

In addition to increasing prognostic confidence, physicians valued CPMs as a means of enhancing their prognostic authority. Several physicians reported a lack of perceived authority as a barrier to prognostication, especially in circumstances where a long‐standing relationship with a patient or family was absent. Physicians felt that this lack of authority was significant and diminished their capacity to influence decision making and that CPMs could augment their credibility and authority by confirming their prognostic estimates. They viewed CPMs as providing a de facto independent confirmatory opinion in EOL care discussions, especially in situations in which trusting relationships with patients and family members were yet to be developed.

Geriatrician 3: I know some of my colleagues use [CPMs] particularly when they are in contact with a patient whom they don't know well. It gives some additional support to what they are saying that it is not just my opinion but we have got this tool we can use and it is saying the same thing I am.

Geriatrician 2: I mean if a relationship with people that goes from between 15 and 30 years, to then talk … is pretty strong, as opposed to the hospitalist who comes in for the first visit and has a zero relationship and I think just them giving their opinion about qualitative things is a much more difficult discussion… it's hard to have these discussions without really having strong quantitative data…

Another perceived value of CPMs was their ability to increase prognostic authority by reducing ambiguity – i.e., the presence of conflicting opinions about prognosis, especially among physicians:

Hospitalist 1: [A tool] would be particularly helpful in situations where I didn't agree with the patient and or the patient didn't know what they wanted or a subspecialist didn't want to do the procedure…where again a big decision was overdue and it might help push us in one direction or the other….Or I thought someone looked just barely alive and [it] told me they had a good chance of survival I'd think – what am I missing here? Maybe I need to scale back my assessment.

Geriatrician 4: The surgeon says he's not a surgical candidate and if you're able to say actually he has a really good survival… I mean I don't think would be swayed necessarily by this alone but it would certainly perhaps fortify one's own resolve in presenting it again as something that should be considered.

Cardiologist 1: I don't want to do it, ‘cause they don't look like they're a great candidate, but I'll do it because they meet the criteria. And then you say but wait, this says they have a 90% chance of being dead in 6 months, I'd say, ‘Praise the lord I'm out of here’….

The point was also made that the authority CPMs provide may even mitigate medicolegal risk if prognostic estimates and any resulting decisions proved incorrect:

Cardiologist 1: I was thinking [that prognostic tool data could be helpful] where the level of litigiousness is higher and the family of that 89‐year old just very well might sue you if they drop dead at the age of 90 and you didn't put in an implantable defibrillator. If you could just say: ‘this says that they didn't qualify,’ then I could see that has having some advantage.

Benefits of CPMs: Enabling patient persuasion

Physicians also saw value in the ability of CPMs to reassure and persuade patients to make medical decisions. Similar to the concept whereby CPMs can increase prognostic authority by providing a de facto independent confirmatory opinion, physicians also viewed CPMs as offering a level of proof that could facilitate patients' agreement on particular courses of action when clinical circumstances seemed clear:

Cardiologist 3: I think it would be most useful if I felt strongly that a patient ought to have a certain intervention but the patients didn't feel that way and I really felt like trying to push the patient in certain direction. I could give them some data that said, you know, look, somebody has done this study and these are the outcomes…

Internist 1: … families think, you know, yeah Granny's old, but she still lives on her own and she's kind of getting by ‘cause we're helping her out. [Families] may have a 5‐year plan in their brain and I may be looking at this patient and saying, man, you know, this ain't 5 years. This is 6–12 months or something like that and having a way to convince people of that may be very helpful.

Physicians' perceptions of the persuasive value of CPMs reflected increased confidence in their own prognostic judgments and therapeutic recommendations when validated by CPMs. Physicians reported that the prognostic confidence provided by CPMs would empower them to take a more directive role in cases in which a right decision truly existed but was somehow unable to be discerned by patients – when patients ‘really just aren't seeing the forest for the trees’, as one oncologist put it.

Cardiologist 3: For the most part, though, I really try to get a sense of what the patient wants and what the family wants … I try to respect that. It's really when I think they're making a mistake in judgment. I think there are situations where sometimes they think they're too old for a procedure but really they don't have a lot of other morbidity and the risks probably are pretty good to have something done. So to me it might be helpful as one other factor that could point them in certain direction.

Oncologist 3: … the model would help me have an evidence base to be able to say to the patient, look, this reliable model says your life expectancy, factoring out the cancer, is already very limited… And it would help them understand the situation and make a treatment decision about the cancer…this model would give me a way to have something to point to help make that case. It would help me help the patient see the situation more clearly; it would help persuade them…it would push them to make a decision.

Importantly, the clinical circumstances raised by each of these physicians were one of low prognostic uncertainty – where future outcomes and the optimal course of action appeared clear.

Risks of CPMs: emotional distress

Although physicians perceived mostly benefits of using CPMs in EOL care, they also perceived two important risks. The first was the potential for prognostic estimates produced by CPMs to cause emotional distress or panic in patients when estimates were not favourable. Consistent with a large body of evidence, most physicians perceived that such prognostic information would be upsetting to patients and families. Physicians saw disclosure of a poor prognosis in numeric terms as particularly distressing given patients' tendency to ‘fixate on the number’, as described previously by one geriatrician and another cardiologist:

Cardiologist 1: You can tell somebody [that] you have two chances in three of being dead in 6 months…you know they walk out of the office wondering what they should do tomorrow.

Several physicians traced the source of distress to patients' general discomfort with the subject of death and dying. One physician expressed concerns that this discomfort might lead some patients to perceive the very act of a physician initiating a prognostic discussion as inappropriate:

Cardiologist 2: And then there would also be…the perception by the patient that before you even fill this out, that the physician has given them the death question. The death exam, you know. There would be some sort of…death test.

Risks of CPMs: prognostic overconfidence

Physicians also expressed concerns that CPMs may promote prognostic overconfidence – that is, excessive prognostic certainty – about future events on the part of patients. They worried that patients would not be able to appreciate limitations in the sensitivity and specificity of a CPM and the applicability of prognostic estimates to individual patients.

Pulmonologist 3: So – I worry that too much specificity in the information given will lead to misunderstandings on some part of the patient or family. You know, you tell them they have a 40% chance of living the next 6 months they go, oh, I'm going to die in the next 6 months and they go and sell all their belongings.

A related concern pertained to the consequences of clinical decisions based on prognostic estimates. One physician expressed the need for caution in the use of prognostic estimates in EOL care decisions because of the outcomes at stake (death due to the withholding or withdrawal of life‐sustaining care):

Critical Care Physician 2: In the ICU there's a very powerful self‐fulfilling prophecy of work which is: that if you prognosticate you have essentially ended their life, because you are dictating that aggressive care won't be taken and if aggressive care is not taken early on, they all die.

This statement acknowledges deeper problems that can result from the use of CPMs in EOL care. The mortal consequences of decisions based on estimates of poor prognosis make the endeavour of prognostication ethically daunting. But they also pose the risk of inflating the apparent accuracy of this endeavour, and thereby reinforcing overconfidence in prognostic estimates.

Another concern related to overconfidence was that prognostic estimates from CPMs could be misused either for financial gain or to ration care at the end of life:

Oncologist 1: I mean [a tool] may even be used to discourage referral. Sort of a stairway to manage healthcare resources but I'm sure it's done a bit. Sort of a rationing tool….

Geriatrician 1: Well, it is hard enough to certify that somebody has a 6‐month prognosis for hospice… So suppose you refer somebody…and he doesn't meet the [tool's] criteria for some reason and they reject him because of that… I just worry that if he doesn't fit in that model then they can exclude it inappropriately.

Interviewer: Alright, so your worry is that then people would use it in a way that denied appropriate hospice care to certain patients.

Geriatrician 1: Primarily that. I suppose you could take it to another extreme and say that somebody is going to go fishing for hospice patients with this model too.

Discussion

This qualitative study yields new insights into physicians' perceptions of the value of CPMs in EOL care. To our knowledge, it is the first study to explore physicians' attitudes towards the use of CPMs in this setting and has important implications regarding the optimal approach to implementing these tools.

Consistent with past studies, our study suggests that lack of prognostic confidence is a primary cause of physicians' reluctance to communicate prognostic information5, 6, 7, 8 and that CPMs may be an effective means of enhancing this confidence. Physicians view CPMs as providing critical evidence that can validate and confirm their own prognostic judgments, enhance their prognostic authority and empower them to recommend clinical actions that they believe are indicated. In these ways, CPMs may increase the extent to which physicians communicate and ultimately use prognostic information in EOL care.

Yet, these same findings also highlight several concerns regarding the use of CPMs. The added prognostic confidence afforded by CPMs could be detrimental if it inhibits shared decision making – that is, if it simply empowers physicians to make decisions without involving patients and encourages persuasion instead of discussion. Physicians in our study were careful to specify that the circumstances in which they would utilize CPMs for the purpose of patient persuasion were ones of low prognostic and therapeutic uncertainty – that is, when the prognosis and right course of action were clear but not discerned by patients. Yet as these physicians also acknowledged, prognostic judgments are sometimes ‘dead wrong’ and the right course of action is often unclear. Our findings thus highlight the need to implement CPMs as part of a more comprehensive effort aimed at ensuring shared decision making in EOL care. This effort requires establishing care processes that go beyond the provision of prognostic information and that include involving patients in decision making, discussing the benefits, harms and uncertainties associated with alternative treatment options, and eliciting patients' values and preferences.

A related need raised by our study is to prevent prognostic overconfidence. The physicians we interviewed recognized that the use of CPMs might promote a level of confidence in prognostic estimates that is unwarranted given the limitations of predictive modelling. Corroborating this concern, past studies have shown that people tend to attribute certainty to numeric estimates and fail to understand fundamental uncertainties inherent to risk information.25, 26 These include ‘aleatory’ uncertainty that arises from irreducible randomness or indeterminacy of all future events27 and ‘epistemic’ uncertainty (also known as ‘ambiguity’) that arises from limitations in the reliability, credibility or adequacy of risk information.27, 28 These uncertainties limit the value of prognostic estimates and need to be effectively communicated to patients in order to prevent prognostic overconfidence in the use of CPMs.29

As study participants noted, furthermore, the need to communicate uncertainty is especially great given that prognostic estimates have potentially serious, mortal consequences. Estimates of poor prognosis can result in death from the withholding or withdrawal of life‐sustaining treatments, while estimates of good prognosis lead to continued treatment towards life‐sustaining goals. Both consequences may be inappropriate if the prognostic estimates themselves are inaccurate1 and raise the need to ensure that clinicians and patients understand the inherent uncertainty of prognostic estimates and the implications of this uncertainty for treatment decisions at the end of life.

Several limitations of our study qualify its conclusions. The study employed qualitative methods, a useful approach to understand the nature and breadth – but not the prevalence or causes – of people's beliefs and attitudes. The study sample consisted of a convenience sample of a relatively small number of physicians from a single institution. The physicians did represent a broad variety of medical specialties but had limited diversity in sociodemographic characteristics and practice experience – factors known to be related to the extent of physicians' disclosure of prognostic information to patients.5, 8, 11, 30 Additional research in more diverse, representative physician populations is needed to assess the validity of our findings. A potential selection bias was also introduced by how physicians were recruited. Participants were identified either by other subjects or by practice leaders. This may have led to an increased sampling of like‐minded individuals interested in prediction models and concerned about prognostic communication, which may limit the generalizability of our findings. Finally, we only studied physicians' hypothetical responses to the concept of using CPMs in EOL care; physicians' actual practices may diverge from these responses. Future research needs to quantify the real impact of CPM use on physicians' perceptions and practices.

Despite these limitations, our data have important implications for the implementation of CPMs in EOL care. This study suggests that physicians are generally receptive to use of these new tools, particularly in situations in which they lack prognostic confidence. At the same time, our data suggest that the added confidence provided by CPMs has potential risks that call for broad and consistent implementation processes aimed at engaging patients in shared decision making in EOL care, and helping patients and physicians communicate effectively about uncertainty in prognostic estimates.

Funding

Dr. Hallen was supported by HRSA grant K01HP20461.

Conflicts of interest

None.