Extent and determinants of error in doctors' prognoses in terminally ill patients: prospective cohort study

Abstract

Objective

To describe doctors' prognostic accuracy in terminally ill patients and to evaluate the determinants of that accuracy.

Design

Prospective cohort study.

Setting

Five outpatient hospice programmes in Chicago.

Participants

343 doctors provided survival estimates for 468 terminally ill patients at the time of hospice referral.

Main outcome measures

Patients' estimated and actual survival.

Results

Median survival was 24 days. Only 20% (92/468) of predictions were accurate (within 33% of actual survival); 63% (295/468) were overoptimistic and 17% (81/468) were overpessimistic. Overall, doctors overestimated survival by a factor of 5.3. Few patient or doctor characteristics were associated with prognostic accuracy. Male patients were 58% less likely to have overpessimistic predictions. Non-oncology medical specialists were 326% more likely than general internists to make overpessimistic predictions. Doctors in the upper quartile of practice experience were the most accurate. As duration of doctor-patient relationship increased and time since last contact decreased, prognostic accuracy decreased.

Conclusion

Doctors are inaccurate in their prognoses for terminally ill patients and the error is systematically optimistic. The inaccuracy is, in general, not restricted to certain kinds of doctors or patients. These phenomena may be adversely affecting the quality of care given to patients near the end of life.

Introduction

Although doctors commonly have to prognosticate, most feel uncomfortable doing so.¹ Neither medical training¹,² nor published literature³,⁴ treat prognostication as important, and prognostic error is widespread.² Unfortunately, prognostic error may have untoward effects on both patient care and social policy.

Parkes showed that doctors' predictions of survival in 168 cancer patients were often erroneous and optimistic,⁵ and these findings were confirmed by subsequent studies.^6–10 However, previous work has been limited by use of small samples of patients and very small samples of prognosticators (typically fewer than four); failure to examine whether certain types of doctors are more likely to err in certain types of patients; and neglect of the possibility of different determinants of optimistic and pessimistic error. Therefore, we conducted a large, prospective cohort study of terminally ill patients to evaluate the extent and determinants of prognostic error.

Participants and methods

Our cohort consisted of all patients admitted to five outpatient hospice programmes in Chicago during 130 consecutive days in 1996. Participating hospices notified us about patients on admission, and we immediately contacted the referring doctors to administer a four minute telephone survey. Of the 767 patients (referred by 502 doctors), 65 did not meet the entry criteria (they were children, were denied hospice admission, or refused to give consent) and 51 died before we were notified (and thus survival predictions would be meaningless). Of the remaining 651 patients, for 66 (10%) we contacted the doctor only after the patient's death (and so could not get meaningful prognoses), for 14 (2%) the doctor refused to participate, and for 67 (10%) the doctor could not be contacted. We thus completed surveys with 365 different doctors caring for 504 patients (504/651=77%). Comparison of these 504 patients with the 147 excluded patients showed no important differences in patient or doctor characteristics. On 30 June 1999 we had dates of death for 486 of the 504 patients (96%). Because data were occasionally missing, not all totals in the analyses are equivalent.

We obtained the patients' age, sex, race, religion, marital status, diagnosis, and comorbidities from the hospice. From the survey, we obtained an estimate of how long the patient had to live; information about the patient, including Eastern Cooperative Oncology Group performance status¹¹ and duration of illness; information about the doctor, including experience with similar patients and self rated dispositional optimism; and information about the doctor-patient relationship, including the duration, recentness, and frequency of contact. We obtained other data on the doctors, such as specialty, years in practice, and board certification from public records. Dates of patients' deaths were obtained from public death registries or the hospices.

We divided the observed by the predicted survival, and deemed prognoses “accurate” if this quotient was between 0.67 and 1.33. Values less than 0.67 were “optimistic” prognostic errors and those greater than 1.33 were “pessimistic.” We conducted analyses using different cut off points or more categories, as well as analyses that treated this quotient as a continuous measure, but these analyses did not contravene the results presented. To evaluate associations between categorical and continuous variables and the trichotomous prognostic accuracy variable, we used χ² tests and analysis of variance respectively. We used multinomial logistic regression to assess the multivariate effect of patient and doctor variables on prognostic accuracy.

Results

The patients had a mean age of 69 (SD 17) years and 225/504 (45%) were men. The diagnosis was cancer in 326 (65%), AIDS in 62 (12%), and other conditions in 116 (23%). The mean duration of disease was 83.5 (135.8) weeks, and the median performance status was 3 (corresponding to >50% of the day spent bedridden). The doctors had a median duration of medical practice of 16 years; 291/363 (80%) were men; 293/365 (80%) were board certified; and 255/345 (74%) rated themselves optimistic. A total of 114/358 (32%) specialised in general internal medicine, 71/358 (20%) in non-oncological internal medicine subspecialties, 61/358 (17%) in oncology, 55/358 (15%) in family or general practice, 27/358 (8%) in geriatrics, and 30/358 (8%) were surgeons or practised other specialties. In the past year, the doctors had had experience caring for a median of five patients with the same diagnosis and had referred a median of eight patients to a hospice. They had known the patient an average of 159 (308) weeks; had 11 (14) contacts in the previous three months; and had examined the patient 14 (29) days before.

Doctors' prognostic estimates

In only 18 of 504 patients did the doctor refuse to predict survival to us. Eighteen of the remaining 486 had missing dates of death, leaving 468 cases referred by 343 doctors for analysis of prognostic accuracy. The figure illustrates the extent of the error. The median observed patient survival was 24 days. The mean ratio of predicted to observed survival was 5.3. The correlation between predicted and observed survival was 0.28 (P<0.01). When an accurate prediction was defined as between 0.67 and 1.33 times the actual survival, 20% (92/468) of predictions were accurate, 63% (295/468) optimistic, and 17% (81/468) pessimistic. When an accurate prediction was defined as between 0.50 and 2.0 times the actual survival, 34% (159/468) of predictions were accurate, 55% (256/468) optimistic, and 11% (53/468) pessimistic. Death occurred within one month of the predicted date for 42% (195/468) of patients, at least one month before the predicted date in 46% (214/468), and at least one month after the predicted date in 13% (59/468) of patients.

The extent of prognostic error varied depending on both observed and predicted survival (table). The longer the observed survival (that is, the less ill the patient), the lower the error, and, conversely, the longer the predicted survival, the greater the error.

Factors associated with prognostic accuracy

Bivariate analyses of the trichotomous accuracy variable and patient attributes showed no important differences with respect to patients' age, sex, race, religion, or marital status. However, cancer patients were the most likely to have overoptimistic predictions (220/301 (67%) v 37/58 (64%) of AIDS patients and 56/109 (51%) of other patients) and the least likely to have overpessimistic predictions (39/301 (13%) v 13/58 (22%) and 29/109 (27%)); AIDS patients were the least likely to have correct predictions (8/58 (14%) v 60/301 (20%) of cancer patients and 24/109 (22%) of others; P<0.01).

Bivariate analyses of the doctor attributes showed no important differences with respect to sex, years in medical practice, board certification, self rated optimism, number of hospice referrals in past year, or number of medically similar patients in the past year. However, doctors in non-oncological medical subspecialties were the least likely to give correct estimates (8/79 (10%) v 11/30 (37%) doctors in surgery or other, 18/76 (27%) in family or general practice, 24/105 (23%) in oncology, and 30/180 (17%) in geriatric or general internal medicine), and oncologists were the least likely to be overpessimistic in their estimates (10/105 (9%) v 21/79 (27%) in other internal medicine subspecialties, 13/67 (19%) in family or general practice, 31/180 (17%) in geriatric or general internal medicine, and 4/30 (13%) in surgery or other; P<0.01).

Among the doctor-patient relationship variables (such as length of professional relationship, number of recent contacts, time since last examination), the interval since last examination was important: overpessimistic predictions were associated with the most recent examinations (7.5 days), overoptimistic predictions with the next most recent examinations (13.8 days), and the correct predictions with the longest interval since physical examination (19.5 days); P<0.05.

The trichotomous prognosis variable was regressed on patients' age, sex, race, diagnosis, duration of disease, and performance status and on doctors' experience, sex, optimism, board certification, specialty, related practical experience, duration of relationship, number of contacts, and interval since last examination (full results available on the BMJ's website). The model showed that doctors' prognostic accuracy was independent of most patient and doctor attributes. However, after other attributes were adjusted for, male patients were 58% less likely to have overpessimistic than correct predictions (odds ratio 0.42; 95% confidence interval 0.18 to 0.99). Doctors in the upper quartile of practice experience were 63% less likely to make optimistic rather than correct predictions (0.37; 0.19 to 0.74) and 78% less likely to make pessimistic rather than correct predictions (0.22; 0.08 to 0.61). Doctors with medical subspecialty training (excluding oncologists) were 3.26 times more likely than geriatricians and general internists to make pessimistic rather than correct predictions (3.26; 1.01 to 10.7). As the duration of the doctor-patient relationship increased, so too did the doctor's odds of making an erroneous prediction—for example, each one year longer that the doctor had known the patient resulted in a 12% increase in the odds of an overpessimistic prediction (1.12; 1.02 to 1.22). Also, as the interval since last physical examination increased, the odds of a doctor making a pessimistic rather than a correct prediction decreased; each day longer resulted in a 3% decrease in the odds (0.97; 0.94 to 0.99).

Discussion

Our study of 365 doctors and 504 hospice outpatients found that only 20% of prognoses were accurate. Most predictions (63%) were overestimates, and doctors overall overestimated survival by a factor of about five. These prognoses were doctors' best guesses about their patients' survival prospects, objectively communicated to the investigators and not to patients themselves. Close multivariate examination showed that most doctor and patient attributes were not associated with prognostic error. However, the tendency of doctors to make prognostic errors was lower among experienced doctors. Moreover, the better the doctor knew the patient—as measured, for example, by the length and recentness of their contact—the more likely the doctor was to err.

These findings have several implications. Firstly, undue optimism about survival prospects may contribute to late referral for hospice care, with negative implications for patients.¹²,¹³ Indeed, although doctors state that patients should ideally receive hospice care for three months before death,¹⁴ patients typically receive only one month of such care.¹⁵ The fact that doctors have unduly optimistic ideas about how long patients have to live may partly explain this discrepancy. Doctors who do not realise how little time is left may miss the chance to devote more of it to improving the quality of patients' remaining life. Secondly, to the extent that doctors' implicit or explicit communication of prognostic information affects patients' own conceptions of their future, doctors may contribute to patients making choices that are counterproductive. Indeed, one study found that terminally ill cancer patients who hold unduly optimistic assessments of their survival prospects often request futile, aggressive care rather than perhaps more beneficial palliative care.¹⁶ Thirdly, our work hints at corrective techniques that might be used to counteract prognostic error. Disinterested doctors, with less contact with the patient, may give more accurate prognoses, perhaps because they have less personal investment in the outcome.¹⁷ Clinicians may therefore wish to seek “second opinions” regarding prognoses, and our work suggests that experienced doctors may be a particularly good source of opinion. Finally, our work suggests that prognostic error in terminally ill patients is rather uniformly distributed. This has implications for doctors' training and self assessment since it suggests that there is not one type of doctor who is prone to error, nor is there one type of patient in whom doctors are likely to err.

What is already known on this topic

Doctors' prognostic estimates are a central element of both patient and physician decision making, especially at the end of life

Doctors' prognostic estimates in their terminally ill patients are often wrong and usually optimistic

What this study adds

A prospective cohort study of 504 terminally ill patients and their 365 doctors found that only 20% of the doctors' predictions were accurate: 63% were overoptimistic and 17% overpessimistic

Multivariate modelling showed that most types of doctors are prone to error, in most types of patients

The greater the experience of the doctor the greater the prognostic accuracy, but a stronger doctor-patient relationship is associated with lower prognostic accuracy

Obtaining prognostic information is often the highest priority for seriously ill patients, eclipsing their interest in treatment options or diagnostic details.¹⁸,¹⁹ And reliable prognostic information is a key determinant of both doctors' and patients' decision making.¹⁶,²⁰,²¹ Although some error is unavoidable in prognostication, the type of systematic bias towards optimism that we have found in doctors' objective prognostic assessments may be adversely affecting patient care.

Figure.

Predicted versus observed survival in 468 terminally ill hospice patients. Diagonal line represents perfect prediction. Patients above diagonal are those in whom survival was overestimated; patients below line are those in whom survival was underestimated

Table.

Doctors' overestimates of patient survival by observed and predicted survival

	% overestimate in survival (mean)	No of patients
Observed survival (days):
1-30	795	251
31-90	288	130
91-180	136	49
>180	71	38
Overall	526	468
Predicted survival (days):
1-30	192	150
31-90	382	144
91-180	501	119
>180	1872	55
Overall	526	468