The Inability to Calculate Predictive Values: an Old Problem that Has Not Gone Away

Abstract

Background

Previous studies suggest that many physicians and medical trainees have trouble calculating the probability that a patient has a condition, also known as the predictive value.

Activity

Two questions from the medical literature were administered to medical students, residents (post-medical school), fellows (post-residency), and faculty physicians.

Results

Only 6% answered both questions correctly. Most commonly, the participants grossly overestimated the probability of disease.

Discussion

Physicians still struggle with the ability to calculate predictive values, a skill that affects all branches of medicine and will become more consequential as new tests are administered to patients at low risk for disease.

Background

It has been nearly 100 years since George W. Peabody, MD, wrote, “Good medicine does not consist in the indiscriminate application of laboratory examinations to a patient, but rather in having so clear a comprehension of the probabilities of a case to know what tests may be of value.” [1] We are now in an era of rapidly evolving new tests (e.g., genetic testing), which can be applied indiscriminately, making Peabody’s statement even more relevant today. A small study published in the New England Journal of Medicine in 1978 demonstrated that physicians have difficulty quantifying a patient’s probability of disease even when given a categorical test result and all other necessary information. [2] In that study, 20 physicians, 20 residents, and 20 medical students at Harvard teaching hospitals were asked the following question: If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s symptoms or signs? Only 18% answered the open-ended question correctly. In July 2013, the same question was asked of a convenience sample of physicians, residents, and medical students (n = 61) at a Boston area hospital with 23% answering correctly. [3]

In another question from the medical archives, Gigerenzer queried 160 gynecologists [4] prior to a continuing-education session with the following scenario: Assume you conduct breast cancer screening using mammography in a certain region. You know the following information about the women in this region:

• The probability that a woman has breast cancer is 1% (prevalence).

• If a woman has breast cancer, the probability that she tests positive is 90% (sensitivity).

• If a woman does not have breast cancer, the probability that she nonetheless tests positive is 9% (false-positive rate).

A woman tests positive. She wants to know from you whether that means that she has breast cancer for sure, or what the chances are. What is the best answer?

• A. The probability that she has breast cancer is about 81%.
• B. Out of 10 women with a positive mammogram, about 9 have breast cancer.
• C. Out of 10 women with a positive mammogram, about 1 has breast cancer.
• D. The probability that she has breast cancer is about 1%.

With only four choices, random guessing would be correct 25% of the time. Yet, only 21% (34 out of 160) answered C, the correct answer.

Using different terminology, the two questions mentioned above asked participants to determine the predictive value, which some term the post-test probability. The predictive value is easily calculated if one is given the pre-test probability, the test accuracy, and the test result. Undoubtedly, these criteria are often unknown when evaluating patients. However, when these criteria are known, such as in the above scenarios, then patient-centered care would be best served if physicians were able to apply the information to the individual patient. The need to understand predictive values impacts all branches of medicine, from primary care to neurosurgery, and is vitally important in the quest for participatory shared medical decision-making. [5, 6] Furthermore, the ability to calculate predictive values will become increasingly consequential as clinicians aim to individualize diagnostic tests and treatments with the evolution of “precision medicine.” [7] Precision medicine is based on the premise that there’s individual variation between patients; each patient has different risks for different conditions. A synonym for “risk” in this context is the pre-test probability, a necessary component in the calculation of predictive values.

Assessment of the ability to calculate predictive values is important for medical science educators teaching clinical decision-making. Replication studies, especially by different researchers and in different environments, are needed for scientific progress. [8] In order to assess the ability of current medical students, residents, and physicians (in family medicine and orthopedic surgery) to calculate predictive values, the above questions were replicated at the University of Minnesota medical school.

Activity

The study involved a convenience sample of 51 medical students, residents (post-medical school), fellows (post-residency), and faculty physicians. The students were all on clinical rotations after having completed at least 2 years of medical school. The two questions were entered into a password-protected secured platform through the University of Minnesota’s Qualtrics system. [9] Three elected to not answer the second question. With the hope of increasing efficiency and making the participants more comfortable, the question from the 1978 paper in NEJM was modified slightly so that participants had multiple choice options. A “Quick-Response” (QR) matrix barcode was provided so that attendees could use their cell phones to access the questions and provide responses. I administered the questions prior to invited presentations (on unrelated topics) at our University’s medical residency training programs between February 26, 2018, and February 27, 2019. Fellows (post-residency) and faculty physicians were lumped together for the analysis. The study was determined exempt from IRB review.

Results

The questions and responses are seen in Figs. 1 and 2. Eleven out of 51 (22%) answered question 1 correctly and 14/48 (29%) answered question 2 correctly. Of note, only 3 of the respondents answered both questions correctly. For question 1, the majority (29/51, 57%) incorrectly assumed that the person had a 95% probability of having the condition, a gross overestimation from the 2% probability. For question 2, the most common answer was that there was a 9 in 10 chance of the woman having breast cancer when the actual probability was only 1 in 10—again, a gross overestimation0.

Fig. 1.

Question 1 with responses

Fig. 2.

Question 2 with responses

Discussion

This assessment of a convenience sample of medical students, residents, and physicians found that few were able to calculate a predictive value. Presented with two questions from previously published studies, correct answers were provided in only 22% and 29% of responses, respectively. Multiple choice options were provided for both questions so some correct answers may have been due to chance. Only 3 of 48 (6%) answered both questions correctly suggesting that few are fluent in the calculations.

The findings were consistent with past studies showing that medical trainees and physicians have difficulty calculating predictive values. In this study, the medical students fared the worst. This may be due to less experience, but, as noted by one of the reviewers, this was surprising given their current medical science education. Because the total number of subjects was small, no formal comparisons were made between the groups.

Since diseases/conditions tend to be rare, such as in the two questions (where the prevalence rates were 1 in 1000 and 1 in 100), the miscalculations tend to grossly overestimate the probability of disease for the individual patient. For example, in question #1, rather than informing someone that their probability of disease was about 2 in 100, the majority would inform the patient that their probability of disease was 95 in 100. And, in question #2, rather than informing a woman that her probability of breast cancer was 1 in 10, the majority would inform the woman that her probability was 9 in 10. In an actual clinical setting, many may prefer to convey more qualitative and less quantitative information (e.g., “you may have…” or “your test results are concerning for…”). The best methods for risk communication are beyond the scope of this manuscript. [6, 10–13] Regardless of one’s communication style, it would be ideal if the physician knew an approximate probability of disease in cases when that information is known.

The effect of miscalculating predictive values is unclear. The shared decision-making process asks physicians to inform patients of their risks and treatment options in an understandable format so that the patient may participate in an informed decision. [14] Driever et al. studied 232 physicians and found that most physicians appreciated the shared decision-making perspective but then often reverted to a paternalistic approach. [15] Inability to calculate predictive values (a primary aspect of clinical statistical literacy) may impede the practice of shared decision-making. [16, 17] Lack of knowledge likely discourages physicians from engaging in specific discussions with patients about the probability of disease. [18] Furthermore, for those who do the calculation inaccurately, the overestimation of risk when a patient has a positive test for a rare condition may be translated into recommendations, such as “more testing or treatment is needed.” It is reasonable that further testing and treatment would be recommended more commonly when the physician believes that a positive test signifies a 95% probability rather than a 2% probability of disease.

The study was limited by the fact that it was done at just one institution and in an informal setting. I encourage others to replicate these questions (and/or similar questions) in larger venues and with more systematic recruitment. If future inquiries also find respondents unable to answer questions correctly, then efforts must focus on education. I agree with recommendations that pre-medical training in statistics would likely be more beneficial than calculus. [3] Once in medical school, training in statistical literacy is needed. This should move beyond the generalities of test qualities (sensitivity and specificity) and, instead, teach students to incorporate prevalence/pre-test probability into their calculations. Novel teaching devices, such as the development of an educational mobile phone application, may prove helpful. An expertly designed mobile phone application with a user-friendly interface may serve as both an effective teaching modality and as a means to assist busy clinicians with calculations and patient-centered communication.

Future studies are needed to determine whether training in quantitative reasoning results in improved physician-patient communication. Other questions for future studies include the following:

• Would a patient choose differently if presented with further testing or treatment options based on different disease probabilities?

• If the patient and provider would choose differently, what might be the effects on medical costs and outcomes including side effects and adverse events?

• Is there a better method to teach physicians how to calculate predictive values?

One thing is clear; we cannot optimize patient-centered care or the shared decision-making process if the patient is provided with incorrect information.

Compliance with Ethical Standards

The study was determined exempt from IRB review by the criteria of the University of Minnesota’s institutional review criteria.

Informed Consent

N/A.

Conflict of Interest

The author declares that he has no conflict of interest.