Statistics and the Rise of Medical Fortunetellers

There was a time when the foretelling of future events was an undertaking of prophets, palm-readers, and weathermen. In recent years, however, the medical profession seems to have embraced this activity with a great deal of enthusiasm. A prime example is the use of the term “predicts” in the titles of journal articles dealing with human subjects. According to a search of the MEDLINE® database, “predicts” appeared a total of only 13 times before 1980.¹ Since then, however, the occurrence of the term in citation titles has increased dramatically. Expressed as a percentage of the annual number of MEDLINE publications, the trend follows a curve that could be described as hyperbolic (Fig. 1).

Fig. 1. Percentage of MEDLINE® publications on human subjects with the term “predicts” in the title.

But hyperbole is also what comes to mind when one examines the claims that authors of such reports make. For example, the article “Cardiac troponin I predicts short-term mortality in vascular surgery patients”² does not provide a method to identify the specific individuals who are headed for the funeral home. Nor does the report “Transesophageal echocardiography predicts mortality in critically ill patients with hypotension”³ offer a means to determine the fate of a given person in shock.

Of course, the skill of prediction has always been one of medicine's defining features. A physician is expected to collect clinical data, formulate a clinical impression, and predict a likely course of events for the patient's illness. Before the advent of modern therapeutics, establishing the prognosis was one of the most important functions a physician was called upon to perform. But in the golden age of bedside medicine,⁴ master clinicians and academicians—rooted in a humble appreciation of the complexity of human biology and behavior—recognized the limitations of their prescience. In those days, the focus of interest was the outcome for the individual patient.

In the current age of population medicine, however, the “case” rarely merits our attention.⁵ The subject about whom a prediction is made is not a particular person but a group deemed suitably homogeneous to be turned into a cohort to whom tests are applied and for whom subsequent clinical events are tallied. The keenly wished-for result is a rate of outcomes sufficiently larger or smaller than the rate for the comparative group, in order to sanction the observation as “predictive” through the use of the much-misunderstood P value.⁶

Invariably, the prediction comes to naught when applied to an individual within the group, but this point is all too frequently overlooked. The diligent minimization of biasing factors, the detached execution of the experiment, and the statistical anointment by the P value in carefully designed studies provide such a sense of objectivity that the temptation to relate the findings to a specific patient becomes irresistible, regardless of the limitations of such an extrapolation.

So we are told that cardiac troponin I predicts death because its elevation in the postsurgical setting is more prevalent among those patients who later died (21%), as compared with those who remained alive (6%), yielding a statistically significant odds ratio of 6.² Yet an elevated troponin-I level in this particular cohort could foretell a fatal outcome only a third of the time (32% “positive predictive value”), a far more modest achievement than the title of the paper leads one to believe. At best, one could say that an elevated troponin-I level indicates an increased risk of death, but the notion of risk is also a statistical concept of elusive applicability to individual circumstances.⁷

Apart from the strengths or weaknesses of statistical relationships, there are also and always practical matters to consider. B-Type natriuretic peptide can never possibly “[predict] benefit from a home-based nurse care in chronic heart failure”⁸ if the nurse in charge of the patient is incompetent. Mathematical relationships may explain group behavior, but a unique reality rules the individual patient.

When clinicians make predictions, they place their reputations at stake, especially if they share their forecasts with their patients or colleagues. The prediction is understood to carry the full weight of its meaning and is rarely uttered explicitly or casually. For academicians and researchers, a predictive declaration has a different audience. Such statements may be aimed at journal editors, funding agencies, news media, or venture capitalists, but to practitioners they sound increasingly hollow. C-Reactive protein (CRP) may be the darling of the medical fortunetellers, but among clinicians, it is sometimes known as “Can't Really Predict.”

The application of statistics to medicine has greatly improved our ability to discern phenomena not otherwise visible to the naked eye. Yet Alvan Feinstein himself, considered by many to be a founder of modern clinical research, was keenly aware of what he called “the distraction of quantitative models”: an over-reliance on mathematical tools and methods imported from nonclinical domains and “applied to goals for which they are not aimed.”⁹

Will misleading claims of insight and inflated pronouncements of prediction continue to permeate the medical literature? I ask, but I dare not answer.