Letter to the Editor: CORR Synthesis: When Should We Be Skeptical of Clinical Prediction Models?

To the Editor,

We would like to thank Drs. Karhade and Schwab [8] for a thoughtful CORR Synthesis review discussing when and why a reader might trust a clinical prediction model and when such models are most useful.

These models call for careful external validation to ensure that they are generalizable [6, 7, 16]. However, the area under the receiver operating characteristic curve, the Brier score, the calibration analysis, and the decision curves [12], known as the “ABCD methods,” may not be sufficient for certain clinical applications.

A high-quality prediction model provides objective suggestions that can inform the shared decision-making process between patient and physician. Together, with the physician’s expertise, the best therapeutic strategies can be tailored based on the suggestions delivered by these models [2, 4, 10]. In some clinical situations, several prediction models may be needed to get the necessary information; a common situation involves decision-making for patients with metastatic bone disease, in which different prediction models have evaluated different time points [5, 9, 13], and consulting the correct model could be the difference between a good decision and a bad one. For example, patients with spinal metastasis and a life expectancy of less than 3 months would likely not choose surgery because the postoperative recovery might take multiple months [3]. In contrast, patients with a life expectancy longer than a year could potentially benefit from an aggressive operation that reduces the local tumor progression and the subsequent revision surgeries [11, 14]. In this example, the timing matters when predicting the potential for therapeutic benefit.

But this only holds true if a prediction model delivers predictions that pass even the most rudimentary logical scrutiny, which is not always the case. For example, if given the same parameters, a prediction model should not estimate survivorship in the long term to be greater than survivorship in the short term. Unfortunately, none of the “ABCD methods” can determine the likelihood of such errant predictions being generated by a particular model. However, we have devised the simple “model consistency (MC)” metric defined as:

$$\text{MC}= \frac{\text{Number of consistent prediction pairs}}{\text{Number of consistent prediction pairs }+\text{ Number of inconsistent prediction pairs }}$$

An MC = 1 indicates the best consistency, whereas an MC = 0 signifies the worst. The MC serves as a performance metric that gives clinicians a more comprehensive idea about the prediction model’s behavior.

We believe that the MC metric has many clinical applications. In particular, it may help to identify when prediction models are going to have important inconsistencies. For example, Basu et al. [1] proposed prediction models to estimate 5-year cardiovascular and all-cause mortality (Table 1). Naturally, the former is lower than the latter; however, their models give inconsistent predictions in certain settings. Those kinds of obvious inconsistencies—it’s impossible for 5-year cardiovascular mortality to be higher than 5-year all-cause mortality—should call our attention to the importance of getting this right. We believe that the MC is a tool that can help us to do so.

Table 1.

Examples of inconsistent prediction results from two widely used models

Parameters entered into the SORG prediction model [11, 15]	Parameters entered into the RECODe prediction model [1]
Primary tumor: slow-growing ECOG PS: 3-4 ASIA impairment score: A Charlson comorbidities: 2 Visceral metastases: No Brain metastases: No Number of spine metastases: 2 Previous systemic therapy: Yes BMI: 27 kg/m2 Hemoglobin: 7 g/dL Platelet count: 127×103/uL Absolute lymphocyte count: 0.91×103/uL Absolute neutrophil count: 0.8×103/uL Creatinine: 0.5 mg/dL International normalized ratio: 1.1 Albumin: 3.9 g/dL Alkaline Phosphatase: 63 IU/L	Age: 60, Gender: Male Ethnicity: African-American Tobacco use: No Blood pressure: 160 mmHg Cardiovascular disease history: Yes Antihypertensive agents: Yes Statins use: Yes Anticoagulants use: Yes HbA1c: 9.5 % Total cholesterol: 180 mg/dL High density lipoprotein: 65 mg/dL Creatinine: 2.0 mg/dL Albumin to creatinine ratio: 300 mg/g
3-month survival as estimated by SORG: 46% *	All-cause mortality as estimated by RECODe: 30% **
1-year survival as estimated by SORG: 70% *	Cardiovascular mortality as estimated by RECODe: 32% **

^*This necessarily is an inconsistent prediction; 3-month survival in the same clinical circumstances should always be the same or higher than 1-year survival, not lower.

^**This, too, is an inconsistent prediction by definition; in the same patient, cardiovascular mortality should not exceed all-cause mortality at the same time point.

SORG = Skeletal Oncology Research Group; RECODe = Risk Equations for Complications Of type 2 Diabetes; ASIA = American Spinal Injury Association Impairment Scale; ECOG PS = Eastern Cooperative Oncology Group performance status.