Implementation of dynamically updated prediction models at the point of care at a major cancer center: making nomograms more like Netflix

Introduction

Prediction models have been widely advocated to aid patient counseling and clinical decision-making. There are two major advantages of statistical predictions in comparison to more traditional approaches such as clinical judgment or general risk categorization. First, there is good empirical data that statistical models outperform clinicians and risk groups. For example, Kattan et al. asked 24 clinicians to predict the risk of a positive bone scan based on the case histories of 25 men who experienced recurrence of prostate cancer after radical prostatectomy but had not received hormone therapy. The concordance index of clinician predictions (0.63) was qualitatively lower than that of a prediction model (0.81), and there was gross variation in the predictions given to individual patients by different clinicians.¹ Prediction models are also superior to risk groups, as they allow greater individualization of care, with clinicians able to modify recommendations according to patient preference. A patient either is in a risk group or not; a quantitative risk estimate from a prediction model can be discussed in terms of the patient's age, comorbidities, and thoughts and feelings about competing endpoints such as side-effects of treatment.

In a paper in Urology entitled “Why can't nomograms be more like Netflix?” we argued that our current approach to medical prediction was limited by using static models, rather than updating models as new data become available². There are many hundreds of papers on prediction modeling published in the urologic literature. In almost all cases, the authors obtained a data set, most often from a single hospital, ran it through some statistical software, obtained coefficients and then treated the model as if it were true for all patients at all times. But times change. A good example of secular trends affecting a model is the well-known Kattan nomogram for predicting prostate cancer recurrence.³ This was originally developed in the late 1990s, based on patients treated in the previous decade. Since that time, we have seen the effects of the stage shift,⁴ changes in grading⁵ and improvements in surgical technique,⁶ such that this nomogram is badly miscalibrated when applied to contemporary patients.⁷

We made the case for updating medical prediction models with new data, similar to how the Netflix movie-recommendation algorithm adjusts its recommendations as users rate movies.² Here we describe the implementation of this approach at Memorial Sloan Kettering Cancer Center (MSKCC).

Dynamic prediction modeling

To create dynamic prediction models at MSKCC, data are downloaded from our clinical databases every 6 – 12 months, automatically imported into the R statistical software and new model coefficients produced. These coefficients are then applied to a small set of test data and the results compared to the previous set of predictions. Any large differences are flagged, allowing the analyst to determine if an error has occurred. If no errors are found, the coefficients are uploaded to a central database of coefficients that can be accessed by a variety of different applications. These include both applications for our publically available website, where members of the public manually enter in variables such as stage and grade, and internal Electronic Health Record applications, where variables are automatically populated from stored data.

One key difference between the modeling approach we take and that used by other analysts who build models, is that we weight our models so that that more recent patients have a greater influence on model coefficients. The weighting factor we use is the inverse square root of the difference between the current year and the year the patient's data were first available (eg, year of surgery for a recurrence nomogram). To illustrate how this works in practice, imagine that a dataset included 100 patients from 2014 and 100 patients from 1990. The weighting for patients treated 2 and 26 years ago would be 0.707 and 0.196 respectively, so that the analysis is conducted as if there were 157 patients from 2014 and 43 from 1990 in the cohort.

The MSKCC prostate models are available at www.nomograms.org/nomograms/prostate. A link from the prediction tool itself provides researchers with the formula for the model, the current coefficients and the date of last update. At the time of writing (August 2016), the coefficients were last updated on May 31^th, 2016. The concordance index for the pre-operative model for recurrence was 0.799, or 0.821 if information on cores is included; discrimination for the postoperative model was 0.841.

Making predictions available at the point of care

Figure 1 shows a typical report accessed by an MSKCC doctor shortly before a patient visits for follow-up after radical prostatectomy. Much of the report focuses on patient-reported outcomes, which are obtained electronically as a routine part of care.⁸ There are also several predictions on the report that doctors can use in the clinical consultation. Along the very bottom of the report are predictions as to the risk of recurrence. These use coefficients from dynamically updated prediction models and themselves are dynamically updated as new information becomes available about the individual patient. The patient in question is free of recurrence 24 months after surgery, modifying the original five-year postoperative prediction of freedom from recurrence of 90% to one of 94%, reflecting the fact that the patient has not experienced recurrence during part of the period from which the total risk of recurrence is derived.

Figure 1.

Example report for an individual patient presenting for routine follow-up after radical prostatectomy at MSKCC.

The report also shows predictions for recovery of urinary and erectile function. These are similarly obtained from prediction models that are dynamic in two ways, via regular updating of model coefficients and through updating of an individual patient's predictions as he progresses through the process of surgical recovery. For instance, if a man is using two urinary incontinence pads per day at 6 months, the probability that he will recover full urinary continence by 1 year is lower than a man using only a safety pad and who occasionally has to rush to the bathroom. These predictions are used at the point of care for counseling and referral. As an example, a man who had poor erectile function 6 months after surgery previously might have been told something generic like “It can take time for the nerves to regrow, and recovery sometimes takes many months.” The doctor can now give a more precise estimate and perhaps make a recommendation like “We haven't seen much recovery thus far, so perhaps we should consider a referral to sexual medicine.”

Conclusion

We have described a novel system to dynamically update prediction models, our attempt to make nomograms more like Netflix. We have not completely followed the Netflix approach, as we chose to use conventional statistical modeling, rather than computer-intensive models. Although this does restrict us somewhat both in terms of the number of models we can make available and our ability to give predictions in the presence of missing data, it does allow us to publish coefficients that other groups can incorporate into research or clinical practice applications. Moreover, not all current MSKCC prediction models for prostate cancer – and no MSKCC prediction models for other cancers - are dynamic, as it takes time to set up the underlying informatics infrastructure. This is particularly a problem for studies that require multicenter collaborations, such as those with low prevalence endpoints like prostate cancer specific mortality. We do plan to expand dynamic prediction modeling at MSKCC for prostate cancer – for instance, prediction of recurrence after radiation therapy – and for other cancer sites.

All of which begs the question: Why do it? Does it make much difference? In the first few years of dynamic prediction modeling, our predictions have not changed much. For instance, the risk of recurrence for a man with extraprostatic extension, Gleason 4+3, PSA of 8, negative surgical margins and no seminal vesicle invasion or lymph node involvement has increased from 34.1% to 34.9%. This may reflect the “reverse stage shift”⁹ occurring at MSKCC and other tertiary care centers. Although this small change makes no practical difference, our view is that although small changes from year to year may be negligible, cumulatively, such changes may produce outcomes that drift far from the predictions provided by a static prediction model. We intend to research the effects of dynamic updating on predictive accuracy over the next few years.