Prediction models for covid-19 outcomes

Abstract

Reasons to be cautious

Robust models that predict the prognosis of coronavirus 2019 (covid-19) are urgently needed to support decisions about shielding, hospital admission, treatment, and population level interventions. With cases increasing in the UK and elsewhere, and winter approaching, such models could have a rapid clinical impact. Two linked articles report on two newly developed covid-19 prediction models. QCOVID is a risk prediction model for covid-19 related mortality for use in the general population (doi:10.1136/bmj.m3731),¹ whereas the 4C mortality score is for use on admission to hospital (doi:10.1136/bmj.m3339).² Notably, these models are of higher quality than others published to date,³ having been developed using ample sample sizes,⁴ with generally appropriate modelling choices, and suitably internally validated and reported.^{5 6} Nevertheless, we sound a note of caution in their use.

QCOVID predicts the risk of catching and dying from (or being admitted to hospital with) covid-19 in the general population.¹ The authors rightly emphasise the fact that predicting separately either the probability of catching covid-19 or the probability of dying from it is not possible, owing primarily to incomplete knowledge of who actually has the disease. However, this conflation causes limitations in the model’s application. The risk of catching covid-19 depends on an individual’s behaviour and the local dynamics of the disease, which are not modelled by QCOVID. These dynamics, such as local disease prevalence, change rapidly. Therefore, calibration of the model is likely to deteriorate rapidly. Moreover, recent data show a shift in the age distribution of cases towards younger people; discrimination of the model may also drop, therefore, as age is a strong predictor. QCOVID is, however, described as a “living” model¹; with regular updating, these problems can be mitigated.⁷

A further challenge is that the predictions made reflect interventions in place at the time the model was developed. A “low risk” prediction generated by the model might reflect active steps taken by similar people in the past to lower their risk, such as shielding. Therefore, using a low risk prediction to support a decision to return to work, for example, is problematic. Explicit separation of baseline risk factors and interventions can, in principle, be achieved through issuing counterfactual predictions, which provide a risk estimate assuming certain interventions.^{8 9} Of particular interest for decision making is the counterfactual prediction generated when no preventive measures (such as shielding) are taken.¹⁰

QCOVID might, with these caveats, be used to inform national guidelines on shielding and employment legislation regarding who can reasonably be required to remain in, or return to, specific work environments. Risk estimates from QCOVID could also inform discussions between clinicians and patients. If challenge trials of covid -19 treatments are to go ahead,¹¹ QCOVID could help scientists to target recruitment and enable potential participants to make informed decisions about their risks from taking part. Should effective vaccines be developed, it could inform decisions about which groups should be prioritised.

The 4C mortality score, calculated at hospital admission, predicts in-hospital mortality among patients with confirmed or likely covid-19.² Here, the authors explicitly suggest that the model should be used for decision support, noting, for example, that “patients with a 4C mortality score falling within the low risk groups (mortality rate 1%) might be suitable for management in the community.” This does not account for the “treatment paradox”: these patients may seem to be at low risk because of the interventions that similar patients in the development cohort received in hospital.¹² Again, counterfactual prediction modelling offers a potential solution.⁹ Furthermore, with patient management, and potentially the disease itself, changing rapidly, the 4C model must also be updated regularly.⁷

Some clinical scenarios exist in which these risk calculators are of limited value. Neither model would help community based clinicians to determine whether patients being video triaged should be seen in person or admitted to hospital. Greenhalgh et al have developed guiding principles,13 but much remains to be done. Future studies could assess the clinical and cost benefits of supplying patients at high risk with equipment to record their vital signs from home (pulse oximeters, blood pressure monitors, thermometers, and peak expiratory flow rate meters), as this could improve the discrimination of any risk assessment tools based on remote triage.

To conclude, the 4C and QCOVID models are likely to be helpful and represent a step forward in the quality of prognosis models for covid-19. Given the rapidly changing nature of the disease and its management, we emphasise the need to update these models regularly and monitor their performance closely over time and space. Care must also be taken when interpreting the predictions generated by these models: they reflect the risk for a patient taking similar measures, and receiving similar care, to similar patients in the past, not the risk to a patient if no actions are taken. Improved data on incident cases of covid-19 will allow greater granularity in prediction. With these caveats, we support the continued validation and impact assessment of these models.