A study protocol of external validation of eight COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting

Abstract

Background

The COVID-19 pandemic has a large impact worldwide and is known to particularly affect the older population. This paper outlines the protocol for external validation of prognostic models predicting mortality risk after presentation with COVID-19 in the older population. These prognostic models were originally developed in an adult population and will be validated in an older population (≥ 70 years of age) in three healthcare settings: the hospital setting, the primary care setting, and the nursing home setting.

Methods

Based on a living systematic review of COVID-19 prediction models, we identified eight prognostic models predicting the risk of mortality in adults with a COVID-19 infection (five COVID-19 specific models: GAL-COVID-19 mortality, 4C Mortality Score, NEWS2 + model, Xie model, and Wang clinical model and three pre-existing prognostic scores: APACHE-II, CURB65, SOFA). These eight models will be validated in six different cohorts of the Dutch older population (three hospital cohorts, two primary care cohorts, and a nursing home cohort). All prognostic models will be validated in a hospital setting while the GAL-COVID-19 mortality model will be validated in hospital, primary care, and nursing home settings. The study will include individuals ≥ 70 years of age with a highly suspected or PCR-confirmed COVID-19 infection from March 2020 to December 2020 (and up to December 2021 in a sensitivity analysis). The predictive performance will be evaluated in terms of discrimination, calibration, and decision curves for each of the prognostic models in each cohort individually. For prognostic models with indications of miscalibration, an intercept update will be performed after which predictive performance will be re-evaluated.

Discussion

Insight into the performance of existing prognostic models in one of the most vulnerable populations clarifies the extent to which tailoring of COVID-19 prognostic models is needed when models are applied to the older population. Such insight will be important for possible future waves of the COVID-19 pandemic or future pandemics.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41512-023-00144-2.

Background

COVID-19 has a large impact worldwide, causing over 6 million deaths [1], various long-term health effects in a large group of individuals, and an increased burden on healthcare providers and medical institutions. Health outcomes of a COVID-19 infection can be severe, particularly in the older population [2]. In the Netherlands, it is estimated that disproportionately high mortality of 88.8% of all COVID-19 deaths occurred in the older population (≥ 70 years of age) even though they make up only 14% of the total population [3]. Similarly, these mortality proportions in older individuals were also relatively high in hospitalized (60%) and nursing home settings (40%) [3].

Hundreds of prognostic models have been developed to quantify (differences in) mortality risk or other outcomes in COVID-19 patients and to identify individuals at greater risk of developing various future health outcomes [4]. COVID-19 prognostic models have been used to facilitate informed shielding decisions by governments [5], identify higher-risk groups requiring ventilatory or critical care support early to enable targeted recruitment for randomized controlled trials [6], and deliver more personalized, risk-based treatments for which effectiveness is known to vary according to disease severity more precisely [7]. However, despite the development of hundreds of prognostic models, only a few models are of high quality and low risks of bias, according to a critical appraisal in a large living systematic review [4]. For those models that were appraised at a low risk of bias, information about their actual performance in external validation studies is scarce. Further, due to the added complexity of health conditions like frailty [8] and multimorbidity [9] in older individuals, we hypothesize that these prognostic models derived for the general adult population will underperform when validated in an older population.

In this protocol, we describe a comprehensive external validation study to evaluate the predictive performance of eight prognostic models in the older population, defined as individuals aged 70 and older, in hospital, primary care, and nursing home settings. The predictive performance of the prognostic models will be evaluated in a different population than they were derived on, being the older population of individuals aged 70 and older compared to the general adult population. One model will be evaluated in all healthcare settings (hospital care, primary care, and nursing home) to assess predictive performance across settings.

Methods

We have adhered to the TRIPOD guidelines checklist for external validation studies [10] in reporting this study protocol (Supplementary file 4).

Selection of COVID-19 prognostic models

In the living systematic review of diagnostic and prognostic prediction models for COVID-19 (www.covid-precise.org) [4], all published prediction models were reviewed using PROBAST (www.probast.org) [11, 12], a quality or risk of a bias assessment tool for prediction model studies. Using results from the fifth update of this review, we have identified all candidate prognostic models that predict the risk of mortality in individuals with COVID-19 infection with uncertain or low risk of bias. Fifteen candidate models met this criterion of which five prediction models (PRIEST [13], CUCAF-SF [14], CUCA-SF [14], and QCOVID [15] for males and females) were not included for validation due to the unavailability of data on certain predictors in the six cohorts of older patients while two prognostic scores (qSOFA [16] and NEWS [17]) were excluded because they express risk of mortality qualitatively rather than as a risk prediction (Supplementary file 1). Eight prognostic models were included for external validation (Fig. 1). Of these eight models, five were COVID-19-specific (GAL COVID-19 mortality model [18], 4C Mortality Score [19], NEWS2 + model [20], Xie model [21], and Wang clinical model [22]) and were developed in adult COVID-19 populations during the pandemic. Three prognostic models were already existing before COVID-19 pandemic and were used for the prediction of in-hospital mortality risk after admission for any respiratory infections or sepsis (APACHE-II [23], CURB65 [24], and SOFA [25]) (Table 1). The details of eight selected models can be found in Supplementary file 2.

Fig. 1.

Flowchart for inclusion of prognostic models for external validation

Table 1.

Overview of selected prognostic models

Model name	Derivation country		Pre-existing or COVID-19 specific	Derivation health care setting	Derivation population	Intended moment of use	Predicted outcome	Predictors		Model type
GAL-COVID-19-mortality model [18]	Spain		COVID-19 specific model	Primary care	Adults (≥ 18 years) with confirmed COVID-19 diagnosis	First presentation with COVID-19 infection at general practitioner	Mortality (no prediction horizon reported)	• Age • Sex • Lymphoma/leukemia • Liver disease • Dementia • Ischemic heart disease • Chronic obstructive pulmonary disease • Diabetes mellitus • Chronic kidney disease		Prediction model
4C-Mortality Score [19]	UK		COVID-19 specific model	Hospital	Adults (≥ 18 years) with confirmed COVID-19 diagnosis	At hospital admission for COVID-19 infection	In-hospital mortality	• Age • Sex • Respiratory rate • Peripheral oxygen saturation on room air • Glasgow Coma Scale • Urea • C-reactive protein • Number of comorbidities (counted as chronic cardiac disease, chronic respiratory disease (excluding asthma), chronic renal disease, liver disease, dementia, chronic neurological conditions, connective tissue disease, diabetes mellitus, HIV or AIDS, malignancy, obesity)		Points-based score
NEWS2 + model [20]		UK	Pre-existing risk stratification score updated for COVID-19 patients	Hospital	Adults (≥ 18 years) admitted to the hospital with a confirmed COVID -19 diagnosis	• At hospital admission for non-nosocomial patients (i.e., community-acquired COVID infection) • At the date of symptom onset for nosocomial patients. If the date of onset was unavailable the date of positive SARS-CoV-2 RT-PCR minus 4 days was used instead	ICU admission or death within 14 days of admission	• Age • Peripheral oxygen saturation • Heart rate • Systolic blood pressure • Body temperature • Alertness • Supplemental oxygen flow rate • Urea • C-reactive protein • Estimated glomerular filtration rate • Neutrophil count • Neutrophil/lymphocyte ratio		Prediction model
Xie model [21]		China	COVID-19 specific model	Hospital	Adults (≥ 18 years) admitted to the hospital with a confirmed COVID -19 diagnosis	At hospital admission for COVID-19 infection	In-hospital mortality	• Age • Lactate dehydrogenase • Lymphocyte count • Oxygen saturation		Prediction model
Wang clinical model [22]		China	COVID-19 specific model	Hospital	Adults (≥ 18 years) admitted to the hospital with a confirmed COVID-19 diagnosis. Pregnant women were excluded	At hospital admission for COVID-19 infection	In-hospital mortality	• Age • History of hypertension • History of heart disease		Prediction model
APACHE-II Score [23]		USA	Pre-existing risk stratification score	Hospital	All newly admitted ICU patients	At admission to ICU	Mortality	• Age • Body temperature • Heart rate • Respiratory rate • Mean arterial pressure • Arterial oxygen tension • pH • Potassium • Sodium • Creatinine • Hematocrit • Leucocyte count • Glasgow Coma Scale • Chronic health evaluation		Points-based score
CURB-65 [24]		UK, New Zealand, The Netherlands	Pre-existing risk stratification score	Hospital	Patients with community-acquired pneumonia	For triage at the emergency department	Mortality (30 days)		• Age • Alertness (new confusion) • Urea • Respiratory rate • Systolic blood pressure • Diastolic blood pressure	Points-based score
SOFA Score [25]		Unclear	Pre-existing risk stratification score	Hospital	All ICU patients	At admission to and during admission at ICU	ICU-mortality		• Arterial oxygen tension • Fraction of inspired oxygen ratio • Glasgow Coma Scale • Mean arterial pressure • Administration of vasopressors • Creatinine • Bilirubin • Platelet count	Points-based score

COVID-19 Coronavirus disease of 2019, ICU Intensive care unit; RT-PCR Reverse transcription polymerase chain reaction, SARS-CoV-2 Severe acute respiratory syndrome coronavirus

Validation cohorts

Data for this external validation study is collected from six cohorts with older individuals presenting with COVID-19 infection in the Netherlands from three settings: a hospital setting (3 cohorts), primary care setting (2 cohorts), and nursing home setting (1 cohort) (Table 2).

Table 2.

Details of study cohorts

Healthcare settings	Name of cohort	Description	Models that are externally validated in the cohort
Hospital setting	CliniCo	6 Hospitals in the region of Nijmegen in the Netherlands	• GAL-COVID-19 mortality model • 4C Mortality Score • NEWS2 + model • Xie model • Wang clinical model • CURB-65 Score • SOFA Score
	COVID-OLD	15 Hospitals throughout the Netherlands	• GAL-COVID-19 mortality model • 4C Mortality Score • NEWS2 + model • Xie model • Wang clinical model • APACHE-II Score • CURB-65 Score • SOFA Score
	COVID-PREDICT	10 Hospitals throughout the Netherlands	• GAL-COVID-19 mortality model • 4C Mortality Score • NEWS2 + model • Xie model • Wang clinical model • CURB-65 Score • SOFA Score
Primary care	JHN/ANH/AHA	General practitioner centers in Utrecht and Amsterdam municipalities in the Netherlands	• GAL-COVID-19 mortality model
	PHARMO	General practitioner centers throughout the Netherlands	• GAL-COVID-19 mortality model
Nursing homes	Ysis	Nursing homes throughout the Netherlands	• GAL-COVID-19 mortality model

Participants

The study participants consist of older individuals (≥ 70 years of age) presenting with highly suspected or reverse transcription polymerase chain reaction (RT-PCR) confirmed COVID-19 from March 2020 to December 2020 in hospital, primary care, and nursing home settings.

Before the widespread use of the RT-PCR test for COVID-19 diagnosis, participants were included using proxy criteria. In the hospital cohorts (CliniCo, COVID-OLD, COVID-PREDICT), the reported respiratory diseases that had COVID-like symptomology are used as an inclusion criterion until 31 March 2020. From April 2020 onwards, a confirmed RT-PCR test for COVID-19 is used as an inclusion criterion. In the primary care cohorts (PHARMO and JHN/ANH/AHA), the participants are included based on free text information and reports of respiratory infections. From June 2020 onwards, ICPC R83.03 for COVID-19 infection is used as an inclusion criterion. The nursing home cohort (Ysis) used RT-PCR as an inclusion criterion. Only participant data on the first presentation or admission of COVID-19 will be included. In the three hospital cohorts, admissions with a duration fewer than 7 days between discharge and readmission will be considered a single hospital admission.

Outcome

All prediction models have mortality as the predicted outcome (Table 1). In all three hospital cohorts, the outcome is defined as in-hospital mortality. In the primary care and nursing home cohorts, the outcome is defined as 28-day mortality.

Predictors

Definitions and timing of the predictor variables for the eight models were extracted from original publications (Table 1). Recorded variables in the cohorts are matched, as closely as possible, to the original predictor measurement procedures (Supplementary file 3).

Statistical analysis

We will externally validate the eight COVID-19 prognostic models in the six cohorts of older patients with COVID-19, aiming to assess their predictive performance when transported from a general adult population to a specific older population. The performance of the GAL-COVID-19 model is assessed across the three healthcare settings. The GAL-COVID-19 mortality model was developed in a primary care setting (general practitioners) and will be validated across different settings (in hospitals, primary care, and nursing homes) [26]. The 4C Mortality Score, NEWS2 + model, Xie model, Wang clinical model, APACHE-II score, CURB-65 score, and SOFA score were developed in hospitalized populations and will be externally validated in the same setting. Evaluation and assessment of the predictive performance of the COVID-19 prognostic models will be performed in each cohort separately. The statistical analysis will be performed in R (version 4.0.0 or later) [27].

Descriptive analysis

Participant characteristics and predictor information will be described in all study cohorts (overall and stratified on mortality outcome status) to identify differences in case-mix between the development and validation study populations [28]. These comparisons give insight into the expected model performance and transportability [26].

Missing data

The missing data will be described to determine possible reasons for and patterns in missingness [29]. Based on these findings, decisions about the handling of the missing values in the statistical analysis will be made. We anticipate that missing data will be handled using multiple imputations by chained equations using the Full Conditional Specification or Joint Modelling (JOMO) [30]. All variables and outcomes in the final prognostic models are included in the imputation model to ensure compatibility. A total of 50 imputed datasets will then be generated as cohorts are expected to have less than 50% missing values for all relevant variables [31].

Assessment of predictive performance

For each prognostic model, we will apply the model according to the authors’ original descriptions and evaluate its predictive performance. We will evaluate discrimination (the model’s ability to distinguish individuals who died after presentation with COVID-19 diagnosis from those who did not) and calibration (the agreement between predicted and observed mortality risks) in each cohort [32]. Discrimination will be assessed in all models by quantifying the area under the receiver operating characteristic curve, i.e., c-statistic [33], and pooled by taking the median c-statistic over imputed datasets and computing dispersion using the interquartile range [34].

Calibration will be assessed by visualizing the calibration of expected versus observed risk using LOESS-smoothed plots on stacked imputed data sets [35]. The GAL-COVID-19 mortality model, NEWS2 + model, Xie model, and Wang clinical model are model equations. For these models, calibration will be assessed in terms of the calibration-in-the-large coefficient and calibration slope [35]. The coefficients are again pooled on a log scale using Rubin’s rules in case of multiple imputed datasets [36]. For each performance measure, for each evaluated model, we will compute the point estimate, standard error, and 95% confidence interval.

Decision curve analysis

Decision curve analyses will be performed to quantify the net benefit achieved by each model for predicting the originally intended endpoint across a range of risk thresholds ranging from zero to one [37].

Updating

Prediction models showing miscalibration will be adjusted using an intercept update, and predictive performance will be re-assessed for the recalibrated model.

Sensitivity analysis

Two sensitivity analyses will be performed to assess the variation in the predictive performance of the eight COVID-19 prognostic models when implemented in different time periods: January 2021 to December 2021 and March 2020 to December 2021. Additionally, predictive performance in estimating the 90-day mortality risk will be evaluated in cohorts that have data available on this outcome (CliniCo, COVID-PREDICT, PHARMO, JHN/ANH/AHA, Ysis).

Sample size

Using national statistics from July 2020 to December 2021, the COVID-related mortality fraction for the older population (≥ 70 years) living at home in the Netherlands (including self-reported COVID-19-positive patients) was around 3% [3]. Although we expect higher mortality in nursing homes, hospitals, and ICU admissions, we will assume that an outcome incidence of 3% is the lowest event fraction that we will encounter for the current prediction model validation study. We take this fraction as a starting point for the sample size calculations, which are based on sample size calculation recommendations for external validation studies by Riley and colleagues [38]. Since we are considering various models, we base our sample size calculation around a general scenario that is reasonably applicable to all cases. Based on the distributions of predicted risk of mortality that were previously reported for some of the cohorts [8, 39], we assume the distribution of the linear predictor to be approximately N (− 3.9,1) [38]. Based on the calculations, a sample size of 754 will be required per cohort to validate an assumably well-calibrated model with a calibration-in-the-large coefficient of 0 and calibration slope of 1, and an assumed target standard error of calibration slope of 0.02. The target standard error of the calibration slope was chosen to ensure the evaluation of calibration with a higher precision than the typically used value of 0.05.

Discussion

This external validation study will assess the predictive performance of pre-existing clinical and COVID-19-specific prognostic models in the older population. Our study will reveal the validity of these COVID-19 prognostic models in one of the most vulnerable populations, the older population, as well as give insight into the requirements for tailored prediction models for the older population in future COVID-19 waves or pandemics.

While previous studies have largely focused on external validation of COVID-19 prediction models in the general adult population [14, 40, 41], this external validation study will focus on evaluating the performance of existing COVID-19 prognostic models specifically in the older population which represents the highest proportion of hospitalized COVID-19 patients and the highest mortality [3].

External validation in multiple cohorts across different healthcare settings allows for the assessment of between-setting heterogeneity in predictive performance and applicability at different points of care in older COVID-19 patients. This is one of the first studies to evaluate prognostic models for COVID-19 disease in individuals living in nursing homes.

Challenges and limitations

There are certain challenges that we anticipate encountering while conducting this research. Definitions of participant inclusion and predictor measurement procedures are expected to differ across cohorts. Although we have carefully planned to collect information on the definition of a COVID-19 infection and predictor measurements for each cohort, we cannot rule out the possibility of heterogeneity occurring across different settings and its effects on model performance [42]. However, this heterogeneity resembles practical conditions encountered in clinical practice and thus provides relevant knowledge on the anticipated performance of the models in clinical practice.

A limitation of this study is that not all low risk of bias COVID-19 prognostic models identified by the living systematic review could be included in this external validation. This was due to the lack of predictor information available in the cohorts. Similarly, predictor measurements and incidence of mortality due to COVID-19 can vary over time due to newer variants, better medical management, and improved vaccination coverage [43]. These temporal changes can potentially limit the predictive performance of the prognostic models being investigated in the current study.

Conclusion

External validation of newer and existing COVID-19 prognostic models can provide evidence about their predictive performance when implemented in the older population as tools of effective risk stratification and in aiding decision-making for targeted and timely clinical interventions.

Abbreviations

PCR

Polymerase chain reaction

APACHE score

Acute Physiology and Chronic Health Evaluation score

NEWS score

National Early Warning Score

SOFA score

Sequential organ failure assessment score

PROBAST

Prediction model Risk of Bias Assessment Tool

RT-PCR

Reverse transcription polymerase chain reaction

JOMO

Joint modeling

LOESS

Locally estimated scatterplot smoothing

Authors’ contributions

AZ performed the methodology, investigation, writing—original draft, and visualization. KL performed the conceptualization, methodology, investigation, writing—review and editing, and supervision. EA and JP performed the writing—review and editing, and data curation for a hospital cohort. JvdB and MTB performed the investigation, writing—review and editing, and data curation for the primary care data. PE, JF, and JG performed the writing—review and editing. KJ performed the writing—review and editing, and data curation for a nursing home cohort. RM performed the investigation and writing—review and editing, and performed the data curation for a hospital cohort. SM performed the project administration and funding acquisition. HPB performed the investigation and writing—review and editing, and performed the data curation for a hospital cohort. BvR performed the investigation and writing—review and editing, and data curation for a hospital cohort. AS performed the investigation and writing—review and editing, and data curation for a hospital cohort. HT performed the investigation and writing—review and editing, and data curation for the primary care data. CM performed the conceptualization, methodology, and writing—review and editing, and supervision. MvS performed the conceptualization, methodology, and writing—review and editing, supervision, and project administration. The authors read and approved the final manuscript.

Funding

This study is supported by grants from the Netherlands Organization for Scientific Research (ZonMW, grant number: 10430102110005).

Availability of data and materials

The datasets generated and/or analyzed during the current study are not publicly available due to confidentiality. COVID-OLD, CliniCo, COVID-PREDICT, PHARMO, and JHN/ANH/AHA are available on reasonable request.

Declarations

Ethics approval and consent to participate

This study will be conducted in accordance with the EU GDPR (General Data Protection Regulation). Due to the fact that this study does not require direct patient or physician involvement, the need for formal ethical reviewing was waived by the local medical research ethics committee (MREC Utrecht, the Netherlands), according to Dutch law.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.