Development and Validation of a Multivariable Risk Prediction Model for COVID-19 Mortality in the Southern United States

Abstract

Objective

To evaluate clinical characteristics of patients admitted to the hospital with coronavirus disease 2019 (COVID-19) in Southern United States and development as well as validation of a mortality risk prediction model.

Patients and Methods

Southern Louisiana was an early hotspot during the pandemic, which provided a large collection of clinical data on inpatients with COVID-19. We designed a risk stratification model to assess the mortality risk for patients admitted to the hospital with COVID-19. Data from 1673 consecutive patients diagnosed with COVID-19 infection and hospitalized between March 1, 2020, and April 30, 2020, was used to create an 11-factor mortality risk model based on baseline comorbidity, organ injury, and laboratory results. The risk model was validated using a subsequent cohort of 2067 consecutive hospitalized patients admitted between June 1, 2020, and December 31, 2020.

Results

The resultant model has an area under the curve of 0.783 (95% CI, 0.76 to 0.81), with an optimal sensitivity of 0.74 and specificity of 0.69 for predicting mortality. Validation of this model in a subsequent cohort of 2067 consecutively hospitalized patients yielded comparable prognostic performance.

Conclusion

We have developed an easy-to-use, robust model for systematically evaluating patients presenting to acute care settings with COVID-19 infection.

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which causes coronavirus disease 2019 (COVID-19), is the third coronavirus this century to cause severe illness in humans (after the more limited outbreaks of SARS-CoV and Middle East respiratory syndrome–coronavirus in the past 2 decades). Case fatality rates have ranged from 0.1% of diagnosed infections in Singapore to 16% in Belgium. True infection fatality rates are believed to be much lower.¹ Since February 2020, the COVID-19 pandemic has infected more than 183 million people around the world and resulted in more than 3.9 million deaths.² Of these, approximately 33 million cases and 600,000 deaths have been in the United States, with a 1.8% case fatality ratio.² Increasing evidence indicates that much of the mortality results from hyperinflammation related to a cytokine release syndrome (or “cytokine storm”).3, 4, 5 Clinical data suggests that persons with diabetes, cardiovascular disease, or chronic lung disease are at higher risk,4, 5, 6, 7 and there may be an associated proinflammatory genotype.⁸

In addition to demographic factors and cardiovascular risk factors, clear patterns have developed in the serologic presentation of patients with COVID-19. These derangements span multiple organ systems and include markers of inflammation, immune regulation, the clotting cascade, and indicators of end-organ function.^{8 , 9} Health care systems have been overwhelmed with a surge of hospital admissions due to the COVID-19 pandemic.¹⁰ Identifying patients at high risk of adverse outcomes at the time of presentation plays an important role in allocating limited resources.4, 7 Various prediction models have been developed to risk stratify patients with COVID-19. However, the majority of these risk prediction models have been found to be at a high risk of bias.^{7 , 11}

A combination of epidemiological factors had made urban Louisiana a nexus of early COVID-19 morbidity and mortality. There were 428,000 cases reported through February 2021, with a case fatality rate of 2.2%.¹² We examined more than 3700 patients hospitalized for COVID-19 within the Ochsner Health network of hospitals between February 2020 and December 2020 to better understand the clinical impact of demographics, laboratory data, and medical therapies. We sought to develop a mortality risk assessment tool using patient-level data that can be applied at the time of presentation to acute care settings, to improve triage, and to identify patients at high risk of adverse outcomes.

Patients and Methods

All patients older than 18 years admitted to Ochsner Health system hospitals with COVID-19 infection throughout Louisiana from March 1, 2020, through April 30, 2020, were enrolled into an observational cohort after approval of all protocols from an independent institutional review board. These patients represented the model derivation cohort. Following creation of the risk model, appropriate limited data were collected on all hospitalized patients with COVID-19 infection from June 1, 2020, through December 31, 2020, resulting in a validation cohort. We collected patients’ demographics, medical history, presenting symptoms, medications, select inpatient therapies, labs, and clinical outcomes. Patients with positive COVID-19 infection who were treated on an outpatient basis were not included in this study.

Clinical outcomes tracked included: intensive care unit patient management, number of ventilator days, maximal number of pressors/inotropes, sequential organ failure assessment score, and significant clinical organ-specific events encompassing cardiac events (myocardial injury, reductions in contractility, and arrhythmias), renal injury, hepatic injury, thrombotic events, and death. Major organ dysfunction was defined as the presence of kidney injury, myocardial injury, hepatic injury, or respiratory injury requiring mechanical ventilation. Acute myocardial injury was considered present if troponin I levels were elevated above the upper limit of normal (ULN), with a 50% change in subsequent level (either increase or decrease) at an interval of 3 to 6 hours. For the model to work as a triage tool, troponin I should have been collected at first contact with acute care settings. Non–acute myocardial injury was defined as troponin elevation above the ULN, but with less than 50% change in subsequent values. Cardiogenic shock was defined as heart failure requiring inotropic or mechanical support. Renal injury was defined based on The Kidney Disease: Improving Global Outcomes criteria for acute kidney injury.¹³ Hepatic injury was considered to be present in patients with elevation in the aminotransferase levels greater than two times the ULN or international normalized ration greater than 1.5 in the absence of underlying liver disease (Table 1 ).

Table 1.

Definition of Outcomes Used in the Study^a

Term Definition
Acute myocardial injury	Troponin I level above ULN with 50% change in subsequent level (increase or decrease) checked at 3 to 6 hour intervals.
Chronic myocardial injury	Troponin I level above ULN with <50% change in subsequent levels.
Cardiogenic shock	Heart failure requiring inotropic or mechanical support.
Acute renal injury	Creatinine elevation 1.5 times baseline.
Acute hepatic injury	Aminotransferase levels greater than 2 × ULN or INR greater than 1.5 in absence of underlying liver disease.
Thrombotic events	New diagnosis of deep vein thrombosis or pulmonary embolism on imaging.

^aINR, international normalized ratio; ULN, upper limit of normal.

Laboratory information was collected on admission and included inflammatory markers, D-dimer, lactate dehydrogenase, ferritin, lactic acid, renal function, and a complete blood count. Arterial and venous blood gas results were recorded when present. Maximum values of these markers during the admission were recorded.

Hospital days were calculated from the date of urgent/emergency presentation culminating in an admission to an inpatient facility until the patient was discharged. Transfers for escalation of care were considered as part of the index admission. Mortality was evaluated during the index hospitalization only. Mortality shortly after discharge from inpatient settings was not included in analysis.

Statistical Analysis

Statistical analysis was performed using SPSS v27.0 (IBM, Armonk, NY, USA), Stata v17.0 (StataCorp, College Station, TX, USA) and RStudio v1.4.1717 (PBC, Boston, MA, USA). Continuous variables were analyzed using two-tailed Student t test or analysis of variance, using an α of 0.05. Proportions were compared using χ² tests. Multiple logistic regression models and Cox proportional hazards models were used for univariate and multivariate analyses of outcome predictors. Continuous predictor variables were converted to categories for an easier application to the risk prediction model. Cutoffs were derived using receiver operating characteristic (ROC) curve analysis using Youden’s index. Missing values for candidate variables were handled using multiple imputation with chained equations under missing at random assumption.

Model Development

Patients admitted between March 1, 2020, and April 30, 2020, were included in the derivation cohort. The risk model was built using stepwise elimination of variables from a comprehensive multivariable model. Potential predictor variables were identified from review of literature and based on availability at first contact with acute care settings. Variables with greater than 50% missing values were removed as the first step (Supplemental Table, available online at http://www.mayoclinicproceedings.org). In the second step, univariate analysis was performed and variables with significance value greater than 0.1 were removed. Next, we checked for collinearity and any factor with variance inflation factor greater than 1.5 was removed. In the final step, a least absolute shrinkage and selection operator (LASSO) regression was applied to minimize overfitting and further minimize potential collinearity of variables using 10-fold cross validation. The LASSO regression was performed to fit models for all lambdas as well as using the one standard error rule to select lambda. Final regularization model with λ (0.017) using the one standard error rule was selected (Figure 1 ). Adjusted odds ratio was used to calculate weights of variables by rounding off the nearest integer. All included variables were adjusted based on age and sex except for age and sex. An ROC curve was constructed to evaluate the discrimination power of the score. The optimal cutoff score was chosen using Youden’s index. Validation of the model was performed using a separate cohort of patients admitted between June 1, 2020, and December 31, 2020. Although the validation cohort was checked for duplicates, we chose to omit patients admitted during May 2020 to minimize inadvertent overlap.

Figure 1.

Least absolute shrinkage and selection operatory regression with coefficient paths.

Results

The mean age for our patient cohort was 63±16 years. The population was evenly split between males (830, 49.6%) and females (842, 50.4%) and was predominantly African American (1168, 71.4%). In this patient population, 1556 (93.1%) of individuals had comorbidity; the most common diagnosis was hypertension (1309, 78.3%), but other conditions such as obesity (973, 58.2%), diabetes (754, 45.1%), and hyperlipidemia (738, 44.1%) also affected a large proportion of hospitalized patients. Tobacco use was present in 589 (37.4%) individuals, and coronary artery disease, chronic kidney disease, heart failure, and chronic obstructive pulmonary disease were each present in 10-20% of the population. The mean sequential organ failure assessment score on admission was 2.33±2.78 (Table 2 ).¹⁴

Table 2.

Clinic Characteristics of COVID-19 Derivation Cohort^a

Characteristic	N=1672
Age, years	63.4±15.8
BMI, kg/m2	32.6±8.7
Female	842 (50.4)
Black	1,168 (71.4)
DM	754 (45.1)
HTN	1,309 (78.3)
HLP	738 (44.1)
Smoking
Never	987 (62.6)
Former	513 (32.6)
Current	76 (4.8)
CAD	261 (15.6)
CHF	247 (14.8)
COPD	199 (11.9)
CKD	350 (20.9)
ESRD	72 (4.3)
Asthma	179 (10.7)
Cirrhosis	24 (1.4)
HIV	16 (0.1)
Sleep apnea	158 (9.5)
Transplant	27 (1.6)
Immunocompromised	54 (3.2)
Presenting symptoms
Fever	1,132 (68.0)
Cough	1,214 (72.7)
Myalgia	460 (27.5)
Diarrhea	472 (28.3)
Nausea	353 (21.1)
Vomiting	205 (12.3)
Anorexia	690 (41.3)
Shortness of breath	1,301 (77.9)
Presenting vitals
Heart rate, beats/min	90.9±15.5
Oxygen saturation, %	94.9±2.6
Systolic blood pressure, mm Hg	131.5±19.2
Diastolic blood pressure, mm Hg	72.1±10.4
Temperature, °F	100.3±1.7
SOFA admission score	2.4±2.8
Presenting lab values
Hemoglobin, g/dL	12.6±2.1
White cell count, cells/L	7.5±3.8
Lymphocyte count, cells/L	1.1±0.6
Platelet, cells/L	220.0±91.0
Creatinine, mg/dL	1.8±2.2
Blood urea nitrogen, mg/dL	26.4±24.5
AST, U/L	63.6±161.4
ALT, U/L	43.6±105.6
Lactate dehydrogenase, U/L	479.3±354.6
Lactate, mmol/L	1.6±1.0
Albumin, g/L	3.2±0.5
D-dimer, μg/mL	2.0±3.3
Troponin I, ng/mL	0.2±1.0
C-reactive protein, mg/L	108.1±88.8
Procalcitonin, ng/mL	1.4±11.5
Ferritin, μg/L	1247.3±1726.1
Peak values
Lactate dehydrogenase	538.1±553.9
Lactate	1.9±1.9
Troponin	0.3±1.4
D-dimer	4.7±6.5
C-reactive protein	159.6±124.6
Procalcitonin	2.8±18.2
Ferritin	1673.2±2718.8
Clinical course
ICU admission	610 (36.5)
Days in ICU	9.9±8.1
Mechanical ventilation	436 (26.1)
Ventilator days	9.9±7.4
Acute myocardial injury	307 (28.4)
DVT	23 (1.4)
PE	45 (2.7)
Stroke	62 (3.7)
New dialysis	110 (6.6)
Acute kidney injury	439 (26.3)
Acute hepatic injury	55 (3.3)
Death	403 (24.2)

Values are n (%) or median ± SD as appropriate.

^aALT, alanine transaminase; AST, aspartate transaminase; BMI, body mass index; CAD, coronary artery disease; CHF, congestive heart failure; CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease; DM, diabetes mellitus; DVT, deep vein thrombosis; ESRD, end-stage renal disease; HIV, human immunodeficiency virus; HLP, hyperlipidemia; HTN, hypertension; ICU, intensive care unit; PE, pulmonary embolism; SOFA, sequential organ failure assessment.

The average length of hospitalization was 11±10 days, and in-hospital mortality was 24.2% (403 patients). Of all admitted patients, 36.5% (610) required intensive care. Major organ dysfunction was present in 42.0% (703) of patients. Renal injury, myocardial injury, and lung injury requiring mechanical ventilation were the most common type of organ dysfunction present, each present in 25% to 30% of the patient sample. Venous thrombosis and hepatic injury were each present in approximately 4% (Figure 2 ). Mortality among patients who required critical care was 48.1% (293 patients), and their mean length of stay was 17±12 days. Mechanical ventilation was required by 26.1% (436) of the patients. In patients requiring mechanical ventilation, the mortality rate and mean length of stay were 59.2% and 20±12 days, respectively. In hospitalized patients, 6.6% (110) required new initiation of dialysis; these patients had a mortality rate of 50% and a mean hospital stay of 18±12 days. Of these patients, 72.7% (80) required mechanical ventilation, mortality was 67.5% (74), and mean hospital length of stay was 20±12 days. Those with acute myocardial injury (ie, troponin greater than the ULN and >50% variation) had a mortality of 40.0% (123) and mean length of stay of 16±13 days. Those with acute hepatic injury (aminotransferases >2 times the ULN) had a mortality of 72.4% (40) and mean length of stay was 13±9 days. Variables associated with shorter time from admission to death were age 60 years and older, acute kidney injury at admission, admission lab values of lactate greater than 2 mmol/L, and procalcitonin >0.25 ng/mL.

Figure 2.

Clinical course in hospitalized coronavirus disease 2019 patients. ICU, intensive care unit.

Significant mortality-based differences in baseline characteristics were present in most of the categories recorded, with the notable exception of race (Table 3 ). Deceased patients were significantly more likely to be taking most of the medications surveyed, except for ibuprofen (Table 4 ). Additionally, those who died were more likely to have been started on therapy with one of the three agents hypothesized to modify mortality during the early pandemic period (ie, hydroxychloroquine, azithromycin, and remdesivir).¹⁴

Table 3.

Demographic and Clinical Differences Between Deceased and Living Patients^a

Variable	Alive n=1260 (%)	Deceased n=403 (%)	P	Unadjusted Hazard ratiob	95% CI
Age, years	61.1±15.9	70.4±13.5	<.001	1.03	1.02-1.03
Female	684 (54.3)	152 (37.7)	<.001	0.68	0.55-0.83
Black	895 (72.5)	265 (67.6)	.16	0.86	0.7-1.04
BMI, kg/m2	33.0±8.8	31.5±8.2	.003	0.98	0.97-0.99
Diabetes	539 (42.8)	210 (52.1)	.02	1.26	1.03-1.53
Hypertension	967 (76.8)	335 (83.1)	.03	1.33	1.02-1.72
Dyslipidemia	530 (42.1)	204 (50.6)	.01	1.29	1.06-1.57
Smoking	808 (67.5)	174 (46.9)	<.001	1.54	1.27-1.87
CAD	170 (13.5)	90 (22.3)	<.001	1.58	1.25-2.00
CHF	160 (12.7)	86 (21.3)	<.001	1.53	1.21-1.95
COPD	126 (10.0)	71 (17.6)	<.001	1.61	1.25-2.08
CKD	221 (17.5)	128 (31.8)	<.001	1.64	1.33-2.02

^aBMI, body mass index; CAD, coronary artery disease; CHF, congestive heart failure; CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease.

^bUnadjusted hazard ratio for mortality with 95% during hospital stay. Significance values from Student t test and χ². Age not categorized for hazard regression.

Table 4.

Medication Differences Between Deceased and Living Patients^a

Variable	Alive n=1260 (%)	Deceased n=403 (%)	P	Unadjusted hazard ratiob	95% CI
Entresto	10 (0.8)	6 (1.5)	.21	1.18	0.53-2.66
Aldosterone antagonist	52 (4.1)	27 (6.7)	.01	1.65	1.11-2.44
Ibuprofen	259 (20.6)	44 (10.9)	<.001	0.58	0.42-0.79
Statins	755 (59.9)	270 (67.0)	.01	0.99	0.81-1.23
OAC	195 (15.5)	76 (18.4)	.13	0.76	0.59-0.98
Other NSAIDs	338 (26.8)	98 (23.8)	.17	0.83	0.66-1.04
Remdesivir	6 (0.5)	5 (1.2)	.10	0.71	0.29-1.72
HCQ	745 (59.1)	326 (80.9)	<.001	1.00	0.78-1.23
Zmax	1010 (80.2)	355 (88.1)	<.001	1.01	0.75-1.37
HCQ with Zmax	764 (53.5)	303 (75.2)	<.001	0.97	0.77-1.22

^aHCQ, hydroxychloroquine; NSAID, nonsteroidal anti-inflammatory drugs; OAC, oral anticoagulant; Zmax, azithromycin. Dosing regimens not available.

^bUnadjusted hazard ratio for mortality with 95% confidence interval during hospital stay. Significance values of χ² analysis.

Multivariable logistic regression modelling examining associations between patient characteristics and mortality found confirmed predictive value of several factors that have been found in previously modeled data.¹⁵ Significant factors associated with death included age older than 60 years, smoking, hypoxemia, thrombocytopenia, acute liver injury, acute myocardial injury, and elevated lactate or procalcitonin (Table 5 ). The final model, termed the ALPACA (AAALLPPPACA) score, is shown in Table 6 . The maximal possible score in the model is 33, the minimum score is 0. At a score of 0, sensitivity is greater than 99% and specificity is less than 10%. At a score of 21, specificity reaches 100% and sensitivity is 5%. As shown in Figure 3 A, a cut-off of 8.5 maximized Youden’s index: 74% sensitivity and 69% specificity, with area under the curve of 0.783 (95% CI, 0.76 to 0.81) (Figure 3A).

Table 5.

Multivariable Logistic Regression Modelling Mortality Associations With Adjusted Odds Ratio^a

Variable	Adjusted odds ratiob	95% CI	p-value	Assigned weightc
AKI	2.3	1.8-3.00	.003	2
Age 60-70 years	2.2	1.5-3.03	<.001	2
Age ≥70 years	4.6	3.4-6.1	<.001	5
Male	2.07	1.63-2.62	<.001	2
Smoking	1.42	1.11-1.83	.005	1
CHF	1.38	1.02-1.88	.03	1
COPD	1.41	1.01-1.96	.04	1
CKD	1.58	1.2-2.07	<.001	2
Hypoxemia (oxygen saturation ≤92%)	3.7	2.6-5.4	.001	4
Lymphopenia	1.8	1.2-2.5	.05	2
Thrombocytopenia	1.6	1.3-2.2	.001	2
AST elevation	1.8	1.3-2.4	.002	2
Lactate elevation	2.4	1.8-3.2	.001	2
Hypoalbuminemia	2.2	1.56-3.2	<.001	2
Elevated procalcitonin	2.9	2.2-3.9	.003	3
Myocardial Injury	1.78	1.3-2.4	.03	2

^aAKI, acute kidney injury; AST, aspartate transaminase; CHF, congestive heart failure; CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease.

^bWeights assigned according to adjusted odds ratio rounded off to the nearest integer.

Table 6.

AAALLPPPACA Risk Stratification Score^a

Factor	Weight
Age
≥60 to <70 years	2
≥ 70 years	5
Acute kidney injury  Creatinine >1.5 times baseline	2
Acute myocardial injury  Troponin I variation of 50% at admission (trend up or down) with at least one value above ULN	2
Lactate ≥2 mmol/L	2
Lymphopenia ≤500 u/uL	2
Pulse oximetry ≤92%	4
Platelet count ≤150,000 u/uL	2
Procalcitonin ≥0.25 ng/mL	3
Albumin ≤2.5mg/dL	2
Comorbities
CHF	1
CKD	2
COPD	1
Male	2
AST ≥80 u/L (or 2 times ULN)	2

^aAST, aspartate transaminase; CHF, congestive heart failure; CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease; ULN, upper limit of normal.

Figure 3.

AAALLPPPACA receiver operating curves (ROCs) for the (A) derivation cohort and the (B) validation cohort. AUC, area under the curve.

The validation cohort comprised 2067 patients (61.5±17.5 years old, 50% women) who were admitted with a positive COVID-19 test between June 1, 2020, and December 31, 2020. The average length of stay was 7.4±8.0 days and in-hospital mortality was 9.6% (198 patients). Among the 95% of these patients for whom renal function data were available, 26.4% (521) had renal injury; and in these individuals, mortality was 17.2% (85 patients). Evidence of myocardial injury was present in 11.9%, but data were incomplete for more than 50% of patients (ie, troponin levels were only measured in 912 patients). Mortality in patients with cardiac injury was 29.3% (267 patients), and the mean length of hospitalization was 12.5 days. Major organ dysfunction was present in 41.9% (867). In comparison with the derivation cohort, this population was significantly younger and had higher rates of liver injury, lower rates of cardiac injury, a significantly higher albumin, and significantly lower mortality and length of stay (Table 7 ). The difference in mortality between derivation and validation cohort reflected the national and international trends in decreasing mortality with COVID-19 illness.^{16 , 17} This was likely due to a combination of factors including younger patient population, lower community prevalence, and use of corticosteroids.^{18 , 19}

Table 7.

Comparison of Validation and Derivation Cohorts^a

	Derivation	Validation	P
Age, years	63.39±15.80	61.58±17.51	0.001
Female Gender	842 (50;4%)	1025 (49.6%)	0.80
CAD	261 (15.6%)	305 (15%)	0.39
CHF	247 (14.8%)	280 (14%)	0.35
COPD	199 (11.9%)	238 (12%)	0.74
CKD	350 (20.9%)	422 (20.4%)	0.10
AKI	439 (26.3%)	521 (26.4%)	0.92
ACS	307 (28.4%)	109 (11.9%)	<0.001
Liver Injury	55 (3.3%)	248 (12.8%)	<0.001
Oxygen saturation %	94.93±2.63	94.86±2.97	0.43
Lactate, mmol/L	1.58±0.99	1.57±0.95	0.72
Leukocytes, u/uL	7.52±3.81	8.41±8.34	<0.001
Platelets, u/uL	219.97±91.00	223.74±98.61	0.23
Procalcitonin, ng/mL	1.37±11.45	1.79±21.47	0.56
Albumin, mg/dL	3.16±0.54	3.34±0.63	<0.001
LOS, days	10.91±10.02	7.39±8.00	<0.001
ALPACA score	7.42±4.17	7.34±5.23	0.61
Organ Injury	690 (41.3%)	867 (41.9%)	0.68
Death	405 (24.2%)	198 (9.6%)	<0.001

^aACS, acute coronary syndrome; AKI, acute kidney injury; CAD, coronary artery disease; CHF, congestive heart failure; CKD, chronic kidney disease; COPD, chronic obstructive disease; LOS, length of stay.

However, the populations' mean ALPACA score and rates of any organ injury were not significantly different, and pre-existing comorbidities were similar between the two cohorts.

The ALPACA score was at least as effective for predicting mortality in the validation cohort as it was in the derivation cohort. The ROC analysis showed an area under the curve of 0.81 (95% CI, 0.78 to 0.84), and maximal discrimination was achieved with a score of 9.5 which yielded sensitivity 0.76 and specificity 0.73 (Figure 3B). A calibration belt plotting the observed outcome against the predicted mortality showed good calibration (Figure 4 ).²⁰ A decision curve analysis was also performed to estimate a clinical net benefit for the prediction model and all the included variables in comparison to default strategies of treating all or no patients (Figure 5 ). Finally, a nomogram was created as a visual model summary tool (Figure 6 ).²¹ The Brier score was excellent (0.073), and the Hosmer-Lemeshow test was nonsignificant (P=.18), indicating good fit of the prediction model.

Figure 4.

Calibration belt plotting observed and expected mortality.

Figure 5.

Decision curve analysis with ALPACA score and individual predictor variables. AKI, acute kidney injury; AMI, acute myocardial infarction; AST, aspartate aminotransferase; CKD, chronic kidney disease; CHF, chronic heart failure; COPD, chronic obstructive pulmonary disease.

Figure 6.

Easy to use nomogram with included predictor variables and assigned scores. Probability for mortality at the bottom can be calculated using total score. AKI, acute kidney injury; AMI, acute myocardial infarction; AST, aspartate aminotransferase; CKD, chronic kidney disease; CHF, chronic heart failure; COPD, chronic obstructive pulmonary disease.

Discussion

In this investigation, we used a wealth of information available from the beginning of the COVID-19 pandemic to construct a risk model for mortality, the ALPACA score. The ALPACA model also performed very well in a subsequent validation cohort. Modeling mortality risk in the COVID-19–infected population is an important aspect of triage in the hospitalized population. Understanding prognosis can be useful to clinicians for bed management, care delivery, and palliative discussions in a disease that has affected an enormous number of patients throughout the world.

In examining patterns of illness between the early pandemic and subsequent waves of illness, several interesting patterns emerge.¹⁹ The first is that the mean age of the population is lower. In numerous studies, the concern was that this indicated that younger patients were increasingly developing more severe infections with COVID-19.²² However, the fact that the prevalence of comorbidities was not significantly different also suggests that those who were ill enough for hospitalization were already suffering from chronic illnesses, and that younger healthy individuals may not have been disproportionately more affected in later stages of the pandemic. Another interesting observation from our data is that the prevalence of end-organ injury did not significantly decrease in the validation cohort, despite decreases in mortality and length of stay. One way to interpret this finding would be that severe infection of organ systems is not an age-dependent phenomenon. In addition, the lower mortality rate may indicate improvements in care that allowed patients to survive COVID-19 infection despite significant end-organ damage during infection.

Our relatively simple prediction model, derived from a manually curated patient-level database, makes ALPACA unique in comparison to others developed over the course of the pandemic. Our independently derived risk factors for mortality, represented in ALPACA, are similar to those used in other COVID-19 risk calculators.^{15 , 16} Additionally, the excellent performance of this model for estimating mortality in the validation cohort, despite changes in age, length of stay, mortality, and other variables, suggests that ALPACA is a robust predictive method. The validation cohort included patients admitted both during “surge” and “non-surge” conditions in Louisiana, and our model performed well.

A systematic review of COVID-19 prediction models found that all available prediction models were at high risk of bias.⁷ A low event rate in the studies’ validation cohorts was one of the major concerns in the majority of these prediction models.⁷ Another study evaluated 22 models (including 17 models developed specifically for COVID-19) and found that no prognostic model offered higher net benefit than univariable predictors (specifically, age and admission oxygen saturation).¹¹ Moreover, 10 of 17 models were developed in the Chinese population; therefore, they may not be as robust in a different population.

Study Limitations

The ALPACA COVID-19 mortality risk stratification score appears effective and robust. However, our data are limited to a single geographic region in the Southeastern United States. Furthermore, although the number of patients examined was significant, this risk model still falls far short of several other studies using much higher numbers of patients, with data sourced via automated algorithms. Another limitation of the study was that we chose not to include mortality shortly after discharge from acute care settings to simplify data collection process. This limits the ability of this model to predict mortality beyond hospital discharge. Lastly, the variation in data collection between the derivation and validation cohorts (specifically, a significantly reduced proportion of troponin testing in the validation cohort) may have introduced some bias. However, the fact that the model performed almost identically in both cohorts does suggest that it is robust.

With the emergence of new SARS-CoV-2 variants and significant time left before widespread vaccination, there still is ample time for developing tools to better triage and manage hospitalized COVID-19 patients. With additional validation, the ALPACA score could be a potent tool to help manage hospital bed shortages, identify proper patient placement, and hopefully to help evaluate novel treatment strategies for seriously ill COVID-19 patients.

Conclusion

The ALPACA score, derived early in the COVID-19 pandemic and validated in late 2020, is a valuable tool for risk-stratifying COVID-19–infected patients for the endpoint of in-hospital mortality.