Abstract
Objective
We sought to derive and internally validate a Sepsis Risk Score (SRS) and a Severe Sepsis Risk Score (SSRS) predicting long-term risks of future sepsis and severe sepsis events among community-dwelling adults.
Design
National population-based cohort.
Setting
United States.
Subjects
30,239 community-dwelling adults ≥45 years old in the national REasons for Geographic and Racial Differences in Stroke (REGARDS) cohort.
Interventions
None.
Measurements and Main Results
Over a median of 6.6 years (IQR 5.1-8.1) of follow-up, there were 1,532 first sepsis (incidence 8.3 per 1000 person-years) and 1,151 first severe sepsis (6.2 per 1,000 person-years) events. Risk factors in the best derived SRS and SSRS included: chronic lung disease, age≥75 years, peripheral artery disease, diabetes, tobacco use, white race, stroke, atrial fibrillation, coronary artery disease, obesity, hypertension, deep vein thrombosis, male sex, high-sensitivity C-reactive protein >3.0 mg/dL, Cystatin C ≥1.11 mg/dL, estimated glomerular filtration rate <60 mL/min/1.73m2, and albumin-to-creatinine ratio >30 mcg/mg. SRS risk categories were: very low (0-3 points; 2.3 events per 1,000 person-years), low (4-6; 4.1), medium (7-9; 6.5), high (10-12; 9.7), and very high (13-38; 21.1). SSRS risk categories were: very low (0-5 points;1.5 events per 1,000 person-years), low (6-9; 3.4), medium (10-13; 6.7), high (14-17; 9.9), and very high (18-45; 22.1). The SRS and SSRS exhibited good discrimination (bootstrapped C Index 0.703 and 0.742) and calibration (p=0.65 and 0.06).
Conclusions
The SRS and SSRS predict 10-year sepsis and severe sepsis risk among community-dwelling adults and may aid in sepsis prevention or mitigation efforts.
Keywords: sepsis, infections, epidemiology, risk prediction
Introduction
Prevention encompasses measures to protect healthy individuals from developing a medical condition or to mitigate the progression of an illness.(1) Prevention can play an important role in the public health management of acute medical conditions. For example, myocardial infarction and stroke were once believed to be untreatable conditions triggered by acute exposures (-e.g., stress or exercise) or random processes.(2, 3) However, the recognition of heart disease and stroke as endpoints of a chronic disease process allowed for a paradigm shift from acute care to the management of a chronic disease process. The identification of treatable risk factors has led to the broad application of risk stratification, detection and reduction strategies, contributing to a more than 70% decline in heart disease and stroke mortality, a feat declared as one the 10 greatest public health achievements of the last century.(4)
Sepsis is a major public health problem associated with over 750,000 hospital admissions, 570,000 Emergency Department visits, 200,000 deaths and $16.7 billion in medical expenditures in the United States (US) annually.(5-8) Much like myocardial infarction and stroke in the distant past, current thinking has conceptualized sepsis primarily as an acute event, with attention focused on its early identification and resuscitation.(9, 10) However, as with other acute conditions, chronic processes such as chronic lung disease, chronic kidney disease and obesity are known to contribute to the long-term likelihood of sepsis.(11, 12) Although counter to current acute care paradigms, the understanding of an individual's baseline susceptibility could present important opportunities to reduce the long-term risk and societal burden of sepsis in the general population.
Important steps in disease prevention include not only the identification of individual risk factors but also the characterization of a person's cumulative risk. A population-based cohort offers the optimal design for these goals, linking baseline individual characteristics with events identified over an extended follow-up period. The 30,239-subject REasons for Geographic And Racial Differences in Stroke (REGARDS) cohort is one of the largest longitudinal population-based cohorts in the United States.(13) In prior studies we have identified a range of sociodemographic factors, modifiable health behaviors, preventable and treatable chronic medical conditions, and biomarkers independently associated with sepsis events occurring over a 10-year follow-up period.(11, 12, 14) In the current study we sought to derive and internally validate a Sepsis Risk Score (SRS) and a Severe Sepsis Risk Score (SSRS) characterizing community-dwelling individuals' long-term risks of sepsis and severe sepsis.
Materials and Methods
Study Design – the REGARDS Cohort
This study was approved by the Institutional Review Board of the University of Alabama at Birmingham.
REGARDS includes 30,239 adults ≥45 years old from the 48 contiguous US states and the District of Columbia. While designed to characterize US geographic and racial disparities in stroke mortality, REGARDS is comprised of community-dwelling adults at a stable phase of health – not just individuals suffering from a stroke.(13) The REGARDS design includes oversampling of individuals from the Southeastern US, with 21% of the cohort originating from the coastal plains of North Carolina, South Carolina and Georgia (the “stroke buckle”), and 35% from the remainder of North Carolina, South Carolina and Georgia plus Tennessee, Mississippi, Alabama, Louisiana and Arkansas (the “stroke belt”). Among REGARDS participants, 42% are African American, 45% are male, and 69% are over 60 years old.
Enrollment of REGARDS participants occurred between 2003-2007, with ascertainment of baseline participant information including medical history, functional status, health behaviors, physical characteristics (height, weight), physiologic measures (blood pressure, pulse, electrocardiogram), current medications, diet, family history of diseases, psychosocial factors and prior residences. The study also collected blood and urine specimens from each participant. REGARDS contacted study participants by telephone at 6-month intervals to identify the date, location and attributed reason for all hospitalizations.
Identification of Sepsis and Severe Sepsis Hospitalization Events
Using the taxonomy of Angus, et al., we identified Emergency Department visits or hospital admissions attributed by participants to a serious infection.(6) Two trained abstractors independently reviewed relevant medical records to identify clinical and laboratory information, confirm the presence of a serious infection on initial hospital presentation and to verify the relevance of the serious infection as a major reason for hospitalization. We included hospitalization events for February 5, 2003 through December 31, 2012.
Sepsis events consisted of presentation to the hospital with a serious infection plus two or more systemic inflammatory response syndrome (SIRS) criteria.(5) Severe sepsis events consisted of the presence of sepsis plus at least one organ dysfunction (defined by the Sequential Organ Dysfunction Assessment [SOFA] for respiratory, renal, hepatic, cardiovascular, hematologic, and neurologic systems) or hypotension (systolic blood pressure ≤90 mm Hg).(15) We defined SIRS, organ dysfunction and hypotension using any asynchronous combination of the most abnormal vital signs and laboratory test values observed during the initial 28 hours of hospitalization, allowing for Emergency Department and up to one full day of inpatient treatment. Because of our focus on “community-acquired” rather than “hospital-acquired” sepsis, we did not utilize clinical data from later points of hospitalization.(16)
Participant Characteristics and Biomarkers
In developing the risk prediction models, we considered candidate sociodemographics, health behaviors, chronic medical conditions and biomarkers that could be reasonably measured at baseline and that exhibited demonstrated or plausible connections with future sepsis events.(11, 12, 14, 17) Participant demographic characteristics included age, sex, race, geographic region, annual household income and education (years of school). We dichotomized age at 75 years because of a higher sepsis risk observed at this threshold. Health behaviors included smoking status and alcohol use. Chronic medical conditions included atrial fibrillation, chronic lung disease, coronary artery disease, diabetes, deep vein thrombosis, dyslipidemia, hypertension, myocardial infarction, obesity, peripheral artery disease and stroke. (Detailed definitions in Supplemental Digital Content -- Appendix 1 .)
We also considered select biomarkers that could be broadly measured in community-dwelling individuals, were available in the REGARDS cohort, and exhibited plausible connections with future sepsis risk. Because prior studies have associated chronic kidney disease with sepsis risk, we evaluated estimated glomerular filtration rate (eGFR) based upon serum creatinine as well as urinary albumin-to-creatinine ratio (ACR).(11, 18, 19) Because of its association with long-term sepsis risk independent of eGFR and ACR, we assessed serum Cystatin C (Cyst-C).(17) We also considered high sensitivity C-reactive protein (hsCRP), which has been associated with long-term sepsis risk.(20) (Additional details in Supplemental Digital Content -- Appendix 1)
Data Analysis
While the primary objective of the study was to develop a sepsis risk prediction model (Sepsis Risk Score - SRS), current clinical initiatives often focus on the identification of severe sepsis.(21) Therefore, we developed a parallel but separate prediction model for severe sepsis (Severe Sepsis Risk Score - SSRS). We followed identical analytic strategies for derivation and internal validation of the SRS and SSRS.
We limited the analyses to first-sepsis and first-severe sepsis events. Using Pearson's chi-square, we compared characteristics between participants who did and did not experience sepsis or severe sepsis events. To identify SRS and SSRS risk factors, we fit a series of multivariable Cox regression models, defining elapsed time as the period from subject enrollment to the first sepsis or first severe sepsis event. Participants not experiencing a sepsis or severe sepsis event were censored on death or at the point of last follow-up (December 31, 2012).
A traditional approach to prediction rule development is split-sample derivation and validation (-e.g., derivation using half of the dataset, validation using the other half), but this approach reduces the available analytic data and may result in overly pessimistic model performance estimates.(22) Therefore, we opted to derive the SRS and SSRS using the entire cohort, assessing discrimination using bootstrap resampling.(23) This approach has been shown to provide greater efficiency than traditional split-sample validation.(22)
Because of a substantial number of missing values for select covariates (Cyst-C 6.8%, hsCRP 6.4%, ACR 4.7%, eGFR 4.5%), we also performed the analyses using multiple imputation.(24, 25) We performed multiple imputation using chained equations with the Stata ‘MI’ suite, generating ten imputed data sets and pooling estimates using Rubin's rules.(26) For prognostic measures involving resampling techniques (e.g. concordance statistics and reclassification indices), we determined the median, minimum, and maximum values across imputations.(26)
For each imputed data set, we first fit a model (Model 1) using backwards elimination of participant demographics, health behaviors and chronic medical conditions. We used p≤0.1 for as a criterion for retention. We then fit a model adding the variables elevated hsCRP, elevated Cystatin C, elevated ACR, and decreased eGFR (Model 2), reflecting a clinical scenario with the availability of select biomarkers measurements. We did not include participant income, education, or region in the models because it was unclear how these variables would be used in clinical practice. We followed the same variable selection and model development strategies for the SRS and SSRS.
Our a priori plan was to select models within each imputation.(27) As identical SRS and SSRS models resulted from each imputation, methods for handling differences in variable selection across imputations were not required. We developed point systems for each model by standardizing beta coefficients for each factor to the lowest coefficient, summing the integer point values, and averaging the points across imputations.(28) For each model, we separated participants into five graded risk score categories.
For the regression model and point system of each SRS and SSRS model, we assessed discrimination using an optimism-adjusted concordance statistic (Harrell's C Index), estimated using bootstrap resampling with 150 replicates (transformed Dxy obtained from validation in the R package ‘RMS’).(23) We assessed model fit and calibration using the Groennesby and Borgan score test (Stata ‘stcoxgof’ command).(29) We also assessed calibration by plotting the observed and expected incidence rates by PI deciles. To assess the improvements in discrimination between SRS Models 1 and 2 and SSRS Models 1 and 2, we used an extension of the Net Reclassification Improvement (NRI) index to survival data (1/2NRI(>0) from R package ‘survIDINRI’), estimating NRI using 200 resampling-perturbations and assessing the values at the midpoint of the observation period (5 years).(30, 31)
We used Stata 12.1 (Stata, Inc. College Station, Texas) and R 3.1.1 for all analyses.
Results
The median observation period among REGARDS participants was 6.6 years (IQR 5.1-8.1). There were 1,532 first sepsis (incidence 8.3 per 1,000 person-years; 95% CI: 7.9-8.7) and 1,151 first severe sepsis (incidence 6.2 per 1,000 person-years; 95% CI: 5.8-6.5) events. Median elapsed times to first sepsis and severe sepsis events were 3.7 years (IQR 1.8-5.6) and 3.8 years (IQR 2.0-5.7), respectively. The most common infections associated with first sepsis and first severe sepsis events were pneumonia, kidney and urinary tract infections, and abdominal infections. (Table 1) Most first sepsis cases fulfilled severe sepsis criteria. Intensive care unit admission and hospital and 30-day mortality were higher for first severe sepsis than first sepsis cases.
Table 1. Infection and organ dysfunction types of 1,532 first sepsis and 1,151 first severe sepsis hospitalizations.
| Hospitalization Characteristic | First Sepsis Hospitalizations (n=1,532) N (%) | First Severe Sepsis Hospitalizations (n=1,151) N (%) |
|---|---|---|
| Infection | ||
| Pneumonia | 603 (39.4) | 468 (40.7) |
| Kidney and Urinary Tract Infections | 261 (17.0) | 211 (18.3) |
| Abdominal | 231 (15.1) | 168 (14.6) |
| Bronchitis, Influenza and other Lung Infections | 138 (9.0) | 76 (6.6) |
| Skin and Soft Tissue | 123 (8.0) | 81 (7.0) |
| Sepsis | 104 (6.8) | 98 (8.5) |
| Fever of Unknown Origin | 29 (1.9) | 21 (1.8) |
| Catheter (IV / Central / Dialysis) | 6 (0.4) | 5 (0.4) |
| Surgical Wound | 10 (0.7) | 6 (0.5) |
| Meningitis | 5 (0.3) | 3 (0.3) |
| Unknown/Other | 22 (1.4) | 14 (1.2) |
| Organ Dysfunction* | ||
| Respiratory | 223 (14.6) | 236 (20.5) |
| Renal | 816 (53.3) | 835 (72.6) |
| Hepatic | 224 (14.6) | 228 (19.8) |
| Cardiovascular | 391 (25.5) | 408 (35.5) |
| Hematologic | 290 (18.9) | 295 (25.6) |
| Neurological | 139 (9.1) | 145 (12.6) |
| Shock | 218 (14.2) | 221 (19.2) |
| Severe Sepsis | 1,117 (72.9) | 1,151 (100.0) |
| Intensive Care Unit Admission | 189 (12.3) | 180 (15.6) |
| Death | ||
| Hospital Mortality** | 141 (9.2) | 138 (12.0) |
| 30-Day Mortality*** | 183 (12.0) | 178 (15.5) |
Organ dysfunction types do not sum to total, as participants may have had multiple organ dysfunctions.
Death on hospital discharge.
Death within 30 days of beginning of hospitalization.
Compared with other REGARDS participants, individuals experiencing a sepsis event were older, more likely to be male and white, and more likely to report lower education and annual income. (Table 2) While sepsis participants were more likely to report prior or current tobacco use, they were less likely to report moderate or heavy alcohol use. Chronic medical conditions were more common in sepsis participants. Sepsis participants were more likely to have abnormal hsCRP, Cystatin C, eGFR and ACR. Similar associations were observed with participants experiencing a severe sepsis event.
Table 2. Baseline patient characteristics.
| Characteristic | No Sepsis (n=28,164) N (%) | Sepsis (n=1,532) N (%) | p-value* | No Severe Sepsis (n=28,545) N (%) | Severe Sepsis (n=1,151) N (%) | p-value* |
|---|---|---|---|---|---|---|
| DEMOGRAPHICS | ||||||
| Age | <0.001 | <0.001 | ||||
| <50 | 1,449 (5.1) | 27 (1.8) | 1,459 (5.1) | 17 (1.5) | ||
| 50-59 | 7,479 (26.6) | 280 (18.3) | 7,583 (26.6) | 176 (15.3) | ||
| 60-69 | 10,659 (37.9) | 529 (34.5) | 10,803 (37.9) | 385 (33.5) | ||
| 70-79 | 6,649 (23.6) | 509 (33.2) | 6,745 (23.6) | 413 (35.9) | ||
| ≥80 | 1,928 (6.9) | 187 (12.2) | 1,955 (6.9) | 160 (13.9) | ||
| Gender | <0.001 | <0.001 | ||||
| Male | 12,556 (44.6) | 780 (50.9) | 12,693 (44.5) | 643 (55.9) | ||
| Female | 15,608 (55.4) | 752 (49.1) | 15,852 (55.5) | 508 (44.1) | ||
| Race | <0.001 | <0.001 | ||||
| White | 16,458 (58.4) | 1,019 (66.5) | 16,726 (58.6) | 751 (65.3) | ||
| Black | 11,706 (41.6) | 513 (33.5) | 11,819 (41.4) | 400 (34.8) | ||
| Education | <0.001 | <0.001 | ||||
| Less than high school | 3,460 (12.3) | 250 (16.3) | 3,518 (12.3) | 192 (16.7) | ||
| High school graduate | 7,253 (25.8) | 417 (27.3) | 7,351 (25.8) | 319 (27.8) | ||
| Some college | 7,515 (26.7) | 436 (28.5) | 7,630 (26.8) | 321 (27.9) | ||
| College or higher | 9,915 (35.2) | 427 (27.9) | 10,025 (35.2) | 317 (27.6) | ||
| Missing | 21 | 2 | 21 | 2 | ||
| Income | <0.001 | <0.001 | ||||
| <$20k | 4,981 (20.2) | 366 (26.7) | 5,069 (20.3) | 278 (27.0) | ||
| $20k-$34k | 6,737 (27.3) | 437 (31.9) | 6,839 (27.4) | 335 (32.6) | ||
| $35k-$74k | 8,401 (34.1) | 409 (29.9) | 8,511 (34.1) | 299 (29.1) | ||
| ≥$75k | 4,540 (18.4) | 157 (11.5) | 4,580 (18.3) | 117 (11.4) | ||
| Missing | 3,505 | 163 | 3,546 | 122 | ||
| Geographic Region | 0.009 | 0.105 | ||||
| Stroke Buckle | 5,896 (20.9) | 321 (21.0) | 5,979 (21.0) | 238 (20.7) | ||
| Stroke Belt | 9,704 (34.5) | 582 (38.0) | 9,855 (34.5) | 431 (37.5) | ||
| Non-Belt/Buckle | 12,564 (44.6) | 629 (41.1) | 12,711 (44.5) | 482 (41.9) | ||
| HEALTH BEHAVIORS | ||||||
| Tobacco Use | <0.001 | <0.001 | ||||
| Current | 4,023 (14.3) | 264 (17.3) | 4,100 (14.4) | 187 (16.4) | ||
| Past | 11,202 (39.9) | 709 (46.5) | 11,350 (39.9) | 561 (49.0) | ||
| Never | 12,833 (45.7) | 551 (36.2) | 12,988 (45.7) | 396 (34.6) | ||
| Missing | 106 | 8 | 107 | 7 | ||
| Alcohol Use | <0.001 | 0.001 | ||||
| None | 17,231 (62.4) | 1,015 (67.6) | 17,478 (62.5) | 768 (67.9) | ||
| Moderate | 9,263 (33.5) | 430 (28.6) | 9,365 (33.5) | 328 (29.0) | ||
| Heavy | 1,120 (4.1) | 57 (3.8) | 1,142 (4.1) | 35 (3.1) | ||
| Missing | 550 | 30 | 560 | 20 | ||
| CHRONIC MEDICAL CONDITIONS | ||||||
| Atrial Fibrillation | 2,343 (8.3) | 206 (13.5) | <0.001 | 2,384 (8.6) | 165 (14.7) | <0.001 |
| Cancer | 2,424 (8.6) | 192 (12.5) | <0.001 | 2,475 (8.7) | 141 (12.3) | <0.001 |
| Chronic Lung Disease | 2,446 (8.7) | 286 (18.7) | <0.001 | 2,536 (8.9) | 196 (17.0) | <0.001 |
| Coronary Artery Disease | 4,808 (17.1) | 426 (27.8) | <0.001 | 4,866 (17.4) | 368 (32.8) | <0.001 |
| Diabetes | 6,210 (22.1) | 496 (32.4) | <0.001 | 6,295 (22.1) | 411 (35.9) | <0.001 |
| Deep Vein Thrombosis | 1,431 (5.1) | 125 (8.2) | <0.001 | 1,457 (5.1) | 99 (8.7) | <0.001 |
| Dyslipidemia | 15,999 (59.0) | 964 (65.5) | <0.001 | 16,220 (59.0) | 743 (67.4) | <0.001 |
| Hypertension | 16,503 (58.6) | 1,048 (68.4) | <0.001 | 16,738 (58.8) | 813 (70.9) | <0.001 |
| Myocardial Infarction | 3,417 (12.4) | 304 (20.3) | <0.001 | 3,459 (12.4) | 262 (23.3) | <0.001 |
| Obesity | 14,940 (53.1) | 929 (60.6) | <0.001 | 15,165 (53.2) | 704 (61.3) | <0.001 |
| Peripheral Artery Disease | 596 (2.1) | 68 (4.4) | <0.001 | 610 (2.1) | 54 (4.7) | <0.001 |
| Stroke | 1,740 (6.2) | 158 (10.3) | <0.001 | 1,765 (6.2) | 133 (11.6) | <0.001 |
| BIOMARKERS | ||||||
| High-Sensitivity C-Reactive Protein | 10,543 (40.0) | 746 (52.8) | <0.001 | 10,725 (40.1) | 564 (53.0) | <0.001 |
| >3.0 mg/dL | ||||||
| Missing | 1,781 | 118 | 1,812 | 87 | ||
| Cystatin-C ≥1.11 mg/dL | 6,555 (25.0) | 644 (45.7) | <0.001 | 6,637 (25.0) | 562 (53.0) | <0.001 |
| Missing | 1,906 | 124 | 1,939 | 91 | ||
| eGFR (creatinine) | 2,942 (10.9) | 308 (21.1) | <0.001 | 2,957 (10.8) | 293 (26.8) | <0.001 |
| <60 mL/min/1.73m2 | ||||||
| Missing | 1,192 | 73 | 1,209 | 56 | ||
| Albumin-to-Creatinine Ratio | 3,939 (14.7) | 371 (25.7) | <0.001 | 4,000 (14.7) | 310 (28.7) | <0.001 |
| ≥ 30 mcg/mg (%) | ||||||
| Missing | 1,316 | 90 | 1,335 | 71 |
From Pearson chi-square test of association.
Sepsis Risk Score (SRS) Model 1 contained 13 independent predictors, of which chronic lung disease and age≥75 years were the most influential variables. (Table 3) In addition to incorporating the biomarkers (abnormal hsCRP, Cyst-C, ACR and eGFR), SRS Model 2 also retained all Model 1 predictors. Chronic lung disease, age≥75 years, peripheral artery disease, elevated hsCRP, elevated Cystatin-C and elevated ACR were the most influential SRS Model 2 variables.
Table 3. Derivation of candidate Sepsis Risk Score (SRS) and Severe Sepsis Risk Score (SSRS) models.
|
|
|
|
||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SEPSIS RISK SCORE (SRS) | SEVERE SEPSIS RISK SCORE (SSRS | |||||||||||
|
|
|
|||||||||||
|
SRS MODEL 1 (Demographics, Health Behaviors, and Chronic Medical Conditions) |
SRS MODEL 2 (Add Biomarkers) |
SSRS MODEL 1 (Demographics, Health Behaviors, and Chronic Medical Conditions) |
SSRS MODEL 2 (Add Biomarkers) |
|||||||||
|
|
|
|
||||||||||
| Variable | HR (95% CI) | β | SRS Points† | HR (95% CI) | β | SRS Points† | HR (95% CI) | β | SSRS Points† | HR (95% CI) | β | SSRS Points† |
|
|
|
|
||||||||||
| Demographics, Health Behaviors and Chronic Medical Conditions | ||||||||||||
| Chronic lung disease | 2.18 (1.92-2.48) | 0.78 | 5 | 2.09 (1.83-2.38) | 0.74 | 5 | 1.93 (1.65-2.25) | 0.66 | 2 | 1.84 (1.58-2.15) | 0.61 | 5 |
| Age ≥75 years | 1.94 (1.73-2.18) | 0.66 | 4 | 1.62 (1.44-1.83) | 0.48 | 3 | 2.23 (1.96-2.54) | 0.80 | 3 | 1.71 (1.50-1.97) | 0.54 | 4 |
| Peripheral Artery Disease | 1.66 (1.29-2.13) | 0.50 | 3 | 1.50 (1.17-1.93) | 0.41 | 3 | 1.57 (1.18-2.09) | 0.45 | 2 | 1.41 (1.06-1.87) | 0.34 | 3 |
| Diabetes | 1.56 (1.40-1.75) | 0.45 | 3 | 1.41 (1.25-1.58) | 0.34 | 2 | 1.74 (1.53-1.98) | 0.55 | 2 | 1.52 (1.33-1.73) | 0.42 | 3 |
| Tobacco use (current) | 1.56 (1.36-1.78) | 0.44 | 3 | 1.40 (1.22-1.61) | 0.34 | 2 | 1.48 (1.26-1.74) | 0.39 | 1 | 1.33 (1.13-1.57) | 0.29 | 2 |
| White Race | 1.45 (1.29-1.62) | 0.37 | 2 | 1.46 (1.31-1.64) | 0.38 | 3 | 1.34 (1.18-1.52) | 0.29 | 1 | 1.34 (1.17-1.52) | 0.29 | 2 |
| Stroke | 1.42 (1.20-1.69) | 0.35 | 2 | 1.30 (1.10-1.54) | 0.27 | 2 | 1.52 (1.26-1.83) | 0.42 | 2 | 1.36 (1.13-1.63) | 0.30 | 2 |
| Atrial Fibrillation | 1.39 (1.19-1.61) | 0.33 | 2 | 1.31 (1.12-1.52) | 0.27 | 2 | 1.42 (1.20-1.68) | 0.35 | 1 | 1.32 (1.11-1.56) | 0.28 | 2 |
| Coronary artery disease | 1.35 (1.19-1.53) | 0.30 | 2 | 1.25 (1.10-1.41) | 0.22 | 2 | 1.54 (1.34-1.76) | 0.43 | 2 | 1.37 (1.20-1.57) | 0.32 | 2 |
| Obesity | 1.34 (1.21-1.50) | 0.30 | 2 | 1.20 (1.07-1.34) | 0.18 | 1 | 1.38 (1.21-1.56) | 0.32 | 1 | 1.21 (1.06-1.38) | 0.19 | 1 |
| Hypertension | 1.31 (1.16-1.46) | 0.27 | 2 | 1.15 (1.02-1.29) | 0.14 | 1 | 1.36 (1.19-1.55) | 0.31 | 1 | 1.15 (1.00-1.31) | 0.14 | 1 |
| Deep Vein Thrombosis | 1.27 (1.05-1.54) | 0.24 | 1 | 1.20 (1.00-1.46) | 0.19 | 1 | 1.31 (1.06-1.63) | 0.27 | 1 | 1.23 (0.99-1.52) | 0.21 | 2 |
| Male Sex | 1.19 (1.07-1.32) | 0.17 | 1 | 1.23 (1.11-1.37) | 0.21 | 1 | 1.43 (1.27-1.62) | 0.36 | 1 | 1.49 (1.32-1.69) | 0.40 | 3 |
| Biomarkers | ||||||||||||
| hsCRP >3.0 mg/dL | 1.48 (1.32-1.65) | 0.39 | 3 | 1.50 (1.31-1.71) | 0.40 | 3 | ||||||
| Cystatin-C ≥1.11 mg/dL | 1.62 (1.43-1.84) | 0.48 | 3 | 1.94 (1.66-2.26) | 0.66 | 5 | ||||||
| eGFR (creatinine) | 1.15 (1.00-1.33) | 0.14 | 1 | 1.36 (1.15-1.59) | 0.30 | 2 | ||||||
| <60 mL/min/1.73m2 | ||||||||||||
| ACR ≥ 30 mcg/mg | 1.45 (1.27-1.64) | 0.37 | 3 | 1.45 (1.26-1.67) | 0.37 | 3 | ||||||
| TOTAL RISK SCORE POINTS | 32 | 38 | 20 | 45 | ||||||||
|
|
|
|
||||||||||
Points calculated as the Cox regression β value divided by the smallest β coefficient and rounded to the nearest integer. Scoring system based on pooled estimates following multiple imputation. Total N = 29,696.
We partitioned each SRS model into five graded risk categories (very low, low, medium, high and very high). (Table 4, Figure 1) The regression and point forms of SRS Model 1 exhibited fair discrimination but poor calibration. (Supplemental Digital Content -- Appendix 2) The regression and point forms of SRS Model 2 exhibited good discrimination and good calibration. SRS Model 2 exhibited higher classification accuracy than SRS Model 1 (NRI=0.180). Observed and predicted sepsis risks were similar for both Model 1 and Model 2 over PI deciles. (Supplemental Digital Content -- Appendix 3) Based on these measures, SRS Model 2 exhibited the best combination of model fit and discrimination.
Table 4. Sepsis risk categories for candidate Sepsis Risk Score (SRS) and Severe Sepsis Risk Score (SSRS) models.
|
|
|
|
||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SEPSIS RISK SCORE (SRS) | SEVERE SEPSIS RISK SCORE (SSRS) | |||||||||||
|
|
|
|||||||||||
|
SRS MODEL 1 (Demographics, Health Behaviors, and Chronic Medical Conditions) |
SRS MODEL 2 (Add Biomarkers) |
SSRS MODEL 1 (Demographics, Health Behaviors, and Chronic Medical Conditions) |
SSRS MODEL 2 (Add Biomarkers) |
|||||||||
|
|
|
|||||||||||
| Risk Category | Points | N (%) |
Events per 1,000 PY (95% CI) | Points | N (%) |
Events per 1,000 PY (95% CI) | Points | N (%) |
Events per 1,000 PY (95% CI) | Points | N (%) | Events per 1,000 PY (95% CI) |
|
|
|
|
||||||||||
| I - Very Low | 0-3 | 5,677 (19.2) |
3.0 (2.5-3.6) |
0-3 | 3,599 (12.1) |
2.3 (1.8-3.1) |
0-2 | 9,145 (30.8) |
1.8 (1.5-2.1) |
0-5 | 7,965 (26.8) | 1.5 (1.2-1.9) |
| II - Low | 4-6 | 9,045 (30.5) |
5.4 (4.9-6.1) |
4-6 | 8,130 (27.4) |
4.1 (3.6-4.7) |
3-4 | 9,122 (30.7) |
5.1 (4.6-5.7) |
6-9 | 8,323 (28.0) |
3.4 (3.0-3.9) |
| III - Medium | 7-9 | 7,973 (26.9) |
8.6 (7.8-9.4) |
7-9 | 7,237 (24.4) |
6.5 (5.8-7.3) |
5-6 | 6,312 (21.3) |
7.4 (6.6-8.4) |
10-13 | 6,075 (20.5) |
6.7 (5.9-7.6) |
| IV - High | 10-12 | 4,429 (14.9) |
13.4 (12.1-14.9) |
10-12 | 4,992 (16.8) |
9.7 (8.7-10.9) |
7-8 | 3,341 (11.3) |
12.4 (10.9-14.1) |
14-17 | 3,807 (12.8) |
9.9 (8.7-11.2) |
| V - Very High | 13-32 | 2,572 (8.9) |
24.1 (21.6-26.9) |
13-38 | 5,738 (19.3) |
21.1 (19.5-22.8) |
9-20 | 1,776 (6.0) |
23.4 (20.4-26.7) |
18-45 | 3,526 (11.9) |
22.1 (20.0-24.4) |
|
|
|
|
||||||||||
Total N = 29,696 participants. PY=person-years; CI= confidence interval.
Figure 1.
Comparison of Sepsis Risk Score (SRS) and Severe Sepsis Risk Score (SSRS) models. For both SRS and SSRS, Model 1 contains participant demographics, health behaviors and chronic medical conditions, and Model 2 adds biomarkers (eGFR, Cystatin-C, ACR, hsCRP). Total N = 29,696.
Severe Sepsis Risk Score (SSRS) Model 1 contained the same 13 independent predictors as SRS Model 1. (Table 3) In addition to incorporating the biomarkers (abnormal hsCRP, Cyst-C, ACR and eGFR), SSRS Model 2 also retained all Model 1 predictors. Compared with SSRS Model 1, SSRS Model 2 exhibited the best combination of model fit and discrimination. (Table 4, Figure 1, Supplemental Digital Content -- Appendix 2, Supplemental Digital Content -- Appendix 3)
Discussion
Current scientific and clinical initiatives conceptualize sepsis as an acute condition.(32) However, as with other acute conditions such as myocardial infarction and stroke, immense public health gains could be appreciated if sepsis were recognized as the preventable end result of a chronic disease process.(33) Important steps in disease prevention include not only the identification of individual factors associated with the condition but also the development of strategies to characterize individuals' cumulative risk. We previously identified a range of individual factors associated with future sepsis events.(11, 12, 17) The current analysis provides important and novel advancements, offering risk prediction tools characterizing an individual's 10-year risk of sepsis and severe sepsis based upon their pattern of comorbidities. There have been no prior similar efforts to estimate or characterize long-term sepsis risk in this fashion. Our study sets the stage for risk prediction and prevention as potential strategies to reduce the public health burden of sepsis.
The clinical application of the SRS and SSRS is in contexts that may not be familiar to critical and acute care practitioners. These risk scores are potentially useful for outpatients to prevent or mitigate the early effects of sepsis. For example, knowledge of an individual's baseline sepsis risk may impact outpatient or early hospital care of infections, prompting earlier antibiotic therapy, or inpatient rather than outpatient treatment. Key elements in the SRS and SSRS such as chronic lung disease, peripheral artery disease, tobacco use, coronary artery disease, obesity, hypertension, deep vein thrombosis and chronic kidney disease, can be modified or optimally managed. Should a novel therapy prove effective at sepsis prevention, the SRS and SSRS could be used to select high risk individuals most likely to benefit from the intervention. For example, hsCRP has been used to identify high cardiovascular risk individuals most likely to benefit from statin therapy.(34)
Our derivation of separate sepsis and severe sepsis risk prediction models illustrates the robustness of the models as well as the potential choices influencing clinical implementation. Some may favor the SRSS because of its higher discrimination and focus on predicting higher acuity severe sepsis events. However, the predictors of the SRS and SSRS were nearly identical; this observation is not surprising as 70% of sepsis events fulfilled severe sepsis criteria. Also, we used clinical data limited to the first 28-hours of hospitalization, and it is possible that some sepsis hospitalizations may have later fulfilled severe sepsis criteria. Therefore, use of the SRS to predict the broader category of sepsis (vs. severe sepsis) events may in fact have clinical utility. Comparison of the SRS and SSRS is important in future external validation efforts.
Prior efforts have described sepsis prediction rules such as the Mortality in Emergency Department Sepsis (MEDS) Score, Predisposition, Infection, Response and Organ failure (PIRO) staging system and Mortality in Severe Sepsis in the Emergency Department (MISSED).(35-37) However, these decision rules are designed for inpatients, focusing upon the outcomes of those already afflicted by sepsis. Our SRS and SSRS offer a very different outpatient application, identifying the long-term risk of sepsis and severe sepsis among community-dwelling adults at a stable phase of health, prior to the onset of disease.
Naturally, additional efforts are needed prior to SRS or SSRS implementation in clinical practice, including validation with an independent data set. However, the current risk scores are highly feasible and could be easily implemented in outpatient settings. While not currently in broad clinical practice, outpatient measurements of hsCRP and Cyst-C (key components of SRS Model 2 and SSRS Model 2) are attainable. The conceptual frameworks of the SRS and SSRS allow for incorporation of additional participant factors or biomarkers to improve risk prediction. For example, in a nested case-control series we have linked markers of endothelial cell activation and inflammation with increased odds of sepsis.(38)
Limitations
REGARDS is not a surveillance study and may have under-identified sepsis events. REGARDS included only African Americans and whites ≥45 years. REGARDS did not include nursing home patients, who may be more vulnerable to sepsis than community dwelling individuals. We did not have information on the presence of select immunosuppressive comorbidities such as human immunodeficiency virus infection or liver disease. We opted not to incorporate information on the use of immunosuppressive medications such as corticosteroids and rheumatologic agents because the use of such medications in REGARDS is low (<3%). While we used biomarkers that were readily available, other biomarkers may have potentially improved model discrimination. We could not account for changes in participants characteristics over time.
Our analysis consisted entirely of medical record-verified community-acquired sepsis and severe sepsis. We are currently linking REGARDS participants to their corresponding Medicare claims, which may allow identification of hospital-acquired or healthcare-associated sepsis.(16)
While we derived the risk prediction models using Cox regression, other techniques are possible (-e.g., Classification and Regression Tree analysis).(39) We note that the current SRS and SSRS exhibited calibration and discrimination comparable to other commonly used clinical risk prediction models.(40) While we conceptualized the SRS and SSRS as having roles in primary prevention, further study is also needed to confirm if risk modification in fact leads to reduced sepsis risk.
Conclusions
We derived a Sepsis Risk Score and a Severe Sepsis Risk Score that accurately predict 10-year risks of sepsis and severe sepsis among community-dwelling adults. The SRS and SSRS may play key roles in community sepsis prevention or mitigation efforts.
Supplementary Material
Acknowledgments
Financial Support: This study was supported by award R01-NR012726 from the National Institute for Nursing Research, UL1-RR025777 from the National Center for Research Resources, as well as by grants from the Center for Clinical and Translational Science and the Lister Hill Center for Health Policy of the University of Alabama at Birmingham. The parent REGARDS study was supported by cooperative agreement U01-NS041588 from the National Institute of Neurological Disorders and Stroke, National Institutes of Health, Department of Health and Human Service. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. Representatives of the funding agencies have been involved in the review of the manuscript but not directly involved in the collection, management, analysis or interpretation of the data.
The authors thank the other investigators, the staff, and the participants of the REGARDS study for their valuable contributions. A full list of participating REGARDS investigators and institutions can be found at http://www.regardsstudy.org and http://www.regardssepsis.org.
Mr. Donnelly is currently supported by grant T32-HS013852 from the Agency for Healthcare Research and Quality, Rockville, MD, USA.
Copyright form disclosures: Dr. Wang received grant funding from the National Institutes of Health (NIH) and received support for article research from the NIH. Dr. Donnelly received support for article research from the NIH. His institution received funding from the AHRQ Predoctoral Fellowship in Health Services Research. Dr. Levitan received funding from Robinson Calcagnie Robinson Shapiro Davis, Inc. His institution received funding from Amgen. Dr. Shapiro received support from Rapid Pathogen Screening, Cumberland pharma, thermo-fischer, and siemens and he received support for article research from the NIH. His institution received support from the NIH. Dr. Howard received support for article research from the NIH. Dr. Safford received support for article research from the NIH and received funding from diaDexus, Inc. Dr. Griffen disclosed that he does not have any potential conflicts of interest.
Footnotes
Dr. Wang, Donnelly, Griffin and Shapiro do not report any related conflicts of interest.
Author Contributions: HEW, NIS, GH, EBL, RG and MMS conceived the study. HEW, MMS, NIS and GH obtained funding. HEW, GH, RG and MMS oversaw data collection. HEW, JPD, RG and EBL conducted the analysis. HEW drafted the manuscript and all authors contributed to its critical review. HEW assumes overall responsibility for the paper.
Conflicts of Interest: Dr. Safford reports the following potential conflicts of interest: Amgen - salary support to study patterns of statin use in Medicare and other large databases; diaDexus - salary support for a research grant on lipids and CHD outcomes; diaDexus - consulting to help with FDA application; NIH, AHRQ - salary support for research grants.
Dr. Levitan receives research support from Amgen.
References
- 1.Katz DL, Ali A. Preventive Medicine, Integrative Medicine, and the Health of the Public. Commissioned paper for Institute of Medicine (IOM) of the National Academies. Summit on Integrative Medicine and the Health of the Public. 2009 In. [Google Scholar]
- 2.Corday E, Corday SR. Advances in clinical management of acute myocardial infarction in the past 25 years. Journal of the American College of Cardiology. 1983;1(1):126–132. doi: 10.1016/s0735-1097(83)80017-5. [DOI] [PubMed] [Google Scholar]
- 3.Marshall RS, Mohr JP. Current management of ischaemic stroke. Journal of neurology, neurosurgery, and psychiatry. 1993;56(1):6–16. doi: 10.1136/jnnp.56.1.6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Centers for Disease C, Prevention. Ten great public health achievements--United States, 2001-2010. MMWR Morb Mortal Wkly Rep. 2011;60(19):619–623. [PubMed] [Google Scholar]
- 5.Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Intensive care medicine. 2003;29(4):530–538. doi: 10.1007/s00134-003-1662-x. [DOI] [PubMed] [Google Scholar]
- 6.Angus DC, Linde-Zwirble WT, Lidicker J, et al. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Critical care medicine. 2001;29(7):1303–1310. doi: 10.1097/00003246-200107000-00002. [DOI] [PubMed] [Google Scholar]
- 7.Wang HE, Shapiro NI, Angus DC, et al. National estimates of severe sepsis in United States emergency departments. Critical care medicine. 2007;35(8):1928–1936. doi: 10.1097/01.CCM.0000277043.85378.C1. [DOI] [PubMed] [Google Scholar]
- 8.Martin GS, Mannino DM, Eaton S, et al. The epidemiology of sepsis in the United States from 1979 through 2000. The New England journal of medicine. 2003;348(16):1546–1554. doi: 10.1056/NEJMoa022139. [DOI] [PubMed] [Google Scholar]
- 9.Dellinger RP, Levy MM, Rhodes A, et al. Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock: 2012. Critical care medicine. 2013;41(2):580–637. doi: 10.1097/CCM.0b013e31827e83af. [DOI] [PubMed] [Google Scholar]
- 10.Yealy DM, Kellum JA, Huang DT, et al. A randomized trial of protocol-based care for early septic shock. The New England journal of medicine. 2014;370(18):1683–1693. doi: 10.1056/NEJMoa1401602. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wang HE, Shapiro NI, Griffin R, et al. Chronic medical conditions and risk of sepsis. PLoS One. 2012;7(10):e48307. doi: 10.1371/journal.pone.0048307. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Wang HE, Griffin R, Judd S, et al. Obesity and risk of sepsis: A population-based cohort study. Obesity (Silver Spring) 2013 doi: 10.1002/oby.20468. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Howard VJ, Cushman M, Pulley L, et al. The reasons for geographic and racial differences in stroke study: objectives and design. Neuroepidemiology. 2005;25(3):135–143. doi: 10.1159/000086678. [DOI] [PubMed] [Google Scholar]
- 14.Wang H, Shapiro N, Deaconsess BI, et al. High-Sensitivity C-Reactive Protein and Risk of Sepsis (abstract) Critical care medicine. 2012;40(12):1–328. 310.1097/1001.ccm.0000424299.0000487210.0000424214. [Google Scholar]
- 15.Vincent JL, Moreno R, Takala J, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive care medicine. 1996;22(7):707–710. doi: 10.1007/BF01709751. [DOI] [PubMed] [Google Scholar]
- 16.Page DB, Donnelly JP, Wang HE. Community-, Healthcare-, and Hospital-Acquired Severe Sepsis Hospitalizations in the University HealthSystem Consortium. Critical care medicine. 2015;43(9):1945–1951. doi: 10.1097/CCM.0000000000001164. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Powell TC, Donnelly JP, Gutierrez OM, et al. Cystatin C and long term risk of community-acquired sepsis: a population-based cohort study. BMC nephrology. 2015;16(1):61. doi: 10.1186/s12882-015-0055-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Hemmelgarn BR, Manns BJ, Lloyd A, et al. Relation between kidney function, proteinuria, and adverse outcomes. JAMA : the journal of the American Medical Association. 2010;303(5):423–429. doi: 10.1001/jama.2010.39. [DOI] [PubMed] [Google Scholar]
- 19.James MT, Hemmelgarn BR, Wiebe N, et al. Glomerular filtration rate, proteinuria, and the incidence and consequences of acute kidney injury: a cohort study. Lancet. 2010;376(9758):2096–2103. doi: 10.1016/S0140-6736(10)61271-8. [DOI] [PubMed] [Google Scholar]
- 20.Wang HE, Shapiro NI, Safford MM, et al. High-sensitivity C-reactive protein and risk of sepsis. PLoS One. 2013;8(7):e69232. doi: 10.1371/journal.pone.0069232. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Critical care medicine. 2013;41(2):580–637. doi: 10.1097/CCM.0b013e31827e83af. [DOI] [PubMed] [Google Scholar]
- 22.Steyerberg EW, Harrell FE, Jr, Borsboom GJ, et al. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. Journal of clinical epidemiology. 2001;54(8):774–781. doi: 10.1016/s0895-4356(01)00341-9. [DOI] [PubMed] [Google Scholar]
- 23.Harrell FE. Regression modeling strategies : with applications to linear models, logistic regression, and survival analysis. New York: Springer; 2001. [Google Scholar]
- 24.Rubin DB, Schenker N. Multiple imputation in health-care databases: an overview and some applications. Statistics in medicine. 1991;10(4):585–598. doi: 10.1002/sim.4780100410. [DOI] [PubMed] [Google Scholar]
- 25.Rubin DB. Multiple imputation after 18+ years. Journal of the American Statistical Association. 1996;91(434):473–489. [Google Scholar]
- 26.Marshall A, Altman DG, Holder RL, et al. Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC medical research methodology. 2009;9:57. doi: 10.1186/1471-2288-9-57. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Vergouwe Y, Royston P, Moons KG, et al. Development and validation of a prediction model with missing predictor data: a practical approach. Journal of clinical epidemiology. 2010;63(2):205–214. doi: 10.1016/j.jclinepi.2009.03.017. [DOI] [PubMed] [Google Scholar]
- 28.Sullivan LM, Massaro JM, D'Agostino RB., Sr Presentation of multivariate data for clinical use: The Framingham Study risk score functions. Statistics in medicine. 2004;23(10):1631–1660. doi: 10.1002/sim.1742. [DOI] [PubMed] [Google Scholar]
- 29.Hosmer DW, Lemeshow S, May S. Applied survival analysis : regression modeling of time-to-event data. 2nd. Hoboken, N.J: Wiley-Interscience; 2008. [Google Scholar]
- 30.Pencina MJ, D'Agostino RB, Sr, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Statistics in medicine. 2011;30(1):11–21. doi: 10.1002/sim.4085. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Uno H, Tian L, Cai T, et al. A unified inference procedure for a class of measures to assess improvement in risk prediction systems with survival data. Statistics in medicine. 2013;32(14):2430–2442. doi: 10.1002/sim.5647. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Dellinger RP, Levy MM, Rhodes A, et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med. 2013;39(2):165–228. doi: 10.1007/s00134-012-2769-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Centers for Disease C, Prevention. Ten great public health achievements--worldwide, 2001-2010. MMWR Morb Mortal Wkly Rep. 2011;60(24):814–818. [PubMed] [Google Scholar]
- 34.Ridker PM, Cannon CP, Morrow D, et al. C-reactive protein levels and outcomes after statin therapy. The New England journal of medicine. 2005;352(1):20–28. doi: 10.1056/NEJMoa042378. [DOI] [PubMed] [Google Scholar]
- 35.Shapiro NI, Wolfe RE, Moore RB, et al. Mortality in Emergency Department Sepsis (MEDS) score: a prospectively derived and validated clinical prediction rule. Critical care medicine. 2003;31(3):670–675. doi: 10.1097/01.CCM.0000054867.01688.D1. [DOI] [PubMed] [Google Scholar]
- 36.Granja C, Povoa P, Lobo C, et al. The predisposition, infection, response and organ failure (Piro) sepsis classification system: results of hospital mortality using a novel concept and methodological approach. PLoS One. 2013;8(1):e53885. doi: 10.1371/journal.pone.0053885. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Sivayoham N, Rhodes A, Cecconi M. The MISSED score, a new scoring system to predict Mortality In Severe Sepsis in the Emergency Department: a derivation and validation study. European journal of emergency medicine : official journal of the European Society for Emergency Medicine. 2014;21(1):30–36. doi: 10.1097/MEJ.0b013e328364a8d4. [DOI] [PubMed] [Google Scholar]
- 38.Wang HE, Shapiro NI, Griffin R, et al. Inflammatory and endothelial activation biomarkers and risk of sepsis: A nested case-control study. Journal of critical care. 2013 doi: 10.1016/j.jcrc.2012.11.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Wong HR, Lindsell CJ, Pettila V, et al. A multibiomarker-based outcome risk stratification model for adult septic shock*. Critical care medicine. 2014;42(4):781–789. doi: 10.1097/CCM.0000000000000106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Muntner P, Colantonio LD, Cushman M, et al. Validation of the atherosclerotic cardiovascular disease Pooled Cohort risk equations. JAMA : the journal of the American Medical Association. 2014;311(14):1406–1415. doi: 10.1001/jama.2014.2630. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.