Abstract
Objective:
To evaluate the screening performance characteristics of existing tools for diagnosis of sepsis during delivery admissions.
Methods:
Electronic Health Record data including vital signs and laboratory results for all delivery admissions with sepsis from 59 nationally distributed hospitals. Sepsis cases were matched by gestational age at delivery in a 1:4 ratio with controls to create a comparison group. Chorioamnionitis sepsis cases were compared to a complete cohort of chorioamnionitis cases without sepsis. Multiple screening criteria for sepsis were evaluated: California Maternal Quality Care Collaborative (CMQCC), systemic inflammatory response syndrome (SIRS), Maternal Early Warning Criteria (MEWC), United Kingdom Obstetric Surveillance System (UKOSS), and the Maternal Early Warning Trigger Tool (MEWT). Sensitivity, false-positive rates and C statistics were reported for each screening tool. Analyses were stratified into cohort 1 which excluded cases of chorioamnionitis/endometritis and cohort 2 which included those cases.
Results:
Delivery admissions of 59 hospitals were extracted for cases of sepsis. Cohort 1 comprised 647 cases of sepsis including 228 cases with end organ injury matched with 2,588 non-sepsis controls. Cohort 2 comprised 14,591 cases of chorioamnionitis/endometritis of which 1,049 had sepsis and 238 had end organ injury. In Cohort 1, the CMQCC and the UKOSS pregnancy-adjusted criteria had the lowest false-positive rates (6.9% and 9.6%) and the highest C-statistics of 0.92 and 0.91, respectively. Although other screening criteria such as SIRS and MEWC had similar sensitivities, it was at the cost of much higher false-positive rates of 21.3% and 38.3%. In Cohort 2 including all chorioamnionitis/endometritis cases, the highest C-statistics were again for CMQCC (0.67) and UKOSS (0.64). All screening tools had high false-positive rates, but the false-positive rate for CMQCC and UKOSS were substantially lower than SIRS and MEWC.
Conclusion:
During the delivery admission, the CMQCC and UKOSS pregnancy-adjusted screening criteria have the lowest false-positive results while maintaining >90% sensitivity rates. Performance of all screening tools was degraded in the setting of chorioamnionitis/endometritis.
Precis:
During delivery admission, California Maternal Quality Care Collaborative and United Kingdom Obstetric Surveillance System pregnancy-adjusted sepsis screening tools have the lowest false-positive results while maintaining >90% sensitivity rates.
Introduction
Maternal sepsis remains a leading cause of death in the United States and worldwide.1,2 Deaths due to sepsis are judged preventable in many maternal mortality review committee reports due to poor recognition leading to delays in treatment.3,4 Early identification and treatment of infection improves outcomes.5 Maternal physiology overlaps with vital sign changes often seen with sepsis.6 This can result in many patients meeting general sepsis screening criteria; these criteria have not been adjusted for pregnancy physiology and the typical changes that occur during the course of labor and delivery. Screening criteria with high false negative rates may delay recognition, while criteria with high false-positive rates will result in overutilization and diversion of resources.
It is unclear how general sepsis screening criteria and pregnancy-adjusted screening criteria perform during the delivery hospitalization. This has been difficult to study, given the rarity of maternal sepsis. Current published studies include single academic centers with limited sample sizes.7,8 The objective of this study was to evaluate the performance characteristics of multiple sepsis screening tools in a large diverse set of United States hospitals during delivery hospitalization to identify the optimal first screening step for assessment of sepsis risk. The secondary objective of this study is to report the frequency and types of end organ injury and the underlying source of infection in patients with maternal sepsis during delivery admission.
Methods
This was a case control study using Electronic Health Record (EHR) data for patients with deliveries between 2016 to 2021 from 59 hospitals. To allow for standardization of data collection, all hospitals were required to be using Epic (Verona, WI) as their electronic health record (EHR) and Epic Clarity as their data storage tool. Hospitals were selected as a convenience sample but do represent diversity in patient demographics, geographic location, types of hospitals (urban, rural, community, university), and delivery volumes. Data including demographic data, vital signs, laboratory data, and ICD-10 (International Classification of Diseases 10th Revision) coding for all diagnoses present on admission and diagnoses and procedures during the encounter) were obtained through Epic Clarity for the delivery hospitalization. Race and Ethnicity was reported in this study and to the funder, National Institutes of Health (NIH), consistent with the Inclusion of Women, Minorities, and Children policy. The race and ethnicity were extracted from the demographic fields in Epic and were presented to describe the study population. We did not analyze the data regarding race or ethnicity. Hospitals were asked to provide institutional information such as delivery volume, teaching hospital status, and self-reported maternal levels of care. The population reported in the 2020–21 census was used to report the size of the community where the hospital was located.9
Duke University Single Institutional Review Board (IRB) approval was obtained through the IRB with all other sites relying on the host site. Written informed consent was waived by the IRB for this retrospective study with deidentified data. The methods and reporting of this study follow the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE).109
The host site created a Structured query language (SQL) (Microsoft Corporation, Redmond, WA) to retrieve data that were then validated for accuracy of identification of infection cases and further modified to improve identification at the host site. Sepsis cases were required to have a minimum length of stay of 48 hours or more to improve accuracy and remove miscoded cases. The host site provided SQL queries to each site to search local Microsoft SQL or Oracle SQL databases hosted on the Epic Clarity database platform. The host site also assisted with modification of the syntax to retrieve de-identified data in identical tables to be collated for analysis.
Delivery hospitalizations were identified using a combination of EPIC Delivery method, standard delivery code set (ICD-10, and DRG codes from the HRSA/AHRQ/Alliance for Innovation on Maternal Health (AIM).11 Sepsis cases were identified using a ICD-10 codes set from the Severe Maternal Morbidity (SMM) code set (updated from CDC codes) now used by states to measure SMM.111 All ICD-10 codes used in this study are reported in Appendix 1, available online at http://links.lww.com/xxx. Sepsis cases were randomly matched in a 1:4 ratio to non-sepsis cases of the same gestational age at delivery. Gestational age matching was conducted because vital signs change during the course of pregnancy and infection and sepsis are associated with prematurity; sepsis cases were matched for gestational age windows (<20 weeks, 20–28 weeks, >28 to 36 weeks and >36 weeks). Controls were selected from the same institution except when the gestational age matched control was unavailable. An appropriate gestational age matched control was then taken from an institution in the same region. Sepsis cases with deliveries at < 20 weeks’ gestation were excluded.
Two cohorts were created for analysis. Cohort 1 excluded patients with chorioamnionitis/endometritis and cohort 2 included them. The rationale for these separate cohorts was that chorioamnionitis and endometritis are clinical diagnoses of variable severity, often not established by bacterial cultures, and are known to have substantial vital sign overlap with SIRS criteria. While American College of Obstetricians and Gynecologists made a distinction in 2017 between Intraamniotic infection/ Chorioamnionitis and fever >38°C in labor without other clinical risk factors, they recommended that both be treated with antibiotics.112,13 Several follow-up studies comparing these two diagnostic criteria found similar maternal and neonatal risks of serious infection complications. 14 Among the 57 facilities in this study there were no standard criteria for chorioamnionitis, and diagnosis was left to the clinical judgment of the clinician.
In both cohorts, end organ injury cases were identified with ICD-10 codes for end organ injury defined by both Angus and Dombrovskiy.15,16 The Sepsis-3 definition requires end organ injury based on SOFA scoring, but the Centers for Medicaid and Medicare Services Core Measure SEP-1: “Early Management Bundle for Severe Sepsis/Septic Shock” still uses the former definition of sepsis (infection with 2 or more SIRS criteria but without needing end organ injury).17 Both ICD-10 codes for sepsis and sepsis with end organ injury were studied.
The primary analysis was to test sensitivity and false-positive rates of sepsis screening criteria among patients during the delivery hospitalization. To test the ability of the tools to identify sepsis against the normal maternal physiology during the delivery hospitalization, we did not include controls with ICD-10 codes indicating hemorrhage due to the known overlap of hemorrhage vital signs with sepsis physiology.
Screening criteria evaluated included systemic inflammatory response syndrome (SIRS), California Maternal Quality Care Collaborative (CMQCC) initial sepsis screen, Maternal Early Warning Trigger Tool (MEWT), United Kingdom Obstetric Surveillance System (UKOSS) obstetric SIRS, and the Maternal Early Warning Criteria (MEWC).18–22 Vital sign and WBC criteria used in these screening tools are shown in Table 1. Several criteria not readily ascertained from EHR extracts including urine output, maternal agitation, confusion or unresponsiveness, shortness of breath, and “ill-appearing”, were not included in this analysis and are also clarified in Table 1. Heart rate, blood pressure, and pulse oximetry were only accepted as abnormal if rechecked within 20 minutes to demonstrate persistent threshold values. Respiratory rates and temperature were classified as abnormal with single values since they would be unlikely to be repeated within 20 minutes. For criteria requiring 2 or more abnormal criteria (SIRS, CMQCC, UKOSS, SIRS, MEWT), the abnormal values were required to be within 2 hours of each other. White blood cell count was required to be within 24 hours of the other abnormal criteria. MEWT had two levels of evaluation: a screen for red flags (severely deranged values), or a screen for two abnormal values that were abnormal but not as severe (yellow alert).19 Either a red or yellow alert was considered positive for this study.
Table 1.
Vital sign and WBC criteria used to screen for sepsis in five widely used tools
| Pregnancy-adjusted screening tools | Screening tools focused more broadly on maternal morbidity |
|||||
|---|---|---|---|---|---|---|
| Criterion | SIRS | CMQCC | UKOSS | MEWC | MEWT (red) | MEWT (yellow) |
| Any two | Any two | Any two | Any one | Any one | Any two | |
| WBC (109 cell/L) | < 4 or > 12 | < 4 or > 15 | < 4 or > 17 | < 4 or > 15 | ||
| Heart rate (beats/min) | > 90 | > 110 | > 100 | < 50 or > 120 | >130 | < 50 or > 110 |
| Respiratory rate (breaths/min) | > 20 | > 24 | > 20 | < 10 or > 24 | >30 | > 24 or < 10 |
| Temperature (°C) | < 36 or > 38 | < 36 or > 38 | < 36 or > 38 | < 36 or > 38 | < 36 or > 38 | |
| Pulse Oximetry (%) | < 95 | <90 | < 93 | |||
| Blood Pressure (mm Hg) | <90 or >160/100 | >160/110 | <85/45 | |||
| Mean arterial pressure (mm Hg) | <55 | |||||
| Exclusions for this study | None | None | None | None | Urine output, maternal agitation, confusion, or unresponsiveness; Patient with preeclampsia reporting a non-remitting headache or shortness of breath | Nurse concern |
Descriptive statistics were calculated and are reported as number and percentage for categorical variables or median and interquartile range for continuous variables. These statistics are reported for sepsis cases and gestational age-matched controls, and separately for sepsis patients with and without end organ injury. Characteristics of contributing hospitals are also summarized and reported as a percentage of contributing hospitals. Sensitivity and false-positive rates are reported for each screening score overall and for each individual component of each score. For each overall score, an exact 95% confidence interval was constructed for sensitivity and specificity using the Clopper-Pearson method and a C-statistic representing the area under the receiver operating characteristic (ROC) curve was calculated. A post-hoc sensitivity analysis was conducted excluding sepsis cases with hemorrhage due to potential overlap of vital sign abnormalities in patients with hemorrhage and sepsis.
Results
Characteristics of the 59 hospitals from 12 states in the study group are presented in Table 2. Hospitals were generally representative of birthing facilities in the United States: all geographic regions were represented; their annual birthing volume distribution matched closely to the U.S. (41% had an annual delivery volume under 1,000); and most (81%) were not teaching hospitals. We identified 1,761 patients with sepsis during the delivery admission from an estimated > 500,000 deliveries (some hospitals could only provide an estimate of the total delivery volume in this time period). Cases were excluded for a hospital length of stay <48hrs or a transfer of care (n=48), or delivery < 20 weeks’ gestation (n=17). Overall 647 sepsis cases and 2,588 control patients without sepsis or chorioamnionitis/endometritis were included in cohort 1. Cohort 2 included all cases of chorioamnionitis/endometritis (n=14,591, 6.5% were endometritis alone) during the delivery admission from the same hospitals. This cohort included 1,049 cases of sepsis (Figure 1). End organ injury codes were identified among 228 (35.2%) patients with sepsis in cohort 1 and in 238 (22.7%) patients in cohort 2. Patient demographics and comorbidity data for both cohorts are presented in Table 3.
Table 2.
Study Hospital Characteristics
| Hospital Characteristics | N=59 |
|---|---|
| Region | |
| Northeast: Massachusetts | 5 (8.5%) |
| South: Arkansas-North Carolina-Texas | 7 (11.9%) |
| Midwest: Illinois-Michigan | 12 (20.3%) |
| Prairie: Iowa-Nebraska-North Dakota | 9 (15.3%) |
| West: California-Washington | 26 (44.1%) |
| Annual Hospital Delivery Volume* | |
| < 500 | 11 (19.0%) |
| 500–999 | 13 (22.4%) |
| 1000–1999 | 13 (22.4%) |
| 2000–3999 | 12 (20.7%) |
| > 4000 | 9 (15.5%) |
| Teaching Status* | |
| Major Teaching Hospital (OB-GYN residency core site) | 11 (19.0%) |
| Non-teaching hospital | 47 (81.0%) |
| City Population Where Hospital is Located | |
| < 50,000 | 17 (28.8%) |
| 50,000–99,999 | 15 (25.4%) |
| 100,000–249,999 | 10 (16.9%) |
| 250,000–499,999 | 6 (10.2%) |
| > 500,000 | 11 (18.7%) |
One hospital each had unknown yearly delivery volume and teaching status
Figure 1.
Study flow chart identifying the two cohorts and the six study groups. Electronic medical record data included vital signs, laboratory results, and International Classification of Diseases 10th Revision (ICD-10) codes were collected for all patients in the six study groups.
Table 3.
Patient characteristics for each of the study populations
| COHORT 1: Cases excluding Chorioamnionitis and Endometritis | COHORT 2: Cases with Chorioamnionitis and Endometritis | |||||
|---|---|---|---|---|---|---|
| Characteristics | Sepsis cases (n=647) | Sepsis cases with End organ injury (n=228) | Comparison (n=2588) (Matched for gestational age) | Sepsis cases (n=1049) | Sepsis cases with End organ injury (n=238) | Chorio/endo without sepsis (n=13542) |
| Age, Mean [IQR] | 30 [25–34] | 31 [27–35] | 32 [27–36] | 30 [25–35] | 32 [28–35] | 30 [26–34] |
| Race and Ethnicity, n (%) * | ||||||
| Asian/Pacific Islander | 121 (18.7 %) | 42 (18.5%) | 321 (12.4%) | 288 (26.5%) | 74 (31.1%) | 2288 (16.9%) |
| Black | 97 (15.0%) | 38 (16.7%) | 392 (15.1%) | 125 (11.9%) | 36 (15.1%) | 2080 (15.4%) |
| Hispanic | 194 (30.0%) | 60 (26.3%) | 647 (25.0%) | 356 (33.9%) | 74 (31.1%) | 3020 (22.3%) |
| White | 198 (30.6%) | 74 (32.5%) | 1096 (42.3%) | 226 (21.5%) | 42 (17.6%) | 5231 (38.6%) |
| Additional races and ethnicities | 28 (4.3%) | 10 (4.4%) | 76 (2.9%) | 50 (4.8%) | 10 (4.2%) | 553 (4.1%) |
| Type of insurance, n (%) * | ||||||
| Commercial | 368 (58.5%) | 112 (52.3%) | 1558 (61.9%) | 684 (65.7%) | 148 (62.2%) | 8791 (69.6%) |
| Medicaid | 243 (38.6%) | 94 (43.9%) | 910 (36.2%) | 345 (33.1%) | 83 (34.9%) | 3517 (27.8%) |
| Gestational age (at time of sepsis), n(%) | ||||||
| < 20 weeks | 0 (0%) | 0 (0%) | 0 (0%) | 9 (0.9%) | 5 (2.1%) | 68 (0.5%) |
| 20 weeks to 27 weeks 6 days | 30 (4.6%) | 26 (11.4%) | 120 (4.6%) | 63 (6.0%) | 21 (8.8%) | 372 (2.7%) |
| 28 weeks to 36 weeks 6 days | 174 (26.9%) | 94 (41.2%) | 696 (26.9%) | 110 (10.5%) | 26 (10.9%) | 827 (6.1%) |
| ≥ 37 weeks | 443 (68.5%) | 108 (47.4%) | 1772 (68.5%) | 867 (82.7%) | 186 (78.2%) | 12119 (89.5%) |
| Mode of delivery | ||||||
| Cesarean delivery | 308 (47.6%) | 136 (59.6%) | 946 (36.6%) | 523 (49.9%) | 129 (54.2%) | 5810 (42.9%) |
| Length of stay (days) Mean [IQR] | 4.7 [3.2–7.7] | 7.4 [4.6–15.8] | 2.7 [2.2–3.5] | 4.4 [3.3–5.8] | 5.0 [4.0–7.1] | 3.5 [2.9–4.6] |
| Cesarean Delivery | 5.9 [4.3–10.6] | 9.1 [5.3–16.6] | 3.5 [3.0–4.4] | 5.1 [4.3–6.9] | 5.7 [4.8–7.9] | 4.5 [3.7–5.3] |
| Vaginal Delivery | 3.6 [2.8–5.6] | 6.0 [3.6–12.6] | 2.4 [1.9–2.8] | 3.5 [2.9–4.4] | 4.2 [3.2–5.1] | 3.0 [2.6–3.6] |
| Preexisting Comorbidities | ||||||
| Obesity | 164 (25.3%) | 64 (28.1%) | 468 (18.1%) | 198 (18.9%) | 42 (17.6%) | 2304 (17.0%) |
| Diabetes mellitus | 31 (4.8%) | 16 (7.0%) | 67 (2.6%) | 23 (2.2%) | 7 (2.9%) | 237 (1.8%) |
| Systemic lupus erythematous | 9 (1.4%) | 6 (2.6%) | 20 (0.8%) | 6 (0.6%) | 2 (0.8%) | 56 (0.4%) |
| Sickle cell disease | 6 (0.9%) | 2 (0.9%) | 3 (0.1%) | 3 (0.3%) | 1 (0.4%) | 36 (0.3%) |
| Chronic renal disease | 7 (1.1%) | 3 (1.3%) | 8 (0.3%) | 4 (0.4%) | 2 (0.8%) | 12 (0.1%) |
| Chronic heart disease | 82 (12.7%) | 67 (29.4%) | 56 (2.2%) | 71 (6.8%) | 43 (18.1%) | 320 (2.4%) |
| Chronic Hypertension | 64 (9.9%) | 41 (18.0%) | 171 (6.6%) | 59 (5.6%) | 16 (6.7%) | 573 (4.2%) |
| Asthma | 195 (30.1%) | 119 (52.2%) | 228 (8.8%) | 168 (16.0%) | 54 (22.7%) | 1126 (8.3%) |
| Obstetric Comorbidities | ||||||
| Gestational Diabetes | 80 (12.4%) | 23 (10.1%) | 302 (11.7%) | 104 (9.9%) | 26 (10.9%) | 1362 (10.1%) |
| Gestational Hypertension | 62 (9.6%) | 23 (10.1%) | 216 (8.3%) | 110 (10.5%) | 28 (11.8%) | 1388 (10.2%) |
| Preeclampsia | 104 (16.1%) | 59 (25.9%) | 281 (10.9%) | 151 (14.4%) | 43 (18.1%) | 1205 (8.9%) |
| Multiple gestation | 20 (3.1%) | 9 (3.9%) | 150 (5.8%) | 29 (2.8%) | 10 (4.2%) | 193 (1.4%) |
| Preterm premature rupture of membranes | 36 (5.6%) | 13 (5.7%) | 245 (9.5%) | 114 (10.9%) | 35 (14.7%) | 854 (6.3%) |
| Stillbirth | 21 (3.2%) | 16 (7.0%) | 56 (2.2%) | 31 (3.0%) | 17 (7.1%) | 234 (1.7%) |
| Preterm delivery | 210 (32.5%) | 123 (53.9%) | 820 (31.7%) | 187 (17.8%) | 54 (22.7%) | 1400 (10.3%) |
Suppressed due to small numbers: Race: American Indian/Alaska Native, Unknown. Insurance status: Medicare, Other Public, Self-pay.
Insurance status missing for 18 sepsis patients and 72 control patients due to data use agreement at one hospital prohibiting sharing insurance status.
Data are n (%) or median (interquartile range)
Performance of screening criteria during the delivery hospitalization in patients with ICD-10 codes for sepsis and sepsis with end organ injury are presented in Table 4. Full sets of vital signs per admission, including temperature, were obtained among Cohort 1 sepsis cases for a median frequency of 36 full sets per case [IQR: 24, 61]; in the non-sepsis controls for a median frequency of 14 full sets per case [IQR:10, 19]; in Cohort 2 chorioamnionitis/endometritis cases with sepsis for a median frequency of 34 full sets per case [IQR: 25, 46]; and for the chorioamnionitis/endometritis without sepsis controls for a median frequency of 24 full sets per case [IQR: 19, 32].
Table 4.
Performance of Screening Tools for Intrapartum Sepsis and Sepsis with End Organ Injury defined by ICD-10 codes
| COHORT 1: Cases excluding chorioamnionitis and endometritis cases | ||||||
|---|---|---|---|---|---|---|
| Sepsis by Diagnosis Codes | Sepsis with End Organ Injury by Diagnosis Codes | |||||
| Screening System | False Positive Rate (95% CI) (in patients without sepsis codes, n=2,588) | Sensitivity (95% CI) (n=647 sepsis cases) | C statistic (95% CI) | False Positive Rate (95% CI) (In patients without sepsis codes, n=912) | Sensitivity (95% CI) (n=228 sepsis cases with end organ injury) | C statistic (95% CI) |
| CMQCC | 6.9% (6.0–8.0) |
90.6% (88.1–92.7) |
0.92 (0.91–0.93) |
9.2% (7.4–11.3) |
96.9% (93.8–98.8) |
0.94 (0.92–0.95) |
| SIRS | 21.3% (19.7–22.9) |
96.9% (95.3–98.1) |
0.88 (0.87–0.89) |
23.9% (21.2–26.8) |
98.7% (96.2–99.7) |
0.87 (0.86–0.89) |
| MEWC | 38.3% (36.5–40.2) |
96.9% (95.3–98.1) |
0.79 (0.78–0.80) |
43.9% (40.6–47.2) |
98.2% (95.6–99.5) |
0.77 (0.75–0.79) |
| UKOSS | 9.6% (8.5–10.8) |
92.0% (89.6–93.9) |
0.91 (0.90–0.92) |
11.6% (9.6–13.9) |
96.1% (92.6–98.2) |
0.92 (0.91–0.94) |
| MEWT (overall) |
15.8% (14.4–17.3) |
79.9% (76.6–82.9) |
0.82 (0.80–0.84) |
19.8% (17.3–22.6) |
90.8% (86.3–94.2) |
0.85 (0.83–0.88) |
| COHORT 2: Cases including chorioamnionitis and endometritis cases | ||||||
| Sepsis by Diagnosis Codes | Sepsis with End Organ Injury by Diagnosis Codes | |||||
| Screening System | False Positive Rate (95% CI) (In patients without sepsis codes, n=13,542) | Sensitivity (95%CI) (n=1049 sepsis cases) | C statistic (95%CI) | False Positive Rate (95% CI) (In patients without sepsis codes, n=13,542) | Sensitivity (95%CI) (n=238 sepsis cases with end organ injury) | C statistic (95%CI) |
| CMQCC | 60.2% (59.3–61.0) |
93.6% (92.0–95.0) |
0.67 (0.66–0.68) |
60.2% (59.3–61.0) |
93.7% (89.8–96.4) |
0.67 (0.65–0.68) |
| SIRS | 86.6% (86.0–87.1) |
99.4% (98.8–99.8) |
0.56 (0.56–0.57) |
86.6% (86.0–87.1) |
99.2% (97.0–99.9) |
0.56 (0.56–0.57) |
| MEWC | 92.3% (91.9–92.8) |
97.7% (96.6–98.5) |
0.53 (0.52–0.53) |
92.3% (91.9–92.8) |
97.9% (95.2–99.3) |
0.53 (0.52–0.54) |
| UKOSS | 67.5% (66.7–68.3) |
95.2% (93.2–96.0) |
0.64 (0.63–0.65) |
67.5% (66.7–68.3) |
95.0% (91.4–97.4) |
0.64 (0.63–0.65) |
| MEWT (Overall) |
45.7% (44.8–46.5) |
78.5% (75.8–80.9) |
0.66 (0.65–0.68) |
45.7% (44.8–46.5) |
87.4% (82.5–91.3) |
0.71 (0.69–0.73) |
In Cohort 1, the CMQCC and the UKOSS obstetric criteria had the lowest false-positive rates and the highest C-statistics (Table 4). Although other screening criteria had slightly higher sensitivity to identify patients with ICD-10 codes for sepsis such as SIRS and MEWC, it was at the cost of higher false-positive rates.
In Cohort 2, the highest C-statistics were again for CMQCC and UKOSS. The false-positive rates for CMQCC (60.2%) and UKOSS (67.5%) were lower than SIRS (86.6%) and MEWC (92.3%). The sensitivity to identify patients with ICD-10 codes for sepsis was above 90% for all screening tools except MEWT. All screening systems had similar sensitivity and screen positive rates (compared to overall sepsis) for patients with ICD-10 coding for sepsis with end organ injury.
Types of infection and end organ injury system affected were reported in Appendix 2 available online at http://links.lww.com/xxx). In the non-chorioamnionitis cohort, the most common type of infection was pyelonephritis followed by pneumonia and viral infections. SARS-CoV-2 infections were a negligible contributor to the chorioamnionitis/endometritis cohort (<1%). COVID-19 was a larger contributor to the non-chorioamnionitis/endometritis cohort (8%) but too small to analyze separately. The most common organ system affected in patients with sepsis and end organ injury was cardiovascular (which included ICD-10 codes for hypotension), followed by renal and hematologic complications. Of note, <1% of the matched controls had a code for an end organ injury, largely thrombocytopenia.
In the sensitivity analysis excluding sepsis cases with hemorrhage, the sensitivity for each screening tools was similar (within 3 percentage points) to the primary analysis (Appendixes 3 and 4, available online at http://links.lww.com/xxx).
Discussion
Pregnancy-adjusted sepsis screening tools (CMQCC and UKOSS) had the highest C-statistics for predicting sepsis during the delivery hospitalization and the lowest false-positive rates in patients without sepsis. A low false positive rate is desired at the first screening step to limit resources expended for the second step which would include laboratory tests for sepsis diagnosis and clinical evaluation for end organ injury. These results underscore the need for pregnancy adjustment of vital sign parameters in the evaluation of sepsis during delivery hospitalizations.
The other pregnancy-adjusted more general maternal morbidity screening tools (MEWC and MEWT) did not perform as well as CMQCC and UKOSS screening tools in our study. They had lower C-statistics for sepsis and higher false-positive rates. These tests have a greater number of screening elements that can each add to the false-positive rate without necessarily improving the sensitivity. The high false-positive rate for SIRS criteria during pregnancy raised concern for inaccurate reporting of the SEP-1 measure. Accordingly, CMS has recently changed their pregnancy SIRS criteria to reflect the pregnancy-adjusted criteria used in the CMQCC screen.
Patients with chorioamnionitis/endometritis posed greater challenges. For all first-step screening approaches based on vital signs alone, false-positive rates for chorioamnionitis without sepsis were high (45.7–92.3%) demonstrating significant overlap between vital sign abnormalities between chorioamnionitis and sepsis. The higher rate of false-positive cases in the setting of chorioamnionitis/endometritis should not be discouraging as these screening tools may still present the opportunity to more rapidly identify severe infection, provide prompt treatment, and possibly prevent sepsis. However, the high false-positive rate should also spur further research to improve the first-step screening approach in patients with chorioamnionitis/endometritis.
In clinical settings, optimal screening strikes a balance between high sensitivity and low false-positive rates to identify the most ill patients with the least number of resources used for false-positive cases. It is important to note that these are screening tools for severe infection and follow-up evaluation needs to be done before a diagnosis of sepsis can be made. Given the low population incidence of maternal sepsis, it is anticipated that the positive predictive value will be low. For that reason, these screening tools are proposed as a first step in a two-step process whereby the first step uses existing clinical data to identify a population at higher risk of serious infection to be followed in a second step by clinical and laboratory evaluation. While 6.9% screen positive for the CMQCC tool appears to be a small number, given the low incidence of sepsis, the majority of false-positive patients will not have sepsis, but they may have a serious infection so a label of “false-positive” may not be appropriate. It may be prudent to increase surveillance of these patients as clinical presentation can rapidly change.23
Previous investigators have evaluated scoring systems to identify patients with clinically deteriorating sepsis or in need of intensive care based on single-institution cohorts with small numbers of patients with the outcome of interest (sepsis or decompensation from sepsis).7,8,23 Others have also noted the limitations of SIRS and MEWS in the setting of chorioamnionitis.24,25 This study concentrated on the general obstetric population to address the issues of recognition and delay. It has strength in large numbers with the outcome of interest analyzed from a diverse national set of hospitals using actual EHR data with comparisons of multiple criteria on the same data sets. Additionally, we presented false-positive rates for multiple screening tools in both cohorts of patients with and without chorioamnionitis. A limitation of the study is the reliance on ICD-10 codes for diagnosis of sepsis. We used a broad set of standard codes for both sepsis and sepsis with end organ injury that are also used by HRSA and the CDC for public health surveillance of maternal sepsis.
Difficulties in early identification of sepsis in pregnant patients has been an important driver of maternal deaths due to sepsis. Early treatment with antibiotics and fluids has been proven effective in reducing morbidity and mortality; delays as minimal as an hour can have differences in mortality, including in pregnant patients.26,27 In this setting, accurate screening tools can be helpful as they provide objective criteria to spur further evaluation. This study demonstrates substantial variation in the performance of five widely used screening systems during the delivery hospitalization.
Supplementary Material
Funding Source:
This work was funded by NIH/NICHD UG3 HD 108053 (Melissa Bauer and Elliott Main). Vesela Kovacheva reports funding from the NIH/NHLBI grants 1K08HL161326–01A1.
Financial Disclosure:
Melissa Bauer reports consulting fees from Institute for Healthcare Innovation. Vesela. Kovacheva reports consulting fees from Avania CRO unrelated to the current work. Paloma Toledo reports speaker fees from Pacira Biosciences, Inc. Kurt Wharton receives consulting fees from Molnlycke.
Footnotes
Each author has confirmed compliance with the journal’s requirements for authorship.
The other authors did not report any potential conflicts of interest.
PEER REVIEW HISTORY
Received July 20, 2023. Received in revised form October 7, 2023. Accepted October 19, 2023. Peer reviews and author correspondence are available at http://links.lww.com/xxx.
Contributor Information
Elliott Main, Department of Obstetrics and Gynecology, Stanford University, Palo Alto, CA
Matt Fuller, Department of Anesthesiology, Duke University, Durham, NC
Vesela P. Kovacheva, Department of Anesthesiology, Perioperative and Pain Medicine, Brigham and Women’s Hospital, Boston, MA.
Rania Elkhateb, Department of Anesthesiology, University of Arkansas for Medical Sciences, Little Rock, AR.
Kristin Azar, Sutter Health Institute for Advancing Health Equity, Sutter Health, Sacramento, CA..
Morgan Caldwell, Department of Anesthesiology, Duke University, Durham, NC
Vanna Chiem, Department of Systems Clinical Informatics, Common Spirit Health, San Francisco, CA.
Mindy Foster, Common Spirit Health, San Francisco, CA
Ronald Gibbs, Department of Obstetrics and Gynecology, Stanford University School of Medicine, Palo Alto, CA
Brenna L Hughes, Department of Obstetrics and Gynecology, Duke University, Durham, NC
Rebecca Johnson, Department of Obstetrics and Gynecology, Baylor College of Medicine, Houston, TX
Nobin Kottukapaly, Wayne State University School of Medicine, Wayne, MI
Melissa G. Rosenstein, Department of Obstetrics, Gynecology, and Reproductive Sciences, University of California, San Francisco, CA
Magdalena Sanz Cortes, Department of Obstetrics and Gynecology, Baylor College of Medicine, Houston, TX.
Laurence E. Shields, Common Spirit Health, San Francisco, CA
Sylvia Sudat, Center for Health Systems Research, Sutter Health, Sacramento, CA
Caitlin D. Sutton, Department of Anesthesiology, Baylor College of Medicine, Houston, TX
Paloma Toledo, Department of Anesthesiology, Perioperative Medicine and Pain Management, University of Miami Miller School of Medicine, Miami, FL..
Austin Traylor, Department of Anesthesiology, Duke University, Durham, NC
Kurt Wharton, Department of Obstetrics and Gynecology, Oakland University William Beaumont School of Medicine, Royal Oak, MI.
Melissa E Bauer, Department of Anesthesiology, Duke University, Durham, NC
References
- 1.Trost SL, Beauregard J, Chandra G, Njie F, Berry J, Harvery A, et al. Pregnancy-Related Deaths: Data from Maternal Mortality Review Committees in 36 US States, 2017–2019. In. Atlanta, GA: Centers for Disease Control and Prevention, US Department of Health and Human Services; 2022. https://www.cdc.gov/reproductivehealth/maternal-mortality/erase-mm/data-mmrc.html. Accessed 7/1/2023. [Google Scholar]
- 2.Commonwealth fund. Accessed 3/1/2023. https://www.commonwealthfund.org/publications/issue-briefs/2020/nov/maternal-mortality-maternity-care-us-compared-10-countries
- 3.Bauer ME, Lorenz RP, Bauer ST, Rao K, Anderson FWJ. Maternal Deaths Due to Sepsis in the State of Michigan, 1999–2006. Obstet Gynecol. 2015;126(4):747–752. doi: 10.1097/AOG.0000000000001028 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Seacrist MJ, Morton CH, VanOtterloo LR, Main EK. Quality Improvement Opportunities Identified Through Case Review of Pregnancy-Related Deaths From Sepsis. J Obstet Gynecol Neonatal Nurs. 2019;48(3):311–320. doi: 10.1016/j.jogn.2019.02.007 [DOI] [PubMed] [Google Scholar]
- 5.Rivers E, Nguyen B, Havstad S, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377. doi: 10.1056/NEJMoa010307 [DOI] [PubMed] [Google Scholar]
- 6.Bauer ME, Bauer ST, Rajala B, et al. Maternal physiologic parameters in relationship to systemic inflammatory response syndrome criteria: a systematic review and meta-analysis. Obstet Gynecol. 2014;124(3):535–541. doi: 10.1097/AOG.0000000000000423 [DOI] [PubMed] [Google Scholar]
- 7.Arnolds DE, Carey KA, Braginsky L, et al. Comparison of early warning scores for predicting clinical deterioration and infection in obstetric patients. BMC Pregnancy Childbirth. 2022;22(1):295. doi: 10.1186/s12884-022-04631-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Stephens AJ, Lee KH, Barton JR, Chauhan SP, Baker AL, Sibai BM. Development of the Sepsis-Associated Adverse Outcomes in Pregnancy Score. Am J Perinatol. 2023;40(2):128–136. doi: 10.1055/a-1884-1001 [DOI] [PubMed] [Google Scholar]
- 9.United States Census Bureau: City and Town Population Totals 2020–2021. Accessed. https://www.census.gov/data/tables/time-series/demo/popest/2020s-total-cities-and-towns.html [Google Scholar]
- 10.von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007;370(9596):1453–1457. doi: 10.1016/j.jclinepi.2007.11.008 [DOI] [PubMed] [Google Scholar]
- 11.Maternal and Child Health Bureau. Federally Available Data (FAD) Resource Document. Rockville, MD: Health Resources and Services Administration. Updated September 2, 2022. Accessed June 6, 2023. https://mchb.tvisdata.hrsa.gov/Admin/FileUpload/DownloadContent?fileName=FadResourceDocument.pdf&isForDownload=False [Google Scholar]
- 12.Committee Opinion No. 712: Intrapartum Management of Intraamniotic Infection. Obstet Gynecol. 2017;130(2):e95–e101. doi: 10.1097/AOG.0000000000002236 [DOI] [PubMed] [Google Scholar]
- 13.Bank TC, Nuss E, Subedi K, Hoffman MK, Sciscione A. Outcomes associated with antibiotic administration for isolated maternal fever in labor. Am J Obstet Gynecol. 2022;226(2):255.e251–255.e257. doi: 10.1016/j.ajog.2021.07.020 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Denoble AE, Wu J, Mitchell CJ, Hughes BL, Dotters-Katz SK. Chorioamnionitis versus intraamniotic infection among preterm deliveries-is postpartum infectious morbidity different? Am J Obstet Gynecol MFM. 2020;2(3):100176 doi: 10.1016/j.ajogmf.2020.100176. [DOI] [PubMed] [Google Scholar]
- 15.Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001;29(7):1303–1310. doi: 10.1097/00003246-200107000-00002 [DOI] [PubMed] [Google Scholar]
- 16.Dombrovskiy VY, Martin AA, Sunderram J, Paz HL. Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003. Crit Care Med. 2007;35(5):1244–1250. doi: 10.1097/01.CCM.0000261890.41311.E9 [DOI] [PubMed] [Google Scholar]
- 17.Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992;101(6):1644–1655. doi: 10.1378/chest.101.6.1644 [DOI] [PubMed] [Google Scholar]
- 18.Mhyre JM, D’Oria R, Hameed AB, Lappen JR, Holley SL, Hunter SK, et al. The maternal early warning criteria: a proposal from the national partnership for maternal safety. Obstet Gynecol. 2014;124(4):782–786. doi: 10.1097/AOG.0000000000000480 [DOI] [PubMed] [Google Scholar]
- 19.Shields LE, Wiesner S, Klein C, Pelletreau B, Hedriana HL. Use of Maternal Early Warning Trigger tool reduces maternal morbidity. Am J Obstet Gynecol. 2016;214(4):527.e521–527.e526. doi: 10.1016/j.ajog.2016.01.154 [DOI] [PubMed] [Google Scholar]
- 20.Gibbs R, Bauer M, Olvera L, Sakowski C, Cape V, Main E. Improving Diagnosis and Treatment of Maternal Sepsis: A Quality Improvement Toolkit. 2020. Stanford, CA: California Maternal Quality Care Collaborative. Updated July 1, 2022. Accessed May 15, 2023. https://www.cmqcc.org/content/sepsis [Google Scholar]
- 21.Acosta CD, Kurinczuk JJ, Lucas DN, Tuffnell DJ, Sellers S, Knight M. Severe maternal sepsis in the UK, 2011–2012: a national case-control study. PLoS Med. 2014;11(7):e1001672. doi: 10.1371/journal.pmed.1001672 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Achten NB, Plötz FB, Klingenberg C, Stocker M, Bokelaar R, Bijlsma M, et al. Stratification of Culture-Proven Early-Onset Sepsis Cases by the Neonatal Early-Onset Sepsis Calculator: An Individual Patient Data Meta-Analysis. J Pediatr. 2021;234:77–84.e78. doi: 10.1016/j.jpeds.2021.01.065 [DOI] [PubMed] [Google Scholar]
- 23.Albright CM, Has P, Rouse DJ, Hughes BL. Internal Validation of the Sepsis in Obstetrics Score to Identify Risk of Morbidity From Sepsis in Pregnancy. Obstet Gynecol. 2017;130(4):747–755. doi: 10.1097/AOG.0000000000002260 [DOI] [PubMed] [Google Scholar]
- 24.Lappen JR, Keene M, Lore M, Grobman WA, Gossett DR. Existing models fail to predict sepsis in an obstetric population with intrauterine infection. Am J Obstet Gynecol. 2010;203(6):573.e571–575. doi: 10.1016/j.ajog.2010.07.040 [DOI] [PubMed] [Google Scholar]
- 25.Edwards SE, Grobman WA, Lappen JR, Winter C, Fox R, Lenguerrand E, et al. Modified obstetric early warning scoring systems (MOEWS): validating the diagnostic performance for severe sepsis in women with chorioamnionitis. Am J Obstet Gynecol. 2015;212(4):536.e531–538. doi: 10.1016/j.ajog.2014.11.007 [DOI] [PubMed] [Google Scholar]
- 26.Bauer ME, Housey M, Bauer ST, Behrmann S, Chau A, Clancy C, et al. Risk Factors, Etiologies, and Screening Tools for Sepsis in Pregnant Women: A Multicenter Case-Control Study. Anesth Analg. 2019;129(6):1613–1620. doi: 10.1213/ANE.0000000000003709 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Kumar A, Roberts D, Wood KE, Light B, Parrillo JE, Sharma S, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit Care Med. 2006;34(6):1589–1596. doi: 10.1097/01.CCM.0000217961.75225.E9 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.