Abstract
Objective:
We compare validation characteristics of four early warning systems for maternal morbidity.
Study Design:
We used a retrospective cohort of severe maternal morbidity cases between January 2016 and December 2016 compared with a cohort of controls. We determined if the modified early obstetric warning score (MEOWS), maternal early recognition criteria (MERC), modified early warning system (MEWS), or maternal early warning trigger (MEWT) would have alerted. We developed criteria to determine which of these alerts was considered clinically “relevant.”
Results:
We reviewed 79 morbidity cases and 123 controls. MEOWS and MERC were more sensitive than MEWS or MEWT (67.1 and 67.1% vs. 19% and 40.5%, p < 0.001); however, MEWT and MEWS were more specific (88.6% MEWT and 93.5% MEWS vs. 51.2% MEOWS and 60.2% MERC, p < 0.001). In the control population, 70% of MEWT alerts still appeared “relevant” to the clinical scenario in contrast to the MEOWS (32%) or MERC systems (31%).
Conclusion:
There are limited comparative data regarding how early warning systems perform in an American population for maternal morbidity. None of the systems performs with high sensitivity and specificity. High-volume, high-acuity units may decide that the lower sensitivity of the MEWT is relatively acceptable when considering the high false trigger rate of the other more sensitive systems. In addition, triggers in the MEWT system were more likely to be clinically relevant even in cases that did not have severe morbidity.
Maternal mortality surveillance in the United States suggests a doubling in the maternal mortality ratio between 1987 and 2014, and almost half of the cases were deemed “preventable.”1–3 Subtle changes in physiologic parameters heart rate (HR), blood pressure (BP), respiratory rate (RR), and level of consciousness often precede acute deterioration, and studies have shown that these can be present for hours prior to the occurrence of a serious adverse event.4 This difficulty is compounded in the obstetric population given that critical illness is relatively rare, and normal pregnancy and childbirth can generate significant vital sign changes which may be perceived as normal.5
Early warning scores are algorithms based on physiologic parameters (HR, BP, RR, temperature, and mental status) in an effort to spur earlier recognitionofmorbidity.6–9 These systems have various acronyms: modified early warning system (MEWS),10 and in obstetrics: modified early obstetric warning score (MEOWS),11,12 maternal early recognition criteria (MERC),5 and maternal early warning trigger (MEWT).13,14 They all require collection of various physiologic parameters at the bedside and use predetermined, standardized criteria to escalate care in a timely fashion for a deteriorating patient.
These systems may use a single vital sign trigger (MERC) or may activate based on a combination of vital signs resulting in a score (MEWS and MEOWS). MEWT incorporates both single and aggregate parameters as well as requires persistence of these abnormalities over time. There is conflicting opinion about the benefit of the simplicity in the single parameter alerts versus the potential for improved detection in the aggregate systems.5,15
There are only limited validation studies in the obstetric population for these systems. They have not been compared simultaneously within a single population. MEWT was recently validated for predicting intensive care unit (ICU) admission.14 MEOWS was validated in one study in the United Kingdom but used different morbidity criteria than are generally used in the United States. MERC has been recently validated in Chicago for an intrapartum cohort of women; however, it did not consider the timing of a morbidity in relationship to the alert in their analysis.16 This group also modified MERC by adding temperature of>38.5°C as an additional trigger. MEWS is an early warning system validated in the non-obstetric patient population and is employed in our hospital in the general wards.17 We aim to retrospectively evaluate the sensitivity and specificity of MEOWS, MERC, MEWS, and MEWT for predicting subsequent severe maternal morbidity as defined by Centers for Disease Control and Prevention International Classification of Diseases, Tenth edition (ICD-10) coding18 in our urban US-based patient population. In addition to sensitivity and specificity, we aim to provide a more extensive evaluation of each tool to describe the burden of the various systems on a clinical care team and the relevance of the triggers for clinical decision-making.
Materials and Methods
The setting for this study is Long Beach Memorial Medical Center in Long Beach, California, a 453 bed acute, tertiary care facility. Long Beach Memorial serves an urban, ethnically diverse community with mixed governmental and commercial payers. Services include a level III neonatal ICU, level IV maternity care, level I trauma center, and critical care units. This site employs residents and fellows in Obstetrics and Gynecology as well as a large number of private practice physicians. The annual delivery volume is approximately 6,000. At the time of this study, there was no formal early warning system or pre-defined set of vital sign triggers for care escalation in use in our obstetrical units with the exception of severe hypertension parameters as recommended by American College of Obstetrics and Gynecology (ACOG) (systolic blood pressure [SBP] of ≥ 160 or diastolic blood pressure (DBP) of ≥ 110). We performed a retrospective validation study of four proposed early warning systems for the prediction of severe maternal morbidity. The study period was January 2016 to December 2016. Cases of severe maternal morbidity based on the ICD-10 codes proposed by Callaghan et al18were identified from our hospital’s cases in the California Maternal Data Center. The California Maternal Data Center is an online tool that generates near real time data and performance metrics and is used by over 200 hospitals in California. Participating hospitals electronically upload discharge data from all delivery admissions each month and this links instantaneously to birth certificate and supplemental clinical data. Morbidity in our cases was then confirmed by local chart review (►Supplementary Table S1, available in the online version). Control patients were randomly selected from our population of delivering women in the Data Center during the same study period. Control patients had no severe maternal morbidity by coding, which was likewise confirmed by chart review.
The Memorial Health Services Institutional Review Board ID # 743–17 approved the study.
We retrospectively reviewed the vital signs from the electronic medical records for each case and control and applied the criteria for MEOWS, MERC, MEWS, and MEWT to determine whether each system would have “alerted” and when it alerted with respect to the time of the morbidity. All vital signs in the electronic record had been reviewed and validated by nursing staff. If an alert occurred, a grace period of 60 minutes was observed prior to recording that an alert occurred again. We recorded if an alert occurred during labor, but excluded vital signs captured in the operating room. The alert needed to occur prior to the morbidity to be included as a true positive alert. Multiple alerts (outside of the 60-minute grace period) were only counted once toward the sensitivity and specificity of the systems. Characteristics of each alert system are listed (►Supplementary Table S2, available in the online version). Demographics, comorbidity, and maternal morbidity data were collected. Comorbidities such as gestational hypertension or preeclampsia with or without severe features were defined by ACOG Task Force criteria. Anemia defined as hemoglobin less than 10 g/dL, prior to admission was confirmed by initial blood counts on date of admission. Other comorbid diagnoses were identified based on prior history noted in physician documentation at time of admission. The total number of system alerts that would have occurred during the entire hospitalization was recorded.
We separately coded whether the alert would have been relevant in caring for either the major morbidity or an alternative diagnosis that would have been important to the treating team such as preeclampsia or chorioamnionitis even if unrelated to the morbidity. This was done to capture how useful the alert system would be for a clinical team separate from its alerting to a morbidity. The criteria for clinical relevance were determined by practicing obstetricians by consensus and described in ►Supplementary Table S3 (available in the online version). The reviewers were asked to determine if the alert would have reasonably altered the actions of a clinical team at the time the trigger occurred (guided by the criteria in the supplemental table). There was some degree of clinical judgement that was left to the discretion of the reviewers. Alerts that occurred after the morbidity were not counted toward the sensitivity or specificity of the system; however, they were included in the overall assessment of clinical relevance.
Finally, we identified how frequently the different systems alerted during a patient’s labor and whether the cases would have been considered “emergencies without warning.” Cases were coded as “emergencies without warning” if the reviewer believed that the event would present as a critical emergency prior to any changes in vital signs. This included episodes where the morbidity of an eclamptic seizure was the chief complaint (without other blood pressure abnormalities) or where immediate hemorrhage occurred either in the operating room or the delivery room at time of delivery and transfusion was immediately started by a team without the need of an additional warning system. This was separately coded because an early warning system would not be expected to help or improve recognition of these particular cases.
A power calculation found that we would need a cohort of 76 cases and 124 controls to detect which of the four available systems used throughout the delivery hospitalization detected maternal morbidity with at least 90% sensitivity and 95% specificity. We selected these measures as we felt the ideal system would be highly sensitive and specific. This calculation was based on published data regarding MEWT sensitivity and specificity for ICU admission (96 and 97%)14 as well as an anticipated morbidity prevalence at our institution of 85 cases/year based on prior data. We sampled an additional 10% of cases of morbidity above the power assuming that morbidity may not actually be identified in the chart review of all cases coded for morbidity.
Separate sub-analyses were planned to review validity for alerts that triggered during labor and also for cases that received a transfusion greater than 12 hours after the bleeding event. In addition, a subanalysis was planned excluding cases of “emergencies without warning.”
Baseline patient and delivery characteristics were compared using chi-squared test for categorical factors and independent t-test for continuous factors. Sensitivity, specificity, and likelihood ratios were calculated for each alert system: MEOWs, MERC, MEWT, and MEWS. The testing characteristics of the four separate systems were compared with the chi-squared test. The planned subanalyses repeated this testing to assess performance of the systems during labor and for cases with transfusions given greater than 12 hours from the initial bleeding event. Additional subanalysis was performed excluding cases of “emergencies without warning.” The total number of system alerts within each delivery hospitalization and the percentage of relevant alerts for each warning system were compared based on generalized linear models analyzed with Poisson regression specified for the overall study population and after stratification by intervention group. The clinical relevance of all four systems was compared across the total study population as well as within both the control and case groups separately with chi-squared test statistic.
Results
Two-hundred two charts were included in the analysis: 79 cases of maternal morbidity and 123 controls who did not experience a severe maternal morbidity during their delivery hospitalization (►Fig. 1).
Fig. 1.
Flow chart of case and control selection. ER, emergency room; ICD-10, International Classification of Diseases, Tenth edition.
►Table 1 describes the baseline characteristics of the two groups. The case group was admitted and delivered at an earlier gestational age. The case group had a higher incidence of comorbidities at admission, including placenta previa, prior cesarean delivery, diabetes, and hypertensive disorders of pregnancy. Cesarean delivery was also more common in the case group (53.2 vs. 38.1%).
Table 1.
Description of patient population by case-control status
| n (valid %) or mean ± SD | Overall n = 202 | Controls n = 123(60.9%) | Cases n = 79 (39.1%) | p-Valuea |
|---|---|---|---|---|
| Patient characteristics | ||||
| Age in years, mean ± SD | 28.9 ± 6.2 | 28.4 ± 5.9 | 29.5 ± 6.7 | p = 0.206 |
| Gestational age weeks, mean ± SD | 38.2 ± 3.2 | 39.0 ± 1.8 | 36.8 ±4.2 | p < 0.001 |
| Race | p =0.104 | |||
| African-American | 26(12.9%) | 18 (14.6%) | 8 (10.1%) | |
| Asian | 19(9.4%) | 13 (10.6%) | 6 (7.6%) | |
| Caucasian | 37 (18.3%) | 22 (17.9%) | 15 (19.0%) | |
| Hispanic | 94 (46.5%) | 58 (47.2%) | 36 (45.6%) | |
| Other | 8 (4.0%) | 1 (0.8%) | 7 (8.9%) | |
| Unknown | 18 (8.9%) | 11 (8.9%) | 7 (8.9%) | |
| BMI, mean ± SD | 31.1 ± 6.0 | 30.7 ± 5.9 | 31.8 ± 6.0 | p = 0.208 |
| Placenta previa | 12 (6.0%) | 2 (1.7%) | 10(12.7%) | p = 0.002 |
| Prior cesarean | 53 (26.4%) | 25 (20.5%) | 28 (35.4%) | p = 0.019 |
| Gestational diabetes | 15 (7.5%) | 6 (4.9%) | 9(11.4%) | p = 0.088 |
| Pregestational diabetes | 6 (3.0%) | 1 (0.8%) | 5 (6.3%) | p = 0.036 |
| Gestational hypertension or preeclampsia (without severe features) | 10(5.0%) | 3 (2.5%) | 7 (8.9%) | p = 0.045 |
| Preeclampsia (with severe features) | 14(7.0%) | 2 (1.6%) | 12 (15.2%) | p < .001 |
| Obesity | 104 (52.8%) | 65 (53.3%) | 39(52.0%) | p =0.861 |
| Asthma | 25 (12.4%) | 18 (14.8%) | 7 (8.9%) | p =0.216 |
| Autoimmune disorder | 14(6.9%) | 6 (4.9%) | 8 (10.1%) | p =0.152 |
| Substance use | 19(9.5%) | 13 (10.7%) | 6 (7.6%) | p =0.469 |
| Chronic hypertension | 8 (4.0%) | 6 (4.9%) | 2 (2.6%) | p =0.407 |
| Anemia at admission (hemoglobin<10 g/dL) | 41 (20.3%) | 23 (18.7%) | 18(22.8%) | p =0.481 |
| Characteristics at delivery | ||||
| G A at delivery, mean ± SD | 38.3 weeks ± 3.1 | 39.1 ± 1.9 | 37.0 ±4.1 | p < .001 |
| Delivery type | p < .001 | |||
| Cesarean | 77 (38.1%) | 35 (28.5%) | 42 (53.2%) | |
| Vaginal | 125 (61.9%) | 88 (71.5%) | 37 (46.8%) | |
Abbreviations: BMI, body mass index; SD. standard deviation.
Note: p <0.05. significant between group difference.
p-Value indicates significance of between group difference using chi-sguared test statistic (Fisher’s exact when less than 5 patients in any cell) for categorical factors and independent f-test for continuous factors (egual variances not assumed with in comparison of average gestational age. Levene’s test significant, p < 0.05).
The most common morbidity in the case cohort was blood transfusion (62 cases, 78.5%). Twenty-one of these patients were given four or more units of packed red blood cells. Twenty-seven (44%) of transfusion cases were delayed (occurring more than 12 hours after the initial bleeding event). ►Table 2 describes the morbidity events captured by ICD-10 coding and confirmed by chart review in addition to other significant metrics of morbidity identified during chart review.
Table 2.
Description of morbidity events among 79 cases
| Morbidity event defined by CDC ICD-10 Criteria19 | n (%) |
|---|---|
| 1. Blood transfusion | 62 (78.5) |
| 2. Pulmonary edema | 12 (15.2) |
| 3. Hysterectomy | 11 (13.9) |
| 4. Adult respiratory distress syndrome | 8 (10.1) |
| 5. Disseminated intravascular coagulation | 7 (8.9) |
| 6. Acute renal failure | 7 (8.9) |
| 7. Eclampsia | 5 (6.3) |
| 8. Sepsis | 4(5.1) |
| 9. Internal injuries of the thorax, abdomen, pelvis | 4(5.1) |
| 10. Cardiac monitoring | 3 (3.8) |
| 11. Shock | 1 (1–3) |
| 12. Myocardial infarction | 1 (1–3) |
| 13. Cardiac arrest | 1 (1–3) |
| Additional morbidity data collected during chart reviewa | |
| 14. Obstetric hemorrhage with ≥4 units packed red blood cells transfused | 21 (26.6) |
| 15. Prolonged postpartum length of stay (≥4 days vaginal delivery or ≥6 days cesarean delivery) | 12 (15.2) |
| 16. Preeclampsia with difficult to control severe hypertension | 10 (12.7) |
| 17. ICU admission (unplanned) | 9(11.4) |
| 18. Maternal mortality | 1 (1–3) |
Abbreviations: CDC ICD-10; Centers for Disease Control and Prevention International Classification of Diseases, Tenth edition; ICU, intensive care unit.
Additional morbidity data was NOT included in the validation analysis.
The performance of the different alert systems to identify Patients who experience a severe morbidity event captured by ICD-10 coding is listed in ►Table 3. MEOWS and MERC were The most sensitive of the four systems (67.1% for both MEOWS and MERC vs. 19% for MEWS, 40.5% for MEWT, p < 0.001). Sensitivity of all systems improved if cases that were deemed “emergencies without warning” were excluded from analysis. In addition, the sensitivity of all systems improved if cases with a presumably obvious hemorrhage resulting in the need for early transfusion were excluded from analysis. The early warning systems might not be expected to be helpful in cases of obvious hemorrhage.
Table 3.
Validity of trigger alert(s) prior to the morbidity to identify patients who experience a severe morbidity event overall and after restriction of case group by defined criteria a, b, and c
| Sensitivity (95% Cl) | Specificity (95% Cl) | LR± Sensitivity/(1- Specificity) | LR− (1- Sensitivity)/Specificity | |
|---|---|---|---|---|
| Overalla | p < 0.001b | p < 0.001 | ||
| MEOWS | 67.1% (55.6, 77.3) | 51.2% (42.1, 60.3) | 1.38 (1.08, 1.74) | 0.64 (0.45, 0.92) |
| MERC | 67.1% (55.6, 77.3) | 60.2% (51.0, 68.9) | 1.68 (1.29, 2.20) | 0.55 (0.39, 0.77) |
| MEWS | 19.0% (11.0,29.4) | 93.5% (87.6, 97.2) | 2.92 (1.30, 6.56) | 0.87 (0.77, 0.97) |
| MEWT | 40.5% (29.6, 52.2) | 88.6% (81.6, 93.6) | 3.56 (2.03, 6.24) | 0.67 (0.55, 0.81) |
| (a) Excluded 25 cases coded “Emergency without warning”c | p < 0.001 | Same as overall | ||
| MEOWS | 74.1% (60.4, 85.0) | 1.52 (1.19, 1.93) | 0.51 (0.31, 0.82) | |
| MERC | 74.1% (60.4, 85.0) | 1.86 (1.42, 2.43) | 0.43 (0.27, 0.69) | |
| MEWS | 22.2% (12.0, 35.6) | 3.42 (1.48, 7.88) | 0.83 (0.72, 0.97) | |
| MEWT | 44.4% (30.9, 58.6) | 3.90 (2.19, 6.95) | 0.63 (0.49, 0.80) | |
| (b) Excluded 52 cases where transfusion delay < = 12 hour | p < 0.001 | Same as overall | ||
| MEOWS | 88.9% (70.8, 97.7) | 1.82 (1.46, 2.28) | 0.22 (0.07, 0.64) | |
| MERC | 96.3% (81.0, 99.9) | 2.42 (1.92, 3.04) | 0.06 (0.01, 0.42) | |
| MEWS | 22.2% (8.6, 42.3) | 3.42 (1.29, 9.04) | 0.83 (0.68, 1.02) | |
| MEWT | 59.3% (38.8, 77.6) | 5.21 (2.90, 9.34) | 0.46 (0.29, 0.73) |
Abbreviations: LR, likelihood ratio; MERC, maternal early recognition criteria; MEOWS, modified early obstetric warning score; MEWS, modified early warning system; MEWT, maternal early warning trigger.
Note: These were events where an early warning system probably would not have improved clinician recognition of severe disease.
System alerts analyzed throughout entire duration of hospitalization prior to morbidity (before, during, and after labor).
p value based on chi-squared test statistic comparing the performance between four alert systems.
“ Emergency without warning” morbidity events included significant bleeding immediately after delivery or events that occurred entirely in the OR setting.
While MEOWS and MERC were most sensitive, they were substantially less specific for a severe morbidity event (51 and 60.2% vs. 93.5% for MEWS and 88.6% for MEWT). The specificity of all of the alert systems was separately evaluated in the context of labor (►Table 4).
Table 4.
Validity of trigger alert(s) during labora to identify patients who experience a severe morbidity event
| Sensitivity (95% Cl) | Specificity (95% Cl) | LR+ | LR− | |
|---|---|---|---|---|
| MEOWS | 36.7% (26.1,48.3) | 64.2% (55.1, 72.7) | 1.03 (0.71, 1.49) | 0.99 (0.80, 1.22) |
| MERC | 34.2% (23.9,45.7) | 69.9% (61.0, 77.9) | 1.14 (0.76, 1.71) | 0.94 (0.77, 1.15) |
| MEWS | 7.6% (2.8, 15.8) | 97.6% (93.0, 99.5) | 3.11 (0.80, 12.09) | 0.95 (0.88, 1.01) |
| MEWT | 13.9% (7.2, 23.6) | 90.2% (83.6, 94.9) | 1.43 (0.66, 3.08) | 0.95 (0.86, 1.06) |
Abbreviations: LR, likelihood ratio; MERC, maternal early recognition criteria; MEOWS, modified early obstetric warning score; MEWS, modified early warning system; MEWT, maternal early warning trigger.
Triggers that occurred outside of labor were not counted.
In review of the study population, both MEOWS and MERC alerted in over 60% of the overall study population (►Fig. 2) and over four times in almost 30% throughout their hospitalization. ►Fig. 2 demonstrates the number of study patients who had alerts and if at least one of the alerts in a hospitalization was considered relevant for clinical care. ►Fig. 3 demonstrates the alerts in the control population. MEOWS and MERC triggered in 49% (60/123) and 40% (49/123) of control patients versus MEWS and MEWT which alerted in 7% (8/123) and 11% (14/123). In addition, the majority of MEWT alerts (70%, 10/14) in the control population were deemed clinically relevant for the detection of a condition that may not have resulted in morbidity, but would still be important for treatment such as preeclampsia or chorioamnionitis. This contrasts with the MEOWS and MERC systems which were clinically relevant in 32% (19/60) and 31% (15/49) of the alerts within the control groups.
Fig. 2.
Number of study patients (cases and controls, n = 202) with alerts by each system and number of study patients with at least one alert considered “clinically relevant.” Dark grey: Study patients (n = 202) with an alert. Grey stipple: Study patients (n = 202) with ≥ 1 clinically relevant alert (p < 0.001). MERC, maternal early recognition criteria; MEOWS, modified early obstetric warning score; MEWS, modified early warning system; MEWT, maternal early warning trigger.
Fig. 3.
Number of control patients (n = 123) with alerts by each system and number of control patients with at least one alert considered “clinically relevant.” Dark grey: Control patients (n = 123) with an alert. Grey stipple: Control patients (n = 123) with ≥1 “clinically relevant” alert (p = 0.027). MERC, maternal early recognition criteria; MEOWS, modified early obstetric warning score; MEWS, modified early warning system; MEWT, maternal early warning trigger.
►Fig. 4 demonstrates the total number of alerts among the study population throughout their hospitalization as well as how many of these alerts were considered relevant to patient care. MEOWS and MERC were relevant in 50% or less of the total number of alerts throughout the population, while MEWS and MEWT were relevant in 82 and 89% of alerts, respectively. ►Fig. 5 demonstrates the total number of alerts among the control population and how many of these were considered relevant to patient care. MEWT was still considered relevant in 62% (18/29) of the total alerts that occurred within the hospitalization for the control group versus MEOWS and MERC which were relevant in 15 (35/229) and 27% (31/113) of the control patient alerts.
Fig 4.
Total number of alerts within the STUDY population (cases + controls) throughout the hospitalization and total number of alerts considered “clinically relevant.” Dark grey: Total number of alerts within the STUDY population. Grey stipple: Total number of alerts within the STUDY population considered “clinically relevant” (p < 0.001). MERC, maternal early recognition criteria; MEOWS, modified early obstetric warning score; MEWS, modified early warning system; MEWT, maternal early warning trigger.
Fig 5.
Total number of alerts within the CONTROL population throughout the hospitalization and total number of alerts considered “clinically relevant.” Dark grey: Total number of alerts within the CONTROL population. Grey stipple: Total number of alerts within the CONTROL population considered “clinically relevant” (p = 0.017). MERC, maternal early recognition criteria; MEOWS, modified early obstetric warning score; MEWS, modified early warning system; MEWT, maternal early warning trigger.
The most frequent MEOWS trigger was a “red” (or single) alert with DBP > 90 mm Hg. Single parameter alerts were most common occurring in88%of the alerts within the MEOWS system. The most frequent MERC alert was oxygenation saturation less than 95% followed by SBP < 90 or > 160 mm Hg. In the MEWS alerts, all components of the MEWS score were recorded in 33% of the total alerts; however, in 87% of MEWS alerts, a significant score of 4was obtained without full vital sign information. The most common initial MEWT alert and sustained trigger was SBP > 155 or DBP> 105 mm Hg.
Conclusion
None of the four tools examined performed with 90% sensitivity and 95% specificity. However, in this population at a delivery unit with significant volume, acuity, and comorbidities, the characteristics of MEWT best meet the needs of our unit. MEWT’s positive predictive value (70%) exceeds MEOWS (47%) and MERC (52%) and makes it more useful in our system despite lower sensitivity (►Supplementary Table S4, available in the online version). MEWT was also clinically relevant in 89% of alerts, whereas MEOWS and MERC were clinically relevant in less than 50%. MEWS was also generally clinically relevant, however, with much lower sensitivity than the MEWT system. We agree with the National Partnership for Maternal Safety in stating that the optimal balance between sensitivity and specificity likely varies between clinical environments and patient populations. Our conclusion may not be generalizable to smaller delivery units that would be best served by choosing a system with better sensitivity. Given that no tool studied had both high sensitivity and specificity, a center must examine their patient population as well as resources in determining if they desire a system with heavier alert burden, however higher sensitivity versus a system with more limited alert burden and higher specificity.
“The National Partnership for Maternal Safety” has advocated use of the MERC5 tool, which has recently been validated in a retrospective study at the University of Chicago. This group reported a sensitivity of 97% (vs. 67% in our study) and specificity of 39% (vs. 60% in our study). There were three important differences in the way the MERC tool was assessed in this study compared to our own. First, the MERC tool was modified to include an alert for a temperature of >38.5F, as would be clinically reasonable. Second, only intrapartum vital signs were assessed. Third, the timing of morbidity at any time throughout the delivery hospitalization was not assessed relative to the occurrence of the intrapartum vital sign trigger. Our data would have yielded similar results for MERC (as originally described without a temperature trigger) if we had analyzed alerts without regard to timing of morbidity, with 92% sensitivity and 60% specificity. In addition, our data adds information about the sensitivity of alerting only when occurring prior to the morbidity event. The alerting burden for women without morbidity in the Chicago group for MERC, restricted to the intrapartum time period, was 50% as compared to our alerting burden for MERC throughout the delivery hospitalization of 40%.”16
The National Partnership for Maternal Safety emphasizes the improved simplicity of a single trigger system such as MERC and potential for increased use and adoption with ease of use. The question of an aggregate versus single parameter system in our population was less pertinent, and in fact the most frequent triggers for MEOWS, MERC, or MEWT were single parameter triggers. More than the question of single versus aggregate systems, the inclusion of a sustained abnormality time parameter (as specified in MEWT) provided a substantial difference in sensitivity and specificity versus systems that capture both sustained and non-sustained abnormalities (MEOWS, MERC, MEWS). This is a feature also demonstrated in other studies.13 Abnormality persistence substantially improved MEWT specificity and “clinical relevance” over MEOWS and MERC; however, it also compromised tool sensitivity.
While we did not investigate the “optimal” physiologic parameters for a tool, there are certain parameters of the MEOWS and MERC systems that trigger at lower degrees of abnormality versus MEWT and MEWS. This contributed to tool relevance particularly if a woman has a diagnosis of chronic hypertension or preeclampsia. A significant portion of MEOWS alerts included DBPs greater than 90 mm Hg, which is less valuable to clinicians once an initial diagnosis is made.
The principal validation study for MEOWS12 performed in the UK reported a sensitivity and specificity of 89 and 79%, respectively (as opposed to our findings of 67% sensitivity and 51% specificity). We used a definition of morbidity based on ICD-10 codes which was proposed in the United States utilizing convenient, administratively derived data,19 which differs somewhat from the morbidity metrics used in the original validation study. In addition, they found a 30% trigger rate in their population, while our overall rate using the MEOWS system was 56% in cases and 49% in our control group. This prior study may have had a healthier baseline population, as there were no ICU admissions or mortality (compared with 9 ICU admissions and 1 mortality in our population). In our population, a trigger rate of almost 50% in patients who never experience a true morbidity would be a significant burden. Throughout the course of our control group’s full hospitalization, MEOWS would have alerted 229 times and only 35 of these were considered “clinically relevant” in our analysis.
MEWS has already been implemented in the non-obstetric populations of many hospital systems, and simply expanding a system already in place to the obstetric population has appeal. However, while it appears to have high morbidity specificity and clinical relevance, its sensitivity was the worst of the tools. The low sensitivity was primarily due to the cut point at which blood pressure acts as a trigger in the MEWS system, requiring SBP to exceed 199 to generate any points. Adjusting this parameter for the obstetric population would likely improve tool performance.
The retrospective nature of this study resulted in significant limitations. Each of these tools has parameters around “patient appearance,” “altered mental status,” and “nursing concern” which were less completely and reliably documented in reviewing the medical record. Therefore, the current analysis reflects primarily the numeric vital sign parameter performance of each tool rather than the incorporation of subjective nursing assessment into the tools. While the retrospective nature is a weakness, prospectively evaluating multiple early warning systems in a single patient population seems impractical and unlikely to be achieved.
Another weakness of our study was the subjectivity in determining criteria for clinical relevance. It is unlikely that every clinician will agree with the parameters set; however, this attention to clinical relevance and the chart level review of each of these cases provides a degree of consideration that rounds out the perspective on the use of these tools as well as an assessment of the degree to which these tools may increase clinical burden. We acknowledge that if these systems had in fact “alerted,” it is possible that a clinical team may have acted more quickly to remedy the alert, which may address the overall alert frequency. The main driver of alert frequency included mild range blood pressure alerts in both the MEOWS and MERC systems. These are not parameters that likely would have been addressed in clinical care beyond the first alerts. This was a reason that clinical relevancy seemed important to consider because as a measure it helps address when an alert should have triggered a specific clinical response (vs. alerts that would not have necessarily required additional action).
An important question is if the widespread application of a maternal early warning system can contribute to reductions in maternal morbidity and mortality in the United States. A recent prospective study utilizing the MEWT system, in which triggers were linked to specific recommendations to clinicians for diagnostic testing and therapy, found a reduction in maternal morbidity over a 1-year period of time in six study hospitals compared with control hospitals.14 Additional studies confirming improvement in maternal morbidity metrics with use of early warning systems are still warranted.
Supplementary Material
Acknowledgment
The authors thank Tricia Morphew, MS of Morphew Consulting LLC for work on statistical analysis and design.
Funding
The Memorial Care Foundation provided funding for the study including statistical analysis of data as well as travel for presentation of the study at the Society of Maternal Fetal Medicine meeting. Statistical design was supported by grant UL1 TR001414 from the National Center for Advancing Translational Sciences, National Institutes of Health (NIH), through the Biostatistics, Epidemiology and Research Design Unit. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Footnotes
Conflict of interest
None declared.
References
- 1.Main EK, Menard MK. Maternal mortality: time for national action. Obstet Gynecol 2013;122(04):735–736 [DOI] [PubMed] [Google Scholar]
- 2.Hirshberg A, Srinivas SK. Epidemiology of maternal morbidity and mortality. Semin Perinatol 2017;41(06):332–337 [DOI] [PubMed] [Google Scholar]
- 3.Centers for Disease Control and Prevention. Pregnancy mortality surveillance system. Available at: http://www.cdc.gov/reproductivehealth/maternalinfanthealth/pregnancy-mortality-surveillance/system.html. Accessed January 13, 2019
- 4.Alam N, Hobbelink EL, van Tienhoven AJ, van de Ven PM, Jansma EP, Nanayakkara PW. The impact of the use of the early warning score (EWS) on patient outcomes: a systematic review. Resuscitation 2014;85(05):587–594 [DOI] [PubMed] [Google Scholar]
- 5.Mhyre JM, D’Oria R, Hameed AB, et al. The maternal early warning criteria: a proposal from the national partnership for maternal safety. J Obstet Gynecol Neonatal Nurs 2014;43(06):771–779 [DOI] [PubMed] [Google Scholar]
- 6.Ludikhuize J, Smorenburg SM, de Rooij SE, de Jonge E. Identification of deteriorating patients on general wards; measurement of vital parameters and potential effectiveness of the modified early warning Score. J Crit Care 2012;27(04):424.e7–424.e13 [DOI] [PubMed] [Google Scholar]
- 7.Moon A, Cosgrove JF, Lea D, Fairs A, Cressey DM. An eight year audit before and after the introduction ofmodified early warning score (MEWS) charts, of patients admitted to a tertiary referral intensive care unit after CPR. Resuscitation 2011;82(02):150–154 [DOI] [PubMed] [Google Scholar]
- 8.Trinkle RM, Flabouris A. Documenting rapid response system afferent limb failure and associated patient outcomes. Resuscitation 2011;82(07):810–814 [DOI] [PubMed] [Google Scholar]
- 9.Maupin JM, Roth DJ, Krapes JM. Use of themodified early warning score decreases code blue events. Jt CommJ Qual Patient Saf 2009; 35(12):598–603 [DOI] [PubMed] [Google Scholar]
- 10.Ludikhuize J, Borgert M, Binnekade J, Subbe C, Dongelmans D, Goossens A. Standardized measurement of the modified early warning score results in enhanced implementation of a rapid response system: a quasi-experimental study. Resuscitation 2014;85(05):676–682 [DOI] [PubMed] [Google Scholar]
- 11.Quinn AC,Meek T,Waldmann C. Obstetric early warning systems to prevent bad outcome. Curr Opin Anaesthesiol 2016; 29(03):268–272 [DOI] [PubMed] [Google Scholar]
- 12.Singh S, McGlennan A, England A, Simons R. A validation study of the CEMACH recommended modified early obstetric warning system (MEOWS). Anaesthesia 2012;67(01):12–18 [DOI] [PubMed] [Google Scholar]
- 13.Hedriana HL, Wiesner S, Downs BG, Pelletreau B, Shields LE. Baseline assessment of a hospital-specific early warning trigger system for reducing maternal morbidity. Int J Gynaecol Obstet 2016;132(03):337–341 [DOI] [PubMed] [Google Scholar]
- 14.Shields LE, Wiesner S, Klein C, Pelletreau B, Hedriana HL. Use of maternal early warning trigger tool reduces maternal morbidity. Am J Obstet Gynecol 2016;214(04):527.e1–527.e6 [DOI] [PubMed] [Google Scholar]
- 15.McNeill G, Bryden D.Doeither earlywarning systems or emergency response teams improve hospital patient survival? A systematic review. Resuscitation 2013;84(12):1652–1667 [DOI] [PubMed] [Google Scholar]
- 16.Arnolds DE, Smith A, Banayan JM, Holt R, Scavone BM. National partnership for maternal safety recommended maternal early warning criteria are associated with maternal morbidity. Anesth Analg 2018. Doi: 10.1213/ANE.0000000000003889 [DOI] [PubMed] [Google Scholar]
- 17.Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified Early Warning Score in medical admissions. QJM 2001; 94(10):521–526 [DOI] [PubMed] [Google Scholar]
- 18.Callaghan WM, Creanga AA, Kuklina EV. Severe maternal morbidity among delivery and postpartum hospitalizations in the United States. Obstet Gynecol 2012;120(05):1029–1036 [DOI] [PubMed] [Google Scholar]
- 19.Callaghan WM, Grobman WA, Kilpatrick SJ, Main EK, D&Alton M. Facility-based identification of women with severe maternal morbidity: it is time to start. Obstet Gynecol 2014;123(05): 978–981 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.