Severe Maternal Morbidity: A Comparison of Definitions and Data Sources

Abstract

Severe maternal morbidity (SMM) is a composite outcome measure that indicates serious, potentially life-threatening maternal health problems. There is great interest in defining SMM using administrative data for surveillance and research. In the United States, one common way of defining SMM at the population level is an index developed by the Centers for Disease Control and Prevention. Modifications to this index (e.g., exclusion of maternal blood transfusion) have been proposed; some research defines SMM using an index introduced by Bateman et al. (Obstet Gynecol. 2013;122(5):957–965). Birth certificate data are also increasingly being used to define SMM. We compared commonly used US definitions of SMM among all California births (2007–2012) using the kappa (κ) statistic and other measures. We also evaluated agreement between maternal morbidity fields on the birth certificate as compared with health insurance claims data. Concordance between the 7 definitions of SMM analyzed was generally low (i.e., κ < 0.41 for 13 of 21 two-way comparisons). Low concordance was particularly driven by the presence/absence of transfusion and claims data versus birth certificate definitions. Low agreement between administrative data–based definitions of SMM highlights that results can be expected to differ between them. Further research on validity of SMM definitions, using more fine-grained data sources, is needed.

Abbreviations

CDC

Centers for Disease Control and Prevention

ICD-9

International Classification of Diseases, Ninth Revision

LOS

length of stay

SMM

severe maternal morbidity

The United States has an alarmingly high rate of maternal mortality compared with other high-resource nations, and it is the only high-resource nation where maternal mortality rates have increased in recent decades. Racial/ethnic inequities in this outcome are substantial and persistent. Maternal death is still a very rare outcome in absolute terms (17.2/100,000 live births between 2011 and 2015 (1)), which highlights the need to better understand maternal health complications that precede it. In recent years, increasing emphasis has also been placed on identifying and studying women with severe maternal morbidity (SMM)—that is, women who have serious, life-threatening complications that presumably could lead to death. SMM carries significant short- and long-term physical, psychological, social, and financial consequences for women, families, and society. In the United States, one common way of defining SMM from administrative data is an index developed by the Centers for Disease Control and Prevention (CDC), which includes pregnancy-related conditions such as eclampsia and amniotic fluid embolism, severe cardiovascular complications like acute myocardial infarction and heart failure, and procedures such as blood transfusion, hysterectomy, and ventilation support (2, 3). The index is based on diagnostic and procedure codes that are typically available in administrative data, enabling population-level surveillance. SMM as measured by this index has also been increasing in the United States in recent years (4).

There is agreement that SMM is an important outcome that merits further study; however, it is measured and defined inconsistently across studies. Detailed review of medical records is currently considered the optimal way to identify true cases of SMM (5–8), but this labor-intensive approach is not feasible for tracking SMM at the population level (9). The CDC index (3) is a common approach to identifying women with SMM using administrative data, and in prior research investigators have examined potential refinements to it in attempts to increase face validity (6, 10). It is not clear that all index subcomponents truly reflect a dire maternal health situation in all circumstances (e.g., blood transfusion (6)), and other factors have been used to increase construct validity (e.g., maternal length of stay (LOS)). Accordingly, modifications to the CDC SMM index have been proposed, such as excluding maternal blood transfusion from the composite index or restricting SMM cases to those with unusually long LOS. Alternatively, some US-based research (11–14) has defined SMM using an index introduced by Bateman et al. (15) in 2013. The Bateman et al. (Bateman) definition focuses on indicators of end organ injury, but even for indicators that overlap with the CDC definition, it uses somewhat different sets of codes (see Web Table 1, available online at https://doi.org/10.1093/aje/kwab077, for details). Finally, there is also increasing interest in using fields on the birth certificate that are likely indicators of SMM (e.g., blood transfusion, uterine rupture) (16–20), despite research suggesting very low sensitivity (21, 22).

Our objective in this study was to compare commonly used US definitions of SMM within a large US population (all California births, 2007–2012). We compared the CDC SMM index and its modifications, the SMM definition developed by Bateman et al. (15), the CDC index adding in codes contained in the Bateman definition, and 2 definitions that incorporate CDC-listed conditions that are available on the birth certificate using the kappa (κ) statistic and other measures of agreement. We also evaluated how each maternal morbidity field on the birth certificate performs in comparison with health insurance claims data. This knowledge will help shed light on whether administrative data–based SMM definitions are comparable to each other. This is important for informing ongoing efforts to monitor and improve maternal health in the United States, facilitate comparisons of findings across studies using varied definitions of SMM, and support refinement of the optimal definition of SMM.

METHODS

This population-based study used the California Office of Statewide Health Planning and Development’s linked birth cohort files from the years 2007–2012. This data source includes data from birth certificates and fetal death records, which were linked with inpatient maternal health insurance claims during the period from pregnancy through 9 months postpartum. We included 3,049,774 births for which the vital record was linked with data from the maternal birth hospitalization. Hospital discharge records contain International Classification of Diseases, Ninth Revision, Clinical Modification, codes related to diagnoses and procedures. We used codes from maternal hospitalizations at birth, as well as information from vital records, to identify women with SMM occurring during the childbirth hospitalization (i.e., we did not address SMM occurring prenatally or postpartum). All study population descriptors (maternal race/ethnicity, education, and age) were derived from vital records.

We calculated the reliability between 7 administrative data–based SMM definitions, some from the literature and some that were refinements based on availability of additional data beyond International Classification of Diseases, Ninth Revision (ICD-9) codes. We describe each measure, its original intended purpose, and how it is measured (i.e., which administrative data sources are used).

Definitions of SMM

CDC index without transfusion.

Prior research has demonstrated low construct validity for SMM cases identified by blood transfusion only, partly due to the fact that the number of units of blood product transfused is not available in ICD-9 codes (i.e., in one study, Main et al. (6) reported that only 26% of these women were deemed to have true severe morbidity). Some amount of blood loss during delivery is physiologically normal, and some experts argue that transfusion of a small amount of blood (e.g., 1 unit) may reflect routine preventive care. Therefore, this first SMM definition utilized the CDC index, excluding transfusion-only cases of SMM. The intention of removing blood transfusion from the index was to exclude false-positive cases (i.e., women with SMM defined by the overall CDC index who did not actually experience a severe and life-threatening complication).

CDC index.

The CDC index of SMM is based on ICD-9 codes and was intended to be used for surveillance and monitoring of SMM across populations and over time using administrative data only (3). Notably, this definition and its derivatives were not intended to be used for comparing facility-level quality of maternity care.

CDC index—prolonged LOS required for diagnoses.

Several studies have used the CDC index but with the added criterion that women who only had diagnosis codes were included only if they had a prolonged LOS after giving birth (≥90th percentile or more than 3 standard deviations above the mean, estimated separately for women with vaginal birth and women with cesarean delivery using hospital discharge data) (2, 6, 23–27). This additional criterion is sometimes used under the rationale that true SMM reflects an acute and medically complex patient prognosis, requiring a significantly longer amount of time in the hospital before a woman’s medical condition can be sufficiently stabilized to enable discharge. We used the criterion of LOS ≥90th percentile for this definition (2 days for vaginal birth, 4 days for primary cesarean birth, and 3 days for repeat cesarean birth in our data). The intention of this added criterion also was to exclude false-positive cases.

CDC index—Bateman modification.

The CDC index with Bateman modification was defined using as a starting point the list of complications from the CDC index, other than blood transfusion (since the Bateman index did not include transfusion). For complications that appear in the Bateman index as well, this definition augments the list of ICD-9 codes to also include codes that appear in the Bateman index but not in the CDC index. The purpose of considering this SMM definition was to assess the degree to which agreement between the CDC SMM index (without transfusion) and the Bateman SMM index might be affected by differing sets of ICD-9 codes used to define the same outcome.

Birth certificate—CDC analog.

Although the standard US birth certificate is intended for registering vital events rather than research, there is interest in using this data source to study SMM because of its accessibility. This composite flags women with any of the following boxes checked on the birth certificate: blood transfusion, eclampsia, anesthetic complications, unplanned hysterectomy, and sepsis. Each of these checkboxes corresponds to an item in the CDC SMM index.

Birth certificate—CDC analog expanded.

This composite variable includes data from checkboxes in the “birth certificate—CDC analog” definition as well as other checkboxes that are also likely to identify women with a serious maternal health complication but do not correspond to specific ICD-9 codes in the CDC SMM index. These additional complications are: other excessive bleeding, admission to the intensive care unit, unplanned operating room procedure following delivery, and seizure during labor (not specified to be an eclamptic seizure, although generally presumed to be).

Bateman definition.

An index was created by Bateman et al. to identify “severe maternal morbidity, defined as the occurrence of acute maternal end-organ injury” (15, p. 958), as the dependent variable for validating an obstetrical comorbidity index. This ICD-9 code–based index has been used to define SMM in prior research (11–14) and was not designed for population-level surveillance or comparing the quality of care between hospitals.

Metrics of concordance between SMM definitions

We calculated the prevalence of each of these definitions of SMM and used several measures of concordance to establish the extent to which they agreed with one another. We calculated the κ statistic, sensitivity, specificity, positive predictive value, negative predictive value, and accuracy. To compare one SMM definition with another, we estimated these measures for all 2-way comparisons between the 7 administrative data–based SMM measures listed above. This enabled assessment of whether the specific SMM metric chosen affects how SMM is measured and reported, which could matter for evaluating studies and tracking SMM trends (especially if concordance between measures is low). We compared SMM metrics using the order above, beginning with the CDC index without transfusion as the first “true SMM” metric and the other 6 as tests, and then the CDC index as the “truth” and the other 5 as tests, and so on.

The κ statistic ranges from −1 to 1, with 0 representing chance. We used the common system to designate the degree of agreement indicated by a given κ value (<0–0.20 = poor; 0.21–0.40 = fair; 0.41–0.60 = moderate; 0.61–0.80 = good; and 0.81–1.00 = very good) (28, 29). We created a heat map to visually depict how the different SMM definitions compared with each other in terms of κ value ranges. We considered the data source (i.e., birth certificate vs. ICD-9 codes) and the presence/absence of transfusion codes (for ICD-9–based definitions) as key factors that differentiated SMM definitions from each other.

Concordance between maternal complications identified using ICD-9 codes and birth certificate checkboxes

In addition, we examined concordance measures for specific conditions listed on the birth certificate that are part of the CDC index and thus have corresponding ICD-9 codes: blood transfusion, eclampsia, anesthetic complications, unplanned hysterectomy, and sepsis. The birth certificate checkbox was considered the test and the ICD-9 code was considered the “truth.”

RESULTS

Among the 3,049,774 births included in these analyses, the frequency of SMM as defined by CDC criteria was 1.3% (Table 1). Women with SMM were more likely to be at least 35 years of age, of lower educational attainment, and non-Hispanic Black or other/missing race/ethnicity (considering “race” as a social construct that is marker for adverse exposures, including racism, that increase risk of SMM (30)).

Table 1.

Characteristics of 3,049,774 Women Who Gave Birth in California During the Period 2007–2012, Overall and by CDC-Defined Severe Maternal Morbidity Status

			SMM Status
Characteristic	Total   (n = 3,049,774)		No SMM   (n = 3,010,021; 98.7%)		SMM   (n = 39,753; 1.3%)
	No.	% a	No.	% b	No.	% b
Maternal race/ethnicity
Non-Hispanic White	829,085	27.2	820,206	98.9	8,879	1.1
Hispanic	1,579,274	51.8	1,558,354	98.7	20,920	1.3
Non-Hispanic Black	174,783	5.7	171,101	97.9	3,682	2.1
Asian	400,422	13.1	395,136	98.7	5,286	1.3
Other/missing data	66,210	2.2	65,224	98.5	986	1.5
Nativity
Foreign-born	1,313,619	43.1	1,296,310	98.7	17,309	1.3
US-born	1,736,155	56.9	1,713,711	98.7	22,444	1.3
Mother’s age, years
<20	267,981	8.8	263,949	98.5	4,032	1.5
20–34	2,233,754	73.2	2,207,312	98.8	26,442	1.2
≥35	547,838	18.0	538,579	98.3	9,259	1.7
Missing data	201	0.0	181	90.0	20	10.0
Mother’s educational level
High school or less	1,512,402	49.6	1,491,329	98.6	21,073	1.4
Some college	699,406	22.9	690,776	98.8	8,630	1.2
College completion	731,877	24.0	723,420	98.8	8,457	1.2
Unknown/missing data	106,089	3.5	104,496	98.5	1,593	1.5
Payment type at birthc
Private health insurance	1,428,773	46.8	1,411,882	98.8	16,891	1.2
Public/government plan	1,512,196	49.6	1,490,851	98.6	21,345	1.4
Uninsured	61,733	2.0	60,902	98.7	831	1.3
Other/missing data	47,072	1.5	46,386	98.5	686	1.5
Parity
0 (nulliparous)	1,182,182	38.8	1,165,351	98.6	16,831	1.4
≥1 (multiparous)	1,854,323	60.8	1,831,757	98.8	22,566	1.2
Missing data	13,269	0.4	12,913	97.3	356	2.7

Abbreviations: CDC, Centers for Disease Control and Prevention; SMM, severe maternal morbidity.

^a Column percentages.

^b Row percentages.

^c Percentages do not sum to 100 because of rounding.

The prevalence of SMM varied from 0.47% (using the birth certificate—CDC analog definition) to 1.30% (using the original CDC index) (Web Table 2). Prevalence was highest for SMM definitions that included blood transfusion (e.g., 1.13% for SMM with the LOS restriction), and adding in additional ICD-9 codes caused the prevalence to increase substantially from the CDC index—no transfusion definition (prevalence = 0.55%) to the CDC index—Bateman modification definition (0.97%). With some notable exceptions, concordance between the 7 definitions of SMM analyzed was low (i.e., poor or fair agreement; κ < 0.41 for 13 of the 21 two-way comparisons). Of the 21 two-way comparisons, κ was in the moderate range (κ = 0.41–0.60) for 4 comparisons and good or very good (κ = 0.61–1.00) for 4 comparisons (Web Table 2). High agreement was observed for SMM definitions that included similar variables (e.g., presence/absence of transfusion in the index), that used similar lists of complications (e.g., those contained in the CDC index or the Bateman index), and that used similar methods of ascertaining outcomes (i.e., ICD-9 codes from the hospital discharge record vs. the birth certificate). For example, very good agreement (κ = 0.81–1.00) was observed for the CDC index and the modified CDC index with the LOS restriction (κ = 0.927), the CDC index—Bateman modification and the Bateman index (both of which excluded transfusion; κ = 0.918), and the birth certificate analog of the CDC index and the expanded birth certificate index (κ = 0.877).

In contrast, low agreement was observed when 1 or more of the following differed: presence/absence of transfusion, inclusion of complications from the CDC index versus the Bateman index, and ICD-9 definition versus birth certificate definition. For example, agreement between the Bateman SMM index and the CDC index was in the fair range (κ = 0.388); these definitions differ based on lists of complications and the presence of transfusion in the index. Agreement between the CDC index without transfusion and the Bateman index (which also does not include transfusion) was higher (κ = 0.579) but still in the moderate agreement range. When the CDC index was compared with the CDC index—Bateman modification, agreement was moderate (κ = 0.492).

Agreement was poor (κ < 0.21) for all comparisons that compared a birth certificate-based SMM definition with any ICD-9–based SMM definition. For example, agreement between the birth certificate analog of the CDC index and the CDC index without transfusion approached the κ value of 0, which indicates agreement no better than chance (κ = 0.086).

These results were driven uniformly by low sensitivity; specificity was consistently above 99%. Sensitivity was particularly low for the birth certificate–based SMM definitions relative to ICD-9–based definitions, never exceeding 15%. Sensitivity was also low for some 2-way comparisons between ICD-9–based SMM definitions, particularly when blood transfusion was included in the variable that was considered the true measurement (sensitivity = 43.5% when CDC index—Bateman modification was the test and the CDC index was used as the truth). The sensitivity of the Bateman SMM index was low in relation to the CDC index (32.3%) but adequate for the CDC index without transfusion (73.2%).

The heat map (Figure 1) depicts the concordance between SMM definitions. All comparisons where κ values were good or very good (i.e., κ ≥ 0.61) were between definitions that shared a data source (i.e., birth certificate vs. ICD-9 codes). For ICD-9–based definitions, good/very good agreement was observed only between definitions that treated transfusion in the same way (i.e., included or excluded). For ICD-9–based definitions that differed with regard to blood transfusion, agreement was either fair or moderate.

Figure 1.

Heat map depicting the degree of concordance (as quantified by κ value ranges) between 7 definitions of severe maternal morbidity (SMM), California, 2007–2012. CDC, Centers for Disease Control and Prevention; ICD-9, International Classification of Diseases, Ninth Revision; LOS, length of stay.

When comparing specific outcomes flagged by birth certificate checkboxes with comparable specific outcomes identified using ICD-9 codes, concordance was also low (Table 2). Although these complications were not ascertained using identical definitions between the data sources, the prevalence of most complications was higher using ICD-9 codes—for example, 0.91% for transfusion (vs. 0.10% on the birth certificate) and 0.09% for hysterectomy (vs. 0.03% for unplanned hysterectomy on the birth certificate). Agreement ranged from κ = 0.013 (for sepsis) to κ = 0.392 (for hysterectomy/unplanned hysterectomy). Considering ICD-9–identified complications as the truth, sensitivity was always below 10%, except for unplanned hysterectomy and hysterectomy (sensitivity = 25.1%). The agreement between “any complication” (considering the 5 complications in this table) was low (κ = 0.123). There were no cases of eclampsia or anesthetic complications identified using the birth certificate checkbox in this population.

Table 2.

Agreement of Data From Birth Certificate Checkboxes With ICD-9-CM Codes in Hospital Discharge Records for Checkboxes That Correspond Directly to Items in the Centers for Disease Control and Prevention’s Severe Maternal Morbidity Index, California, 2007–2012

			Concordance Metric
Complication	Prevalence Based on Birth Certificates, %	Prevalence Based on ICD-9-CM Codes in Discharge Records, %	κ	Sensitivity, %	Specificity, %	PPV, %	NPV, %	Accuracy, %
Transfusion	0.10	0.91	0.143	8.1	99.9	70.1	99.2	99.1
Eclampsia	0.00	0.07	—a	0.0	100.0	—	99.9	99.9
Anesthetic complications	0.00	0.01	—	0.0	100.0	—	99.9	99.9
Unplanned hysterectomy	0.03	0.09	0.392	25.1	100.0	89.9	99.9	99.9
Sepsis	0.35	0.04	0.013	6.7	99.7	0.8	100.0	99.6
Any complication	0.47	1.04	0.123	9.4	99.6	20.8	99.1	98.7

Abbreviations: ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification; NPV, negative predictive value; PPV, positive predictive value.

^a The κ value and PPV could not be calculated because no cases were identified on the basis of the birth certificate.

DISCUSSION

This analysis demonstrated that agreement between administrative data–based definitions of SMM was generally low, especially when blood transfusion was treated differently by SMM definitions (an expected finding (6)). Inclusion of different maternal complications between SMM definitions (e.g., between the CDC index and Bateman index) contributed to the low agreement between SMM definitions observed here (e.g., often below κ < 0.61). Further, different ICD-9 codes were used to define the same condition (e.g., acute myocardial infarction, shock) between some definitions (i.e., CDC and Bateman), and this also contributed to low levels of agreement. When the CDC index without transfusion was augmented to include the additional codes flagged by the Bateman index, the resulting SMM definition had moderate-to-high agreement with both the CDC index without transfusion (κ = 0.72) and the Bateman index (κ = 0.92). Together, this highlights that many factors significantly impact SMM measurement and lead to low agreement between variable definitions, with implications for surveillance and research.

Agreement between the SMM composite as captured on the birth certificate and traditional claims-based SMM definitions was universally poor (not much better than chance). Concordance between specific SMM complications (e.g., transfusion, sepsis) as measured on birth certificates and in health insurance claims was also poor. Because birth certificate fields are increasingly being used to track and study maternal health, more attention should be paid to understanding what meaningful information they provide, if any. The poor agreement was explained by universally low sensitivity; therefore, if future work demonstrates that SMM identified through this means represents true SMM cases, these fields may be used to augment claims-based SMM definitions, but they should not be understood as valid markers for SMM on their own.

These findings inform several discussions in maternal health research. Multiple federal agencies are collaborating on a careful review of SMM codes to ensure consistency across all programs and uses. Among administrative data–based definitions that are common in the United States, the CDC-based definition has been most extensively validated, and prior work has established that transfusion-only cases do not reliably identify true SMM cases (in the absence of additional data like revenue codes, permitting calculation of the number of units of blood transfused as a proxy for severity) (6). If the preferred measure is the CDC index without transfusion, then our study suggests that none of the other administrative data–based SMM metrics are sufficiently similar (the only exception would be the CDC index—Bateman modification, which was intended to check the impact of different ICD-9 codes rather than SMM-defining conditions). Efforts to refine the CDC SMM index are currently under way, and which conditions are considered SMM powerfully affects its prevalence and which women are considered to have experienced SMM. Perhaps less broadly appreciated is the discrepancy between which ICD-9 codes are used to define a given maternal complication (e.g., in the CDC index vs. the Bateman index); this too was found to affect how and for whom SMM was recorded. Moving forward, researchers and practitioners need to assess the specific codes used to flag SMM, determine which of them should be used for this purpose, and be aware of variability across studies.

Poor concordance between birth certificate- and claims-based SMM definitions is notable but is perhaps unsurprising, given the very different provenance of these data sources. Birth certificate completion varies by state, birth setting, health system, and hospital. Checkboxes are generally completed by the attending health-care provider, a nurse, or a birth certificate clerk; via worksheets given to parents; or by some combination of these methods (31, 32). In contrast, ICD-9 codes are sometimes assigned by the provider, but they are also assigned by billing coders who review the medical record for evidence supporting application of the code to that particular case. These differences probably contribute to the discrepancies observed here; more research is needed to understand how data provenance and collection affect validity and reliability.

The limitations of our study include analyzing data from only 1 state during a time period before SMM measurement and quality improvement were national priorities. Given that birth certificate completion differs between states, it will be important to conduct similar studies in other states to determine whether these findings are generalizable. We focused on a number of US-based definitions of SMM that rely on administrative data, but other definitions exist, some relying on data that are not routinely available at the population level. For example, the SMM definition of the American College of Obstetricians and Gynecologists and other definitions rely on at least 4 units of blood being transfused (33, 34), and the World Health Organization relies on vital signs and laboratory results to indicate system failures (35). The validity of SMM measures should also be examined in the current era of maternal health measurement and quality improvement. Although we used the most recent linked data available, recent maternal health quality-of-care initiatives may be associated with increased recording of maternal complications (36, 37), potentially affecting the validity of measured SMM over time. Measurement of SMM will also be affected by the transition from the ICD-9 to the International Classification of Diseases, Tenth Revision, which occurred after our study period. All SMM definitions examined here relied on some form of administrative data; there is also a need for further studies comparing SMM as flagged in these data sources with a contemporaneous clinical recording of these conditions (while acknowledging that there is no consensus as to what clinicians regard as “true” SMM when looking at the medical record, and even less is known about patients’ perspectives on this definition) (5, 6, 38). Even so, such validation studies will be critical to interpreting population-level studies on SMM. Finally, we have focused only on measurement of SMM; research on patient perspectives on what constitutes SMM and how the validity of SMM measurement might affect associations with other variables (e.g., as in a bias analysis) is needed.

SMM is an essential metric for tracking maternal health at the population level, for increasing knowledge of what causes complications during and after birth, and ultimately for preventing progression to maternal death. These are all high-priority goals in the United States and around the globe. Rigorous analysis of SMM and evidence-based improvement of maternal health requires consistent and valid measurement of SMM and its component complications.