Validation of Selected Items on the 2003 U.S. Standard Certificate of Live Birth: New York City and Vermont

Abstract

Objective

We assessed the validity of selected items on the 2003 revised U.S. Standard Certificate of Live Birth to understand the accuracy of new and existing items.

Methods

We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of select variables reported on the birth certificate using the medical record as the gold standard for a representative sample of live births in New York City (n=603) and Vermont (n=664) in 2009.

Results

In both sites, sensitivity was excellent (>90%) for Medicaid coverage at delivery, any previous live births, and current method of delivery; sensitivity was moderate (70%–90%) for gestational diabetes; and sensitivity was poor (<70%) for premature rupture of the membranes and gestational hypertension. In both sites, PPV was excellent for Medicaid coverage, any previous live births, previous cesarean delivery, and current method of delivery, and poor for premature rupture of membranes. In both sites, almost all items had excellent (>90%) specificity and NPV.

Conclusion

Further research is needed to determine how best to improve the quality of data on the birth certificate. Future revisions of the birth certificate may consider removing those items that have consistently proven difficult to report accurately.

The U.S. Standard Certificate of Live Birth serves as a legal document and a national and state data source for monitoring maternal and infant health. Assessments of the validity of items on the birth certificate can inform researchers and maternal and child health professionals on how best to use and interpret the birth certificate data for surveillance, public health practice, and research purposes. Previous validation studies of the 1989 version of the U.S. birth certificate documented high specificity for most items and high sensitivity or agreement for a limited number of items, including maternal demographics, delivery method, and infant birthweight, and low-to-moderate sensitivity or agreement for obstetric and medical risk factors.^1–5 The sensitivity of some items varied by maternal characteristics; indicators from birth certificates of infants born to Hispanic mothers and those not proficient in English had some of the lowest sensitivity estimates.⁴

In 2003, a new revision of the U.S. Standard Certificate of Live Birth was released by the National Center for Health Statistics (NCHS) with some items from the 1989 revision and new items added. The revised certificate incorporated suggestions from a committee that included state vital registration staff, clinicians, medical and public health researchers, and representatives of national organizations. A criticism of the revised certificate is that it would have benefited from additional input from perinatal epidemiologists with clinical experience and that studies with large samples to assess reliability and validity were needed.⁶

Since the release of the 2003 birth certificate, uptake of its use by states has been gradual; however, it was mandated that all jurisdictions adopt its use by 2014. Validation of new or revised items on the 2003 birth certificate has been limited, with few published studies. Studies include the examination of obstetric estimate of gestational age in California⁷ and two additional states;⁸ pre-pregnancy weight and height in Florida;⁹ Medicaid coverage for delivery in Iowa,¹⁰ California,¹¹ and two additional states;⁸ and gestational diabetes in several states.^5,8,12–15

The most comprehensive evaluation of the 2003 birth certificate, conducted by NCHS, in a total of eight hospitals in two states, found wide variation in the quality of data by item and hospital. We sought to expand the current evidence of the validity of items on the 2003 birth certificate by conducting a validation study in two geographically and demographically distinct jurisdictions—New York City (NYC) and Vermont. We evaluated items that were less studied and for which the prevalence was sufficient and medical records could be considered a valid gold standard. We evaluated a combination of new, revised, and unchanged items.

METHODS

We used the Pregnancy Risk Assessment Monitoring System (PRAMS) for this analysis, specifically PRAMS respondents from NYC and Vermont who delivered during a 5- to 8-month period in 2009. PRAMS used the 2009 birth certificate records as its sampling frame, and each month a questionnaire was mailed to a stratified (<2,500-gram infants vs. ≥2,500-gram infants) systematic sample of 100–300 women who recently had delivered a live-born infant. Women who did not respond to the first questionnaire were sent up to two additional questionnaires. If the questionnaires were not completed and returned, attempts were made to contact the women by telephone.

The PRAMS weighted response rates were 67.3% for NYC for women who delivered from January 1 through June 4, 2009 (n=31,844/47,342), and 82.0% for Vermont for women who delivered from January 1 through August 31, 2009 (n=3,091/3,771). The response rate was calculated by dividing the weighted number of women who returned the questionnaire or were interviewed by the total weighted number of women who were sampled during the study period. The samples were weighted for sample design, nonresponse, and noncoverage. By using these weights in the analysis, the samples represented all live births among NYC and Vermont residents during the respective study periods. Nonresponse weights were computed separately within strata (<2,500-gram infants vs. ≥2,500-gram infants) by examining response rate differences among those women with differing marital status, education, race/ethnicity, age, parity, and start of prenatal care. Groups with statistically significantly different response rates were correspondingly assigned different response weights. Noncoverage weights adjusted for births that were not included in the sampling frame. The frame noncoverage weights were derived by comparing birth files for a year of births with the calendar year birth files that states provided to the U.S. Centers for Disease Control and Prevention. Omitted records are usually due to late processing and are evenly scattered across the state. All analyses were conducted with weighted data.

The sample from NYC included 603 PRAMS respondents who delivered in any of the city's 41 birthing hospitals from January 1 to June 4, 2009. The sample from Vermont included 664 PRAMS respondents who delivered from January 1 through August 31, 2009, in any of the state's 12 birthing hospitals or in one New Hampshire hospital close to Vermont's border. Data abstracted from the hospital record were considered the gold standard. Data were abstracted from the mothers' prenatal record, when available; the hospital delivery record; and the infant hospital record. Every woman in the sample had at least one type of record available at the hospital. In NYC, 94.3% had a prenatal record, 99.5% had a hospital delivery record, and 99.3% had an infant record. For Vermont, the corresponding percentages were 98.4%, 99.8%, and 99.7%. The data collectors were trained in record abstraction by four authors (Patricia Dietz, Jennifer Bombard, Candace Mulready-Ward, and Lucinda England) and two additional staff members. To evaluate the reliability of record abstraction, approximately 25 medical records in NYC and in Vermont were re-abstracted and compared. Errors in abstractions (estimated to be <3% for all variables) were noted and then reviewed with the abstractors.

We evaluated items on the birth certificate where there was sufficient sample size to estimate the validity measures, with the exception of gestational age, which was evaluated in a separate analysis.¹⁶ We evaluated Medicaid coverage, private insurance coverage, any diabetes in pregnancy, gestational diabetes, any previous live births, previous cesarean section, any hypertension in pregnancy, gestational hypertension, premature rupture of membranes (PROM) (≥12 hours before delivery), induced labor, augmented labor, and delivery method. The 2003 revised birth certificate had five new items (i.e., health insurance, gestational diabetes, previous cesarean delivery, augmented labor, and induced labor). In Vermont, Medicaid coverage is indicated by a check box on the birth certificate under “Principal Source of Payment for Delivery” along with the options for private insurance, self-pay, and other; in NYC, it is listed under “Financial Coverage, Primary Payer.” Due to small numbers, we could not evaluate categories other than Medicaid or private insurance. Diabetes, hypertension, and previous cesarean delivery were listed under “Risk Factors of Pregnancy” on the birth certificate form for both sites. Gestational diabetes and pre-pregnancy diabetes (i.e., diagnosis prior to the pregnancy) were listed as separate items with separate check boxes. Due to small numbers, we could not evaluate pre-pregnancy diabetes, but we did combine both gestational and pre-pregnancy diabetes to evaluate any diabetes. Both pre-pregnancy and gestational hypertension were listed as separate items with separate check boxes; again, due to small numbers, we could not evaluate pre-pregnancy hypertension, but we were able to evaluate any hypertension by combining pre-pregnancy and gestational hypertension. We assessed PROM (≥12 hours before delivery), induced labor, and augmented labor, which have separate check boxes. “Method of Delivery” included a separate check box for vaginal/spontaneous, vaginal/forceps, and vaginal/vacuum, which we combined into one group.

To describe the characteristics of the sample, we examined age, race/ethnicity, marital status, and education using data from the birth certificate. We examined pre-pregnancy insurance status and participation in the Special Supplemental Program for Women, Infants and Children (WIC) using data self-reported by the mother on the PRAMS questionnaire. We calculated prevalence and 95% confidence intervals (CIs) for items abstracted from the medical record and reported on the birth certificate. We considered differences in prevalence estimates to be statistically significant if the CIs did not overlap. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with 95% CIs for categorical variables. Sensitivity is the probability that the birth certificate will indicate a condition among those mothers or infants whose medical records show they have that condition (true positive divided by the sum of true positives and false negatives). Specificity is the probability that the birth certificate will indicate the absence of a condition among those mothers or infants whose medical records show that they do not have the condition (true negatives divided by the sum of true negatives and false positives). PPV is the probability that a birth certificate indication of a condition is correct when compared with the medical record (true positive divided by the sum of true positives and false positives). NPV is the probability that a birth certificate indication of the absence of a condition is correct when compared with the medical record (true negative divided by the sum of true negatives and false negatives). We categorized sensitivity, specificity, PPV, and NPV as excellent (>90%), moderate (70%–90%), or poor (<70%).² All analyses were stratified by site (NYC and Vermont). Records with information missing from the birth certificate or the medical record were excluded from analysis of the particular item. We used SAS^® version 9.2¹⁷ and SUDAAN^® version 10.1¹⁸ for analyses.

RESULTS

The women included in the NYC and Vermont samples had similar age distributions but differed by all other demographic characteristics (Table 1). Women from NYC were more likely than women from Vermont to be non-Hispanic black, Hispanic, or of another race; enrolled in WIC; uninsured before pregnancy; unmarried; and to have had less than a high school education (p<0.05).

Table 1.

Characteristics of mothers who gave birth in NYC (January 1–June 4, 2009) and Vermont (January 1–August 31, 2009): PRAMS respondents

^aPercentages weighted to adjust for sample design, nonresponse, and noncoverage. Sample size varies for each characteristic due to missing values.

NYC = New York City

PRAMS = Pregnancy Risk Assessment Monitoring System

WIC = Special Supplemental Nutrition Program for Women, Infants, and Children

Maternal, pregnancy, and infant characteristics

For both NYC and Vermont, there were no statistical differences (criterion non-overlapping CIs) between reported prevalence based on the medical record and the birth certificate of most maternal, pregnancy, and infant characteristics (Table 2). The exceptions for NYC included hypertension (any and gestational—estimates from the medical record were higher than estimates from the birth certificate) and augmented labor (medical record estimates were higher than birth certificate estimates). For Vermont, the exception was PROM (medical records were higher than birth certificate estimates).

Table 2.

Comparison of birth certificate and medical record prevalence of insurance coverage, pregnancy history, complications, and method of delivery for infants born in NYC (January 1–June 4, 2009) and Vermont (January 1–August 31, 2009)

^aPercentages are weighted to adjust for sample design, nonresponse, and noncoverage; numbers are unweighted. Sample size varies for each item due to missing values.

^bDid not include women with more than one type of insurance and self-insured due to small numbers

^cNew item on the 2003 birth certificate

NYC = New York City

CI = confidence interval

Sensitivity and specificity

NYC and Vermont both had excellent sensitivity for Medicaid coverage at delivery, any prior live births, and delivery method (vaginal or cesarean) (Table 3). Additionally, Vermont had excellent sensitivity for private insurance and previous cesarean delivery. Both NYC and Vermont had moderate sensitivity for gestational diabetes; Vermont had additional items with moderate sensitivity including any hypertension, any diabetes, induced labor, and augmented labor. NYC had moderate sensitivity for private insurance at delivery. Both NYC and Vermont had poor sensitivity for PROM and gestational hypertension. Additionally, NYC had poor sensitivity for prior cesarean section, any diabetes, any hypertension, induced labor, and augmented labor. Specificity was excellent (>90%) for all items for both NYC and Vermont, except for Medicaid insurance at delivery in NYC (86.7%) (Table 3).

Table 3.

Sensitivity and specificity of insurance coverage, pregnancy history, complications, and method of delivery reported on the birth certificate for infants born in NYC (January 1–June 4, 2009) and Vermont (January 1–August 31, 2009)

^aData are weighted to adjust for sample design, nonresponse, and noncoverage. Sample size varies for each item due to missing values.

^bDid not include women with more than one type of insurance and self-insured due to small numbers

^cNew item on the U.S. 2003 birth certificate

NYC = New York City

CI = confidence interval

PPV and NPV

For both sites, the PPV was >90% for Medicaid coverage, any previous live births, previous cesarean delivery, and delivery method, and, for NYC only, for any diabetes and gestational diabetes (Table 4). For Vermont, the PPV was >90% for any hypertension and induced labor. The PPV was <70% for PROM for both sites and for induced labor in NYC. All other indicators had moderate PPVs. The NPV was >90% for all items except Medicaid coverage (89.1%), induced labor (89.4%), and augmented labor (73.2%) in NYC, and for PROM (89.4%) in Vermont.

Table 4.

PPV and NPV of insurance coverage, pregnancy history, complications, and method of delivery reported on the birth certificate for infants born in NYC (January 1–June 4, 2009) and Vermont (January 1–August 31, 2009)

^aData are weighted to adjust for sample design, nonresponse, and noncoverage. Sample size varies for each item due to missing values.

^bDid not include women with more than one type of insurance and self-insured due to small numbers

^cNew item on the U.S. 2003 birth certificate

PPV = positive predictive value

NPV = negative predictive value

NYC = New York City

CI = confidence interval

Summary

The Figure summarizes the sensitivity, specificity, PPVs, and NPVs for all items by site. Any previous live birth as well as vaginal and cesarean deliveries were >90% for all validity measures in both sites.

Figure.

Sensitivity, specificity, PPV, and NPV^a of insurance coverage, pregnancy history, complications, and method of delivery on the birth certificate for infants born in NYC (January 1–June 4, 2009) and Vermont (January 1–August 31, 2009)

^aBlack = excellent (>90%), gray = moderate (70%–90%), and light gray = poor (<70%). Sample size varies for each characteristic due to missing values.

^bDid not include women with more than one type of insurance and self-insured due to small numbers

^cNew item on the U.S. 2003 birth certificate

PPV = positive predictive value

NPV = negative predictive value

NYC = New York City

DISCUSSION

We found that one new item on the revised 2003 birth certificate (Medicaid coverage) had excellent (or almost excellent) validity (sensitivity, specificity, PPV, and NPV); one new item (gestational diabetes) had moderate sensitivity but excellent (or almost excellent) specificity, PPV, and NPV; and three new items (previous cesarean delivery, induced labor, and augmented labor) had inconsistent results between the two sites. Overall, almost all items had excellent specificity and NPV, which was influenced by the low prevalence of the items included in this evaluation. Even items that had large discrepancies between the birth certificate and the medical record, such as hypertension in NYC and PROM in Vermont, had excellent or near-excellent specificity and NPV.

Our results for Medicaid coverage are consistent with two previous state-based studies—one conducted in California,¹¹ which found an overall agreement of 95.8% between birth certificate and women's self-report on a survey; and the other in Iowa,⁹ which found a sensitivity of 86.3% and specificity of 91.9% for birth certificates using matched Medicaid claims records as the gold standard. Our sensitivity estimates for Medicaid coverage, however, were generally higher than that found in an NCHS study of eight hospitals in two states, where sensitivity for Medicaid coverage in each hospital ranged from <50% to >90%.⁸

The 2003 birth certificate included two items, pre-pregnancy diabetes and gestational diabetes, that had previously been reported as one item (any diabetes). We were unable to assess pre-pregnancy diabetes due to small numbers, but we were able to assess the validity of gestational diabetes. A systematic review of four birth certificate validation studies that assessed gestational diabetes reported sensitivity estimates ranging from a low of 45.8% in a sample of hospitals in Ohio³ to a high of 83.3% among a random sample of births in four counties in New York State.¹³ Our sensitivity estimates (70.3% and 75.7%) fell within this range and were higher than the sensitivity estimates from the NCHS study, which used the same gold standard (medical records) and found an average sensitivity estimate of 57.7% among four hospitals in one state and an average sensitivity estimate of 58.6% among four hospitals in another state.⁸ Our PPV estimate for gestational diabetes for NYC (91.0%) was higher than the other four studies, and our PPV estimate for Vermont (82.5%) was consistent with one and higher than the other three.^5,13–15 Specificity was consistently excellent for our study and the other studies.

The 2003 birth certificate included two items—pre-pregnancy hypertension and gestational hypertension—that had previously been reported as one item (any hypertension). We were unable to assess pre-pregnancy hypertension due to small numbers, but we were able to assess the validity of gestational hypertension. Similar to another study that found low sensitivity for gestational hypertension, 57.7% in one state and 58.6% in another,⁸ our sensitivity estimates were 33.4% and 62.5%. Thus, the evidence so far suggests that gestational hypertension has poor sensitivity on the birth certificate. Any hypertension performed slightly better on sensitivity in Vermont (75.5%) but not in NYC (38.7%).

Previous cesarean section is a new check box in the “Risk Factors in this Pregnancy” section on the 2003 birth certificate; on the prior version, it was included as a check box under delivery method, with primary cesarean section or repeat cesarean section. Two studies have assessed “repeat cesarean”; one conducted among births in four counties in New York State and the other conducted among births in 11 health plans.^13,15 Both documented excellent sensitivity, specificity, PPV, and NPV, a finding that is consistent with our finding for “previous cesarean section” for Vermont births; however, for NYC births, sensitivity was poor and the other three measures were excellent. The NCHS study found variability in the sensitivity for this item as well, with a sensitivity of 82.1% averaged among four hospitals in one state and 62.5% in the other state.⁸

In the 2003 revised birth certificate, separate fields were added for induced and augmented labor. As for induced labor, our results for sensitivity varied greatly from 55.0% in NYC to 84.7% in Vermont; the NCHS study found a sensitivity of 86.0% among hospitals in one state and 45.9% among hospitals in another state. This finding was similar for augmented labor, where our results for sensitivity were 37.1% in NYC and 89.2% in Vermont; the NCHS study found a sensitivity of 69.7% among hospitals in one state and 37.2% among hospitals in another state.⁸ One suggestion to improve reporting of these two items is a field to check if the woman was in spontaneous labor on admission to the hospital, and another if the delivery was elective, with subcategories for planned induction and scheduled cesarean delivery.⁶ Overall, differences in results found by site and other studies could be a result of a number of factors, such as how recently the new birth certificate was implemented, how well staff were trained, how complete and available the medical records were, and the year of the study.

For items on the 2003 U.S. Standard Certificate of Live Birth that remained unchanged, our findings confirm results from previous studies: the birth certificate accurately captured delivery method, but sensitivity was poor for PROM.^1–5,8 Also, consistent with previous studies, we found that birth certificate items had excellent specificity.^1–5 Findings from our study and others suggest that some items (e.g., PROM) are difficult to report on the birth certificate, while others (e.g., cesarean delivery) have excellent validity and are useful for surveillance purposes. PRAMS also collected information from the mothers on self-reported gestational diabetes, any hypertension, and preterm PROM. Validity on these three items based on self-report was similar to the validity found on the birth certificate.¹⁹

This study demonstrates that the sensitivity and PPV of birth certificate items can differ greatly by site for some items such as previous cesarean delivery, any hypertension, induced labor, and augmented labor. This finding suggests that certain items (e.g., those with excellent validation in some sites but poor validation in others) have the potential for improvement through training. NYC and Vermont also differ on several factors including demographics, number of birthing hospitals, health-care delivery, and volume of deliveries. These differences might have affected the results.

Strengths and limitations

One strength of this evaluation was that the inclusion of samples from two geographically diverse populations provided the opportunity to assess the validity of the variables in a rural, predominantly white population and in a multiethnic urban population. In addition, population-based samples included births from all birthing hospitals in 41 hospitals for the NYC sample and 13 hospitals for Vermont; as such, sensitivity and specificity should be generalizable to all women who delivered live infants in Vermont and potentially in NYC.

This study was also subject to several limitations. For one, NYC had a low response rate on the PRAMS questionnaire, and while the weights applied in this analysis adjusted for nonresponse, the weighted data may not be representative of all women who delivered a live birth. Other limitations include possible medical record abstraction errors and the inability to assess validity for individual hospitals due to small sample sizes. Additional evaluations are needed to assess data quality in other reporting areas.

CONCLUSIONS

Findings from this effort will help inform jurisdictions of the strengths and limitations of using birth certificate data for surveillance and research purposes. The results of this evaluation suggest that health coverage data from birth certificates is valid for Medicaid, as well as data on any previous live birth and delivery method. However, gestational diabetes on the birth certificate had moderate sensitivity, suggesting that prevalence estimates based on birth certificate data will under-estimate the true prevalence. Some items (e.g., PROM) had low sensitivity and PPV. Further research is needed to determine how best to improve the quality of data on the birth certificate, and investigations are needed at both the hospital and jurisdiction level. Others have called for more funding for total continuous data quality improvement programs for state vital statistics agencies.⁶ In addition, future revisions of the birth certificate should consider removing those items that have consistently proven difficult to report accurately.