Risk Prediction for Adverse Pregnancy Outcomes in a Medicaid Population

Abstract

Background: Despite prior efforts to develop pregnancy risk prediction models, there remains a lack of evidence to guide implementation in clinical practice. The current aim was to develop and validate a risk tool grounded in social determinants theory for use among at-risk Medicaid patients.

Methods: This was a retrospective cohort study of 409 women across 17 Cincinnati health centers between September 2013 and April 2014. The primary outcomes included preterm birth, low birth weight, intrauterine fetal demise, and neonatal death. After random allocation into derivation and validation samples, a multivariable model was developed, and a risk scoring system was assessed and validated using area under the receiver operating characteristic curve (AUROC) values.

Results: The derived multivariable model (n=263) included: prior preterm birth, interpregnancy interval, late prenatal care, comorbid conditions, history of childhood abuse, substance use, tobacco use, body mass index, race, twin gestation, and short cervical length. Using a weighted risk score, each additional point was associated with an odds ratio of 1.57 for adverse outcomes, p<0.001, AUROC=0.79. In the validation sample (n=146), each additional point conferred an odds ratio of 1.20, p=0.03, AUROC=0.63. Using a cutoff of 20% probability for the outcome, sensitivity was 29%, with specificity 82%. Positive and negative predictive values were 22% and 85%, respectively.

Conclusions: Risk scoring based on social determinants can discriminate pregnancy risk within a Medicaid population; however, performance is modest and consistent with prior prediction models. Future research is needed to evaluate whether implementation of risk scoring in Medicaid prenatal care programs improves clinical outcomes.

Introduction

Compared with privately insured women, the prevalence of pregnancy risk factors among women on Medicaid is high,^1,2 resulting in a disproportionate share of costs associated with preterm birth and other adverse pregnancy outcomes. Despite these disparities, a majority of low-income women served by Medicaid still deliver at ≥37 weeks gestation, and high resolution tools to predict who is at greatest risk of an adverse outcome within this population are lacking.

In recent years, statewide initiatives to expand Medicaid prenatal care coverage and case management programs have increasingly applied a social determinants model of health to address pregnancy outcome disparities. The social determinants model emphasizes the powerful influences of lifestyle and the conditions in which people live and work.^3–5 Prenatal case management seeks to address social determinants through a range of strategies including psychosocial, nutritional, and health promotion assessment; counseling and referral to public health and social services; and overall care coordination. Several evaluation studies of regional or statewide prenatal case management programs have demonstrated reduction of low birthweight or preterm births among Medicaid recipients after program implementation.^6–11 Although the available literature reports a wide range of effect sizes, at least one recent randomized controlled trial of prenatal case management in New York demonstrated a >50% reduction in the adjusted odds of low birthweight.¹²

The current study objective was to develop and validate a risk-scoring tool theoretically grounded in a social determinants model to quantify pregnancy risk within a Medicaid population. Given limited case management resources available, our overall goal was to provide a system for allocating high-quality intensive care coordination, social support, and material resources to those patients at highest risk for adverse outcomes, as well as to assist case managers in developing and implementing individualized plans of care. To date, several scoring systems to predict risk of adverse pregnancy outcomes have evaluated the prognostic utility of demographic factors and obstetrical history including prior preterm birth; these scoring systems have demonstrated a wide range of accuracy, with no clear indication for implementation of any one system in clinical practice.^13–15 More recently, risk models including cervical length as well as various biomarkers have shown promise in increasing prognostic performance.^16–22 However, this research has not yet yielded a risk assessment tool for use in clinical practice; moreover, the majority of these prediction models have been derived using a general population, which may overinflate their ability to discriminate risk within an a priori at-risk population of Medicaid recipients.

For the current study, we hypothesized that a prediction tool grounded in social determinants would have improved performance in discriminating pregnancy risk among at-risk women on Medicaid by incorporating the effect of life experiences such as homelessness, domestic violence, and substance abuse in addition to standard sociodemographic and clinical factors.

Material and Methods

We conducted a validation study to evaluate a risk scoring tool for low-income pregnant women. This study was approved by the University of Cincinnati Institutional Review Board, protocol 2013-2443, including a waiver of consent based on minimal risk to subjects and feasibility of obtaining consent from the entire sample, which would preclude study completion.

Patients and setting

The study included women receiving obstetrical care from the Community Women's Health (CWH) program, University of Cincinnati Health system who delivered an infant between September 2013 and April 2014. Encompassing 17 community health centers located across Greater Cincinnati and Hamilton County, the CWH program provides all routine ambulatory obstetric and gynecologic care for the Cincinnati Health Department, representing a unique collaboration between an academic center and the city's health department, with 750–850 births each year. Approximately 96% of patients are Medicaid insured. Since 2012, the program has utilized a care curriculum focused on timely access to care, standardized assessments for social risks, and rapid referral to programs and community services.

Data sources and variables

All risk factor data was collected by clinicians and registered nurse (RN) case managers using a standardized risk form developed in collaboration with the study team and program staff. Each form was completed using patient interviews and medical record review during routine assessment at entry into prenatal care. A panel of clinical factors was selected based on existing literature supporting their a priori relevance to pregnancy outcomes. These included aspects of prior obstetrical history (i.e. history of preterm birth, history of low birthweight delivery, and history of intrauterine fetal demise).^23–25 Interpregnancy interval was based on maternal report of interval from last prior live birth to current pregnancy and categorized as either <18 or ≥18 months.²⁶ Twin gestation and known short cervical length, defined as <25 mm,²⁷ was also included in the assessment. Additional clinical factors known to be associated with adverse pregnancy outcomes included prepregnancy body mass index, hypertensive disorders (essential and pregnancy induced), recent history of substance abuse (including alcohol dependency), and current tobacco use.^28–30 Because of literature suggesting that comorbidities during pregnancy may adversely affect pregnancy outcomes, a single indicator variable representing other medical conditions such as asthma and rheumatoid arthritis was also included.

The risk form incorporated standard demographics such as maternal race, age, marital status, and employment status, in addition to mental health conditions, homelessness, history of childhood rape or abuse, and developmental delay. As a potential marker of disorderly lifestyle and otherwise higher-risk status, late entry into prenatal care (defined as >28 weeks gestation) was also included as a social determinant, particularly given known associations with worse pregnancy outcomes.⁴⁰

Pregnancy outcomes for each woman were collected through medical chart review at the delivery hospital. We derived a composite outcome measure representing an adverse outcome if one of four events occurred: preterm birth, infant birthweight <2500 g, intrauterine fetal demise, or neonatal death. These outcomes were selected based on their substantial contribution to neonatal and pediatric health and healthcare spending, as well as known high rates of occurrence regionally in comparison with national rates. Our decision to combine them into one composite measure was based on a common set of social and biological risk factors, as well as the fact that pregnancy case management is not focused on prevention of any one singular outcome.

Statistical analysis

A primary objective of this analysis was to validate the accuracy of our derived risk assessment tool. Validation helps address the question of whether a prognostic model has applicability for a wider population of patients and is therefore an important step in assessing its appropriateness for implementation in clinical practice.³² Toward this objective, we used data exclusion techniques to validate the derived model within the same study cohort. Using the random sample function in STATA Version 11.1 (StataCorp, College Station), the cohort was first split into derivation and validation samples.³³ As with similar studies, the proportion of the cohort allocated to the derivation sample was approximately two-thirds versus one-third allocated to validation,^34,35 in order to provide a sufficient sample size for model development based on a fixed alpha 0.05, power 80%, and an estimated correlation coefficient for exposures between cases and controls of 0.2.³⁶ Next, univariable logistic regression was used in the derivation sample to test the univariate association between each individual factor and the primary composite outcome. Multivariable logistic regression including all factors then tested the adjusted association of each variable with the outcome. Model calibration was assessed using the Hosmer-Lemeshow statistic. Discrimination was quantified with the area under the receiver operating characteristic curve (AUROC). The model was then reduced in a stepwise process by excluding variables with univariate p-values>0.15 using the log likelihood ratio test. The cutoff value of p-value>0.15 was intended to avoid rejection of potentially important variables in the absence of adjustment for confounding.³⁷ In addition to using statistical criteria, variables were selected into multivariable modeling based on available evidence supporting the variable's importance for pregnancy outcomes. Based on prior evidence suggesting that risk factors may have differential impact for primiparous versus multiparous women, we made an a priori decision to examine the statistical significance of adding interaction terms between primiparous status with identified key risk factors using likelihood ratio tests. We also examined any modifying effects of maternal race for these factors.

The predictive ability of the resulting reduced model was assessed two ways. First, an unweighted score was calculated based on assigning 1 point for each retained risk factor. Second, a weighted score was calculated using point values 1–3 for each factor based on their regression coefficients. AUROC values for the weighted and unweighted risk scores were compared. To demonstrate lack of over-fitting we internally validated these models (i.e., with the derivation sample) using a 10-fold cross validation method. Finally, we validated performance of the risk score based on sensitivity, specificity, positive predictive values, and negative predictive values using the separate validation sample.

Results

A total of 409 women had a completed risk assessment form and documented pregnancy outcome during the study timeframe; 263 were assigned to the derivation sample. Their demographic characteristics were as follows: mean maternal age was 25 years, 70% were unmarried, 67% reported unemployment, 59% were of black race, 30% did not complete high school, 20% were smokers, 17% reported drug use within the past year, 10% had a history of prior preterm delivery, and 10% had an interpregnancy interval of less than 18 months.

The remaining 146 records were used for validation of the prediction model. As shown in Table 1, there were no statistically significant differences in characteristics or pregnancy outcomes between the derivation and validation cohorts.

Table 1.

Percentage of Characteristics and Pregnancy Outcomes Within Derivation and Validation Cohorts

	Derivation cohort, % (n=263)	Validation cohort, % (n=146)	Chi-squared p-value
Maternal characteristics
Age<20 years	14.8	14.4	0.39
Unmarried	69.6	68.5	0.82
Unemployed	67.3	66.4	0.86
Primiparous	35.6	28.9	0.16
No high school degree	29.7	32.2	0.59
Maternal race			0.94
Black	59.2	57.0
White	28.5	30.5
Other	12.3	12.5
Tobacco use	19.8	26.0	0.14
Substance abuse within past year	17.5	23.3	0.16
History of childhood rape or abuse	5.7	5.5	0.93
Mental health condition	8.4	6.9	0.58
Homeless	1.9	0.7	0.33
Developmental delay	1.1	0.7	0.65
Hypertensive disorders	1.9	2.0	0.91
Other medical conditions	8.4	8.2	0.96
Body mass index>40	6.9	6.2	0.79
Twin gestation	0.8	2.0	0.25
Short cervix	0.0	0.7	0.18
Prenatal care entry>28 weeks	1.9	2.7	0.58
Interpregnancy interval<18 months	10.3	12.3	0.52
Prior low birth weight infant	4.6	4.8	0.92
Prior preterm birth	10.3	8.2	0.50
Prior intrauterine fetal demise/ neonatal death	4.6	6.9	0.33
Outcomes
Preterm birth	8.4	8.9	0.85
Birth weight<2500 grams	11.4	13.0	0.63
Intrauterine fetal demise/ neonatal death	0.4	0.7	0.67
Composite adverse outcome	15.2	15.8	0.88

Prediction model derivation

In univariable logistic regression, the following factors had a univariate p-value<0.15: prior low birthweight infant, prior preterm birth, prenatal care entry >28 weeks, body mass index (BMI) >40, recent history of substance abuse, tobacco use, childhood abuse or rape, other medical conditions, and black race. Three additional variables—short cervix, twin gestation, and short interpregnancy interval—did not meet this statistical criterion but were determined to be important on the basis of empiric evidence.

The initial AUROC value associated with the logistic regression model containing all potential factors was 0.81. Stepwise elimination of variables with univariate p-values>0.15 and otherwise not deemed empirically important resulted in a reduced model shown in Table 2. As shown, the addition of primiparity as an effect modifier was statistically significant for BMI>40 [adjusted odds ratio (AOR) 43.18; 95% confidence interval (CI) 4.29–434.26 among primiparous women], as well as for other medical conditions [AOR 49.49; 95% CI 5.73–427.78 among primiparous women]. For multiparous women, neither BMI>40 nor other medical conditions were significant factors. Conversely, black race was associated with an increased AOR of 15.8 (95% CI 3.35–74.89) among multiparous women but was not a significant factor among primiparous women. The model also contained a statistically significant interaction term between maternal race and smoking status, such that tobacco use was associated with a higher odds of adverse pregnancy outcomes among non-black women (AOR 6.56, 95% CI 1.22–35.14), but not among black women.

Table 2.

Multivariable Logistic Regression Model of Adverse Pregnancy Outcomes with Selected Predictors (n=263)

Predictor	Adjusted odds ratio of adverse outcomea (95% confidence interval)	Points assigned for weighted risk score
Black race
Primiparous	0.58 (0.10–3.50)	1
Multiparous	15.8 (3.35–74.89)	3
Tobacco use
Black race	0.65 (0.21–2.02)	1
White or other race	6.56 (1.22–35.14)	2
Substance abuse within past year	1.85 (0.70–4.91)	1
History of childhood rape or abuse	5.14 (1.17–22.57)	2
Other medical conditionsb
Primiparous	49.49 (5.73–427.78)	3
Multiparous	0.91 (0.17–5.01)	1
Body mass index>40
Primiparous	43.18 (4.29–434.26)	3
Multiparous	0.87 (0.15–5.04)	1
Twin gestation	21.91 (0.41–1177.01)	3
Prenatal care entry>28 weeks	43.22 (4.62–404.17)	3
Interpregnancy interval<18 months	2.81 (0.86–9.20)	1
Prior preterm birth	5.47 (1.80–16.59)	2

Short cervical length was omitted from the model due to small numbers and perfect collinearity with having the outcome.

^aValues in bold indicate p-values<0.05.

^bIndicator variable representing a range of non-obstetrical conditions ascertained at the time of assessment, examples of which include asthma, rheumatoid arthritis, and history of positive tuberculin Tb test.

The AUROC value for this reduced model was 0.85. Based on a 1-point assignment for each risk factor in this reduced model, the range of unweighted risk scores within this derivation cohort was 0–6. While 29% had a risk score of 0; 35% had a score of 1; 21% had a score of 2; 12% had a score of 3; and the remaining 3% had a score of 4 or higher. When univariable logistic regression was used to test the association of unweighted risk score with the primary outcome, each additional risk point was associated with an odds ratio of 2.23 (95% CI 1.62–3.07), model AUROC value 0.76 (see Fig. 1A).

FIG. 1.

Receiver operating characteristic (ROC) curves for unweighted and weighted risk scores in the derivation sample, n=263. (A) ROC curve and corresponding area under the receiver operating characteristic curve (AUROC) statistic for the unweighted risk score in the derivation sample. (B) AUROC curve statistic for the weighted risk score in the same (derivation) sample.

We also applied weights to these risk factors based on multivariable regression coefficients and to account for the interaction effects based on race and primiparity (see Table 2). Factors with regression coefficient values of 0–3 were scored as 1, those with values 3–7 were scored as 2, and those with values ≥7 were scored as 3. The range for the resulting weighted risk scores for the derivation sample was 0–10; 27% of the cohort had a score of zero, 44% had a score of 1–3, 25% had a score 4–6, and the remaining 4% had a score of 7–10. Univariable logistic regression demonstrated improved predictive performance of this weighted risk score, with each additional point associated with an odds ratio of 1.57 (95% CI 1.32–1.84), model AUROC value 0.79 (Fig. 1B).

As a sensitivity analysis, the weighted risk model was applied separately to both preterm birth and low birthweight as outcomes; AUROC values were >0.75 and p-values<0.001 in both models. Finally, internal validation using 10-fold cross validation demonstrated an AUROC was 0.75, suggesting that the model was not overly specified (Fig. 2A).

FIG. 2.

Receiver operating characteristic curves derived using validation procedures. (A) AUROC curve and statistic for the weighted risk score using 10-fold cross validation with the derivation sample, n=263. (B) AUROC curve statistic for the weighted risk score in the validation sample, n=146. Black squares indicate sensitivities and specificities for various cutoffs defining higher risk of the outcome.

Model validation

When this weighted risk model was applied to the validation cohort, the distribution of weighted scores were similar to the derivation cohort, such that 24% had a score of zero, 49% had a score of 1–3, 18% had a score of 4–6, and 8% had a score of 7–10. Univariable logistic regression demonstrated statistical significance of the risk score, with each additional point conferring an odds ratio of 1.20 for adverse outcomes, 95% CI 1.02–1.44. However, the AUROC value for this model was only 0.63 (Fig. 2B). Table 3 demonstrates model sensitivity, specificity, and positive and negative predictive values based on various cutoffs for higher probability of the outcome, also indicated in Figure 2. These were not substantively changed when the sample was stratified by primiparous status (data not shown).

Table 3.

Model Sensitivity, Specificity, and Positive and Negative Predictive Values in the Validation Sample (n=146)

Cutoff for higher outcome probability	Sensitivity	Specificity	Positive predictive value	Negative predictive value
20%	29%	82%	22%	85%
25%	23%	93%	42%	86%
30%	21%	95%	45%	87%
35%	13%	98%	60%	85%

Finally, we conducted an exploratory analysis of the prediction model failures in this validation cohort [i.e., women who had a low weighted risk score (<4) but who nevertheless experienced an adverse pregnancy outcome]. For the 101 women with a score <4, of whom 13 (12%) experienced an adverse outcome, no statistically significant differences were observed in outcomes by maternal social and medical characteristics, including history of prior preterm birth, late entry into prenatal care, or maternal race (all p-values>0.05; data not shown).

Discussion

Profound disparities in birth outcomes for socially disadvantaged Medicaid patients make this population a critical focus for targeting interventions during pregnancy. In Hamilton County, Ohio, where approximately 40% of births are insured under Medicaid, the infant mortality rate among Medicaid recipients is 15.1 deaths/1000 live births, compared with a rate of 5.9 among the privately insured population.³⁸ RN case management, community health worker programs, and home visiting represent key strategies to address such disparities and have received increased investment since passage of the Patient Protection and Affordable Care Act. However, these resources remain a challenge to allocate within under-resourced health care settings that serve women on Medicaid. Assessment tools that may help assign risk status within this higher risk population are therefore of great interest to payers and providers. In this study of a single city's health department system spanning 17 health centers, we developed a risk score model for pregnancy outcomes that accounted for effects of primiparity and race. Results demonstrated low sensitivity and positive predictive value, which indicates that a low score did not reliably rule out a higher likelihood of preterm birth or other adverse outcome. However, the specificity of the risk score model was reasonable, suggesting that women with high scores were appropriately assigned as being at high risk. In this way, risk scoring may have some utility for this subgroup in facilitating the rapid integration of care coordination, social support, and material resources into clinical plans of care.

While prior studies have developed scoring systems to predict preterm birth risk at an individual level, results have had limited capability for clinical implementation.¹⁵ Even models derived from large samples have failed to provide high predictive performance.^13,14,17,18 More recent efforts have focused on biomarkers such as pregnancy-associated plasma protein A, fetal fibronectin, and placental growth factor to augment the predictive capacity of demographic and clinical risk factors; however, this body of research has not yet yielded a clear approach for implementation in clinical practice.^19–22

As with prior studies, important risk factors in our model included race and obstetric history. Our inclusion of childhood abuse in the model is also consistent with recent literature demonstrating the importance of childhood history as a significant risk factor.³⁹ To our knowledge, the interaction effects of risk factors with race and parity included in our risk prediction model have not been incorporated into previous clinical scoring systems. To prevent prior pregnancy factors from overshadowing the importance of other risk factors, our model accounted for differences in weighting of factors based on primiparous status; consequently, model performance was not significantly worse or better for women with prior pregnancies. That tobacco use was differentially associated with higher risk based on race likely reflects racial and cultural differences in smoking patterns.⁴⁰ The significant interaction between race and multiparity is also consistent with the concept of weathering among women of black race.⁴¹

Our results and the work of others underscore the difficulty of accurately assigning risk of preterm birth to an individual woman who, on the basis of race, income, or other conditions is at higher risk compared with the general population. Continued efforts to develop, validate, and implement prognostic models in clinical care are a critical area of research and may be strengthened by further investigation of key interactions between sociodemographic and biologic factors (i.e., epigenetic influences). However, in the absence of a well-performing risk prediction model, one practical implication may be that the delivery of high quality, evidence-based, and systematic approaches to social determinants of health may be necessary for all women within an at-risk population. The utility of this approach may, in part, be reflected in our study sample, given that the preterm birth rate for the entire cohort was 8.6%, which is lower than the reported average for women in Hamilton County.⁴² While issues of selection bias discussed above are likely contributors to this finding, this finding may also be related to the CWH program itself, in which quality improvement initiatives over the past 3 years have focused on a bundle of interventions addressing social determinants of pregnancy health, including early prenatal care access, pregnancy spacing, and smoking cessation.⁴³ Further expansion of RN case management and other care coordination resources may be critical to achieving maximal program impact and improving outcomes through interventions that are tailored to each woman's unique set of risk factors.

Strengths and limitations

A primary limitation of this study is precision of the fitted parameters due to small sample size, particularly as only a portion of the data was used for model derivation. Another limitation of the study is generalizability of findings given the regional population represented. Accuracy of the predictor and outcome data is dependent on the quality of documentation in the medical record, which is likely to vary somewhat across clinics and patients. Also, reporting of certain predictors may be subject to social desirability bias. Particularly for sensitive issues such as history of rape or abuse, the validity of these data may be a function of the way patients were asked the questions. Moreover, dichotomizing certain predictors such as prior preterm birth ignores the potential additive effect of multiple prior adverse events. Given that risk factor assessment was performed by providers in the context of clinical care, women with a high number of risk factors were more likely to receive intensive resources, including ongoing RN case management and a dedicated community health worker or home visitor. Among all 409 women, as an example, weighted risk scores were on average 1.6 points higher among women referred to a community health worker compared with those who were not (p<0.001). This may have potentially mitigated their risk for poor outcomes, and as a result, our analysis may underestimate the associations between risk factors and adverse pregnancy outcomes. However, as a sensitivity analysis, we did include referral to services in our risk score prediction model, and the effect size and AUROC value were not significantly changed. Our cohort also only includes women who sought prenatal care; we would expect differences in risk among women who did not seek care. This selection bias may also contribute to an underestimation of model predictive ability. Finally, use of a composite outcome measure for the primary analysis may be considered a limitation of the study given the potential for different causal pathways. However, results were not substantively changed when we applied the prediction model separately for preterm birth and low birth weight.

Strengths of the study include our use of a separate validation sample, multiple clinic sites represented, and the recent nature of this data. Also, rather than using birth certificate data which can produce inaccuracies in gestational age classification, we relied on clinician chart review to derive pregnancy outcome data.⁴⁴ Finally, our approach using real-time data collection enabled the availability of risk factors not captured in administrative data sources.

Conclusions

With an at-risk population, a risk score incorporating social determinants as well as clinical and demographic factors was consistent with limited performance of previous models. Future research may focus on development and validation of more sensitive risk tools for use within a Medicaid population, as well as the impact of implementing risk tools in the setting of pregnancy case management programs.

Acknowledgments

The authors acknowledge the contribution of Meg James, other program staff, and the patients at the Community Women's Health program at the University of Cincinnati and the Cincinnati Department of Health. Dr. Goyal was supported by the Building Interdisciplinary Research Careers in Women's Health program (5K12HD051953-07), co-funded by the Office of Research on Women's Health and the Eunice Kennedy Shriver National Institute of Child Health and Human Development, administered by the Center for Clinical and Translational Science and Training at University of Cincinnati. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author Disclosure Statement

No competing financial interests exist.