Applying machine learning methods to psychosocial screening data to improve identification of prenatal depression: Implications for clinical practice and research

Introduction

Prenatal depression is one of the most common morbidities in the perinatal period with an overall prevalence between 7% and 17% (Miller et al., 2021; Underwood et al., 2016). Prenatal depression is associated with increased risk for adverse birth outcomes and postpartum depression (Accortt et al., 2015; Bauman et al., 2020), which contribute to developmental delays in offspring (Davalos et al., 2012). As awareness of perinatal depression has grown, routine screening for prenatal depression is now recommended by leading health organizations (American College of Obstetricians and Gynecologists, 2018). Nevertheless, depression screening is not universally conducted in prenatal care, and even when it is, depression is under-detected (Miller et al., 2019).

Empirical research implicates psychosocial vulnerabilities such as socioeconomic deprivation, inadequate prenatal care, unintended pregnancy, and stress as significant contributors to adverse perinatal outcomes, including depression (Abajobir et al., 2016; Hain et al., 2016; O'Hara & Wisner, 2014), and indicates that systematic screening for these types of vulnerabilities can enable early identification and intervention. However, like screening for depression, screening of psychosocial vulnerabilities is lacking. Impediments to effective psychosocial screening and depression screening include provider and patient burden, patient reluctance to disclose, and lack of sufficiently sensitive instruments (Baker et al., 2020; Connell et al., 2018; Oni et al., 2020; Williams et al., 2016); all of which can be lessened with implementation of the PROMOTE screening instrument (Preis et al., 2022).

The PROMOTE is a newly developed screening instrument based upon theoretically relevant, evidence-based, and clinically meaningful knowledge about risk for adverse perinatal outcomes (Preis et al., 2021; Preis et al., 2022). Contextual constructs captured in the PROMOTE such as social determinants of health, social resources, stress, and health behaviors (Table 1) are assessed via single items, a method which has been affirmed to be of value when assessing psychosocial vulnerabilities in the prenatal population (Sagrestano et al., 2002). The PROMOTE is comprehensive yet brief, and in 3-5 minutes, patients can report 19 different vulnerabilities such as emotional distress, health behaviors and substance use, social support, and socioeconomic conditions. Through this self-report method using single items, the PROMOTE systematically collects patient information in a direct but non-judgmental way which is preferred by patients and is as accurate as provider questioning (Midmer et al., 2004; Quispel et al., 2014; Webster & Holt, 2004). By removing barriers to health care provider screening (e.g., lack of time, feeling unqualified to ask) and impediments to patient disclosure (e.g., embarrassed to answer face to face, less burdensome than full-length questionnaires), The PROMOTE also reduces the chance that a patient’s vulnerability will be overlooked. Recent quantitative and qualitative research suggests that the PROMOTE is well-accepted both by patients (Preis et al., 2021) and providers (Preis et al., 2022).

Table 1.

Contextual factors to assess in comprehensive prenatal psychosocial screening

Social Determinants of Health	Social resources	Stress	Behavioral
low income limited education housing conditions employment]	family and friends’ social support marital/relationship status partner support	perceived stress emotional distress stressful life situations interpersonal violence psychiatric medications	pregnancy planning substance use healthy behaviors

Technologically sophisticated strategies can provide high resolution assessments and enable the development of accurate prediction models for personalized clinical care (Shazly et al., 2022). There is a critical need for methods that can improve identification of vulnerable women that are at-risk for adverse pregnancy outcomes. In the current report, we illustrate how vulnerabilities identified in the PROMOTE can be used to classify women at risk for under-detected outcomes (depression) using Machine Learning (ML) methods. Precise mathematical algorithms can be used to evaluate a patient’s risk for prenatal depression and highlight which vulnerabilities, separately and together, contribute to their risk. While attempts to predict perinatal depression using ML have recently been published, they are limited either due to their reliance on electronic health records (EHR) data which often do not systematically document important contextual risk factors (Hochman et al., 2021; Y. Zhang et al., 2020), or due to their reliance on lengthy questionnaires which are not feasible to use in clinical settings (Andersson et al., 2021; W. Zhang et al., 2020). In this respect, the PROMOTE serves two purposes: 1) it improves screening of psychosocial vulnerability; and 2) it provides data for ML prediction. The ML algorithms will expand the use of the PROMOTE from a comprehensive psychosocial screening instrument to a sophisticated clinical decision support tool that will enable providers to identify highly vulnerable childbearing women accurately and simply in real-time clinical practice. The aim of the current study is to demonstrate how the PROMOTE can classify probability for prenatal depression and prioritize the contributing psychosocial vulnerabilities (i.e., risk factors).

Methods

Setting and study population

We conducted a retrospective medical chart review of all patients who completed the PROMOTE during their first visit to an outpatient prenatal clinic associated with a large university hospital between June 2019 and November 2020. Routinely, all patients were asked to arrive 15 minutes in advance of their prenatal care appointment. At check-in, women received a clipboard with identical English or Spanish forms, which include the PROMOTE and the Edinburgh Postnatal Depression Scale (EPDS)(Cox et al., 1987), to be completed before their appointment. The PROMOTE and EPDS were then reviewed by the provider to identify psychosocial vulnerabilities. Both forms were subsequently scanned into EHR. The Institutional Review Board at Stony Brook University granted ethical approval for abstraction of EHR and entry into REDCap data collection software and approved a waiver of informed consent for this minimal risk study (IRB2019-00512).

Measures

The PROMOTE includes 19 items assessing the following contextual risk constructs: unplanned pregnancy; current employment; educational level; financial state; residential stability; current relationship; family support; partner support; stress; healthy behaviors; past year alcohol use, tobacco use, prescription misuse, illicit drug use, and major life events; lifetime interpersonal abuse; current emotional problems and psychiatric medication; and past year opioid use (Table 1). A detailed description of the items, their wording, and the response options can be found elsewhere (Preis et al., 2021). All of the PROMOTE items were included as predictors in the ML model.

Maternal age, gestational week, race/ethnicity, and whether forms (e.g., PROMOTE, EPDS) were completed in English or Spanish were also abstracted from EHR and included as predictors in the ML model.

The Edinburgh Postnatal Depression Scale (EPDS) is a well-validated instrument that can be used to assess depression prenatally or postnatally (Cox et al., 1987). It includes 10 statements about feelings and thoughts experienced in the past two weeks that are answered on a 4-point frequency scale from 0 to 3. Scores are calculated by summing responses (range 0 to 30) with higher scores indicating more probable depression. Several cut off scores are offered for the EPDS to indicate likelihood of clinically relevant depression (Levis et al., 2020). In the current study, we chose a cut off score of 13 or higher, or any positive response to the self-harm item, which is highly indicative of clinically relevant depression. We dichotomized EPDS scores so that 0 = low risk for depression (sum score 0-12) and 1 = high risk for depression (sum score 13-30 and/ or 1-3 on item 10 indicating self-harm). The internal consistency of the EPDS in this study was high (α = 0.84).

Statistical analysis

In ML, data-driven algorithms are developed to classify cases (patients) as belonging to an outcome group (depression). We used ML algorithms that are trained in a supervised manner to classify depression risk on a subset of data and then tested on the remaining data. Data used to produce the algorithm are referred to as ‘training data’ and data used to test the performance of the algorithm are referred to as ‘test cases’. In the current study, we used balanced imputed data (see below), including 114 patients classified as high risk and 114 classified as low risk. Training data included all the cases except for one from each group (113:1 train to test ratio), which were used as test cases. This procedure was repeated until each patient was used as a ‘test case’. The ML classification of the excluded patient/test case was used to evaluate the performance of the algorithms that were developed using the training data.

Depression affects a minority of individuals, and therefore the EPDS data were imbalanced with a greater number of patients scoring in the ‘low risk’ than ‘high risk’ category. Imbalanced data can lead to inaccurate performance of ML algorithms, which is why mathematical balancing of data is commonly done before training the ML algorithms. Therefore, we created balanced samples comprised of equal numbers of patients categorized as low risk or high risk. To match the smaller number of high risk cases, the appropriate number of low risk cases were randomly selected to be included in these training data samples. As expected, balancing the data improved prediction accuracy of the model (see Supplemental Material 1).

Missing values of individual items in the data ranged from 0.0% (language) to 9.8% (opioid use). To replicate real-world clinical use, in which missing data from individual patients are common, we imputed missing item values when necessary. Specifically, we adopted iterative imputation which models each predictor (‘item’) as a function of the other predictors. This method first fits a regression model by treating an item as output and other items as inputs, then the regression model is used to estimate the missing values of this item (Azur et al., 2011). Performance of the algorithms was similar with missing data or imputed data (see Supplemental Material 1).

We examined the ability of ML algorithms to accurately classify women based on their PROMOTE responses as being at high risk for depression or at low risk for depression based on their EPDS score. Results of the classification of 50 training data/test case samples were aggregated (using the median of the results) to produce performance matrices. Rates of True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN) classification were determined and from these, Sensitivity, Specificity, Positive Predictive Value (PPV), Negative Predictive Value (NPV), F1, and Accuracy were calculated using the following equations, with higher values indicating better performance:

$$\begin{array}{c}\text{Sensitivity}=\frac{TP}{TP+FN}\text{PPV}=\frac{TP}{TP+FP}\hfill \\ \text{Specificity}=\frac{TN}{TN+FP}\text{NPV}=\frac{TN}{TN+FN}\hfill \end{array}\text{F1}=2\ast \frac{\text{Sensitivity}\ast \text{PPV}}{\text{Sensitivity}+\text{PPV}}\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$$

The performance of different ML methods including random forests, naïve Bayes, decision trees, AdaBoost, SVM, kNN, and logistic regression was compared using these metrics. Based on these comparisons, the random forest method was chosen to be used in the remaining analyses (Supplemental Material 2).

We then calculated the probability of being classified as ‘high risk for depression’ for each patient. For a given random forest, the probability of depression was computed as the proportion of decision trees that indicate depression. We adopted a multiple random forests model, which consists of 50 different random forests tests that produced the probability of depression by calculating the median of the probabilities of depression provided by the 50 random forests.

Finally, we assessed the ‘importance’ of each PROMOTE construct/item in the classification of depression risk. Constructs were ranked according to their importance using recursive feature elimination (RFE), which assesses the decrease in accuracy when a single item is randomly removed (Bishop & Nasrabadi, 2006). This decrease indicates how much the prediction depends on that item. Then, we assessed the impact of that item on probability of depression risk. The method for obtaining importance values is based on game theory; description can be found elsewhere (Molnar, 2020). All of the ML methods were implemented using the Scikit-learn 1.1.1 Python package (Pedregosa et al., 2011).

Results

Sample and data characteristics

Out of 1844 patients who completed the PROMOTE, 1715 had a complete EPDS in their medical chart from the first prenatal visit and were included in the current analysis. Of these 1715 patients, n = 114 (6.6%) were considered at high risk for depression (EPDS = 13-30) and n = 1601 (93.4%) were considered at low risk for depression (EPDS < 13). Participants were on average 30.6 (SD = 5.8) years old. Additional characteristics and distribution of PROMOTE constructs can be found in Table 2.

Table 2.

Patients’ contextual factors based on PROMOTE responses (N = 1,715)

	n (valid %)		n (valid %)
Employment		Perceived stress
Full time	871 (51.7)	Low (1-3)	1327 (79.3)
Part-time	235 (13.9)	High (4-5)	346 (20.7)
Homemaker	259 (15.4)	Alcohol (>3 drinks/day)
Unemployed	221 (13.1)	No	1164 (69.2)
Student	54 (3.2)	Yes	519 (30.8)
Other	46 (2.7)	Tobacco
Race and ethnicity		No	1429 (84.4)
White-non-Hispanic	901 (52.9)	Yes	264 (15.6)
Black-non-Hispanic	147 (8.6)	Prescription drugs
Asian-non-Hispanic	132 (7.8)	No	1658 (97.9)
Other-non-Hispanic	57 (3.3)	Yes	36 (2.1)
Hispanic (any race)	465 (27.3)	Illegal drugs
Education completed		No	1579 (93.4)
High school or less	651 (40.5)	Yes	112 (6.6)
Some college or more	956 (59.5)	Any kind of opioids
Financial state		No	1493 (96.5)
Below average	233 (14.7)	Yes	54 (3.5)
Average or above	1352 (85.3)	Major life events
Residence*		No	1422 (86.6)
Stable living conditions	1304 (77.5)	Yes	220 (13.4)
Unstable living conditions	378 (22.5)	Emotional difficulty
Current relationship		No	1490 (90.1)
Married or cohabiting	1532 (95.2)	Yes	164 (9.9)
No or Some relationship	78 (4.8)	Psychiatric medication
Planned pregnancy		No	1556 (94.0)
No	950 (60.3)	Yes	99 (6.0)
Yes	626 (39.7)	Positive health behaviors
Abuse		Little (1-3)	422 (25.4)
No	1601 (97.3)	Much (4-5)	1242 (74.6)
Yes	50 (3.0)

Note: Percentages do not include missing values. Stable living conditions = Homeowner or renting; Unstable living conditions = With family or friends, Group residence, or Homeless/shelter.

Performance and probability assessment

As can be seen in Table 3, the random forests method performed well with acceptable performance metrics (Accuracy = 0.80; Sensitivity = 0.75; Specificity = 0.81; PPV = 0.79; NPV = 0.97).

Table 3.

Performance confusion matrix using random forests (N = 1,715)

		True classbased on EPDS Scores
		High risk for depression	Low risk for depression
Random Forests Classification	High risk for depression	TP=75	FP=310	Positive Predictive Value (PPV) 78.7%†
	Low risk for depression	FN=39	TN=1291	Negative Predictive Value (NPV) 97.1%
		Sensitivity 74.6%†	Specificity 80.6%	Classification Accuracy 79.7%

Legend: EPDS- Edinburgh Postnatal Depression Scale; TP- True Positive; TN- True Negative; FP-False Positive; and FN-False Negative.

^†Since these metrics are highly affected by data imbalance, the current value reflects the Balanced-Iterative Imputation and not the TP, FN, and FP numbers in the current table (See Supplemental Material 1).

We used the classification results of 50 random forests to calculate the probability of depression risk for individual patients based on their EPDS score. Figures 1a-1f illustrate the probability of being classified at high risk for depression for individual patients based on applying 50 random forests tests where patients in red are those who were classified at high risk for depression and patients in blue are those who were classified at low risk for depression. Each bar in the figures represents the number of random forests tests that produced a specific probability for that patient to be at high risk for depression. For example, Figure 1a is a patient that had an EPDS score indicating high risk for depression for which most of the random forests tests resulted in high probability of being classified as high risk for depression; the median random forests probability was 92%. As another example, Figure 1f is a patient that had an EPDS score indicating low risk for depression. For this patient, almost all random forests tests indicated very low probability of being classified at high risk for depression; the median random forests probability was close to 0%.

Figures 1a-1f. Classification of individual patients for high risk for depression based on 50 random forests.

EPDS- Edinburgh Postnatal Depression Scale; TP- True Positive; TN- True Negative; FP-False Positive; and FN-False Negative. Red represents patients that scored at high risk for depression on the EPDS and blue represents patients that scored at low risk for depression on the EPDS.

Figure 2 displays the algorithmically determined probability of individual patients to be classified at high risk for depression (based on the median of each patient’s probability). As can be seen, the number of patients at high risk for depression (in red) that are classified by the algorithm as at high probability for depression risk is negatively skewed, indicating that the random forests algorithm very frequently assigned high probability of classifying these patients at risk; 73.7% of the high risk patients had an algorithm-projected probability of depression risk of 50% or higher. Similarly, the number of low risk patients (in blue) is positively skewed, and most frequently they were classified by the algorithm as having low probability of depression; only 21.1% of the low risk patients had an algorithmically projected probability of depression of 50% or higher.

Figure 2. Probability of depression in the balanced datasets.

Item/construct importance

Using RFE methods, the ten most important constructs from the PROMOTE to predict depression risk, listed in declining order of importance were: perceived stress, emotional problems, family support, age, major life events, partner support, unplanned pregnancy, current employment, lifetime abuse, and financial state. The impact of these items on the classification of depression is presented in Figure 3. The colors represent the value of an item where blue reflects low values and red reflects high values. The horizontal-axis shows the impact of the item on the classification of depression. For example, the cluster of blue points on the top line represent patients with low perceived stress. Their distribution suggests that low stress has a small inverse impact on the probability of depression. That is, low perceived stress is modestly associated with reduced probability of depression. In contrast, the distribution of the red points on this line, which represent patients with high perceived stress, indicates that high stress has a relatively large impact on increasing the probability of depression (i.e., high stress is strongly related with the probability of being classified as high risk for depression). As an additional example, a high level of partner support does not have much impact on probability of depression whereas a low level of partner support is strongly associated with greater probability of being classified as high risk for depression.

Figure 3. Impact of constructs on depression.

Discussion

The PROMOTE screening instrument was successful in predicting the likelihood of depression. The contextual risk factors that are included in the PROMOTE can be used both to identify specific vulnerabilities (e.g., high stress, low social support, financial strains, unplanned pregnancy) and to predict further perinatal outcomes, in this case, depression. Study findings indicate that data gathered through the PROMOTE can be used by a ML algorithm to classify patients’ probability of being at high risk for prenatal depression (using EPDS score as the criterion). By adapting these ML techniques to clinical care, we can help providers identify at-risk patients and detect what contextual factors are contributing to their risks, thus improving intervention and care delivery

Many of the contextual psychosocial and behavioral vulnerabilities which are important for the well-being of pregnant patients are not regularly assessed in prenatal care. The PROMOTE, which helps overcome screening barriers (Preis et al., 2022), facilitates detection of vulnerable patients and can be used to predict future or underreported outcomes. Because depression is an under-detected outcome (Bauman et al., 2020; Miller et al., 2019), the ML algorithm can assist in identifying patients that would not have been flagged by the EPDS (i.e., predicted to be a ‘false positive’ by the ML algorithm), and provide an additional means to identify vulnerable patients. Further research using clinical interviews can help understand the clinical vulnerability of these ‘false positive’ cases flagged by the PROMOTE. Moreover, the PROMOTE can help providers understand what in the patient’s life is making them more vulnerable. For example, high stress, unplanned pregnancy, and less family support were independent risk factors for depression. These findings corroborate previous research on risk factors for mental health difficulties in the perinatal period (Abajobir et al., 2016; Field, 2017; O'Hara & Wisner, 2014). A patient burdened by such vulnerabilities might not screen positive on the EPDS; however, they would still benefit from further inquiry. The PROMOTE and ML algorithm would flag them as at high probability of depression since they are likely distressed, even if they did not score high on the EPDS. This is an improvement over usual screening for depression since it does not merely identify depression risk, but also reveals actionable factors that contribute to that risk. Once depression risk is identified, relevant and effective treatment can be offered (Cuijpers et al., 2021).

The prediction accuracy of the PROMOTE-based ML algorithm was as good or even better on some metrics than previous published work predicting prenatal or postpartum depression. This work has important implications for future clinical practice and research, especially when considering the usability of the PROMOTE with its relatively few items compared to previous work relying on lengthy questionnaires, which are valuable but less practical for clinical use (Andersson et al., 2021; W. Zhang et al., 2020). Other work predicting perinatal depression has utilized EHR data (Hochman et al., 2021; Y. Zhang et al., 2020) such as anxiety symptoms, which while generating high probability because of the strong relationship of anxiety to depression, is not informative in identifying vulnerability or risk factors. The analytic strategies that were developed in this study can serve as a blueprint for future investigations of other prediction models. Combining discrete contextual risk factors collected by the PROMOTE with EHR data can produce more robust and accurate prediction models. This can help providers assess the probability of developing perinatal adverse outcomes and increase the delivery of precision medicine. Once the algorithms are trained on previously collected data, they can be applied in real-time to assess a patient’s risk during the prenatal care appointment and promote personalized prenatal care. For example, Figure 4 illustrates the potential individualized output from our methods to highlight the risk for multiple adverse outcomes. Our methods and results also have important research relevance in understanding resilience, specifically, why some women that have several risk factors and high probability of depression remain healthy.

Figure 4. Potential output of machine learning algorithms in clinical use.

While this study includes a large sample of patients and sophisticated methodology, it does have several limitations. The study sample is representative of the region in which the study was conducted, thus, the generalizability to other groups of childbearing patients is limited. We used the EPDS as our benchmark for depression and while it is a widely used instrument to screen for prenatal depression, it is not a clinical interview that can definitively diagnose depression. Relatedly, the study relied on self-reported data which are susceptible to social desirability and other threats to validity. Future studies could extend our findings by testing the ML models in more generalizable community populations of pregnant patients, to predict other physical and mental health outcomes in pregnancy and the postpartum period (including comparing to clinical interviews), and to predict depression in non-pregnant populations.

In conclusion, the PROMOTE has great value, both as a screening instrument to detect vulnerability in prenatal care, and as a tool to be used to predict risks for underreported or future outcomes. By applying ML algorithms to the data collected by the PROMOTE and EHR data, patients’ risk for adverse perinatal outcomes can be better predicted, propelling individually tailored patient-centered care to improve maternal and infant health.