Testing a machine-learning algorithm to predict the persistence and severity of major depressive disorder from baseline self-reports

Abstract

Heterogeneity of major depressive disorder (MDD) illness course complicates clinical decision-making. While efforts to use symptom profiles or biomarkers to develop clinically useful prognostic subtypes have had limited success, a recent report showed that machine learning (ML) models developed from self-reports about incident episode characteristics and comorbidities among respondents with lifetime MDD in the World Health Organization World Mental Health (WMH) Surveys predicted MDD persistence, chronicity, and severity with good accuracy. We report results of model validation in an independent prospective national household sample of 1,056 respondents with lifetime MDD at baseline. The WMH ML models were applied to these baseline data to generate predicted outcome scores that were compared to observed scores assessed 10–12 years after baseline. ML model prediction accuracy was also compared to that of conventional logistic regression models. Area under the receiver operating characteristic curve (AUC) based on ML (.63 for high chronicity and .71–.76 for the other prospective outcomes) was consistently higher than for the logistic models (.62–.70) despite the latter models including more predictors. 34.6–38.1% of respondents with subsequent high persistence-chronicity and 40.8–55.8% with the severity indicators were in the top 20% of the baseline ML predicted risk distribution, while only 0.9% of respondents with subsequent hospitalizations and 1.5% with suicide attempts were in the lowest 20% of the ML predicted risk distribution. These results confirm that clinically useful MDD risk stratification models can be generated from baseline patient self-reports and that ML methods improve on conventional methods in developing such models.

Heterogeneity in major depressive disorder (MDD illness course complicates clinical decision-making. Clinicians have consistently identified absence of guidance on how to deal with this variation as a critical gap in personalizing MDD treatment.^1–4 However, efforts to address this problem by finding useful prognostic subtypes based on empirically-derived symptom profiles^{5, 6} or biomarkers^7–9 have so far yielded disappointing results. A potentially promising complementary approach would be to apply machine learning (ML) methods to baseline data on symptoms and other easily-assessed clinical features to develop first-stage prediction models of subsequent depression course and treatment response^{10, 11} that could be expanded to target and examine incremental prognostic effects of novel biomarkers among patients who could not be classified definitively with the inexpensive first-stage prediction models.

Although ML methods have been used successfully to develop risk prediction schemes in other areas of medicine,^{12, 13} applications to depression have so far relied on small samples and thin predictor sets, failing to realize the full potential of the methods.^{14, 15} A recent exception is a study carried out among 8,261 respondents with lifetime DSM-IV major depressive disorder (MDD) in the WHO World Mental Health (WMH) surveys.^{16, 17} Retrospective reports about parental history of depression, temporally primary comorbid disorders, and characteristics of incident depressive episodes were used to predict retrospectively-reported subsequent depression persistence (number of years with episodes), chronicity (number of years with episodes lasting most days), hospitalization for depression, and work disability due to depression. K-means cluster analysis of the 4 predicted risk scores found a parsimonious three-cluster solution with the high-risk cluster (32.4% of cases) accounting for 56.6–72.9% of high persistence, chronicity, hospitalization, and disability.

While useful as a proof of concept, the WMH results were based on retrospective reports. A prospective validation is reported here that uses the WMH models to predict subsequent MDD persistence, chronicity, and severity in a sample of 1,056 respondents with lifetime DSM-III-R MDD in the 1990–1992 US National Comorbidity Survey (Survey 1)¹⁸ who were re-interviewed 10–12 years later in the 2001–1003 National Comorbidity Survey Follow-Up (Survey 2).¹⁹ ML model results are compared to results based on more conventional logistic regression models to determine whether ML methods improve on conventional methods.

METHODS

Sample

Survey 1 was a community epidemiological survey of common DSM-III-R disorders among English-speaking residents of the non-institutionalized civilian US household population ages 15–54 (n=5,877 respondents; 82.4% response rate).¹⁸ Respondents were paid $25 for participation. Recruitment-consent procedures were approved by the human subjects committee of the University of Michigan. Interviews were conducted face-to-face in respondent homes after obtaining verbal informed consent. Survey 2 attempted to re-interview all baseline respondents considered here 10–12 years later using recruitment-consent procedures identical to Survey 1 other than a $50 incentive. These procedures were approved by the human subjects committees of both Harvard Medical School and the University of Michigan. Interviews were again conducted face-to-face in respondent homes after obtaining verbal informed consent. The 5,001 Survey 2 respondents (87.6% of living targeted Survey 1 respondents) were administered an expanded version of the baseline interview assessing onset-course of disorders between the two surveys. A non-response adjustment weight corrected for baseline differences between Survey 2 respondents and non-respondents conditional on Survey 1 responses. Analyses reported here use the weighted data from the 1,056 Surveys 1–2 panel respondents who met lifetime criteria for major depressive disorder in Survey 1.

The baseline assessment of DSM-III-R disorders

Survey 1 assessed DSM-III-R disorders with the World Health Organization’s Composite International Diagnostic Interview (CIDI) Version 1.1, a fully-structured lay-administered interview that assessed common mental disorders using DSM-III-R criteria.²⁰ Syndromes assessed included major depressive episode, mania-hypomania, 6 anxiety disorders (generalized anxiety disorder, panic disorder, agoraphobia, specific phobia, social phobia, PTSD), and 5 externalizing disorders (conduct disorder, alcohol abuse, alcohol dependence, drug abuse, drug dependence)). Blinded SCID²¹ clinical reappraisal interviews in a probability sub-sample found good concordance with DSM-III-R/CIDI diagnoses.²⁰ Respondents with lifetime MDD were asked whether their first lifetime episode “was brought on by some stressful experience” or happened “out of the blue.” DSM-III-R Criteria A–D MDE symptoms were then assessed for this incident episode. Family History Research Diagnostic Criteria questions²² were used to determine parental history of depression.

Outcome measures

Depression persistence, chronicity, and severity were assessed in Survey 2 with a computerized version of CIDI 3.0 using “pre-loaded” information about Survey 1 responses to guide follow-up questioning. Respondents with Survey 1 lifetime MDD were asked to review the depressive symptoms reported in Survey 1, update subsequent episodes and symptoms using a life history calender, and answer four summary questions about subsequent episodes: In how many years since baseline did the respondent have a depressive episode lasting 2+ weeks (referred to below as persistence) and an episode lasting most days throughout the year (referred to below as chronicity)? Was the respondent ever hospitalized for depression since baseline? Was the respondent currently disabled (at least 50% limitation in ability to perform paid work) because of depression? A fifth Survey 2 outcome measure was whether the respondent attempted suicide at any time since baseline.

Analysis methods

Predicting the outcomes in the WMH surveys

The predictors in the WMH Surveys included temporally primary comorbid lifetime disorders, parental depression, MDD incident episode symptoms, and other information about the incident episode (age-of-onset and if the episode was triggered or endogenous). The outcomes were MDD persistence-severity (number of years since age-of-onset with episodes lasting 2+ weeks and lasting most days throughout the year, each standardized to a 0–100% range in relation to number of years between age-of-onset and age-at-interview), whether respondents were ever hospitalized for depression after their first episode, and whether respondents were disabled at the time of interview because of their depression. The ML methods used to develop the models included ensemble regression trees²³ and 10-fold cross-validated penalized regression,²⁴ both of which were designed to avoid over-fitting. These methods are described elsewhere.^{16, 17}

Between 9 and 13 predictors available at baseline in Surveys 1–2 emerged as significant in each WMH model, including measures of individual symptoms and symptom clusters in the incident episode, whether that episode was triggered or endogenous, parental history of depression, and various measures of temporally primary comorbid anxiety and externalizing disorders (some of them depending on age-of-onset). A more detailed discussion of the final WMH models is available elsewhere.^{16, 17}

In order to evaluate whether models based on ML methods improve prediction in an independent dataset more than models based on conventional methods, we also estimated a logistic regression model for each outcome in the WMH data that included 23 predictors: the 9 DSM-III-R Criterion A symptoms of MDD, a measure of whether the episode was triggered or endogenous, parental history of depression, and 11 measures of the temporally primary comorbid anxiety and externalizing disorders that were also available in Survey 1. To the extent that the ML methods stabilize estimates, we would expect predictions based on these methods to outperform predictions based on logistic regression despite the ML models contained fewer predictors (9–13) than the logistic models (23).

Assigning WMH predicted risk scores to Survey 1 respondents

Risk scores based on the logistic models were generated in Survey 1 using the WMH coefficients and the Survey 1 predictors. This direct estimation method could not be used for the ML models, though, as Survey 1 did not assess a number of significant predictors in the ML models (symptoms of anxious-depression and mixed episodes in incident episodes, comorbid obsessive-compulsive disorder, intermittent explosive disorder, and oppositional-defiant disorder). We addressed this problem by imputing ML risk scores to Survey 1 respondents from a consolidated dataset that combined WMH respondents and Surveys 1–2 respondents. The dataset included all predictors in common across the surveys along with the 4 ML predicted risk scores. The latter 4 scores had valid values for WMH cases and missing values for Survey 1 cases. Multiple imputation was applied to this dataset to generate 10 predicted scores on each missing variable to each Survey 1 respondent using SAS 9.2 proc mi.²⁵ Modal imputed values were assigned to each Survey 1 respondent for purposes of analysis. As these scores were strongly correlated across outcomes, a single composite ML predicted risk score was then constructed for each respondent by averaging across the four scores after transforming to percentiles.

Validating the prediction models

Survey 2 outcomes were predicted from risk scores based on the ML and logistic models applied to the Survey 1 data. The Survey 2 outcomes included high (top 10%) MDD persistence and chronicity in the 10–12 years between the two surveys, hospitalization for depression and attempted suicide during those years, and disability due to depression at the time of Survey 2. Area under the receiver operating characteristic curve (AUC) was calculated for each Survey 2 outcome separately for the ML and logistic models. Sensitivity (SN; the percent of respondents with the outcome classified by the predicted risk scores as having high risk), positive predictive value (PPV; the percent of respondents predicted to have high risk who experienced the outcome), and likelihood-ratio positive (LR+; the relative proportions of respondents who experienced the outcome among those classified as having or not having high risk) were also calculated for the 20% and 33% of Survey 1 respondents with highest and lowest ML imputed composite risk scores. Standard errors of SN, PPV, and LR+ were estimated using the Taylor series method with SUDAAN²⁶ to adjust for design effects in the Surveys 1–2 panel.

RESULTS

Outcome distributions

One-third (37.9%) of the 1,056 Survey 1–2 respondents had at least one depressive episode in the 10–12 years between surveys (Table 1). Mean (se) number of years in episode was 2.0 (0.2) and the 90^th percentile was 9 years. Roughly half the respondents with episodes (16.7% [1.5] of all respondents) reported episodes lasting most days throughout one or more years, with a mean (se) of 0.8 (0.1) and a 90^th percentile of 4 such years. A strong correlation (polychoric) was found between number of years in episode and number of years with episodes lasting most days throughout the year (r_p=.61).

Table 1.

Distributions and Polychoric/Tetrachoric correlations among the outcomes in the Survey 1–2 panel (N =1,056)

	Distribution		Correlations with indications of severity
	Est	(se)	Hospitalized	Suicide attempt	Disability
I. Number of years since Survey 1 with episodes lasting 2+ weeks
Any (%)	37.9	(1.7)	.49	.34	.49
Number (mean)	2.0	(0.2)	.46	.38	.49
High persistence (90th percentile [9+ years]) (%)	9.7	(1.5)	.46	.47	.53
II. Number of years since NCS with episodes lasting most days
Any (%)	16.7	(1.5)	.23	.22	.49
Number (mean)	0.8	(0.1)	.30	.30	.53
High chronicity (90th percentile [4+ years]) (%)	8.4	(1.0)	.29	.29	.58
III. Severity
Hospitalized for MDD since Survey 1 (%)	5.8	(1.1)	--	.84	.76
Suicide attempt since Survey 1 (%)	4.5	(0.8)	.84	--	.51
Disabled due to MDD at Survey 2 (%)	3.2	(0.6)	.76	.51	--

Hospitalization for depression in the years between Surveys 1 and 2 was reported by 5.8% (1.1) of Survey 2 respondents and attempted suicide by 4.5% (0.6). Current disability due to depression was reported by 3.2% (0.6) of Survey 2 respondents. Correlations (tetrachoric) among these three severity indicators were r_t=.51–.84. Correlations (polychoric) between number of years in episode and the severity indicators were r_p=.38–.49. Correlations (polychoric) between number of years in episodes lasting most days throughout the year and the severity indicators were r_p=.30–.53.

Associations of the Survey 1 risk scores with Survey 2 outcomes

AUCs of the Survey 1 ML and logistic risk scores with Survey 2 outcomes were .71 and .68, respectively, predicting high persistence, .63 and .62, respectively, predicting high chronicity, .73 and .65, respectively, predicting hospitalization, .74 and .69, respectively, predicting disability, and .76 and .70, respectively, predicting attempted suicide. (Table 2) The AUCs of the ML scores were somewhat higher than those of the logistic regression scores for all five outcomes despite the ML scores being based on models that used only 9–13 predictors compared to 23 predictors in the logistic models and the fact that the ML predicted values were based on multiple imputation rather than direct estimation.

Table 2.

Area under the receiver operating characteristic curve (AUC) of Survey 1 risk scores based on machine learning (ML) models and logistic regression models predicting Survey 2 outcomes (N =1,056)

	AUC of risk scores based on
	ML models	Logistic models
High persistence	.71	.68
High chronicity	.63	.62
Hospitalization	.73	.65
Disability	.74	.69
Suicide attempt	.76	.70

Operating characteristics of the composite imputed risk score

The 20% of Survey 1 respondents with highest ML composite imputed predicted risk scores accounted for 38.1% of high persistence in the years between the two surveys, 34.6% of high chronicity, 40.8% of hospitalizations for depression, 55.8% of disability due to depression, and 55.8% of attempted suicides. (Table 3) Sensitivities were substantially higher (49.7–70.7%) in the 33% of Survey 1 respondents with highest predicted risk scores. Positive predictive values of the outcomes in the 20% of respondents with highest predicted risk scores were in the range 8.8–18.3% (i.e., 1.8–3.0 times the positive predictive values in the remaining 80% of the sample), while positive predictive values were 6.3–17.5% in the 33% of respondents with highest predicted risk (i.e., 1.5–2.2 times the positive predictive values in the remaining 67% of the sample).

Table 3.

Sensitivity, positive predictive value, and likelihood-ratio positive of Survey 1 risk scores based on machine learning (ML) models in the upper and lower 20% and 33% of the risk distribution predicting Survey 2 outcomes (N =1,056)

	High Persistence		High Chronicity		Hospitalization		Disability		Suicide attempt
	Est	(se)	Est	(se)	Est	(se)	Est	(se)	Est	(se)
I. Sensitivity
Highest 20%	38.1	(4.2)	34.6	(7.3)	40.8	(6.8)	55.8	(6.6)	55.8	(6.9)
Highest 33%	62.2	(4.8)	49.7	(6.7)	66.6	(3.2)	68.3	(5.8)	70.7	(3.8)
Lowest 33%	10.5	(3.7)	16.7	(5.9)	11.8	(4.2)	9.7	(2.9)	10.6	(2.7)
Lowest 20%	5.6	(1.8)	15.9	(5.8)	0.9	(0.9)	7.4	(3.0)	1.5	(0.7)
II. Positive predictive value
Highest 20%	18.3	(3.5)	14.4	(3.2)	13.1	(3.0)	8.8	(1.8)	12.5	(2.7)
Highest 33%	17.5	(2.8)	12.2	(2.3)	12.5	(2.6)	6.3	(1.5)	9.3	(1.7)
Lowest 33%	3.1	(1.3)	4.2	(1.5)	2.3	(1.1)	0.9	(0.5)	1.4	(0.7)
Lowest 20%	2.7	(1.0)	6.7	(2.4)	0.3	(0.3)	1.2	(0.7)	0.3	(0.2)
III. Likelihood-ratio positive
Highest 20%	2.1	(0.4)	1.8	(0.4)	2.2	(0.5)	2.9	(0.6)	3.0	(0.5)
Highest 33%	2.0	(0.2)	1.5	(0.2)	2.1	(0.2)	2.1	(0.3)	2.2	(0.2)
Lowest 33%	0.3	(0.1)	0.5	(0.2)	0.3	(0.1)	0.3	(0.1)	0.3	(0.2)
Lowest 20%	0.3	(0.1)	0.8	(0.3)	0.0	(0.0)	0.4	(0.2)	0.1	(0.0)

The ML predicted risk scores were also useful at the low end of the distribution, as seen most vividly in the fact that the 20% of Survey 1 respondents with lowest predicted risk accounted for only 0.9% of all hospitalizations and 1.5% of all attempted suicides in the 10–12 years between surveys. This means that low ML predicted risk scores can be used as rule-outs for these outcomes (LR+=0.0–0.1). Sensitivities for other outcomes in this 20% of respondents with lowest predicted risk were 5.6–15.9%, while those of the 33% of respondents with lowest predicted risk were 9.7–16.7%. Positive predictive values of the outcomes in the 20% of respondents with lowest predicted risk were 0.3–6.7% (i.e., 0.0–0.8 times the positive predictive values in the remaining 80% of the sample), while positive predictive values were 0.9–4.2% in the 33% of respondents with lowest predicted risk (i.e., 0.3–0.5 times the positive predictive values in the remaining 67% of the sample).

DISCUSSION

Four important limitations of the WMH models should be noted before discussing results. First, MDD was assessed with a fully-structured diagnostic interview rather than a semi-structured clinical interview. Second, the models were developed in a cross-sectional sample using retrospective reports that could have been biased. Third, because the data were retrospective, predictors were limited in two important ways: the predictors for comorbid disorder did not include those with first onsets subsequent to first onset of MDD; no predictors were included for MDD course subsequent to first onset. Both these types of predictors would normally be available to clinicians interested in evaluating differential patient risk for MDD persistence-severity. Because of these limitations, we would expect the performance of the WMH models to be lower bounds on the performance of models with a more complete set of predictors. Fourth, only a limited set of machine learning methods was used to develop the WMH models. Because of these limitations, it would be useful to replicate and expand the model development and validation process illustrated here in prospective clinical samples using consistently-administered semi-structured clinical interviews with a more complete set of predictors using additional machine learning algorithms (e.g., naïve Bayesian, random forests, support vector machines)²⁷ using an optimal combined suite of algorithms to maximize cross-validated prediction accuracy.²⁸

Within the context of these limitations, the validation exercise reported here confirmed the predictive value of the kinds of self-report variables included in the WMH ML models over a 10–12 year follow-up period in an independent sample of the US household population. We also showed that prediction accuracy (AUC) of the ML models was consistently higher across all study outcomes (.63–.76) than a more conventional logistic model (.62–.70) despite the logistic model including 23 predictors and the ML models 9–13 predictors. This finding illustrates the value of ML methods in stabilizing predictions to avoid over-fitting in a training dataset (i.e., the WMH sample) so as to improve prediction in independent samples.

A question can be raised how well the WMH ML composite risk score prediction accuracy compares to previous attempts to predict long-term depression persistence-severity. Only a handful of relevant comparison studies exist over a follow-up period of 10+ years in samples of initially depressed patients^{29, 30} or community residents.^{31, 32} These studies were all quite small (n=87–424) and none reported AUC. However, AUC can be computed post hoc from two of these studies. The first was a 50-year follow-up of 293 community respondents classified post hoc as having had baseline DSM-IV MDD, 20 of whom subsequently died by suicide.³² A composite measure of baseline depression severity predicted subsequent suicide with .69 AUC compared to .76 for the validated AUC of the most comparable Survey 2 outcome (attempted suicide). The second comparison study followed 313 outpatients with initial diagnoses of MDD 1, 4, and 10 years after baseline and predicted persistent depression over that time period from 10 baseline depressive symptoms plus 10 baseline measures of self-concept, social function, and coping. The AUC of .70 is quite similar to the .71 AUC for the most comparable Survey 2 outcome of high persistence.

In making these comparisons, it is important to remember that the AUCs in these other studies were not validated in independent samples. As noted above, AUC estimates in the Surveys 1–2 panel were about 10% lower than in the WMH sample. Shrinkage would be expected to be even greater in the earlier studies because of their much smaller samples than in the broadly-representative WMH sample of 8,261 respondents. Prediction models in the two comparison studies might consequently yield validated AUCs below .60 in independent samples. AUCs in that range are considered small based on conventional guidelines, while WMH ML AUCs would typically be considered moderate.^33–35

It is noteworthy that AUC of the ML models in Surveys 1–2 was similar to widely-used risk models in other areas of medicine.^{36, 37} For example, the .73 mean AUC of the ML models over the four Survey 2 outcomes other than high chronicity is similar to the .74 average AUC of the Framingham Risk Score of coronary heart disease, one of the most widely-used prediction scores in medicine, across 79 different validation studies,³⁸ and higher than the AUCs (typically below .70) of models to predict the course of breast cancer.³⁹ Nonetheless, these AUCs are only moderate, which means that predictions based on such models could not be used to make definite rule-ins and could be used to make definite rule-outs only for risks of hospitalization and suicide attempts in the lowest 20% of the composite risk distribution. But this level of precision could be useful in defining bands of differential risk warranting variation in clinical attention. Tiered risk assessments of this sort are becoming increasingly important in other areas of medicine.^40–42

Given that predictions based on models of the sort evaluated here would most realistically be used to help clinicians identify patients who might more profit from more intensive treatment (e.g., long-term maintenance therapy), the vast majority of whom present for treatment of recurrent rather than incident episodes, an obvious future direction should be to go beyond the WMH model focus on incident episodes to develop expanded models in the Surveys 1–2 panel focused on recurrent episodes. Such an expansion could evaluate the incremental value of including new predictors for course of MDD between onset and time of Survey 1, secondary comorbid disorders, and other variables found to be important in previous studies of the course of depression (e.g., childhood family adversities, history of traumatic stress exposure, comorbid physical disorders, social networks-support, personality). We plan to implement this kind of expansion in future work with the Surveys 1–2 panel.

Beyond our own work with these data, though, it would be useful to develop an interview schedule to assess the full set of self-report predictors found in the WMH data and in the earlier studies reviewed above to use in future depression treatment trials. Such an instrument, if administered at trial baseline, could be used as part of a principled approach to study heterogeneity of treatment effects.^{43, 44} An even more promising extension given the small size of most depression treatment trials might be to administer this same instrument to a large observational sample of patients beginning depression treatment, follow these patients to assess treatment response, and analyze these data to develop a robust model predicting heterogeneity of treatment effects. In addition to providing an a priori representation of predicted treatment response for use in subsequent controlled trials, such a model could be useful in targeting depressed patients with high risk of treatment-resistance at the beginning of treatment who might warrant the substantial investment currently being made in large pragmatic trials to determine the value of expensive baseline biomarker assessments in guiding depression treatment targeting.^{8, 9} It would also be valuable in this context to evaluate the incremental value of promising biomarkers to prediction over and above the level of prediction accuracy achieved in a model based only on baseline self-reports.⁴⁵

Risk stratification data from a large observational study of this sort could also be analyzed using an extension of the innovative statistical approaches recently developed to study comparative effectiveness in observational studies.⁴⁶ The potential value of such an approach is supported both by evidence that treatment effect size estimates in appropriately-analyzed observational studies are comparable to those in controlled trials⁴⁷ and by the existence of numerous replicated predictors of heterogeneity of depression treatment effects in existing trials.^{14, 15, 48, 49} The use of an expansion of our model in this way would address two important problems in previous research on heterogeneity of depression treatment effects: the small sample sizes of depression treatment trials;^{50, 51} and the fact that most such trials assess only a small number of potential treatment effect modifiers, thus providing no principled basis for using pooling across trials to develop the kind of fine-grained multivariate models of heterogeneity of treatment effects that will eventually be needed to guide personalized depression treatment planning.⁴⁴ The results presented in the current report, while only taking a first step in this direction, provide strong support for the potential value of this possible extension.