Efficiently identifying individuals at high risk for treatment resistance in major depressive disorder using electronic health records

Abstract

Background:

With the emergence of evidence-based treatments for treatment-resistant depression, strategies to identify individuals at greater risk for treatment resistance early in the course of illness could have clinical utility. We sought to develop and validate a model to predict treatment resistance in major depressive disorder using coded clinical data from the electronic health record.

Methods:

We identified individuals from a large health system with a diagnosis of major depressive disorder receiving an index antidepressant prescription, and used a tree-based machine learning classifier to build a risk stratification model to identify those likely to experience treatment resistance. The resulting model was validated in a second health system.

Results:

In the second health system, the extra trees model yielded an AUC of 0.652 (95% CI: 0.623–0.682); with sensitivity constrained at 0.80, specificity was 0.358 (95% CI: 0.300–0.413). Lift in the top quintile was 1.99 (95% CI: 1.76–2.22). Including additional data for the 4 weeks following treatment initiation did not meaningfully improve model performance.

Limitations:

The extent to which these models generalize across additional health systems will require further investigation.

Conclusion:

Electronic health records facilitated stratification of risk for treatment-resistant depression and demonstrated generalizability to a second health system. Efforts to improve upon such models using additional measures, and to understand their performance in real-world clinical settings, are warranted.

Introduction

The range of potential therapeutic options for treatment-resistant major depressive disorder (TRD) has broadened substantially beyond electroconvulsive therapy (ECT), for decades the gold standard in this setting (Pagnin et al., 2004; UK ECT Review Group, 2003). Recent developments include repetitive transcranial magnetic stimulation (rTMS), intravenous or intranasal ketamine, as well as atypical antipsychotic augmentation of standard antidepressants (Carter et al., 2020; Mantovani et al., 2012; Marcantoni et al., 2020). For the ~1/3 of individuals who do not benefit adequately from standard antidepressant treatments (Rush et al., 2006), these newer options offer potential opportunities to avoid the disability and chronicity typically associated with treatment resistance; however, each of these emerging strategies carry burdens which exceed those of standard first line antidepressants. As such, these more burdensome treatments are typically reserved for patients who have failed two – and in real-world settings often many more – standard treatments. Given recommendations about adequacy of acute treatment duration in depression (Kennedy et al., 2016), this would suggest a minimum of 6 months to establish treatment resistance, a period during which the intrinsic burden of depression symptoms and the varied morbidities of the syndrome persist.

If high-risk individuals could be identified earlier in their course, they might be prioritized for specialized evaluation and more intensive treatment (Simon and Perlis, 2010). At minimum, this could afford an opportunity to improve clinical outcomes. In addition, tools to stratify risk might enable more efficient investigation of novel therapeutics, by enabling enriched trials of interventions for treatment resistance in individuals most likely to benefit from them. To facilitate this identification, we utilized longitudinal data from the electronic health record of 2 large academic medical centers. We sought to apply standard machine learning methods to develop generalizable models to estimate risk for TRD among a large outpatient cohort, and validated these models in a large cohort from a second health system, as a starting point for more sophisticated approaches that might also incorporate biomarkers or other measures.

Methods

Study overview and cohort description

The study cohort included patients between 18–80 years old seen in the ambulatory networks of two academic medical centers in Eastern Massachusetts with an electronic prescription of one of 9 widely-used antidepressants (Table S1) and a diagnosis of major depressive disorder (MDD) or depressive disorder, not otherwise classified between March 2008 and December 2017. Diagnosis was defined as one or more instances of the International Classification of Diseases, Ninth Revision (ICD9) codes 296.2x, 296.3x, 311, or Tenth Revision (ICD10) codes f32.x and f33.x (Table S2). Patients with 2 or more schizophrenia or bipolar disorder diagnostic codes (Tables S3 and S4) were excluded for diagnostic ambiguity (Figure S1).

Health records data for the relevant cohort were extracted to generate a data mart using i2b2 server software (i2b2, Boston, MA, USA) (Murphy et al., 2007). The data mart included sociodemographic features (age, gender, race and ethnicity), insurance information, diagnostic and procedure codes, inpatient medication administrations, and outpatient medication electronic prescriptions which were mandated during the period examined. The Mass General Brigham institutional review board approved the study protocol and waived the requirement for informed consent since this investigation used de-identified data and no human subjects contact was required.

Outcome definition

The primary study outcome was TRD, defined as 2 or more distinct antidepressant prescriptions in the first year after the index antidepressant (i.e., a total of at least 3 distinct antidepressants). We excluded individuals with any recorded prescriptions of antidepressants before March 2008, 6 months after electronic prescribing became mandatory in these hospital systems, to ensure we were observing index prescriptions. We also required that patients have at least 1 code at some points at least 12 weeks after their index antidepressant prescription. In a secondary analysis, we included features from the 4 weeks following the index prescription in addition to the features recorded in the 26 weeks before the index prescription, as a means of understanding whether available clinical data early in treatment course would improve outcome prediction.

Health records encoding and feature definitions

We encoded features for each patient from the 26-week window preceding the index antidepressant prescription; this threshold was selected a priori to balance the need for some prior features to facilitate prediction, with the desire to make predictions for as many individuals as possible. Features included sociodemographic variables, namely age, gender, and race; diagnostic codes, including both ICD-9 and ICD-10 codes; CPT codes capturing laboratory tests and procedures; and medications defined by Unified Medical Language System RxNorm codes(Bennett, 2012; “RxNorm,” n.d.). All coded data were represented as count variables representing the number of times a feature was recorded in the specified time interval.

In a secondary analysis considering features from the 4 weeks following the index antidepressant prescription, we include the same features described above in both the 26-week window before index prescription, and in the 4-week window following it separately. That is, we included a feature for each code in each ontology (e.g. RxNorm, ICD) in the period before the index prescription, and in the 4 weeks following index antidepressant prescription.

A priori, we included all codes that occurred in 100 different patients in the feature time window in site A. For the main analysis, we selected 961 of the original 36,318 features. For the secondary analysis including the 4 weeks after index antidepressant prescription, we selected 1,357 from 72,593 features, retaining separate features for codes recorded in the 26 weeks before the index prescription, and the 4 weeks after.

Prediction task

For model training, we randomly assigned the 22,006 eligible participants from site A to a training set (70%), validation set (10%) and testing set (20%) with labels stratified across the different sets of data (that is, each set has approximately the same number of participants with the relevant outcome). We repeated this 5 times, randomly splitting the patients into these sets each time, and trained a separate model for each of these training sets. (Repeating these analyses sampling 10 times did not yield meaningfully different results.) Our reported results are averaged across the 5 models and accompanying test sets, unless otherwise specified. All site B participants (12,869) were held out to test our generalization performance. (For the secondary analysis including data from 4 weeks after index prescription, site A had 21,774 patients, and site B had 12,755 after excluding participants experiencing outcome in the 4-week window following index antidepressant prescription).

Classification methods and metrics

We trained a commonly-used classifier, extra trees (ET), and for comparison, a logistic regression (LR) model. For both, we used the open-source Scikit Learn toolkit implementation (Pedregosa et al., 2011). We tuned hyperparameters to achieve the best AUC on the site A validation set using grid search over the options specified below. For each of the 5 random train/valid/test splits of the dataset, the hyperparameter search was repeated to choose the best hyperparameter setting for that train/validation set.

For LR, we used l2 regularization, and we searched over 1 hyperparameter: the regularization strength, denoted “C”, (C={20 points between 1e-5 and 1 spaced linearly on a log-base-10 scale}). For ET, we searched over three hyperparameters: the fraction of features used in each split (max_features={0.04, 0.16, 0.64}), the minimum number of samples at leaf nodes (min_samples_leaf={16, 64, 256}), and the number of estimators (n_estimators={16, 64, 256}). (The regularization chosen for each trained model in each analysis is listed in Table S5.) For both models, features were z-scored so as to have similar magnitudes for all dimensions, and we set the class labels to have equal weight in the optimization.

We reported model performance using AUC, and specificity, PPV and NPV with the threshold chosen to set sensitivity as close to 0.80 as possible (i.e., to define a model with 80% sensitivity for treatment-resistance). We computed these metrics on the independent dataset from site B, as well as the held-out test set from site A. We averaged results over 5 random train/valid/test splits of the site A data (for which a model was trained for each), and we then took 500 bootstrap samples of the points in each test set to compute the mean and the 95% confidence intervals (i.e., scores at 2.5 and 97.5 percentiles) for each metric. We evaluated the 5 models trained on site A on site B using the same bootstrapping procedure. A separate threshold was chosen for each of the bootstrap samples when computing specificity, PPV and NPV.

Results

Characteristics of the cohort in the two sites are summarized in Table 1. A total of 1,079 individuals out of 22,006 individuals (4.90%) in the site A cohort were prescribed more than 2 antidepressants within the first year of the index prescription, and 520/12,869 individuals (4.04%) among those from Site B. Sociodemographic features including age, sex, and race/ethnicity were similar between those with and without TRD at both sites. Rates of TRD were greatest among individuals receiving mirtazapine or duloxetine (Table S1) as index antidepressant at site A (14% and 11% respectively), while in site B, the TRD outcome was more evenly distributed across treatments, ranging from 4% to 8%.

Table 1:

Cohort sociodemographic data at site A and site B.

	Count/Mean full sample	%/Std dev. full sample	Count/Mean TRD sample	%/Std dev. TRD sample
Site A	22006	-	-	-
TRD Prevalence	1079	4.90	-	-
Gender: F	14531	66.0	718	66.5
Gender: M	7475	34.0	361	33.5
Race: Asian	575	2.6	26	2.4
Race: Black	995	4.5	46	4.3
Race: Hispanic	1226	5.6	52	4.8
Race: Other	1303	5.9	68	6.3
Race: White	17907	81.4	887	82.2
Age	47.3	14.9	46.3	14.5
Site B	12869	-	-	-
TRD Prevalence	520	4.04	-	-
Gender: F	9245	71.8	375	72.1
Gender: M	3624	28.2	145	27.9
Race: Asian	207	1.6	11	2.1
Race: Black	1073	8.3	38	7.3
Race: Hispanic	1533	11.9	75	14.4
Race: Other	721	5.6	34	6.5
Race: White	9335	72.5	362	69.6
Age	48.1	14.5	46.4	14.2

Table 2 reports classifier performance metrics for the primary analysis, predicting TRD based on features drawn from the 26 preceding weeks (6 months) of clinical data. Metrics for extra trees are reported both on an independent testing sample from Site A, and the full Site B cohort, as well as for both classes of models (Table 2; The most predictive features in both ET and LR models are presented in Table S6). For the primary analysis, the ET model achieved an AUC of 0.652 (0.623 – 0.682) in site B; constraining sensitivity to be as close as possible to 0.80, specificity was 0.358 (0.300–0.413), NPV 0.977 (0.972–0.981), and PPV 0.050 (0.044–0.056). By comparison, performance was somewhat poorer using logistic regression (Table 2), with an AUC of 0.614 (0.591 – 0.637). (For individual features and model weights or coefficients, please see Supplemental Table 6.) Incorporating an additional 4 weeks after initial prescription did not meaningfully improve prediction (Table 2).

Table 2:

Classifier performance metrics for both models in each site in the main analysis and the secondary analysis. PPV and NPV are reported for the threshold setting sensitivity as close as possible to 0.80.

Analysis	Model	Site	AUC	AUC 95% CI	Speci ficity	Specificity 95% CI	PPV	PPV 95% CI	NPV	NPV 95% CI
Main Analysis	Logistic Regression	A	0.646	0.602 – 0.693	0.349	0.254 – 0.435	0.060	0.050 – 0.071	0.971	0.961 – 0.978
		B	0.614	0.586 – 0.642	0.315	0.241 – 0.377	0.047	0.041 – 0.053	0.974	0.966 – 0.979
	Extra Trees	A	0.666	0.609 – 0.723	0.382	0.253 – 0.529	0.063	0.050 – 0.081	0.973	0.960 – 0.982
		B	0.652	0.623 – 0.682	0.358	0.300 – 0.413	0.050	0.044 – 0.056	0.977	0.972 – 0.981
Secondary Analysis: + 4 weeks after	Logistic Regression	A	0.650	0.595 – 0.701	0.374	0.262 – 0.486	0.050	0.039 – 0.063	0.978	0.969 – 0.985
		B	0.631	0.598 – 0.661	0.353	0.287 – 0.407	0.039	0.034 – 0.044	0.982	0.977 – 0.985
	Extra Trees	A	0.659	0.606 – 0.715	0.407	0.314 – 0.515	0.052	0.042 – 0.065	0.980	0.974 – 0.985
		B	0.648	0.603 – 0.682	0.378	0.300 – 0.461	0.041	0.035 – 0.048	0.983	0.978 – 0.986

We next examined AUC for the primary ET model stratified by gender and race (Table 3) to establish performance in subsamples of the cohort. The AUC in site B for white patients (0.657; 0.621–0.695) and non-white patients (0.639; 0.593–0.684) was similar, as were AUCs for female patients (0.654; 0.621–0.686) and male patients (0.647; 0.590–0.704). Other characteristics of discrimination were likewise similar between subgroups.

Table 3:

Classifier performance metrics stratified by race and gender for the extra trees and the logistic regression model in the main analysis.

Model	Site	Subgroup	AUC	AUC 95% CI	Specificity	Specificity 95% CI	PPV	PPV 95% CI	NPV	NPV 95% CI
Logistic Regression	A	White	0.646	0.600 – 0.691	0.347	0.217 – 0.459	0.061	0.047 – 0.076	0.970	0.956 – 0.979
		NonWhite	0.655	0.536 – 0.773	0.361	0.115 – 0.568	0.058	0.026 – 0.099	0.972	0.944 – 0.987
		Male	0.644	0.571 – 0.718	0.350	0.180 – 0.488	0.061	0.036 – 0.085	0.970	0.952 – 0.982
		Female	0.650	0.590 – 0.709	0.346	0.197 – 0.490	0.060	0.045 – 0.078	0.970	0.952 – 0.981
	B	White	0.614	0.582 – 0.645	0.313	0.229 – 0.381	0.045	0.038 – 0.052	0.974	0.966 – 0.980
		NonWhite	0 615	0.568 – 0.661	0.317	0.171 – 0.437	0.052	0.041 – 0.065	0.970	0.948 – 0.980
		Male	0.597	0.543 – 0.646	0.304	0.199 – 0.421	0.046	0.036 – 0.058	0.972	0.957 – 0.982
		Female	0.622	0.587 – 0.654	0.316	0.231 – 0.389	0.047	0.041 – 0.054	0.974	0.964 – 0.979
Extra Trees	A	White	0.661	0.607 – 0.720	0.385	0.247 – 0.534	0.065	0.049 – 0.087	0.972	0.960 – 0.981
		NonWhite	0 675	0.527 – 0.802	0.311	0.101 – 0.600	0.055	0.025 – 0.101	0.964	0.916 – 0.986
		Male	0.662	0.580 – 0.766	0.375	0.169 – 0.585	0.064	0.038 – 0.093	0.971	0.940 – 0.985
		Female	0.671	0.597 – 0.730	0.381	0.233 – 0.511	0.063	0.046 – 0.085	0.973	0.957 – 0.982
	B	White	0.657	0.621 – 0.695	0.355	0.286 – 0.430	0.048	0.041 – 0.055	0.978	0.972 – 0.982
		NonWhite	0.639	0.593 – 0.684	0.342	0.220 – 0.433	0.054	0.044 – 0.066	0.973	0.961 – 0.981
		Male	0.647	0.590 – 0.704	0.350	0 230 – 0.453	0.049	0.039 – 0.061	0.976	0.965 – 0.983
		Female	0.654	0.621 – 0.686	0.359	0.287 – 0.425	0.050	0.043 – 0.058	0.977	0.971 – 0.981

Finally, we examined calibration in the ET model. Figure 1 illustrates the rate of TRD for each risk quintile for the main analysis, computed on the basis of predicted probabilities. The horizontal dashed line represents the mean risk of developing the outcome. The lift in the top quintile was 1.99 (1.76–2.22) for site B, and 2.12 (1.73–2.50) for the site A testing set with the ET model.

Figure 1:

Outcome rate for each quintile by predicted risk for the main analysis in each site with the extra tree and logistic regression models.

The dotted black line shows the mean outcome across all quintiles. The error bars show the standard deviations. From this, lift can be ascertained.

Discussion

In this investigation of 34,877 individuals with a diagnosis of MDD initiated on standard antidepressant treatment, we developed models able to stratify risk for treatment-resistance. Performance of these models is modest compared to prediction tools in other areas of medicine (Castro et al., 2020), but similar in magnitude to other treatment outcome prediction models relating to antidepressant treatment. Among the specific features associated with greater risk in both models are initial use of non-SSRI antidepressants; rather than reflecting differential efficacy per se, this result likely suggests other patient characteristics such as comorbidity that might lead the clinician to select alternate treatment. The inclusion of clinical features from early in treatment, which we hypothesized might improve predictions by giving indications of propensity to tolerate medications and/or placebo-like response, did not meaningfully improve these predictions.

Since one of us published the first machine learning model of treatment-resistant depression using the STAR*D data set (Perlis, 2013), multiple additional studies have used those data, alone or in concert with additional clinical trials, to try to improve performance (Iniesta et al., 2016; Nie et al., 2018). Such clinical trial-based studies generally face three major challenges: they use research measures that would be challenging to deploy at scale in clinical settings; they are prone to overfit when they incorporate large numbers of features relative to the number of observations; and they are unlikely to reflect real-world clinical populations. Subsequent efforts using clinical cohort studies also reported promising results, but still rely on systematic/structured assessment and have unclear generalizability (Kautzky et al., 2017). In general, despite a proliferation of machine-learning efforts, larger-scale and more systematic investigations tend to yield less optimistic estimates of discrimination.

The present results are not directly comparable to prior work in individual clinical trials, as this study relies solely on artifacts of routine clinical care – i.e., secondary use of data available in the electronic health record at time of prescription. In particular, rates of TRD are far lower than the 1 in 3 commonly cited based upon the STAR*D effectiveness study(Rush et al., 2006). Perhaps unsurprisingly, the resulting discrimination is also lower in absolute terms than that previously reported using STAR*D data(Perlis, 2013). On the other hand, by examining model characteristics in a cohort drawn from a second health system, these estimates are likely to represent a pragmatic estimate of how claims code-based models will perform in real-world settings. Beyond the advantage of generalizability, we find that the performance metrics we measure are not markedly different across sociodemographic subgroups, suggesting less evidence of bias than some other machine learning models, although this remains an important consideration for future work. (One notable exception is the finding of greater rates of treatment resistance among individuals in one health system beginning non-SSRI antidepressants; this may reflect differences in care systems rather than biology per se, with some clinical settings adopting particular antidepressants in higher-risk or more comorbid patient populations.)

In interpreting our results, it is important to consider the limitations inherent in electronic health records. First, we rely on recorded prescriptions, not medication fills, so cannot estimate adherence, which may contribute substantially to apparent treatment resistance. In prior work examining antidepressant blood levels in discarded samples, we found detectable antidepressant levels in more than 80% of participants with electronic prescriptions (Roberson et al., 2016). Second, we cannot exclude antidepressant prescriptions or other treatment received outside of these health systems; as such, it is possible that some individuals have greater treatment resistance than we detect, or that some of those identified as responsive were misclassified. Similarly, although we exclude individuals with antidepressant treatment in the 26 weeks prior to index prescription, in order to maximize the likelihood of studying treatment-naïve individuals, some patients undoubtedly received past treatment. In aggregate, all of these factors should tend to diminish model performance and may impact portability, but are unlikely to introduce systematic bias.

We note that, as with any study using naturalistic data, numerous a priori decisions are required in defining predictors and outcomes. For example, we exclude ICD9 code 300.4 (dysthymia) on the basis of prior observations that the diagnosis may be less reliable(Castro-Rodríguez et al., 2015) and that these individuals are less likely to be treated with antidepressants, while retaining 311 as it is widely used in primary care settings and included in many code-based studies of depression(Simon et al., 2015)(Simon and Savarino, 2008)(Bobo et al., 2020)(Adekkanattu et al., 2018)(Pilon et al., 2019)(Nguyen et al., 2008). While consistent with prior definitions, the use of diagnostic codes beyond those strictly reflecting major depressive disorder could impact validity of these results. We also limited predictors to facts observable 26 weeks prior to initial prescription, in an effort to balance inclusivity (i.e., being able to make predictions for as many individuals as possible, favoring shorter lead-in) with the need for at least some facts in order to generate predictions in the first place (i.e., having sufficient lead-in to observe facts that might impact outcome, favoring longer lead-in); the optimal window depends entirely on how and where such models might be applied and is likely to differ between care settings. We examined 12 month follow-up to minimize loss to follow-up; prior work examines models for predicting discontinuation after treatment initiation(Pradier et al., 2020). In particular, we cannot distinguish when one episode ends and another begins, reflecting the more fluctuating and chronic course of illness often observed in real-world settings(Perlis et al., 2012). And, while definitions of treatment-resistance vary, we selected the need for a 3^rd antidepressant (i.e., 2 prior treatments without remission) to reflect the most common. Altering any of these to fit alternate clinical use-cases, such as a desire to make predictions with less lead-in, or to predict lesser degrees of treatment resistance, would be reasonable strategies for future work. Likewise, whether the metrics we report suggest sufficient performance to warrant clinical application depends entirely on clinical context; there is no ‘magic threshold’ at which a test is good enough(Perlis, 2011; Uher et al., 2012).

Incorporation of additional measures could improve prediction substantially: the use of research measures appears to yield more performant risk predictions(Perlis, 2013), and incorporation of features derived from clinical free text also improves prediction (McCoy et al., 2018, 2016), although free text-based models may be more challenging to transfer from one health system to another. In fact, one application of these models might be to identify higher-risk individuals who then receive more comprehensive assessment to derive more reliable predictions(Perlis et al., 2018). Particularly for biomarkers such as neuroimaging (Drysdale et al., 2017), clinical risk stratification might represent a first step to enrich for individuals more likely to benefit from intensive assessment. Integration of electronic health record data with such assessments, or with other data types such as genomic data, merits further study. Particularly for higher-cost or more intensive interventions, a stepped approach to assessment still offers likely advantages over the trial-and-error approach (Maxfield and Zineh, 2021) to treatment resistance currently applied.

Highlights.

• Risk for treatment resistance in depression can be estimated at time of first antidepressant.

• With a sensitivity of 80%, specificity for treatment resistance was 36% in a second health system.

• Adding data from early in treatment did not improve prediction.

• Even modest improvement in prediction may enable targeted treatments.

Abbreviations:

TRD

Treatment-resistant major depressive disorder

ECT

Electroconvulsive therapy

rTMS

Repetitive transcranial magnetic stimulation

MDD

Major depressive disorder

ICD9

International Classification of Diseases, Ninth Revision

ICD10

International Classification of Diseases, Tenth Revision

ET

Extra trees classifier

LR

Logistic regression

Data Availability:

Our institutional review has indicated that these data cannot be redistributed.