Abstract
Objective
To develop and test predictive models of discontinuation of behavioral health service use within 12 months in transitional age youth with recent behavioral health service use.
Data sources
Administrative claims for Medicaid beneficiaries aged 15–26 years in Connecticut.
Study design
We compared the performance of a decision tree, random forest, and gradient boosting machine learning algorithms to logistic regression in predicting service discontinuation within 12 months among beneficiaries using behavioral health services.
Data extraction
We identified 33,532 transitional age youth with ≥1 claim for a primary behavioral health diagnosis in 2016 and Medicaid enrollment of ≥11 months in 2016 and ≥11 months in 2017.
Principal findings
Classification accuracy for identifying youth who discontinued behavioral health service use was highest for gradient boosting (80%, AUC = 0.86), decision tree (79%, AUC = 0.84), and random forest (79%, AUC = 0.86), as compared with logistic regression (71%, AUC = 0.71).
Conclusions
Predictive models based on Medicaid claims can assist in identifying transitional age youth who are at risk of discontinuing from behavioral health care within 12 months, thus allowing for proactive assessment and outreach to promote continuity of care for younger persons who have behavioral health needs.
Keywords: emerging adults, machine learning, mental health services
What is known on this topic
Many youth and young adults with behavioral health needs drop out or experience significant disruption in continuity of care.
Discontinuation of behavioral health in transitional age youth (i.e., persons aged 15–26 years) has been examined at the organizational and system levels, but surprisingly little is known about person‐level characteristics that are associated with service discontinuation.
What this study adds
Using administrative Medicaid claims data, we developed and tested predictive modeling approaches to identify transitional age youth with behavioral health diagnoses who will discontinue behavioral health service use within 12 months.
This study provides preliminary evidence for predictive modeling that can assist providers and policy makers with correctly identifying 80% of the time which transitional age youth will remain engaged with the behavioral health service system or discontinue service use.
1. INTRODUCTION
Adolescence and early adulthood represent a pivotal time in the development of functional life skills and the onset of mental and substance use disorders. In the United States, half of all lifetime behavioral health disorders started by the age of 14 years, and 75% began by the age of 24 years. 1 Roughly one of two adolescents with a mental disorder receive some sort of behavioral health care, but the rate of service utilization decreases to less than 1 in 3 in young adulthood (30%). 2 , 3 The immediate and long‐term behavioral health needs during this period for transitional age youth (i.e., persons aged 15–26 years) are frequently marked by a need to transition from child and adolescent mental health services (CAMHS) to adult mental health services (AMHS) upon turning 18 years old. Regrettably, this transition is frequently characterized by a significant disruption in continuity of care resulting in unmet treatment need with up to 60% of CAMHS youth failing to transition successfully to AMHS or disengaging from care. 4 , 5 , 6
Consensus is growing that this age‐based cutoff for transitioning from CAMHS to AMHS does not provide sufficient flexibility for differences in young persons' maturity, readiness, functioning, or life context. 7 Furthermore, prior reviews of services and programs intended to facilitate successful CAMHS to AMHS transitions suggest that little or poor quality data exist for their effectiveness. 8 , 9 , 10 A recent 2018 review of 87 articles identified a list of core elements at the policy and organizational level that have yet to be empirically supported but shed light on possible solutions to system‐level issues between the separate funding structures, service philosophies, and target populations for the CAMHS and AMHS. 11
The importance of “bridging the gap” in service transitions for transitional age youth is increasingly being recognized and examined more systematically. 12 , 13 , 14 Nevertheless, what has yet to be sufficiently explored in this population is person‐level characteristics associated with disengagement from behavioral health treatment. Predictive analytics and machine learning are increasingly being applied to the behavioral health field to assist with improving the accuracy of predicting a variety of clinical outcomes. 15 At its core, predictive analytics encompasses a variety of complex statistical techniques that capitalize on advances in data storage capacity, computer processing power, and mathematical algorithms to identify patterns in data and to estimate, or “predict,” future outcomes. In recent years, multiple machine learning approaches to predictive modeling using complex algorithms have been developed to extract multitudinous variables from large datasets for advanced statistical and probabilistic techniques and are increasingly applied to the behavioral health field. 16 , 17
This paper describes the efforts of the Connecticut Behavioral Health Partnership (CT BHP) to develop a predictive analytic model of discontinuation of behavioral health service use in transitional age youth to better serve this vulnerable population. The CT BHP is an inter‐department partnership between the Department of Children and Families (DCF), Department of Social Services (DSS), and Department of Mental and Addiction Services (DMHAS) to create an integrated public behavioral health service system for children, adults, and families with Medicaid. The Administrative Service Organization for the CT BHP, Beacon Health Options, manages the behavioral health care for more than 800,000 Medicaid members by coordinating clinical management processes across the system of care.
The purpose of this report is to describe the development, initial validation, and preliminary results from a machine learning model using predictive analytic algorithms to identify predictors that a transitionally aged youth with recent behavioral health service use will discontinue using behavioral health services within the next 12 months. This project emerged from preliminary analyses conducted by the CT BHP that observed significant reductions in behavioral health service use among Connecticut Medicaid enrollees between their 17th and 18th birthday, suggesting that as many as one in three were disengaging from the service system at the point of transitioning from the CAMHS into the AMHS. The information from this project will be used to help inform a transitional assistance program designed by the CT BHP to help ensure young people in the CAMHS and AMHS receive continuity of care.
2. METHODS
2.1. Study population
The cohort of transitional age youth was extracted from Medicaid administrative claims data from 2016 to 2017. The initial data extract included all Medicaid members who were aged 15–26 years old as of December 31, 2016. Of the 170,685 members who met the age criteria, the following characteristics were used to select members according to the following two criteria. First, the presence of at least one paid claim in 2016 with a behavioral health diagnosis in the primary position (n = 44,314). Second, Medicaid enrollment of ≥11 months in 2016 and ≥11 months in 2017, which occurred in 76% of the members who met the age and behavioral health criteria. No other inclusion or exclusion criteria were used. This yielded a final cohort of 33,532 transitional age youth aged 15–26 with at least one behavioral health need (i.e., at least one paid claim with a behavioral health diagnosis in the primary position during 2016) who were Medicaid enrolled for at least 22 months during the 2016 observation period and 2017 prediction window.
2.2. Features
Behavioral health authorization or claims data from 2016 to 2107 provided three broad classes of variables (i.e., features) that included demographic, clinical, and service use information. Over 150 potential features were extracted for the current analysis. Beneficiary characteristics included demographic information (e.g., age, gender, race, ethnicity, county, benefit package, and catchment region); number of days until/since turning 18 years old; DCF involvement*; and eligibility days by month. Service utilization costs included the following: total medical, behavioral health, dental, and pharmacy cost by month; total service utilization costs by each level of care; and total behavioral health pharmacy category costs. Total costs were also extracted for primary and secondary behavioral health category costs based on the Clinical Classification Software (CCS, v2019.1), which sorts claims data into clinically meaningful categories based on diagnoses and procedures. Specific CCS behavioral health categories included the following behavioral health diagnoses: adjustment disorders; anxiety disorders; attention‐deficit, conduct, and disruptive behavior disorders; delirium, dementia, and amnestic and other cognitive disorders; developmental disorders; disorders usually diagnosed in infancy, childhood, or adolescence; impulse control disorders not elsewhere classified; mood disorders; personality disorders; schizophrenia and other psychotic disorders; alcohol‐related disorders; substance‐related disorders; suicide and intentional self‐inflicted injury; and miscellaneous mental health disorders (CCS codes 650–662, 670). Clinical variables were further categorized by the following: count of both medical and behavioral health chronic conditions; number of days since last behavioral health paid claim; number of days since last behavioral health authorization, and number of days since last behavioral health pharmacy claim.
2.3. Outcome
Discontinuation was defined as having a behavioral health need at any point during the 2016 observation window, but having no paid claims for a behavioral health diagnosis in the primary position during the 2017 prediction window.
2.4. Machine learning algorithms
The study population was partitioned into three separate samples to create mutually exclusive files to train, validate, and test predictive models. Stratified random assignment ensured that each of the three files was equally representative in terms of occurrence of the outcome variable (i.e., discontinuation from behavioral health services) as well as demographic factors (e.g., age, gender, DCF involvement) and clinical and service use characteristics (e.g., behavioral and medical comorbidity, total cost for behavioral health). The training sample (40%, n = 13,412) was used to develop predictive models, the validation sample (30%, n = 10,059) was used to tune all models, and the testing sample (30%, n = 10,061) was a “hold out” to provide unbiased evaluations of the final models.
Based upon the type of data and available variables (i.e., administrative claims data) as well as the overall goal of the predictive model and recommendations in the literature, 18 three machine learning approaches were compared with the more traditional statistical approach of logistic regression. The three machine learning algorithms were as follows: (1) decision tree, (2) random forest, and (3) gradient boosting. Full description of these machine learning approaches is beyond the scope of this article (see Kuhn & Johnson 19 for more information). A fundamental step to the three predictive modeling approaches is to determine the optimal number of predictor variables by eliminating redundant, or highly inter‐correlated, predictors (i.e., multicollinearity) and irrelevant variables that do not provide incremental information to predicting the outcome. Techniques to reduce the number of predictor variables are clinically and statistically driven to achieve independence of observations and to create the best fitting predictive model that produces generalizable, interpretable, and accurate results. For the traditional logistic regression model, predictors were selected using backward elimination. No interaction or polynomial terms were entered into the logistic regression model.
The three machine learning approaches were assessed relative to logistic regression according to the degree of model fit and the optimal number of predictors that produced the most accurate prediction of treatment discontinuation. A key measure used to assess model performance was classification accuracy, which involves maximizing the number of predicted members who truly dropped out of the service system (i.e., true positives, sensitivity), the number of predicted members who stayed in the service system (i.e., true negative, specificity), and minimizing the inverse of both (i.e., false positives and false negatives). This classification accuracy approach was the main method used to assess which predictive model performed the best and thus which predictive model to use when applying to new data. To determine which of the four predictive models was statistically superior, the misclassification rate was used on the validation dataset as the selection criterion for the “winning” model. A supplemental measure of discriminative accuracy was the area under the receiver operator characteristic curve statistic (i.e., AUC). AUC values range from 0.5 (accuracy no better than chance) to 1.0 (perfect accuracy), with higher values indicating better prediction accuracy. Research suggests that AUC values greater than 0.70 are good predictive models.
All analyses were conducted using the SAS Enterprise Miner† (SAS EM), which is a software tool that focuses primarily on predictive modeling. SAS EM allows for multiple models to run, using different algorithms with different scenarios and predictors, to be evaluated in terms of their predictive results.
3. RESULTS
Table 1 provides descriptive information about the characteristics of the stratified random samples extracted for the training, validation, and testing partitions. No significant differences were observed. Table 2 lists the variables that were selected after model optimization from the over 150 candidate variables. Across all models, number of days since the last behavioral health pharmacy claim or behavioral health claim emerged as predictors, whereas demographic variables were primarily only selected in gradient boosting and random forest models.
TABLE 1.
Demographic composition of samples used to develop predictive models of disengagement from behavioral health care within 12 months among emerging adults with documented behavioral health need
| Characteristics | Training data (n = 13,412) | Validation data (n = 10,059) | Testing data (n = 10,061) | |||
|---|---|---|---|---|---|---|
| N | % | N | % | N | % | |
| Age in years (M ± SD) | 20.05 ± 3.66 | 20.06 ± 3.66 | 20.04 ± 3.66 | |||
| Gender | ||||||
| Female | 7189 | 53.6% | 5452 | 54.2% | 5483 | 54.5% |
| Male | 6223 | 46.4% | 4607 | 45.8% | 4578 | 45.5% |
| Race/Ethnicity | 0 | |||||
| Caucasian | 5164 | 38.5% | 3903 | 38.8% | 3843 | 38.2% |
| African American | 1703 | 12.7% | 1308 | 13.0% | 1298 | 12.9% |
| Hispanic | 2843 | 21.2% | 2092 | 20.8% | 2163 | 21.5% |
| Asian | 161 | 1.2% | 111 | 1.1% | 141 | 1.4% |
| Native American | 40 | 0.3% | 30 | 0.3% | 30 | 0.3% |
| Pacific Islander | 13 | 0.1% | 10 | 0.1% | 10 | 0.1% |
| Multiracial | 335 | 2.5% | 241 | 2.4% | 252 | 2.5% |
| Unknown | 3152 | 23.5% | 2364 | 23.5% | 2324 | 23.1% |
| DCF involvementa | ||||||
| Yes | 751 | 5.6% | 593 | 5.9% | 543 | 5.4% |
| No | 12,661 | 94.4% | 9466 | 94.1% | 9518 | 94.6% |
| # of days until/since 18th birthday (M ± SD) | 962.0 ± 1231.2 | 927.3 ± 1233.2 | 936.9 ± 1229.0 | |||
| # of primary behavioral health chronic conditions (M ± SD) | 1.7 ± 1.0 | 1.7 ± 1.0 | 1.6 ± 1.0 | |||
| # of primary medical chronic conditions (M ± SD) | 0.9 ± 1.4 | 0.9 ± 1.3 | 0.9 ± 1.4 | |||
| # of secondary behavioral health chronic conditions (M ± SD) | 0.9 ± 1.1 | 0.9 ± 1.3 | 0.9 ± 1.4 | |||
| # of secondary medical chronic conditions (M ± SD) | 0.7 ± 1.2 | 0.7 ± 1.1 | 0.7 ± 1.2 | |||
| Total behavioral health pharmacy cost (M ± SD) | $905.4 ± 2523.7 | $894.2 ± 2474.8 | $820.9 ± 2500.6 | |||
| Total behavioral health primary diagnosis claim cost (M ± SD) | $7561.8 ± 27,629.1 | $7965.7 ± 30,646.0 | $8030.2 ± 32,324.8 | |||
| Total medical primary diagnosis claim cost (M ± SD) | $3371.6 ± 14,124.3 | $2990.0 ± 9831.1 | $3403.0 ± 14,019.7 | |||
| Total dental cost (M ± SD) | $364.6 ± 650.3 | $373.6 ± 639.8 | $373.1 ± 677.5 | |||
| # of days since last behavioral health pharmacy claim (M ± SD) | 81.3 ± 97.9 | 79.9 ± 94.9 | 83.8 ± 97.7 | |||
| # of days since last behavioral health claim (M ± SD) | 92.3 ± 105.6 | 92.9 ± 105.2 | 92.7 ± 104.3 | |||
| # of days since last authorization (M ± SD) | 163.3 ± 107.1 | 162.2 ± 107.2 | 163.3 ± 107.0 | |||
| # of group home days (M ± SD) | 1.5 ± 20.6 | 1.9 ± 23.3 | 1.4 ± 19.6 | |||
| # of residential treatment days (M ± SD) | 1.0 ± 15.3 | 1.0 ± 15.0 | 0.5 ± 9.8 | |||
DCF‐involvement“ includes any youth who is involved with the Department of Children and Families through any of its mandates.
TABLE 2.
Selected variables of discontinuation of behavioral health care within 12 months among transitional age youth with recent behavioral health service use, by predictive model type a
| Prediction model | ||||
|---|---|---|---|---|
| Predictor variables | Decision tree | Gradient boosting | Random forest | Logistic regression b |
| Demographic | ||||
| Gender | • | • | ||
| Race/ethnicity | • | • | • | |
| Age (# of days until/since 18th birthday) | • | • | ||
| Benefit package | • | • | ||
| County | • | • | ||
| DCF status | • | • | • | |
| Clinical | ||||
| # of behavioral health chronic conditions (primary) | • | • | ||
| # of medical chronic conditions (primary) | • | • | ||
| # of behavioral health chronic conditions (secondary) | • | • | • | |
| # of medical chronic conditions (secondary) | • | • | ||
| Service utilization | ||||
| Total behavioral health pharmacy cost | • | • | • | |
| Total pharmacy cost | • | • | ||
| Total behavioral health primary diagnosis claim cost | • | • | ||
| Total medical primary diagnosis claim cost | • | • | ||
| Total dental cost | • | • | ||
| # of days since last behavioral health pharmacy claim | • | • | • | • |
| # of days since last behavioral health claim | • | • | • | • |
| # of days since last authorization | • | • | ||
| # of group home days | • | • | ||
| # of residential treatment center days | • | • | ||
Variable selection based on chi‐square logworth for decision tree model, selection frequency for gradient boosting and random forest models, and absolute value of coefficient for logistic regression model.
Variables were selected using backward elimination. No interaction or polynomial terms were entered into the logistic regression model.
3.1. Model performance
In terms of classification accuracy, the gradient boosting predictive model demonstrated the lowest misclassification rate by correctly classifying 80% of the time which transitional age youth would remain engaged in (i.e., true negative) or discontinue (i.e., true positive) behavioral health service use (Table 3). The misclassifications rates for both the random forest and decision tree models were very similar (21%) to the gradient boosting model, whereas logistic regression had the highest (29%). Similar results were observed for AUC values with gradient boosting (0.86), random forest (0.86), and decision tree (0.84) having higher values compared with logistic regression (0.71). A primary factor in misclassification derived from logistic regression was lower rates of identifying true positives of service discontinuation. For example, logistic regression correctly identified 11% of true positive cases of discontinuation, whereas gradient boosting identified 66%. Alternative approaches to logistic regression did not improve performance (Supporting Information).
TABLE 3.
Model fit statistics by dataset and predictive model
| Prediction model | |||||
|---|---|---|---|---|---|
| Dataset type | Model fit statistics | Decision tree | Gradient boosting | Random forest | Logistic regression |
| Training | Misclassification rate | 0.21 | 0.19 | 0.19 | 0.28 |
| Area under the curve | 0.86 | 0.88 | 0.88 | 0.72 | |
| Sum of squared errors | 3802.04 | 3481.35 | 3154.00 | 5090.61 | |
| Average squared error | 0.14 | 0.13 | 0.13 | 0.21 | |
| Gini coefficient | 0.71 | 0.76 | 0.76 | 0.44 | |
| Validation | Misclassification rate | 0.21 | 0.20 | 0.21 | 0.28 |
| Area under the curve | 0.84 | 0.86 | 0.86 | 0.71 | |
| Sum of squared errors | 2872.69 | 2727.12 | 2534.87 | 3881.07 | |
| Average squared error | 0.14 | 0.14 | 0.14 | 0.21 | |
| Gini coefficient | 0.68 | 0.72 | 0.72 | 0.42 | |
| Test | Misclassification rate | 0.20 | 0.20 | 0.21 | 0.29 |
| Area under the curve | 0.84 | 0.85 | 0.85 | 0.71 | |
| Sum of squared errors | 2843.99 | 2682.50 | 2562.77 | 3856.33 | |
| Average squared error | 0.15 | 0.14 | 0.14 | 0.21 | |
| Gini coefficient | 0.67 | 0.71 | 0.70 | 0.41 | |
4. DISCUSSION
These preliminary results demonstrated that predictive modeling approaches using demographic, clinical, and service use variables extracted from Medicaid administrative data can correctly classify 80% of the time which transitional age youth will remain engaged or discontinue behavioral health service use within 12 months. Although traditional logistic regression performed satisfactorily, overall, in predicting discontinuation, models using more complex machine learning algorithms improved the rate of classification accuracy by roughly 13% compared with traditional logistic regression (i.e., 80% accuracy compared with 71%). The specific predictive modeling approach that yielded the highest predictive accuracy, gradient boosting, identified 18 variables that contributed to the identification of discontinuation of behavioral health care.
There are several strengths of this preliminary analysis. To our knowledge, this is the first predictive modeling of discontinuation of behavioral health in transitional age youth. Unlike prior qualitative analyses and thematic reviews that have examined transitions and disruptions in CAMHS and AMHS samples, 20 , 21 , 22 our approach used variables that are commonly available in structured formats in administrative data. Also, predictive modeling classifiers, as compared to regression‐based ones, frequently incorporate unexpected predictor variables and interactions that have not been previously reported in the literature. 23
These preliminary results will be further evaluated to determine their predictive accuracy as new data or new methods to measure variables become available. Furthermore, alternative machine learning techniques will be evaluated, such as an ensemble or super learner approach that “stacks” or combines multiple algorithms into a single algorithm. 24 This innovative approach returns a prediction value at least equal to the best performing single algorithm and is often significantly superior than that value. 25 , 26 , 27 An ensemble approach to predictive modeling has been highly recommended for risk stratification in mental health research. 28 , 29
While these preliminary findings suggest that predictive modeling was effective, some caution is warranted due to the following limitations. First, data used to develop and validate the model were based on youth who were essentially continuously enrolled in Medicaid (i.e., 11 of 12 months); therefore, these results may not generalize to beneficiaries with longer lapses in Medicaid coverage. Second, some percentage of youth may have discontinued services due to clinical improvement and no longer needed behavioral health care. Third, the 12‐month time horizon of discontinuation of behavioral health service use does not account for differences in shorter‐term versus longer‐term risk. Although longer time horizons have some policy importance, models that allow for shorter time horizons (e.g., 30 days, 90 days, and 6 months) potentially offer more utility for timely clinical intervention. Fourth, the current models did not include electronic health record (EHR) data, although research demonstrates that the inclusion of administrative claims and EHR data yields more accurate prediction results. 30 , 31 Fifth, this analysis is limited to Medicaid reimbursable services and does not include nontraditional services (e.g., pastoral counseling, animal‐assisted therapy). Finally, variations in receipt of behavioral health services have been associated with insurance type; 32 , 33 therefore, these results may not generalize to transitional age youth with commercial insurance.
Despite the limitations, this study demonstrated that it is feasible to identify 80% of the time which transitional age youth will remain engaged or become disengaged from the mental health service system by using variables derived from routinely collected administrative data. This ability to accurately identify “high risk” individuals has important policy and practice implications. For example, it can inform early intervention and identification of persons who may benefit from proactive assessment and outreach to address barriers to care (e.g., transportation). Also, accurate prediction of discontinuation may promote care continuity, which in turn can prevent potentially avoidable and costly emergency department and inpatient encounters. The results from this project will continue to be evaluated and revised in the hope of developing a transitional assistance program that will target service providers, youth, and families to help young people in the CAMHS and AMHS receive continuity of care.
Supporting information
Appendix S1: Supporting Information
ACKNOWLEDGMENTS
This work was funded in part by the State of Connecticut, Behavioral Health Partnership partner agencies, which include The Department of Social Services, The Department of Children and Families, and the Department of Mental Health and Addiction Services. This publication does not express the views of these partner agencies or the State of Connecticut. The views and opinions expressed are those of the authors.
Bory C, Schmutte T, Davidson L, Plant R. Predictive modeling of service discontinuation in transitional age youth with recent behavioral health service use. Health Serv Res. 2022;57(1):152‐158. 10.1111/1475-6773.13871
Footnotes
“DCF‐involvement” includes any youth who is involved with the Department of Children and Families through any of its mandates. This includes youth committed to DCF through child welfare, and those dually committed. It also includes youth for whom the department has no legal authority, but for whom DCF provides assistance through its voluntary services, family with service needs, and in‐home child welfare programs. Please note that there are exceptions to the “Out‐of‐home Committed” status; however, the majority of youth with this status are out‐of‐home.
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the United States and other countries.
REFERENCES
- 1. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age‐of‐onset distributions of DSM‐IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62(6):593‐602. [DOI] [PubMed] [Google Scholar]
- 2. Copeland WE, Shanahan L, Davis M, Burns BJ, Angold A, Costello EJ. Increase in untreated cases of psychiatric disorders during the transition to adulthood. Psychiatr Serv. 2015;66(4):397‐403. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Costello EJ, He J‐p, Sampson NA, Kessler RC, Merikangas KR. Services for adolescents with psychiatric disorders: 12‐month data from the National Comorbidity Survey–Adolescent. Psychiatr Serv. 2014;65(3):359‐366. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Pottick KJ, Bilder S, Vander Stoep A, Warner LA, Alvarez MF. US patterns of mental health service utilization for transition‐age youth and young adults. J Behav Health Serv Res. 2008;35(4):373‐389. [DOI] [PubMed] [Google Scholar]
- 5. Cappelli M, Davidson S, Racek J, et al. Transitioning youth into adult mental health and addiction services: an outcomes evaluation of the youth transition project. J Behav Health Serv Res. 2016;43(4):597‐610. [DOI] [PubMed] [Google Scholar]
- 6. Paul M, Ford T, Kramer T, Islam Z, Harley K, Singh SP. Transfers and transitions between child and adult mental health services. Br J Psychiatry. 2013;202(s54):s36‐s40. [DOI] [PubMed] [Google Scholar]
- 7. McGorry P, Bates T, Birchwood M. Designing youth mental health services for the 21st century: examples from Australia, Ireland and the UK. Br J Psychiatry. 2013;202(s54):30‐s35. [DOI] [PubMed] [Google Scholar]
- 8. Embrett MG, Randall GE, Longo CJ, Nguyen T, Mulvale G. Effectiveness of health system services and programs for youth to adult transitions in mental health care: a systematic review of academic literature. Adm Policy Ment Health. 2016;43(2):259‐269. [DOI] [PubMed] [Google Scholar]
- 9. Scholz B, Battams S, Platania‐Phung C, Happell B. Transitioning from adolescent to adult mental health services: an integrative literature review. Issues Ment Health Nurs. 2019;40(2):97‐111. [DOI] [PubMed] [Google Scholar]
- 10. Paul M, Street C, Wheeler N, Singh SP. Transition to adult services for young people with mental health needs: a systematic review. Clin Child Psychol Psychiatry. 2015;20(3):436‐457. [DOI] [PubMed] [Google Scholar]
- 11. Cleverley K, Rowland E, Bennett K, Jeffs L, Gore D. Identifying core components and indicators of successful transitions from child to adult mental health services: a scoping review. Eur Child Adolesc Psychiatry. 2020;29(2):107‐121. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Singh SP, Tuomainen H. Transition from child to adult mental health services: needs, barriers, experiences and new models of care. World Psychiatry. 2015;14(3):358‐361. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Tuomainen H, Schulze U, Warwick J, et al. Managing the link and strengthening transition from child to adult mental health Care in Europe (MILESTONE): background, rationale and methodology. BMC Psychiatry. 2018;18(1):167. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Signorini G, Singh SP, Marsanic VB, et al. The interface between child/adolescent and adult mental health services: results from a European 28‐country survey. Eur Child Adolesc Psychiatry. 2018;27(4):501‐511. [DOI] [PubMed] [Google Scholar]
- 15. Hahn T, Nierenberg A, Whitfield‐Gabrieli S. Predictive analytics in mental health: applications, guidelines, challenges and perspectives. Mol Psychiatry. 2017;22(1):37‐43. [DOI] [PubMed] [Google Scholar]
- 16. Shatte AB, Hutchinson DM, Teague SJ. Machine learning in mental health: a scoping review of methods and applications. Psychol Med. 2019;49(9):1426‐1448. [DOI] [PubMed] [Google Scholar]
- 17. Khondoker M, Dobson R, Skirrow C, Simmons A, Stahl D. A comparison of machine learning methods for classification using simulation with multiple real data examples from mental health studies. Stat Methods Med Res. 2016;25(5):1804‐1823. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Fernández‐Delgado M, Cernadas E, Barro S, Amorim D. Do we need hundreds of classifiers to solve real world classification problems? J Mach Learn Res. 2014;15(1):3133‐3181. [Google Scholar]
- 19. Kuhn M, Johnson K. Applied Predictive Modeling. Vol 26. New York: Springer; 2013. [Google Scholar]
- 20. Appleton R, Connell C, Fairclough E, Tuomainen H, Singh SP. Outcomes of young people who reach the transition boundary of child and adolescent mental health services: a systematic review. Eur Child Adolesc Psychiatry. 2019;28(11):1431‐1446. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Hill A, Wilde S, Tickle A. Review: transition from Child and Adolescent Mental Health Services (CAMHS) to Adult Mental Health Services (AMHS): a meta‐synthesis of parental and professional perspectives. Child and Adolescent Mental Health. 2019;24(4):295‐306. [DOI] [PubMed] [Google Scholar]
- 22. Nguyen T, Embrett MG, Barr NG, et al. Preventing youth from falling through the cracks between child/adolescent and adult mental health services: a systematic review of models of care. Community Ment Health J. 2017;53(4):375‐382. [DOI] [PubMed] [Google Scholar]
- 23. Weng SF, Reps J, Kai J, Garibaldi JM, Qureshi N. Can machine‐learning improve cardiovascular risk prediction using routine clinical data? PLoS One. 2017;12(4):e0174944. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Polley E, van der Laan M. Super Learner in prediction. UC Berkeley Division of Biostatistics Working Paper Series. Working Paper 266, May 2010, http://biostats.bepress.com/ucbbiostat/paper266.
- 25. Acion L, Kelmansky D, van der Laan M, Sahker E, Jones D, Arndt S. Use of a machine learning framework to predict substance use disorder treatment success. PLoS One. 2017;12(4):e0175383. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Bhatt S, Cameron E, Flaxman SR, Weiss DJ, Smith DL, Gething PW. Improved prediction accuracy for disease risk mapping using Gaussian process stacked generalization. J R Soc Interface. 2017;14(134):20170520. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Rosellini AJ, Dussaillant F, Zubizarreta JR, Kessler RC, Rose S. Predicting posttraumatic stress disorder following a natural disaster. J Psychiatr Res. 2018;96:15‐22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Kessler RC, van Loo HM, Wardenaar KJ, et al. Testing a machine‐learning algorithm to predict the persistence and severity of major depressive disorder from baseline self‐reports. Mol Psychiatry. 2016;21(10):1366‐1371. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Shrestha A, Bergquist S, Montz E, Rose S. Mental health risk adjustment with clinical categories and machine learning. Health Serv Res. 2018;53:3189‐3206. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Kharrazi H, Chi W, Chang H‐Y, et al. Comparing population‐based risk‐stratification model performance using demographic, diagnosis and medication data extracted from outpatient electronic health records versus administrative claims. Med Care. 2017;55(8):789‐796. [DOI] [PubMed] [Google Scholar]
- 31. Kan HJ, Kharrazi H, Leff B, et al. Defining and assessing geriatric risk factors and associated health care utilization among older adults using claims and electronic health records. Med Care. 2018;56(3):233‐239. [DOI] [PubMed] [Google Scholar]
- 32. Lynch S, Witt W, Ali MM, et al. Follow‐up care after behavioral health‐related hospitalization for children and adolescents. Community Ment Health J. 2020;56(8):1419‐1428. [DOI] [PubMed] [Google Scholar]
- 33. Ali MM, Sherman LJ, Lynch S, Teich J, Mutter R. Differences in utilization of mental health treatment among children and adolescents with Medicaid or private insurance. Psychiatr Serv. 2019;70(4):329‐332. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Appendix S1: Supporting Information