Generalizing the Prediction of Bipolar Disorder Onset Across High Risk Populations

Abstract

Objective

Risk calculators (RC) to predict clinical outcomes are gaining interest. A RC to estimate risk of bipolar spectrum disorders (BPSD) could help reduce the duration of undiagnosed BPSD and improve outcomes. Our objective was to adapt a RC previously validated in the Pittsburgh Bipolar Offspring study sample (BIOS; Hafeman et al., 2017) to achieve adequate predictive ability in both familial high risk and clinical high risk youth.

Method

Participants (aged 6–12 at baseline) from the Longitudinal Assessment of Manic Symptoms (LAMS) study (N=473) were evaluated semi-annually. Evaluations included a KSADS interview. After testing a RC that closely approximated the original, we made modifications to improve model prediction. Models were trained in the BIOS sample and tested in LAMS. The final model was then trained in LAMS participants, including family history of BPSD as a predictor, and tested in the familial high-risk sample.

Results

Over follow-up, 65 youth newly met criteria for BPSD. The original RC identified youth who developed BPSD only moderately well (AUC = 0.67). Eliminating predictors other than the KSADS screening items for mania and depression improved accuracy (AUC=0.73) and generalizability. The model trained in LAMS, including family history as a predictor, performed well in the BIOS sample (AUC=0.74).

Conclusion

The clinical circumstances under which the assessment of symptoms occurs impacts RC accuracy; focusing on symptoms related to the onset of BPSD improved generalizability. Validation of the RC under clinically-realistic circumstances will be an important next step.

Introduction

Generalizing the prediction of bipolar disorder onset across populations Bipolar spectrum disorders (BPSD) often emerge during adolescence^1,2. However, early symptoms can be hard to distinguish from typical behavior and other disorders³. Recently, interest has grown in the prodromal period, prior to the onset of an episode of [hypo]mania or depression^4–16 in hopes that, with better characterization of the prodrome, early identification of BPSD symptoms will improve. Currently, misdiagnosis and lack of diagnosis are common; on average people with bipolar disorder receive mental health services for many years before receiving an accurate diagnosis and appropriate treatment^2,17,18. Delays in treatment have significant consequences for both short- and long-term functioning^18,19. Improving clinicians’ ability to differentiate symptomatic young people who are going to develop a BPSD from those who are symptomatic, but unlikely to have a [hypo]manic episode, could help to reduce misdiagnosis and improve outcomes in this population.

The majority of people who develop a BPSD will experience symptoms prior to the onset of an initial mood episode⁹. Although assessing for these symptoms provides valuable information about an individual’s risk for BD^{5,10,11,13–15,20}, it is a challenge for clinicians to translate the presence/absence of subclinical symptoms into an actionable prediction of BPSD risk^21,22. Although the process of evidence-based assessment, using an actuarial approach, will provide a posterior probability of BD diagnosis, this method focuses on current diagnostic status, not future prediction^23,24. In the case of BPSD, prediction is particularly valuable since preventing or forestalling the onset of a major mood episode could have important implications for an individual’s ability to meet developmental milestones and for the trajectory of their illness. Additionally, the treatment guidelines for BPSD are different from unipolar depression or ADHD, diagnoses that often precede a BPSD diagnosis^25,26. If a BPSD trajectory is probable, an alternative treatment strategy might be indicated.

Inspired by other medical fields, in which patient health information is used to predict future disease state, a risk calculator (RC) for estimating future risk of developing a BPSD among offspring of parents with BPSD was developed²⁷. This RC provides an estimate of the probability of BPSD onset within five years, taking into account the presence of symptoms that are common during the prodromal period⁹, as well as the parent’s age of BD onset, and offspring global functioning and age.

Although BD offspring are at elevated risk for BPSD, a large proportion of people who develop BPSD do not have an affected first degree relative^1,28,29. Familial high-risk studies are distinct from clinical high-risk studies, in which participants are designated at-risk due to subclinical mood symptoms. The goal of this investigation was to evaluate the performance of the published RC, developed for a familial high-risk sample, predicting onset of BPSD (defined as BD I, BD II, BD NOS [BD NOS was diagnosed based on the criteria defined in the Course and Outcome of Bipolar Youth study], or cyclothymic disorder) in a clinical high-risk population. Because the familial high-risk and the clinical high-risk samples varied in important ways, including the fact that the former was recruited through their parents (who had BPSD) and the latter were seeking treatment for their own symptoms, a secondary goal was to adapt the existing RC to achieve adequate predictive ability in the familial high-risk sample and to then cross-validate the RC in the clinical high-risk sample, in order to best serve high-risk youth broadly.

Method

Participants

Participants, aged 6–12 years, were from the Longitudinal Assessment of Manic Symptoms study (LAMS), a multisite, longitudinal study of treatment-seeking youth who were experiencing elevated symptoms of mania (ESM)³⁰. The majority of youth (N=621) were experiencing some manic symptoms at baseline, as indicated by a score of 12+ on the Parent General Behavior Inventory 10-Item Mania scale (PGBI-10M³¹). Eighty-six demographically-matched youth who did not have manic symptoms (PGBI-10M scores <12) were also enrolled. Of these 707 children, 685 were admitted to the longitudinal sample. Participants and their caregiver completed evaluations approximately every six months for up to eight years.

Measures

The Kiddie Schedule for Affective Disorders and Schizophrenia for School-Age Children–Present and Lifetime Version

The Kiddie Schedule for Affective Disorders and Schizophrenia for School-Age Children–Present and Lifetime Version with additional items from the Washington University St. Louis Kiddie Schedule for Affective Disorders (KSADS-PL-W^32–34) was administered by a trained interviewer to youth and caregivers separately. Inter-rater reliability for KSADS diagnoses was good, K=0.82³². KSADS diagnoses informed the research diagnosis, which was determined by a licensed clinical psychologist or psychiatrist. Items from the KSADS modules for depression and mania were used as the measure of prodromal mood symptoms in the RC.

Parent General Behavior Inventory 10-Item Mania Rating scale

Parent General Behavior Inventory 10-Item Mania Rating scale (P-GBI-10M³⁵). The P-GBI-10M assesses manic symptoms and effectively distinguishes youth with BD from other treatment-seeking youth³⁶.

Screen for Child Anxiety Related Emotional Disorders

Screen for Child Anxiety Related Emotional Disorders (SCARED^37,38) is a self-report of child anxiety symptoms. Total scores range from 0 to 82, with high scores indicating more anxiety symptoms; total scores of 25 or above are often used as the clinical cut-off^37,39.

Child’s Global Assessment Scale

Child’s Global Assessment Scale (C-GAS⁴⁰) is a measure of youth functioning, rated by a clinician. We considered baseline C-GAS scores as a potential predictor in the RC.

Procedure

All youth and their caregivers provided assent or consent. Youth and their caregivers were interviewed separately using the KSADS. All caregivers and youth also completed questionnaires about mental health symptoms, functioning, and related topics. DSM-IV-TR diagnoses were determined by a licensed clinical psychologist or child psychiatrist.

Participants were re-evaluated every six months. At follow-up appointments, the youth and caregiver were again interviewed with the KSADS to determine any change in diagnostic status. Additionally, all caregivers and those youth over age 11 completed follow-up questionnaires. Only youth who did not meet criteria for any BPSD at baseline were included in this study. Onset of BPSD was based on the date of the first follow-up appointment at which they met criteria for a BPSD.

Analytic Plan

The process of developing the familial high-risk RC is described in detail elsewhere²⁷. We took a similar approach when implementing the modifications described below; we used Cox proportional hazards regression to model time to BPSD onset, accounting for the clustering of assessments within participant. The resulting model was then used to formulate a risk calculator equation, which is a function of the input predictor variables and estimated model coefficients capable of estimating BPSD onset risk. The product of the equation from an individual’s specific symptoms scores and characteristics, results in the predicted probability of that individual meeting criteria for a BPSD in the future. The Pittsburgh Bipolar Offspring Study (BIOS)⁴¹, in which the familial high-risk calculator was developed, had adequate follow-up data to test cumulative hazard within five years; therefore, we focused on the same time period for the modification analyses. We calculated bootstrapped 95% confidence intervals for each model to test for statistical significance and to evaluate incremental prediction improvements between models. Models were trained in the BIOS sample and externally validated in LAMS, other than where noted.

The process of evaluating and adapting the familial high-risk calculator for a clinical high-risk sample was conducted in steps. First, because the LAMS sample and the BIOS sample have different designs, the RC had to be modified to accommodate the available predictors. The BIOS RC included a sum of the KSADS depression rating scale (DRS) and mania rating scale (MRS) items that are associated with the onset of BPSD⁹ (MRS items were elation, irritability, decreased need for sleep, unusually energetic, increase in goal-directed activity, motor hyperactivity, grandiosity, accelerated speech, racing thoughts, poor judgment, inappropriate laughter, people seeking, increased productivity, distractibility, and mood lability; DRS items were depressed mood, irritability, negative self-image, fatigue, difficulty concentrating, psychomotor agitation, insomnia, daytime sleepiness, anorexia, weight loss, and suicidal ideation), the Children’s Affective Lability Scale self-report (CALS⁴²), the SCARED³⁸, the CGAS⁴⁰, parent age of mood disorder onset, and child current age. LAMS did not include the CALS and the P-GBI-10M was not collected in BIOS. Additionally, only a subgroup of LAMS participants had a parent with BPSD. Consequently, the RC tested in the present study did not include mood lability or parental age of mood disorder onset as predictors. The BIOS RC was time-dependent with predictions for the onset of BPSD within a five-year period. However, because the BIOS follow-ups were every two years and LAMS follow-ups were every six months, time-dependent predictions led to biased predictions (i.e., BIOS times to event were never <2 years, whereas LAMS times to event could be as short as 6 months). Therefore, we used lifetime diagnosis as the observed outcome when computing AUC (as opposed to computing a time-dependent AUC), which is not impacted by adjusting the time horizon of predicted values since all test predictions shift proportionally (see Table S1 for time-dependent results, available online).

In order to further explore the potential to predict onset of BPSD across samples, we also trained a version of the best-fitting model on youth in the LAMS sample and externally validated in BIOS (including both the high-risk offspring and controls).

As a final step, we created calibration plots for the best fitting, bootstrapped models from each sample to determine how well predicted probabilities of BPSD onset matched actual probabilities, as well as reported Spiegelhalter’s z test results.

Results

The LAMS sample included 473 youth who did not have a BPSD diagnosis at baseline. The mean PGBI 10M score was 11.3 ± 6.9. Over the follow-up period, 65 youth newly met criteria for BPSD (BD I=19, BD II=5, BD NOS=40, cyclothymic disorder=1). These youth had significantly higher PGBI-10M, MRS, and DRS scores than those who did not develop BPSD, and they were also slightly (but non-significantly) younger and more likely to be female sex and have an anxiety diagnosis (Table 1).

Table 1.

Demographic and Clinical Characteristics of Longitudinal Assessment of Manic Symptoms Study (LAMS) and Bipolar Offspring Study (BIOS) Samples at Baseline

Variable	BIOS (N=412)	LAMS (N=473)	Test Statistic	p-value
Mean Age at Intake	10.6 (3.7)	9.3 (1.9)	t=6.44	<0.0001
Sex (% Female)	202 (49.0%)	141 (29.8%)	χ2=34.27	<0.0001
Race (% White)	341 (82.8%)	322 (68.1%)	χ2=25.29	<0.0001
SCARED (Child Report)a	20.0 (0.5)	17.1 (0.4)	F=16.91	<0.0001
CGASb	74.3 (0.5)	62.3 (0.5)	F=285.97	<0.0001
MRSc	18.3 (0.2)	18.8 (0.2)	F=2.72	0.0991
DRS	17.6 (0.3)	17.6 (0.3)	F=0.04	0.8504
Baseline Diagnoses
Major Depressive Disorder (%)	74 (18.0)	121 (25.6)	χ2=7.44	0.0064
Anxiety (%)	136 (33.0)	195 (41.2)	χ2=6.35	0.0117
ADHD (%)	121 (29.4)	401 (84.8)	χ2=279.44	<0.0001
DBD (%)	89 (21.6)	330 (69.8)	χ2=204.92	<0.0001
Developed Bipolar Disorder (%)	54 (13.1)	65 (13.7)	χ2=0.08	0.7823
Mean Age of BPSD Onset (SD)	15.3 (4.5)	11.0 (2.7)	t=6.14	<0.0001

Note: SCARED, CGAS, KMRS, and KDRS linear mixed models all featured significant interactions with mean-centered age (p-values<0.0001; least-square group means and standard errors are reported) such that:

^aGroup contrast increases over time.

^bGroup contrast contracts over time.

^cLAMS had higher scores than BIOS at younger ages and lower scores at older ages.

ADHD = attention-deficit/hyperactivity disorder; CGAS = Child Global Assessment Scale; DBD = disruptive behavior disorder; PGBI 10M = Parent General Behavior Inventory 10-item mania scale; SCARED = Screen for Child Anxiety Related Emotional Disorders;

Compared to the familial high-risk sample, the LAMS sample was significantly younger, less likely to be White, more likely to be male sex, and more likely to have baseline diagnoses of MDD, anxiety, ADHD, and DBD (Table 2). Furthermore, LAMS youth had significantly lower baseline SCARED (indicating lower symptoms of anxiety) and CGAS scores (indicating poorer functioning). This pattern changed over follow-up, with the difference in SCARED scores increasing over time and the difference in CGAS scores decreasing. The two samples did not differ significantly in average MRS or DRS scores at baseline; however, among LAMS participants, MRS and DRS scores decreased with age, whereas among BIOS participants, MRS and DRS scores increased with age. Lastly, among the youth who developed BPSD, those from LAMS had significantly younger age of onset than those in BIOS.

Table 2.

Baseline Demographic and Clinical Characteristics of Longitudinal Assessment of Manic Symptoms Study (LAMS) Youth by Bipolar Spectrum Disorder Outcome

Variable	No BPSD (n=408)	BPSD at follow-up (n=65)	Test Statistic	p
Mean Age at Intake	9.3 (1.9)	8.9 (1.9)	t=1.78	0.0750
Sex (% Female)	115 (28.2)	26 (40.0)	χ2=3.74	0.0531
Race (% White)	276 (67.7)	46 (70.8)	χ2=0.25	0.6160
Baseline PGBI 10M	10.9 (6.9)	14.0 (6.8)	t=3.26	0.0012
Baseline SCARED (Child Report)a	17.0 (0.5)	18.6 (1.4)	F=1.18	0.2773
Baseline CGASa	62.6 (0.5)	60.1 (1.2)	F=3.66	0.0562
Baseline MRSa	18.3 (0.2)	22.9 (0.5)	F=72.48	<0.0001
Baseline DRSa	17.5 (0.3)	20.0 (0.7)	F=11.07	0.0009
Major Depressive Disorder (%)	102 (25.0)	19 (29.2)	χ2=0.53	0.4678
Anxiety (%)	161 (39.5)	34 (52.3)	χ2=3.82	0.0507
ADHD (%)	344 (84.3)	57 (87.7)	χ2=0.50	0.4813
DBD (%)	284 (69.6)	46 (70.8)	χ2=0.04	0.8498

Note: ADHD = attention-deficit/hyperactivity disorder; BPSD = bipolar spectrum disorder; CGAS = Child Global Assessment Scale; DBD = disruptive behavior disorder; PGBI 10M = Parent General Behavior Inventory 10-item mania scale; SCARED = Screen for Child Anxiety Related Emotional Disorders;

^aSatterthwaite approximation was used to account for differing variances.

The RC originally developed in a familial high-risk sample (predictors MRS, DRS, SCARED, CGAS, age; could not include CALS or parent age of mood disorder onset) discriminated only moderately well between youth who developed BPSD and those who did not in the LAMS sample (AUC=0.67), corresponding to a medium-sized effect (d=.62)⁴⁴. We also tested this RC in only LAMS participants who had a family history of BPSD, but the predictions did not substantially improve (AUC=0.70). See Table 3. This RC was also significantly less effective at predicting BPSD in the BIOS sample than the original RC, which was time-dependent and included parental age of mood onset and a measure of mood lability (AUC=0.76 vs 0.69).

Table 3.

Comparison of Models Trained in One Sample and Tested in the Other

Predictors	Bootstrap internal time-independent AUC (95% CI)	External time-independent AUC
Model outcomes; trained in BIOS, tested in LAMS
MRS, DRS, SCARED, CGAS, Age	0.69(0.64,0.74)	0.67
MRS, DRS, Age	0.67(0.61,0.72)	0.74
MRS and DRS screening items, Age	0.65(0.59,0.71)	0.76
MRS and DRS screening items	0.67(0.62,0.72)	0.73
Model outcomes; trained in LAMS, tested in BIOS
MRS and DRS screening items, Age	0.78(0.75,0.81)	0.63
MRS and DRS screening items	0.74(0.71,0.77)	0.70
MRS and DRS screening items, Family history of mania, age	0.78(0.76,0.81)	0.66
MRS and DRS screening items, Family history of mania	0.75(0.72,0.78)	0.74

Note: AUC = area under the curve; BIOS = Pittsburgh Bipolar Offspring Study; CGAS = Child Global Assessment Scale; DRS = depression rating scale; FHx = parental history of mania; LAMS = Longitudinal Assessment of Manic Symptoms Study; MRS = mania rating scale; SCARED = Screen for Child Anxiety Related Emotional Disorders

In order to improve model prediction, we eliminated the SCARED and C-GAS scores; the BIOS sample was recruited based on parental history of BPSD, not symptoms; therefore, in BIOS, anxiety symptoms and global functioning were helpful for identifying youth experiencing psychiatric symptoms, which put them at increased risk of BPSD onset. However, we hypothesized that in LAMS, a treatment-seeking sample, these predictors – that are not specific to BPSD – would have limited utility for differentiating at-risk youth. As a sensitivity analysis, we also tested the RC among only the LAMS participants who were ESM+; this did not significantly affect its performance.

Eliminating the SCARED and CGAS from the model improved its predictive ability (time-independent AUC=0.74). In plots of SCARED, CGAS, MRS, and DRS scores, differences in symptom trajectories by sample and diagnostic outcome (BPSD or not) are clear; of note, SCARED scores start high and fall in the non-BPSD LAMS youth, whereas the MRS and DRS scores increase and the CGAS decreases in those who develop BPSD. Figure 1.

Figure 1.

Symptom Plots by Sample and Bipolar Spectrum Disorder Outcome⁶³

Note: Censoring refers to when a case had not yet developed a bipolar diagnosis by the last wave of follow-up.

We also tested a model with just the screening items of the MRS and DRS. Administering the MRS and DRS is time consuming. Reducing the number of items facilitates using the RC clinically. The symptoms tested were unusually energetic, irritability, elation, decreased need for sleep, depressed mood, difficulty concentrating, and insomnia. This change further improved the predictions (AUC=0.76).

Across models, aging was associated with higher risk in BIOS (the coefficient associated with the time-varying age variable was positive). However, in LAMS, risk declined with aging (negative time-varying age coefficient). This is consistent with the fact that LAMS participants were younger (aged 6–12) and selected for having ESM at baseline. LAMS concentrated on cases likely to be clinically at risk at baseline and, as such, their risk (and symptoms) were most elevated closest to the time of enrollment, in contrast to the BIOS sample, whose risk increased as they aged. To better understand the change in risk as participants aged, we examined pubertal status as a time-varying predictor (measured by the Petersen Pubertal Development Scale;⁴⁵ to determine whether puberty could be playing a role, but puberty was not a significant predictor in either sample. We tested a model trained in BIOS using only manic and depressive symptoms associated with onset of BPSD, but excluding age. The estimated model coefficients and resulting risk calculator equation are shown below:

$$\mathit{\text{Estimated}}\mathit{\text{Risk}}=1-{0.97675}^{exp\left(0.1084*MRS+0.1260*DRS\right)}$$

It externally validated in LAMS (time-independent AUC=0.73), though it did not internally validate as well in BIOS (time-independent AUC=0.67).

As a last step, we trained the final model (predictors: MRS and DRS screening items) both with and without age in LAMS participants and then externally-validated the models in BIOS in order to get an estimate of potential generalizability in new samples. The model that included only the core symptoms of mania and depression externally validated the best (time-independent AUC=0.70). Training the model in LAMS and validating in BIOS (including both the offspring participants and offspring of community controls; n=720) also enabled the use of family history of BPSD as a predictor. Family history was not included in the BIOS-trained models because nearly all of the youth in the BIOS sample who developed BPSD had a parent with BPSD; therefore the family history variable alone would discriminate well between converters and nonconverters, but this result would not generalize well outside of the BIOS sample. The model that included family history of BPSD and the reduced symptoms of mania and depression externally validated well (time-independent AUC=0.74) and performed similarly in LAMS (AUC=0.75) (see Table 3). Additionally, we calculated the sensitivity, specificity, and positive predictive value (PPV) for each model and for both internal and external validation. The model including only reduced symptoms of mania and depression had good sensitivity and adequate specificity in external validation (LAMS-tested sensitivity=0.81, specificity=0.50; BIOS-tested sensitivity=0.76, specificity=0.52); meaning that in both samples, about 80% of youth who developed BD were correctly identified as at-risk, whereas about half of youth who did not develop BD were identified as low-risk. The results from the models including only reduced symptoms were the most similar of the models tested across the two samples (i.e., there was no large decrement in performance from one sample to the other as was found with other models tested; see Table S2, available online)

In order to test model calibration, we plotted model-predicted probability of developing BPSD against the actual probability in both BIOS and LAMS (see Figure 2). The BIOS-trained 5-year model predictions (after bootstrap) were well-calibrated to the observed BIOS data (Spiegelhalter’s z=−1.61, p=0.108), but tended to underestimate risk in the LAMS data (z=−4.06, p <0.0001); because LAMS participants were evaluated every six months, onset of BPSD could appear to occur sooner than in BIOS where follow-ups were every two years. The LAMS-trained 5-year model predictions were not as well-calibrated, underestimating risk in the LAMS data (after bootstrap; z=−4.59, p<0.0001) and overestimating risk in BIOS (z=−6.02, p<0.0001). However, the LAMS-trained model performed significantly better when estimating one-year risk (after bootstrap; z=−0.41, p=0.679), which is not surprising since LAMS assessments occurred every six months. See Figure S1, available online.

Figure 2.

Calibration Plots⁶⁴

Note: Plots show that the Pittsburgh Bipolar Offspring Study (BIOS) Model underpredicts risk in Longitudinal Assessment of Manic Symptoms (LAMS); the LAMS model under-predicts risk in LAMS and overestimates risk of conversion in BIOS

Discussion

There is growing interest in predicting the onset of [hypo]manic or depressive mood episodes, particularly among youth at elevated risk for BPSD^{4,5,9,13,15,16,46–48}. The results of our study emphasize the fact that specific population attributes have important implications for prognostication, and that the type of risk (e.g., clinical, familial) matters when considering the probability of BPSD. The original familial high-risk RC²⁷, which achieved good diagnostic discrimination (5-year AUC, 0.76), performed only moderately when applied to a clinical high-risk sample (time-independent AUC = 0.67). Through modifications, we developed a RC that can help to predict the onset of BPSD in youth who are at familial or clinical high-risk. The final RC included only the KSADS screening items for depression and mania.

The accuracy of the new RC (in either sample) is not as strong as the original full RC (bootstrapped); it was necessary to make a trade-off on within-sample accuracy to accommodate differences in available predictors and achieve worthwhile predictions across samples. This is consistent with other efforts to build predictive models; although results can be very precise in the training sample, the model often falls short when applied to a new, external sample²³. The BIOS RC was developed specifically for youth who have a parent with BPSD and, as we found, the factors associated with BPSD onset in this population may not be strong predictors in other populations. Although it is very possible that in another familial high-risk sample that the original RC would perform well, our goal here was to increase generalizability of the RC across both familial and clinical high risk youth.

The youth from the LAMS study were treatment-seeking and the majority had high levels of symptomatology at baseline, which led to poor estimates of risk when the original RC was tested. Reducing the number of predictors to include only mood symptoms increased generalizability by reducing bias from non-specific symptoms and overall functioning. Although the calibration plot suggests that estimated one-year risk scores for youth in the LAMS sample were still too high on average, this reduced RC performed well among individuals who were at familial high-risk, many of whom were psychologically healthy. The youth who stand to benefit the most from a RC are those who present at a clinic with symptoms resembling those in the LAMS sample (i.e., experiencing numerous symptoms such as irritability, distractibility, high energy). These youth are highly impaired, but the nonspecific nature of many of their symptoms can make it challenging for clinicians to know whether the course of the illness is likely to progress to intense symptoms that are consistent with a BPSD diagnosis or to persist in a more chronic fashion, which might be more indicative of another diagnosis, like ADHD. Incorporating the individual’s predicted probability of developing BPSD from the RC with other clinical information could help a clinician and family decide on a course of treatment. Still, further refinement of this model, perhaps by integrating other data related to BPSD risk, will be important to improve the RC clinical utility.

Contrary to expectations, age proved to be an unreliable predictor and it was not included in our final RC. In the LAMS sample, risk declined as participants aged. This may at first appear inconsistent with other studies that show risk increasing through adolescence⁴⁹. However, as noted, most LAMS youth were highly symptomatic at baseline, so although in community samples or familial high-risk samples age is positively correlated with risk, our results suggest that among youth seeking services for potentially prodromal mood symptoms, risk may decrease as time passes and they do not meet criteria for BPSD. Therefore, among youth with significant mood symptoms, age may not help with prediction and it could interfere if a clinician discounts the probability of BPSD in children or young adolescents.

Relying solely on symptoms of BPSD to predict BPSD could limit the RC utility in individuals who are asymptomatic. However, our results – along with other efforts to define the prodrome⁹ – suggest that youth who develop BPSD are unlikely to be asymptomatic in the period during which it is possible to make a prediction. Additionally, beyond simply counting the presence of symptoms, the RC weights each mania and depression item when estimating the probability of BPSD onset, enabling more precise predictions based on the specific symptoms the youth is experiencing. In the LAMS data, predictions were most accurate in the short term (one-to-two years after baseline). Although there is a desire to predict BPSD before any symptoms are apparent, so that preventive interventions can be administered, a year is enough time for a course of therapy to reduce the probability of a full mood episode (e.g.,^50–53. Importantly, predicting BPSD a year or two out is also soon enough to motivate families to make changes; the impact of preventive interventions is often limited by low recruitment and retention^54–56 perhaps because it is hard for families to commit the time, effort, and money required when the feared outcome is far away and/or there is no current signal (i.e., symptoms) indicative of consequences to come⁵⁷. Thus, a prediction one or two years in the future may lead to the right balance of motivation to change behavior and adequate time to alter course, and certainly indicates careful monitoring (Youngstrom et al., 2017).

It is important to keep in mind that the AUCs (0.67 in BIOS, 0.73 in LAMS) achieved with the new RC fall in a range that is clinically informative, but not diagnostic. About 80% of youth likely to develop BPSD were identified as at-risk, but the false positive rate was high, resulting in relatively low positive predictive power (the proportion of positive tests that are “true” positives). Although the RC provides valuable information about future risk for the onset of BPSD and can help to identify those in need of close follow-up, it cannot take the place of ongoing assessment for those at risk. Although the ultimate goal would be to develop a model that could predict, with great precision, the individuals who were on a trajectory to develop BPSD and the polarity and intensity of their index episode, there are limitations of the data and the methods that preclude this outcome. Our current study, which includes 65 new BPSD cases, was underpowered to predict onset with any specificity regarding the nature of the index episode or subsequent course (i.e., BD I, II, NOS).

Methodologically, we were limited by differences in the design of LAMS and BIOS. For example, the measures collected were not the same, so we were not able to exactly replicate the original RC. Although our results suggest that the inclusion of a mood lability scale (general evidence of psychopathology, not specific to BPSD) would probably not help, there may be other variables that could have helped improve the prediction, but were not collected. For example, parental age of mood disorder onset was the strongest predictor in the original familial high-risk RC, but this information was not collected in LAMS. Additionally, the follow-up appointments for the two studies were on a different schedule; BIOS assessments were two (or more) years apart, precluding predictions over shorter intervals, whereas LAMS assessments were every six months, and predictions were more accurate over shorter horizons. In a clinical setting, it is likely that data could be collected more frequently (i.e., weekly in psychotherapy appointments or at least quarterly in medication management appointments). Testing and validating a RC under more realistic clinical circumstances is an important next step. Our final RC included only mania and depression screening items from the KSADS. This means that a relatively short assessment would yield the necessary information to predict an individual’s risk of BPSD, which improves clinical utility over the previous RC. However, this is also a limitation; relying on a relatively small set of symptoms, summarized from youth and caregiver report, for this prediction may result in biased estimates. Psychiatry aspires to implement RCs with the same success that other fields have achieved, but this will require sources of objective data (e.g., actigraphy data indexing activity and circadian function or polygenic risk scores specific to BPSD^58,59) to overcome the recall biases and inaccuracies inherent to self- or caregiver-reported mood^60–62.

It is possible to reliably identify individuals at high-risk to develop BPSD based on early symptoms, but the clinical circumstances under which the assessment of early symptoms occur impacts prediction accuracy and has implications for the period during which the predication is valid. This study built on the RC previously tested in a familial high-risk sample, broadening its utility to include treatment-seeking youth at clinical high-risk. Greater generalizability was achieved by focusing solely on symptoms related to the onset of BPSD. Integration of additional diagnostically specific factors may improve the generalizability of future risk calculators, and it may be necessary to use RCs developed for use in specific high risk populations (i.e., familial or clinical) in order to maximize predictive accuracy, rather than aiming for maximum generalizability. Validation of the RC in a new, treatment-seeking population will be an important next step in determining whether integrating risk predictions from the RC improves outcomes for youth with mood symptoms the way that patients at risk for other negative outcomes (e.g., myocardial infarction²⁷) have benefited. Ideally, information collected during an intake appointment would be entered into the RC, providing an estimate of the probability of developing BPSD in the next 24 months (or other timeframe for future RCs). Based on this information, the clinician, in consultation with the family, could decide on a treatment plan most likely to reduce risk while balancing resource needs and side effects.