Generalizability of Evidence-Based Assessment Recommendations for Pediatric Bipolar Disorder

Abstract

Bipolar disorder is frequently clinically diagnosed in youths who do not actually satisfy DSM-IV criteria, yet cases that would satisfy full DSM-IV criteria are often undetected clinically. Evidence-based assessment methods that incorporate Bayesian reasoning have demonstrated improved diagnostic accuracy, and consistency; however, their clinical utility is largely unexplored. The present study examines the effectiveness of promising evidence-based decision-making compared to the clinical gold standard. Participants were 562 youth, ages 5-17 and predominantly African American, drawn from a community mental health clinic. Research diagnoses combined semi-structured interview with youths’ psychiatric, developmental, and family mental health histories. Independent Bayesian estimates relied on published risk estimates from other samples discriminated bipolar diagnoses, Area Under Curve=.75, p<.00005. The Bayes and confidence ratings correlated r_s =.30. Agreement about an evidence-based assessment intervention “threshold model” (wait/assess/treat) had K=.24, p<.05. No potential moderators of agreement between the Bayesian estimates and confidence ratings, including type of bipolar illness, were significant. Bayesian risk estimates were highly correlated with logistic regression estimates using optimal sample weights, r=.81, p<.0005. Clinical and Bayesian approaches agree in terms of overall concordance and deciding next clinical action, even when Bayesian predictions are based on published estimates from clinically and demographically different samples. Evidence-based assessment methods may be useful in settings that cannot routinely employ gold standard assessments, and they may help decrease rates of overdiagnosis while promoting earlier identification of true cases.

Bipolar disorder is a high stakes diagnosis. Untreated, it often follows a recurrent and progressively worsening course, with increased risk of substance use, incarceration, and death by suicide or other causes (Goodwin & Jamison, 2007). If bipolar is misdiagnosed as depression or attention-deficit/hyperactivity disorder, there are delays initiating effective mood stabilizing treatments (Kowatch, Fristad, et al., 2005). There also are concerns that the options selected instead could worsen the course of the bipolar disorder (Akiskal, et al., 2003; Altshuler, et al., 1995; cf. Carlson, 2003; Hirschfeld, et al., 2002; cf. Joseph, Youngstrom, & Soares, 2009; Scheffer, Kowatch, Carmody, & Rush, 2005). Earlier identification may lead to better outcomes (DelBello, Adler, Whitsel, Stanford, & Strakowski, 2007; Findling, et al., 2003; Fristad, Verducci, Walters, & Young, 2009), delay or prevent relapse (Findling, et al., 2007), and afford more titrated or benign interventions (Miklowitz & Chang, 2008). The median lag exceeds five years between when mood symptoms begin to cause problems versus when a clinician arrives at a bipolar diagnosis in youths (Marchand, Wirth, & Simon, 2006) as well as adults (Hirschfeld, Lewis, & Vornik, 2003; Lish, Dime-Meenan, Whybrow, Price, & Hirschfeld, 1994). To close this gap, early identification has been promoted both in scholarly research (Miklowitz & Chang, 2008; Youngstrom, Findling, Youngstrom, & Calabrese, 2005) and in the popular media (Kluger & Song, 2002; Papolos & Papolos, 2002).

There is concern that the pendulum may have swung too far in favor of early diagnosis, though (Healy, 2006; Parens, Johnston, & Carlson, 2010). Clinical diagnoses of pediatric bipolar disorder (PBD) have risen steeply in the U.S. (Blader & Carlson, 2007), with some estimates showing a 40-fold increase in diagnoses over the last decade (Moreno, et al., 2007). Although evidence supports the validity of research diagnoses of bipolar disorder in terms of phenotypic, genetic, structural, functional, neurocognitive, and treatment response similarities to bipolar disorder in adults (Geller & Luby, 1997; Youngstrom, Birmaher, & Findling, 2008); and longitudinal studies are demonstrating considerable developmental continuity (Birmaher, et al., 2009; Geller, Tillman, Bolhofner, & Zimerman, 2008), the crucial issue from a public health and consumer point of view is whether the research diagnoses and clinical diagnoses identify the same cases.

Clinical diagnosis itself is a complex enterprise, fraught with potential biases and sources of error (Garb, 1998). Clinicians tend to rely on unstructured interviews and observations that have proven problematic (Neisworth & Bagnato, 2004). Clinicians also are prone to cognitive biases that often result in suboptimal diagnostic decisions (Croskerry, 2002; Galanter & Patel, 2005). More than 100 studies over the last 50 years have consistently demonstrated that clinical judgment is no better than simple actuarial algorithms, and often significantly worse (Ægisdóttir, et al., 2006; Grove, Zald, Lebow, Snitz, & Nelson, 2000; Meehl, 1954).

The situation appears even worse with regard to clinical diagnoses of bipolar disorder. A recent meta-analysis comparing clinical diagnoses to those based on structured interviews found that bipolar disorder was an outlier even compared to the general trend of mediocrity in diagnostic reliability, with a kappa of less than .10 (Rettew, Lynch, Achenbach, Dumenci, & Ivanova, 2009). Unfortunately, it is also very difficult to assess because of the phenomenology of the illness (Bowring & Kovacs, 1992; Youngstrom, Findling, et al., 2005) as there is substantial overlap in symptomatology (Leibenluft, Charney, Towbin, Bhangoo, & Pine, 2003; Weller, Danielyan, & Weller, 2004) making it hard to tease apart bipolar symptoms from symptoms of more prevalent diagnoses, such as Attention Deficit Hyperactivity Disorder (ADHD). In addition, comorbidity complicates diagnosis, as youths frequently meet criteria for multiple psychiatric disorders (Axelson, et al., 2006; Kowatch, Youngstrom, Danielyan, & Findling, 2005). Consequently, a “typical” presentation of bipolar in isolation is rare, and clinicians may focus on the comorbid condition and neglect PBD (Youngstrom, Findling, et al., 2005). Further, varied presentations of PBD threaten the reliability of diagnostic impressions. For example, classic bipolar I disorder can present as florid mania, severe depression, a mixed state, or as normal functioning, depending on the mood state.

In addition to PBD being arguably one of the most difficult diagnoses to make correctly, broader practice issues of training, burden, and reimbursement can make research diagnostic instruments, such as structured and semi-structured interviews, impractical for use in many clinical settings. As a result, many common assessment methods are not evidence-based (EB). Thus, the difficulty diagnosing PBD is compounded by complex presentation of illness, contextual constraints common to real-world clinical practice, and clinicians’ cognitive vulnerabilities.

Clinical diagnoses of bipolar disorder in adults also have waxed and waned in popularity, without any evidence of a corresponding change in the base rate of the condition (Zuckerman, 1999). There is debate about whether the contemporary rise in clinical diagnoses is a correction from previous underdiagnosis—similar to historical changes in patterns of diagnosis for depression (Kovacs, 1989), versus being evidence of overdiagnosis, perhaps due to nonspecific use of the “bipolar” label to include a broad group of youths who have marked irritability and aggression, but may not otherwise meet criteria for mania (Leibenluft, et al., 2003). Overall, it appears many youth diagnosed with bipolar do not actually have the disorder while many true cases of bipolar go undiagnosed (Ghaemi, Sachs, Chiou, Pandurangi, & Goodwin, 1999).

Pediatric Bipolar Disorder and Evidence-Based Assessment

Clinical decision-making could benefit from evidence based (EB) assessment and decision-making strategies in diagnosing high-stakes conditions such as PBD. An alternative to traditional clinical assessment is taking an actuarial approach that quantifies the risk of illness using Bayesian reasoning. Moreover, Jaeschke, Guyatt, & Sackett (1994) recommend using a “nomogram” as a simple, practical method for combining information about risk with the “diagnostic likelihood ratios” associated with test results or other clinical findings. Nomograms are charts scaled in a way that eliminates the need for multiplication or division while working with probabilities. The nomogram can correctly combine information (i.e., base rate, familial risk, and test score) into consistent (less spread in opinion), unbiased (neither systematically over- or under-estimating risk), and efficient (using a parsimonious amount of information to arrive at the posterior probability) estimates (Jenkins, Youngstrom, Washburn, & Youngstrom, 2011). This estimate, the Bayesian posterior probability, can be used in the assessment of PBD to determine the likelihood that a youth has PBD, and to guide next steps in clinical care (Youngstrom, Freeman, & Jenkins, 2009). An example of what a nomogram looks like and information about appropriate times for using this tool are provided in the Appendix (Jenkins, et al., 2011; Youngstrom & Duax, 2005, provide worked examples).

Although the nomogram is central in EB medicine (Guyatt & Rennie, 1993; Straus, Richardson, Glasziou, & Haynes, 2005), it is not widely utilized in mental health. In two recent special issues devoted to EB assessment, the guest editors (Mash & Hunsley, 2005) and one of the invited commentaries (McFall, 2005) advocated for the adoption of this type of framework, but only three papers mentioned related approaches (such as Receiver Operating Curves and diagnostic likelihood ratios) and only one described the nomogram approach. Recent evidence suggests that taking a Bayesian approach can significantly increase diagnostic accuracy and consensus among community clinicians as well as significantly decrease overdiagnosis of PBD (Jenkins, et al., 2011). Specifically, clinicians who learned how to use a nomogram (i.e., training took less than 30 minutes) and then participated in a clinical vignette exercise (i.e., testing proficiency in skill post-training) were less likely to overestimate the risk of PBD and improved in consistency and accuracy (Jenkins, et al., 2011). Despite the fact that the actuarial strategies are not widely used in mental health settings, they have started gaining attention.

A second contribution of EB medicine is providing a framework for mapping the assessment results onto a decision-making “threshold model.” The threshold model maps the probability estimate onto a continuum with two major thresholds: the Wait-Test Threshold, and the Test-Treat Threshold. Probabilities lower than the Wait-Test Threshold are sufficiently small that the diagnosis is considered “ruled out” and diagnostic assessment is discontinued. The clinician adopts a “wait and see if anything changes” approach. If the probability rises above the Test-Treat Threshold, then the diagnosis is treated as if present, and treatment begins. In between the two thresholds is the zone where intensive assessment is indicated, until the additional information pushes the probability below the Wait-Test or above the Test-Treat Threshold. Using actuarial methods, like the nomogram, in conjunction with an EB intervention “threshold model” allows clinicians to incorporate patient preferences into the decision-making process, by negotiating adjusted threshold points (Straus, et al., 2005). The nomogram can also help anchor the likelihood of a bipolar diagnosis and inform next steps in clinical care (Youngstrom, et al., 2009, provide a worked example). It is possible that taking this approach can help avoid unnecessary assessment batteries (i.e., cost-benefit analysis) as well as protect against prematurely ruling out bipolar disorder.

Evidence-based approaches to assessment could help considerably. Improving the accuracy of diagnoses can simultaneously reduce the number of false positives, averting the overdiagnosis of bipolar disorder, while also diminishing the delay between onset and correct identification. At present, several different measures had demonstrated diagnostic validity in research samples, but it is unknown how well the performance of these tools would generalize to new settings. Psychometric performance tends to “shrink” when cross-validated in new samples, and the amount of shrinkage can change depending on new sample characteristics. In addition, the Bayesian approach recommended by EB medicine involves some simplifications, such as using scoring thresholds, which may also change performance across samples.

Aims of the Study

Although the current research on actuarial strategies is encouraging (Jenkins, et al., 2011; Youngstrom, et al., 2009), to date no research has examined the degree to which the published actuarial estimates from one sample generalize to new samples. A crucial next step is to examine how well the EB assessment recommendations in the literature might extrapolate to new clinical settings. In the present study, we tested four main hypotheses to examine the effectiveness of actuarial methods for assessing real-world cases with potential PBD. First, we predicted a positive relationship between Bayesian estimates (actuarial estimates of having PBD based on the combination of family history and test score on the Parent General Behavior Inventory) and research diagnoses as well as best estimate clinical probabilities. Using the Bayesian estimates to predict diagnoses provides one means of estimating “shrinkage” or reduced validity when applying the weights in a new sample. Second, we predicted a positive relationship between Bayesian estimates and empirical estimates based on logistic regressions using the same predictors as the Bayesian model. The clinical ratings provide a second criterion against which to measure the extent to which a simple algorithm maps onto expert evaluation using additional sources of information, and the regression estimates provide a criterion for quantifying the “shrinkage” when using published weights compared to weights that are statistically optimal for the new sample.

Third, we anticipated that applying an EB assessment intervention “threshold model” (wait/assess/treat) to clinical ratings and Bayesian estimates would show clinically significant agreement between the two methods (e.g., Cicchetti, et al., 2006). Fourth, we examined potential moderators of agreement between the clinical and actuarial approaches. We predicted that type of bipolar would statistically moderate agreement between the two approaches.

Method

Participants

Youth participants (N=562) were a consecutive case series recruited from an urban community mental health center with four urban sites (Youngstrom, Youngstrom, & Starr, 2005) as part of a larger project (R01 MH066647, PI: E. Youngstrom). The majority (80%) of youth were enrolled by their mothers to participate in the study. See Table 1 for youth participant demographics and diagnostic characteristics.

Table 1.

Participant Demographic and Diagnostic Characteristics (N = 562)

Characteristic	n (% of full sample)	% Within Bipolar Subset
Caregiver
Marital Status
Single Parent (Never Married)   Married   Separated/Divorced   Widowed	326 (58%) 112 (20%) 96 (17%) 28 (5%)	59% 25% 4% 9%
Education
Partial High School   High School Graduate or GED   1-3 Years College or Trade School   4 Year Degree   Some Graduate School	163 (29%) 197 (35%) 169 (30%) 28 (5%) 17 (2%)	30% 30% 36% 3% 1%
Reported Annual Income
< $5,000   $5,000 to $14,999   $15,000 to $29,999   $30,000+	174 (31%) 214 (38%) 124 (22%) 51 (9%)	25% 41% 23% 11%
Youth
Gender: Male	341 (61%)	59%
Average Age	10.6 (SD = 3.4)	10.5 (SD = 3.7)
Race/Ethnicity
Black	495 (88%)	80%
White	37 (7%)	9%
Hispanic	8 (1%)	4%
Other	22 (4%)	7%
Average # of Axis I Diagnoses	3.8 (SD = 1.7)	4.7 (SD = 1.6)***
LEAD Diagnoses on Bipolar Spectrum	72 (13%)	--
Bipolar I	16 (3%)	--
Bipolar II	7 (1%)	--
Cyclothymic Disorder	22 (4%)	--
Bipolar NOS	27 (5%)	--
LEAD Diagnoses All Not Bipolar Spectrum
Unipolar Depression (MDD, dysthymia, adjustment-depressed mood)	168 (30%)	--
ADHD or disruptive behavior without mood disorder	272 (48%)	--
Residual (anxiety, PTSD, psychotic disorders, no Axis I)	49 (9%)	--
Any ADHD (no Bipolar diagnosis)	315 (64%)	74%
Global Assessment of Functioning Score	52.7 (SD = 8.4)	47.0 (SD = 6.2)***

^***Bipolar group different from rest of sample p < .0005, two-tailed; all other comparisons between bipolar and non-bipolar cases not significant, uncorrected p value > .05.

Youths were included if they were between 5 years 0 months and 17 years 11 months of age and both the youth and the primary caregiver were available for the assessment. Youth were excluded if they (or their respective caregivers) could not communicate orally at a conversational level in English to complete the interview, had a pervasive developmental disorder, or there was suspected moderate, severe, or profound mental retardation. The same assessment procedures were administered to all eligible participants.

Measures

Reference Standard: Semistructured Diagnostic Interview Using the Schedule of Affective Disorders and Schizophrenia for Children (KSADS)

The Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime (KSADS-PL) (Kaufman, et al., 1997) combined with the mood disorders module from the Washington University KSADS (WASH-U-KSADS; Geller, Zimerman, et al., 2001) was administered to all participants and their families. The KSADS is the most widely used semi-structured diagnostic procedure for investigations of PBD (Nottelmann, et al., 2001). Bipolar I (BP-I), Bipolar II (BP-II), cyclothymic disorder, and Bipolar Not Otherwise Specified (BP-NOS) diagnoses were made in accordance with DSM-IV diagnostic criteria (American Psychiatric Association, 2001) including a strong emphasis that mood symptoms needed to represent a clear change in functioning, and follow an episodic presentation. The most frequent reason for diagnosing BP NOS was failure to meet strict DSM duration criteria, requiring four days for a hypomanic episode and seven days or hospitalization for mania or mixed episodes (American Psychiatric Association, 2001; Leibenluft, et al., 2003), consistent with emerging data about the duration of mood episodes in clinical and epidemiological samples in youths and adults (see Youngstrom, 2009, for review). Research assistants received extensive training prior to administering KSADS (K>.85 at item level on ten cases) (Youngstrom, Meyers, et al., 2005).

Mini International Neuropsychiatric Interview

(MINI) (Sheehan, et al., 1998). The MINI is a brief (about 20 minutes) structured, reliable diagnostic interview covering common DSM-IV and International Classification of Diseases (ICD-10) psychiatric disorders. The MINI has been validated against the Structured Clinical Interview for DSM diagnoses (SCID-P) (Lecrubier, et al., 1997; Otsubo, et al., 2005; Sheehan, et al., 1998) and the Composite International Diagnostic Interview (CIDI; Kadri, Agoub, Gnaoui, & Alami, 2005; Lecrubier, et al., 1997) in several languages. It was consistently faster than both other interviews.

The primary caregivers completed the MINI first about themselves, and then repeated the interview focusing on first degree relatives of the youth proband; using a modified family History – Research Diagnostic Criteria format (Andreasen, Endicott, Spitzer, & Winokur, 1977).

Parent General Behavior Inventory

(PGBI; Youngstrom et al., 2001). The PGBI has parents of youths 5 to 18 years old rate the severity of depressive, manic, and mixed mood symptoms using a 0 to 3 Likert scale. The Hypomanic/Biphasic Scale has 28 items, alpha of .93 in this sample. Higher scores on the PGBI indicate a higher probability that a given youth has a bipolar disorder. The present study uses risk estimates (Straus, et al., 2005) associated with PGBI scores based on a completely independent sample with higher average SES and more mood disorders in the referral stream, published in Youngstrom et al. (2004).

Procedure

Clinical Assessment

All study procedures were approved by the Institutional Review Boards of University Hospital of Cleveland, Case Western Reserve University, and Applewood Centers. Parents or guardians provided written consent and all youths gave written assent. Highly trained research assistants administered the Kiddie Schedule for Schizophrenia and Affective Disorders (KSADS; Kaufman, et al., 1997) diagnostic interview to all participant families.

Parents completed a series of questionnaires while youths were interviewed, including the PGBI. Similarly, youths completed questionnaires while the parents completed the KSADS and family history interviews. KSADS interviewers were blind to rating scales.

After the client and his/her family completed the psychiatric evaluation, research diagnoses were reached through a Longitudinal Expert Evaluation of All Data (LEAD; Spitzer, 1983) conference. Specifically, all cases were reviewed by an expert consensus team, which always consisted of at least one licensed psychologist in addition to the rest of the members of the interview team for the given family. LEAD diagnoses were based on: (1) results from the KSADS; (2) developmental history; (3) family history of mental illness; and, (4) psychiatric history, including any current diagnoses. Clinical chart reviews often provided information regarding youths’ developmental and psychiatric histories. For each diagnosis, the expert clinician assigned a confidence rating based on how likely he/she viewed the diagnosis given all of the available information. These diagnoses and corresponding confidence ratings (likelihood of illness from 0-100%) represent the closest criterion to a gold standard in clinical assessment that field currently has; these ratings, the LEAD confidence ratings, were compared to the Bayesian estimates.

Actuarial Assessment

Procedures for actuarial assessments involved four steps which are described in more detail below: (1) Determine the prevalence or starting base rate for the study population; (2) Obtain family history of bipolar illness and test scores on the PGBI; (3) Translate family history of bipolar illness from the MINI and test scores on the PGBI for these cases into diagnostic likelihood ratios (DLRs) using published estimates from prior samples and research (Youngstrom, Frazier, Demeter, Calabrese, & Findling, 2008; Youngstrom, Meyers, et al., 2005); (4) Use Bayesian methods to combine the base rate of PBD and the DLRs for family history and PGBI scores to provide the probability of a bipolar diagnosis (i.e., the Bayesian estimate). The Bayesian estimates were probabilities that theoretically could range from 0 to 100, and in the present sample ranged from 0.004 to .746. These were the Bayesian estimates of the likelihood of a bipolar diagnoses based on combing the likelihood rations for prevalence rate (6%), family history of mood disorder, and test score on the Parent General Behavior Inventory (PGBI) questionnaire.

Prevalence

The present study estimated prevalence rates for PDB on the basis of a literature review, as recommended in EB medicine books (Guyatt & Rennie, 2002; Straus, et al., 2005), giving precedent to well-conducted research over local estimates. A recent meta-analysis of 12 epidemiological studies found an average rate of 2% for bipolar disorder in youths under age 19 (Van Meter, Moreira, & Youngstrom, 2011). In outpatient clinical populations, evidence suggests prevalence estimates between 0.6 to 15%, depending on the diagnostic instrument, clinic specialization, and referral source (Geller, Craney, et al., 2001; Lewinsohn, Klein, & Seeley, 1995; Strober, Schmidt-Lackner, Freeman, Bower, & al, 1995). Because prevalence of bipolar disorder varies substantially by type of setting, it is important to consider the starting base rate in light of clinical context. To determine the starting base rate for the present study, we used benchmarks from the literature that approximated local conditions (e.g., Youngstrom, 2007; Youngstrom, et al., 2009, Table 2) --similar to how clinicians could access base rate information if they were to take an actuarial approach in their clinical practice. Two published estimates indicated a base rate of 6% for bipolar spectrum disorders in an outpatient clinic, which by some standards is conservative (Pavuluri, Birmaher, & Naylor, 2005).

Table 2.

Regression model examining whether bipolar subtype moderates agreement between Bayesian estimates and LEAD ratings (N=562)

Predictor	B	SE	p	Part r
Bayesian Estimate	25.4	4.8	.000	.14
LEAD Diagnostic Status
BP-I Yes or No	78.9	7.3	.000	.29
BP-II Yes or No	82.6	10.7	.000	.21
Cyclothymic Disorder Yes or No	78.1	6.0	.000	.34
Interaction Terms (Diagnosis * Actuarial Bayesian Estimate)
BP-I	-11.4	23.2	.622	-.01
BP-II	-27.4	32.5	.399	-.02
Cyclothymic Disorder	-34.8	18.5	.061	-.05
Constant	2.2	0.9	.015

Note. The significance of the interaction terms was the direct test of the hypothesis; none were significant, p > .05. Overall model R² = .61, adjusted R² = .61, p < .0005.

Diagnostic Likelihood Ratios (DLRs)

A DLR quantifies the extent to which new information, such as a test result or risk factor, changes the odds of a diagnosis. A DLR divides the percentage of cases with bipolar showing that result by the percentage of cases without bipolar that would also have similar test results. This is identical to dividing the diagnostic sensitivity by the false alarm rate (Pepe, 2003). Family history and PGBI test scores can be translated into DLRs and plotted on the middle column of a probability nomogram.

For the present study, we conducted a search using the terms “pediatric bipolar disorder” and “sensitivity and specificity” and “diagnostic likelihood ratios” to find the DLRs for PGBI test scores, and “bipolar disorder” and “risk estimate” and “systematic review” to find DLRs for family history of bipolar disorder. See Youngstrom et al. (2004) for the DLRs associated with test scores on the PGBI.

The present investigation concentrated on direct interview information about first degree relatives. For a biological parent (or other first degree relative) with bipolar disorder, the EB familial risk estimate = 5 (Hodgins, Faucher, Zarac, & Ellenbogen, 2002; Tsuchiya, Byrne, & Mortensen, 2003; Youngstrom & Duax, 2005).

Results

Less than 2% of data points were missing. Given the small amount, missing data were excluded listwise. This approach provides less bias than pairwise deletion and is adequately suited for small amounts of missing data (Allison, 2002).

Agreement between LEAD Diagnoses, Confidence and the Bayesian Estimates

Each case received a Bayesian estimate based on DLRs for their age group on the PGBI as well as family history. Receiver Operating Characteristic (ROC) analyses comparing Bayes estimates to the LEAD diagnoses of bipolar disorder found an Area Under the Curve (AUC) of .75, p<.00005. This indicated a marginal degree of shrinkage from the diagnostic efficiency of the PGBI, with AUCs of .83 in the original report (Youngstrom, et al., 2004) and .78 in the present sample. Bayesian estimates based on the combination of family history and test score on the PGBI using a base rate of 6% correlated r_s = .30 with the LEAD confidence ratings, p<.00005, indicating a medium positive relationship (Cohen, 1988). This suggests that clinicians agree with the Bayesian estimate more often than is typically the case comparing clinical versus structured diagnoses of bipolar disorder (Rettew, et al., 2009). On the other hand, these are not redundant or identical pieces of information, as the LEAD confidence rating also integrated other information not included in the Bayesian estimate.

Generalizability of Actuarial Approach

Pearson’s correlation quantified how well Bayesian estimates using independent, published DLRs generalized to logistic regression estimates using optimal weights for the sample. The two estimates were highly correlated, r = .81, p<.0005, indicating a strong positive relationship (Cohen, 1988); see Figure 1.

Figure 1.

Generalizability of the nomogram estimates based on published likelihood ratios compared to logistic regression estimates optimized for new sample (N=562).

Agreement about Next Clinical Action

Cohen’s kappa coefficient tested if applying an EB assessment intervention “threshold model” (wait/assess/treat) to LEAD confidence ratings and Bayesian estimates showed clinically significant agreement between the two assessment methodologies. Results indicate a κ = .24, p<.0005, when test-wait threshold and test-treat thresholds were set at 20% and 90%, respectively, indicating “fair” agreement (per Cicchetti, 2006).

Figure 2 superimposes the Wait-Test and Test-Treat Thresholds on the scatter plot displaying Bayesian estimates and LEAD confidence ratings. The Bayesian estimates were generally more conservative, almost never crossing the treatment threshold, and often falling in the low or indeterminate ranges for cases where the LEAD confidence ratings were high. This pattern suggests: (a) the Bayesian approach by itself would not be sufficient to initiate treatment, at least not with the current set of inputs; (b) additional information captured in the LEAD process may be necessary to cross the treatment threshold; and, (c) the Bayesian approach is unlikely to generate “false positives” that would be exposed to unwarranted treatments.

Figure 2.

Agreement between clinician confidence ratings and Bayesian nomogram estimates based on published likelihood ratios.

Note: The data are skewed for the clinician confidence ratings. The distribution is zero-inflated, with 0 being the modal response because the clinicians assigned a low probability of bipolar disorder to the majority of the cases.

Potential Moderators of Agreement between Nomogram and Clinical Confidence

Clinician confidence was significantly higher for cases with bipolar I or II (M = 84.4) versus the softer spectrum (M = 76.7), t(60.60) = 3.12, p = .003. An OLS regression approach tested whether type of bipolar moderated agreement between LEAD confidence ratings and Bayesian estimates. Specifically, the Bayesian predicted the LEAD confidence rating, along with dummy codes for bipolar type, and interaction terms for bipolar type with Bayesian estimates. None of the interaction terms were significant, indicating that type of bipolar did not statistically moderate agreement between LEAD confidence ratings and Bayesian estimates; see Table 2.

Discussion

The overarching goal of the present study was to compare the current gold standard for clinical assessment of pediatric bipolar disorder -- a LEAD diagnosis integrating a KSADS interview with collateral information and treatment history -- to an innovative actuarial approach. We also examined agreement about next clinical action using the threshold model developed in EB medicine (Straus, et al., 2005). Additional analyses examined potential moderators of agreement between the Bayesian and clinical approaches, including whether agreement was higher for fully syndromal cases (i.e., bipolar I) versus other bipolar spectrum presentations.

Consistent with hypotheses, Bayesian estimates derived from published risk estimates showed clinically meaningful diagnostic efficiency even when generalized to a new sample with different clinical and demographic characteristics. The AUC for the Bayesian estimates shrank compared to the original published estimates, but still remained large (AUC .75) and highly significant. Also as hypothesized, LEAD confidence ratings and Bayesian estimates—based on a much more circumscribed set of variables--showed medium-sized correlation. Clinician confidence integrated substantially more information via a LEAD process: clinicians’ LEAD confidence ratings reflect findings from the KSADS interview, detailed family history, and clinical chart information. The relationship between LEAD confidence ratings and Bayesian estimates would likely differ if clinicians did not have this additional information (see Jenkins et al., 2011). Given that LEAD confidence ratings tended to be higher than Bayesian estimates, clinicians may have been more confident in their bipolar diagnoses because of the additional “supporting” information from the KSADS.

As hypothesized, Bayesian estimates using published estimates were highly correlated with logistic regression estimates optimized for the present sample, indicating a high degree of generalizability. These findings are enhanced by the fact that the present sample is substantially different in terms of demography and SES as well as clinical referral patterns from the prior research on the assessment of PBD (cf. Youngstrom, et al., 2004). Whereas most prior work has relied on middle class, predominantly white participants with high rates of mood disorder seeking services in specialty clinics at academic centers (Hodgins, et al., 2002), the present sample was low income, predominantly underserved ethnic minority families seeking services at a community mental health center, mostly for attention problems and disruptive behavior disorders. The high correlation between new regression estimates and Bayesian estimates using published weights provides strong confirmation that the EB assessment recommendations generalize across a wide range of demographic and clinical facets.

When a clinical action threshold model was applied to LEAD confidence ratings and Bayesian estimates, these different approaches evidenced fair agreement (Cicchetti, et al., 2006; Landis & Koch, 1977). When they disagreed, the actuarial approach consistently recommended a more conservative approach--waiting or active assessment--rather than indicating active treatment. This finding has important clinical implications. Gold standard clinical assessments are often not feasible in real-world practice (Garb, 1998). For example, issues related to insurance reimbursement and staff training can make it difficult to implement a comprehensive assessment such as the KSADS (Eisman, et al., 1998). Moreover, many clinical settings lack the training and supervision resources to ensure acceptable administration of semi-structured diagnostic interviews, which can result in unreliable diagnostic impressions and inappropriate treatment plans. Results suggest that the actuarial approach could be implemented using published likelihood ratios, rapidly and inexpensively identifying cases for further assessment without overdiagnosing or over-treating bipolar disorder. There also is precedent for third party payors such as Medicaid reimbursing for KSADS or other follow-up procedures when they are clinically indicated rather than being administered to all comers.

Contrary to study predictions, the type of bipolar did not significantly moderate the level of agreement between LEAD confidence ratings and Bayesian estimates (i.e., the interaction terms were not significant). This finding is surprising given that clinician confidence was significantly higher about “fully syndromal” bipolar I or II versus “softer spectrum” cyclothymic or bipolar illness cases, consistent with findings from vignette studies (Dubicka, Carlson, Vail, & Harrington, 2008). Clinicians are less confident when diagnosing bipolar that does not fit into current categorical definitions of the disorder, consistent with current scientific debate about the diagnostic status of the “NOS” cases (e.g., Axelson, et al., 2006; Dubicka, et al., 2008; Findling, 2005; Leibenluft, et al., 2003; Parens, et al., 2010). Apparently, type of bipolar may affect clinicians’ diagnostic confidence; but it does not influence the degree to which actuarial and clinical assessment methodologies agree.

It is noteworthy that the present sample was predominantly African American. Given the well-documented difficulties assessing bipolar using other methods in minority samples -- especially African Americans -- (DelBello, Lopez-Larson, Soutullo, & Strakowski, 2001; Neighbors, Trierweiler, Ford, & Muroff, 2003), the effectiveness of the actuarial approach is all the more impressive. Actuarial assessment methods may help decrease the rate at which clinicians misdiagnose African Americans and other minority groups by encouraging more systematic and thorough assessment, as well as preventing or adjusting faulty clinical judgment or differences in initial description of the presenting problem (Jenkins, et al., 2011). For example, if an individual presents with a family history of mental illness and/or receives a positive test score on a bipolar screening instrument, the actuarial approach generates the same probability of bipolar disorder regardless of race/ethnicity. If there were evidence that demographic characteristics moderated risk or test performance, then the actuarial approach could incorporate optimal weights to make the appropriate adjustments.

Limitations

A large portion of the bipolar spectrum cases had diagnoses of cyclothymic disorder and bipolar NOS. By definition, these cases did not present with distinct episodes of week-long mania or sufficient severity to require hospitalization. However, the research diagnoses required an episodic presentation and that symptoms be a clear change in functioning if they were counted towards a mood diagnosis. The majority of cases with NOS or cyclothymic diagnoses failed to meet criteria for bipolar I or II due to insufficient duration of the activated mood states, but virtually all had multiple episodes, and a mean episode duration well in excess of other research definitions (Birmaher, et al., 2006). The emphasis on episodicity and differentiating mood from other possible sources of symptoms increases the specificity of the bipolar diagnoses. It also is important to understand how findings generalize across the bipolar spectrum, as the cyclothymic and NOS cases are proving to be more frequent in epidemiological studies (Merikangas, et al., 2010; Van Meter, et al., 2011) as well as clinical settings (Axelson, et al., 2006), and continue to be associated with high degrees of impairment (Birmaher, et al., 2009; Findling, et al., 2010; Findling, et al., 2005).

It is possible that the conservative approach used in defining risk due to family history of bipolar illness may have lowered estimates of agreement. Some research suggests that individuals with familial probands of bipolar disorder may be as much as ten times more likely to manifest the illness (Smoller & Finn, 2003). Using a smaller likelihood ratio for family risk contributed to the Bayesian approach yielding lower risk estimates. This is not a general limitation of the EB assessment approach: In fact, a strength is that the EB assessment methods can flexibly integrate new information. Also, clinicians have the option of performing “sensitivity analyses,” where they examine the effects of changing the inputs on the range of resulting probabilities. These “what if” scenarios inform the clinician about how changes to assumptions, or the addition of new evidence, would change diagnostic probabilities.

Another caveat is that family history information sometimes was inaccurate or incomplete. Most family history information was gathered from participants’ mothers’ using a semi-structured interview. Collateral report, or a family history method, is less accurate than direct interview of all relatives. The rate of bipolar disorder reported in the other family members was lower than the rates found in epidemiological studies in the general U.S. population (Merikangas & Pato, 2009), suggesting that the interview did not demonstrate high diagnostic sensitivity to bipolar conditions in fathers or second degree relatives. However, these same factors of low awareness of bipolar history in other family members, along with potential bias, are likely to apply to most clinical evaluations as well.

Some of the results are also dependent on starting values, such as the base rate of bipolar disorder or the choice of values for the Wait-Test and Test-Treat thresholds. This is intrinsic to using a Bayesian approach, but it is likely to be disconcerting at first because it seems both unfamiliar and potentially subjective (Gigerenzer & Goldstein, 1996). In fact, the Bayesian methods are often more consistent, efficient, and unbiased than clinical judgment in synthesizing information (Jenkins, et al., 2011), and these advantages remain even when using inaccurate starting values. Additionally, EB assessment methods have will be self-correcting over time: Using the techniques will change diagnostic patterns to converge on the true base rate even when starting from inaccurate values (Meehl, 1954). The potential for subjectivity in choice of thresholds also is a potential strength, as it makes it possible to have meaningful discussions about costs and benefits with the patient and family (Kraemer, 1992; Straus, et al., 2005; Swets, Dawes, & Monahan, 2000). Finally, clinician confidence is not completely equivalent to clinical judgment as operationalized in some other studies (cf. Meehl, 1954).

Future Directions and Clinical Implications

Dissemination efforts are more successful when procedures are acceptable, feasible, and likely to permit adherence (Arndorfer, Allen, & Aljazireh, 1999). Better understanding of clinician perspectives about actuarial assessment along with identifying potential practice barriers can enhance acceptance, feasibility, and adherence. Before attempting to create large scale change in current decision-making practice, researchers should first pilot EB tools with providers. It will be important to understand if clinicians on the front line are willing to adopt actuarial approaches in real-world practice. Recent research suggests that a majority of MH professionals who received a brief nomogram training (< 30 minutes) endorsed using it in practice (Jenkins, et al., 2011). Qualitative methodologies may shed light on clinicians’ attitudes and impressions that can inform future education and training of the nomogram.

More work needs to evaluate barriers related to more general policy considerations (e.g., supervision, clinical training models) as well as consumer preferences (Suppiger, et al., 2009). Documenting and addressing these obstacles could accelerate uptake of EB assessment methods. Changes in “packaging” and interpretation software may also accelerate uptake. Innovative technology combined with EB decision-making tools might greatly appeal to audiences that find statistics intimidating. Using technology to expedite the delivery of EB mental health services is a rapidly growing niche (Bucholz, et al., 1991; Erdman, et al., 1992; Finfgeld, 1999; Kobak, et al., 1997).

Another area of work will be studying the effects of varying the Wait-Test and Test-Treat Thresholds, as well as incorporating patient preferences about risks and benefits. Some of the controversy around the bipolar diagnosis has focused on the consequences of erroneously prescribed pharmacological treatment (DSM5.org, accessed January 19, 2011), rather than the underlying validity of the diagnosis, which has accumulated substantial evidence (Geller & Tillman, 2005; Youngstrom, Birmaher, et al., 2008). Psychopharmacological agents used to treat bipolar disorder carry a range of potential side effects extending from the unpleasant to the life-threatening. The perceived adversity of any treatment’s costs and side effects will naturally vary on an individual basis. More work is needed to understand how the interaction of clinicians’ recommendations and patient preferences map on to the threshold model. Adjusting the Test-Treat Threshold provides a mechanism for clinicians to directly address the potential for harm by literally setting the bar higher, requiring greater confirmatory evidence before initiating the riskiest or most expensive treatments (Straus, et al., 2005).

EB assessment offers a fast and frugal approach to gathering information about bipolar disorder in youngsters. Combined with a two-threshold (Wait-Test, Test-Treat) model, the current generation of EB assessment tools can increase detection of bipolar without increasing the rate of false positives who are then exposed to unnecessary treatments. Using brief, free/public domain tools can identify a limited number of cases for more intensive assessment and follow up. Medicaid and other payors have been found to reimburse this additional evaluation when medical necessity has been demonstrated. Upgrades in the tools and data available will further enhance the accuracy of this approach without requiring a change to the interpretive framework. Present findings also indicated that risk estimates derived from the published literature would generalize well, even to new samples with markedly different demographic and clinical characteristics. The EB approach could likely yield immediate improvements in many clinical settings, which would be further refined as new tools, risk factors, and moderators of assessment and treatment performance are identified.

Appendix.

Figure A1.

Nomogram for Combining Probability with Diagnostic Likelihood Ratios

Note. A nomogram is a particularly helpful tool for quantifying the risk of bipolar disorder when certain warning signs are present (Youngstrom et al., 2009). Warning signs can include family history of bipolar disorder, a high score on a parent report questionnaire that is sensitive to manic symptoms, and/or a youth’s clinical presentation of decreased need for sleep, elevated, expansive mood, and grandiosity, or possible psychotic features.