Development of a Composite Primary Outcome Score for Children with Attention-Deficit/Hyperactivity Disorder and Emotional Dysregulation

Abstract

Objective: Study goals were to (1) provide a rationale for developing a composite primary outcome score that includes symptom severity for attention-deficit/hyperactivity disorder (ADHD) and emotional dysregulation, plus symptom-induced impairment; (2) demonstrate weighting methods to calculate the composite score using a sample of children diagnosed with ADHD and aggression; and (3) identify the optimal weighting method most sensitive to change, as measured by effect sizes.

Methods: We conducted secondary data analyses from the previously conducted Treatment of Severe Childhood Aggression (TOSCA) study. Children aged 6–12 years were recruited through academic medical centers or community referrals. The composite primary outcome comprised the ADHD, oppositional defiant disorder, disruptive mood dysregulation disorder, and peer conflict subscales from the Child and Adolescent Symptom Inventory (CASI), a DSM (Diagnostic and Statistical Manual)-referenced rating scale of symptom severity and symptom-induced impairment. Five weighting methods were tested based on input from senior statisticians.

Results: The composite score demonstrated a larger (Cohen's d) effect size than the individual CASI subscales, irrespective of the weighting method (10%–55% larger). Across all weighting methods, effect sizes were similar and substantial: approximately a two-standard deviation symptom reduction (range: −1.97 to −2.04), highest for equal item and equal subscale weighting, was demonstrated, from baseline to week 9, among all TOSCA participants. The composite score showed a medium positive correlation with the Clinical Global Impressions-Severity scores, 0.46–0.47 for all weighting methods.

Conclusions: A composite score that included severity and impairment ratings of ADHD and emotional dysregulation demonstrated a more robust pre–post change than individual subscales. This composite may be a more useful indicator of clinically relevant improvement in heterogeneous samples with ADHD than single subscales, avoiding some of the statistical limitations associated with multiple comparisons. Among the five similar weighting methods, the two best appear to be the equal item and equal subscale weighting methods.

Introduction

Attention-deficit/hyperactivity disorder (ADHD) in youth is common and impairing (American Psychiatric Association 2013b; Polanczyk et al. 2014; Thomas et al. 2015), often leading to poor long-term outcomes, even with standard treatment (Klein et al. 2012). Assessing treatment effectiveness is complicated by the complexity and heterogeneity in ADHD symptom expression beyond the three hallmark diagnostic symptoms of ADHD: inattention, hyperactivity, and impulsivity. Given the high prevalence of emotional symptoms in ADHD, Faraone et al. (2019) propose that emotional dysregulation be considered as a fourth symptom domain for individuals with ADHD. Emotional dysregulation broadly encompasses constructs such as emotional impulsivity and deficient emotional self-regulation, characterized by lack of control over strong emotions, and high levels of emotional volatility (Faraone et al. 2019). The significant role of emotional dysregulation in ADHD is supported by investigations into temperament types, leading to a call for an irritable ADHD subtype (Karalunas et al. 2014, 2019), further supported by findings that irritability is associated with ADHD genetic risk (Nigg et al. 2020).

Together with aggression, irritability is among the most common reasons children are referred for psychiatric evaluation; it is found in more than 40% of emergency department psychiatric evaluations and more than 20% of pediatric psychiatry outpatient evaluations (Peterson et al. 1996). Clinically impairing irritability affects 25%–50% of children with ADHD (Shaw et al. 2014; Barkley 2015; Fernández de la Cruz et al. 2015; Mulraney et al. 2018; Karalunas et al. 2019). Aggression and irritability symptoms are defining characteristics of comorbid oppositional defiant disorder (ODD) and disruptive mood dysregulation disorder (DMDD) and also distinguish an ADHD presentation that is distinct from other ADHDs (Fernández de la Cruz et al. 2015; Karalunas et al. 2019).

A common problem faced by researchers who conduct randomized controlled trials (RCTs) is deciding on the optimal outcome measure sensitive to treatment response. The decision can be particularly challenging when working with clinical populations who exhibit symptoms of multiple co-occurring disorders, as in ADHD (Jensen et al. 2001; Volk et al. 2005). In the context of symptom heterogeneity, measures that examine only the hallmark diagnostic features of ADHD may fail to capture broader symptom changes such as irritable, angry, and aggressive behaviors. Another consideration was including symptom severity and impairment as components of the outcome measure. Typically, rating scales focus on severity based on the number of reported symptoms, with less attention to impairment levels (Gadow et al. 2013). While severity and impairment overlap, they are two different nonequivalent constructs (Kaat et al. 2013). To address these concerns, we used a Diagnostic and Statistical Manual (DSM)-referenced measure that captured both symptom severity and impairment ratings to accurately monitor clinical change in response to an intervention.

Although not typically used, a composite outcome measure is not novel. In the Multimodal Treatment Study of ADHD, a composite treatment outcome based on the Swanson, Nolan, and Pelham Rating Scale (Swanson et al. 2005) surpassed the sensitivity of individual subscales in detecting an overall effect in secondary analyses (Conners et al. 2001). In the same sample, another composite score combining symptoms of inattention, hyperactivity/impulsivity, and ODD showed a greater effect size than the individual subscales (Swanson et al. 2001). In both of these secondary analyses, the composite score development and its use were post hoc, in contrast to planned a priori use.

Three lines of research guided our goal to develop a holistic, composite, primary outcome score to measure change in the broad symptom domain of emotion dysregulation in children with ADHD. First, impairment and poor outcomes in children with ADHD are driven as much by emotional dysregulation (characterized by irritability and anger) as by the official defining characteristics of ADHD (inattention, hyperactivity, and impulsivity) (Gadow et al. 2014; Lenzi et al. 2018). Second, evidence is growing for an irritable subtype of ADHD that is not explained by comorbid ODD or DMDD (Karalunas et al. 2014, 2019). Third, results from an RCT of micronutrients in children with ADHD demonstrated improvements in behavioral domains of irritability, anger, and aggression and in the cognitive domain of inattention (Rucklidge et al. 2018).

We hypothesized that a composite score would strengthen the effect size and avoid the need to adjust for multiple comparisons (e.g., Bonferroni corrections) when analyzing the primary outcome between treatment groups. To develop a composite measure, we conducted secondary analyses of data from the Treatment of Severe Childhood Aggression (TOSCA) study.

This article describes the steps taken to (1) develop the composite score, inclusive of ADHD and emotional dysregulation; (2) demonstrate five weighting methods using TOSCA study data; and (3) determine the optimal weighting to detect an overall difference in effect size.

Methods

To develop a composite measure of ADHD and co-occurring emotional and behavioral dysregulation symptom severity and impairment in children with ADHD, which could also demonstrate subsequent symptom change following an intervention, we tested a continuous composite outcome score comprising five subscales from the Child and Adolescent Symptom Inventory-IV (CASI-IV): (1) Inattention; (2) Hyperactivity–Impulsivity, (3) ODD, (4) DMDD, and (5) the Peer Conflict Scales (Gadow and Sprafkin 2015). The combination of these subscale severity ratings and impairment scores into a single composite primary outcome was intended to identify change in both the severity of and impairment from a participant's symptoms.

TOSCA study

To determine the strength and magnitude of this composite outcome, data from a population of interest, the TOSCA study (Aman et al. 2014) (NCT00796302), were used to determine the appropriate weighting method for the CASI subscales that would be most sensitive to symptom change. The TOSCA study was a four-site National Institute of Mental Health-funded investigation that enrolled children aged 6–12 years, who met criteria for ADHD and disruptive mood (ODD or conduct disorder as per the Nisonger Child Behavior Rating Form, a factor-analytically derived standardized scale) (Aman et al. 2008), and who also had severe aggression resulting in physical damage to person or property (Aman et al. 2014). Study treatment included parent training for disruptive behavior plus simultaneous open-label stimulant medication for 9 weeks, representing the acute trial phase. For children whose behavior did not improve by the end of 3 weeks, either risperidone (an antipsychotic) or placebo was added from week 4 through week 9. The primary outcome of the TOSCA study was the Disruptive Total Score (combined Oppositional and Conduct Problem scores) on the Nisonger Child Behavior Rating Form. TOSCA parents completed the CASI subscales as secondary measures (Gadow et al. 2014). These data were used to demonstrate the properties of the composite outcome score and test the sensitivity of different weighting methods. Data were collected with informed consent/assent procedures and documents approved by the respective local Institutional Review Boards (Gadow et al. 2014). We included only those TOSCA participants who had data at baseline and at the end of the RCT (week 9; n = 137).

Child and Adolescent Symptom Inventory

Parents rated their children's psychiatric symptoms with the CASI (Sprafkin et al. 2002; Gadow and Sprafkin 2015). Individual items are grouped by disorders and presented at a fifth grade reading level to be easily understood by parents. Items correspond one-to-one with symptoms in the DSM of Mental Disorders (American Psychiatric Association 1994, 2013a). Symptoms were rated on a frequency of occurrence scale: 0 (never), 1 (sometimes), 2 (often), and 3 (very often). For each subscale, parents also indicated symptom impairment by answering, “How often do the behaviors in [the particular subscale] interfere with the child's social or academic functioning?” The response options were the same as above. Numerous studies indicate that CASI subscales demonstrate satisfactory psychometric properties and are sensitive indicators of response to intervention (Gadow 2016).

The CASI composite score comprised the following five symptom severity subscales, plus a score for the impairment rating in each subscale, resulting in six subscale scores.

• Subscale 1: measured inattention symptoms: nine items.

• Subscale 2: measured hyperactivity/impulsivity symptoms: nine items.

• Subscale 3: measured ODD symptoms: eight items.

• Subscale 4: measured DMDD: one item in the CASI-IV used in the TOSCA study (counted twice to approximate the CASI-5, which has two DMDD items).

• Subscale 5: measured verbal and physical aggression toward peers: nine items.

• Impairment rating measured symptom-induced impairment as a summary score pooling the impairment item response from each of the subscales.

Clinical Global Impressions-Severity

The Clinical Global Impressions (CGI) scale was developed as a global measure to reflect the clinician's overall impression of a patient's symptom severity and treatment response in the context of clinical studies (Guy 1976). The CGI-Severity (CGI-S), rated 1–7, from “normal, not at all ill” to “among the most extremely ill,” reflects the clinician's impression of a patient's illness at the time of assessment, relative to other study patients. Completed by the study clinician, the CGI-S rated participant illness severity at baseline and week 9. These ratings were compared to investigate whether one weighting method was more sensitive than another in the context of change in illness severity, irrespective of treatment allocation.

Statistical analyses

Baseline characteristics of TOSCA participants were examined with mean and standard deviation (±SD) for continuous variables and n (%) for categorical variables. To compare effect size magnitude following treatment, we examined individual CASI subscale item mean scores versus composite scores using Cohen's d, calculated as week 9–baseline, which constituted the initial study period. The scores were calculated irrespective of the treatment group to which the TOSCA subjects were assigned. The mean item score was used because it is comparable across different measures, is understood without knowing the denominator, and allows for computing of a mean score if an item score is missing.

To investigate the most sensitive weighting method, we used two standardized analytical techniques. First, we calculated a Pearson correlation coefficient to examine the correlation between change in the CGI-S and change in the CASI weighted score, week 9–baseline. Second, we used change score analysis to examine the change in the CGI-S for all participants, week 9–baseline, per SD change in the CASI weighted scores to determine if the weighting methods differed substantially from one another. All analyses were conducted using STATA 15 (StataCorp 2017).

The sensitivity of five weighting methods was used to examine change in overall item mean scores using the CASI composite score and TOSCA data as follows:

Eighty/20 weighted sum

This weighting method assigned 80% of the weight to the symptom subscales and 20% to the impairment questions. The mean of the five subscales was weighted at 80%. The mean of the impairment questions was weighted at 20%.

Seventy/30 weighted sum

This weighting method assigned 70% of the weight to the symptom subscales and 30% to the impairment questions. The mean of the five symptom subscales was calculated and weighted at 70%. The mean of the impairment questions was weighted at 30%.

Sixty/40 weighted sum

This weighting method assigned 60% of the weight to the symptom subscales and 40% to the impairment questions. The mean of the five subscales was weighted at 60%. The mean of the impairment questions was weighted at 40%.

For the 80/20, 70/30, and 60/40 weighting methods, weighted composite scores at baseline and, again, at week 9 were calculated by adding the two weighted scores.

Equal subscale weighting (1/6th)

This weighting method assigned equal weight to each of the five subscale means and to the four impairment questions combined into one mean score, making a total of six areas. Each area was assigned a weight of 1/6, 16.67%, or 0.1667 so that the total weight summed to 1% or 100%. Weighted composite scores at baseline and at week 9 were calculated by adding each of the six areas.

Equal item weighting (1/42nd)

This weighting method assigned equal weight to each item mean of the 42 items that compose the 5 subscales and four impairment questions. Inattention consisted of 9 items, Hyperactivity 9 items, ODD 8 items, DMDD 2 items, Peer Aggression 10 items, and impairment 4 items (9 + 9 + 8 + 2 + 10 + 4 = 42). Each item was assigned a weight of 2.38% or 0.0238. A weighted composite score at baseline and at week 9 was calculated by adding each of the 42 items. In other words, this was a simple item mean of the 42 items.

Results

Demographic and clinical variables

Of the 168 participants at baseline, data were available for 137 at both baseline and week 9. The average age of participants in the TOSCA study was ∼9 years; 77% were male; 58% (n = 80) were white/Caucasian; and 29% (n = 39) were black/African American. Twenty-seven percent of parents (n = 36) reported an annual family income of >$60,000 (Table 1).

Table 1.

Treatment of Severe Childhood Aggression Baseline Participant Demographic and Clinical Data (N = 137)

Demographic variables	Mean ± SD or N (%)
Age, years	8.9 ± 2.0
Male	105 (77)
Child race
AI/AN	1 (0.7)
Asian	1 (0.7)
African American	39 (28.5)
White	80 (58.4)
Other	16 (11.7)
Parent incomea
≤60,000	96 (72.7)
>60,000	36 (27.3)
Clinical variables: CASI subscales—baseline meansb
ADHD Inattentive	2.4 ± 0.6
ADHD Hyperactive	2.2 ± 0.7
ODD	2.3 ± 0.6
DMDD	1.2 ± 0.9
Peer Conflict Scale	1.5 ± 0.9
Impairment rating	1.9 ± 0.6

^aFive participants missing income data.

^bRange = 0–3; 2 = clinical range.

ADHD, attention-deficit/hyperactivity disorder; CASI, Child and Adolescent Symptom Inventory; DMDD, disruptive mood dysregulation disorder; ODD, oppositional defiant disorder; SD, standard deviation.

Individual CASI subscale effect sizes

Effect sizes (using Cohen's d) for individual CASI subscales ranged from −0.92 to −1.80. The largest effect size among the individual CASI subscales, week 9–baseline, was for inattention at −1.80 (95% CI: −2.07 to −1.53); the smallest effect size was observed in DMDD at −0.92 (95% CI: −1.12 to −0.72) (Table 2).

Table 2.

Effect Sizes for Child and Adolescent Symptom Inventory Individual Subscales: Week 9 – Baseline (N = 137)

CASI individual subscale	Effect sizea(95% CI)
ADHD Inattention	−1.80 (−2.07 to −1.53)
ADHD Hyperactivity	−1.77 (−2.04 to −1.50)
ODD	−1.75 (−2.01 to −1.48)
DMDD	−0.92 (−1.12 to −0.72)
Peer Conflict	−1.39 (−1.62 to −1.15)
Symptom-induced impairment	−1.58 (−1.83 to −1.32)

^aCohen's d.

ADHD, attention-deficit/hyperactivity disorder; CASI, Child and Adolescent Symptom Inventory; DMDD, disruptive mood dysregulation disorder; ODD, oppositional defiant disorder.

Comparison of weighting methods

The effect sizes were similar and substantial across all weighting methods, showing an approximate 2SD decrease (improvement) in mean CASI composite scores, week 9–baseline, among all TOSCA subjects, with a range of −1.97 to −2.04 (Table 3).

Table 3.

Effect Sizes for Mean Change in Weighted Child and Adolescent Symptom Inventory Scores (95% CI) (N = 137)

Weighting method	Mean change in CASI scores between week 9 and baseline	Effect sizeafor mean change in CASI scores: week 9–baseline
80/20	−0.98 (−1.07 to −0.90)	−1.97 (−2.26 to −1.69)
70/30	−0.86 (−0.93 to −0.79)	−1.97 (−2.26 to −1.69)
60/40	−0.74 (−0.80 to −0.67)	−1.97 (−2.26 to −1.69)
Equal subscale weighting (1/6th)	−1.22 (−1.32 to −1.11)	−1.97 (−2.26 to −1.68)
Equal item weighting (1/42nd)	−1.26 (−1.36 to −1.15)	−2.04 (−2.34 to −1.75)

^aCohen's d.

CASI, Child and Adolescent Symptom Inventory.

Using Pearson's r to examine the correlation between change in the CGI-S and change in the CASI weighted composite score, a positive medium-sized correlation was found in a narrow range from 0.46 to 0.47 for all five weighting methods (data not shown). Similarly, the mean change in CGI-S per SD change in CASI scores was in a narrow range: β = 0.53–0.54 units (95% CI: 0.36–0.72) for all five weighting methods (Table 4), indicating that the weighting methods produced very similar results to one another.

Table 4.

Mean Change in Clinical Global Impressions-Severity per Each Standard Deviation Change in Weighted Child and Adolescent Symptom Inventory Scores (N = 137)

Weighting method	β (95% CI)	Standard deviation change in CASI scores: week 9–baseline
80/20	0.53 (0.36–0.70)	0.50
70/30	0.53 (0.36–0.70)	0.44
60/40	0.53 (0.36–0.70)	0.37
Equal subscale weight (1/6th)	0.54 (0.37–0.72)	0.62
Equal item weight (1/42nd)	0.54 (0.37–0.71)	0.62

As would be expected, each of the weighting methods had strengths and weaknesses relative to one another, as summarized in Table 5.

Table 5.

Pros and Cons of Each Weighting Method

Weighting method	Pros	Cons
80/20 70/30 60/40	Gives impairment items greater weight than the number of items	Arbitrary assignments of weights to subscales and impairment questions
Equal subscale weight 1/6th	Weights each of the subscales, plus the impairment questions, equally for easy interpretation	Weights the subscales equally, irrespective of number of items within each subscale
	Reflects clinical importance for each of the subscales	Slightly larger variability in composite scores compared with the 80/20, 70/30, and 60/40 methods
Equal item weight 1/42nd	Weights each item within a subscale equally	Weights the subscales based on number of items, with lower weighting assigned to the subscale with fewer items
	Most intuitively interpretable	Slightly larger variability in composite scores compared with the 80/20, 70/30, and 60/40 methods
		Not based on clinical importance

Discussion

This article describes the first study to create a composite score using both symptom severity and impairment while considering emotional dysregulation as a treatment target in ADHD populations. Results supported the hypothesis that a higher magnitude of change post-treatment would be detected with a composite score than with a single subscale, given the heterogeneity of ADHD symptoms.

The composite CASI outcome score demonstrated meaningful and substantive change following treatment. In comparison with the individual subscale effect sizes ranging from −0.92 to −1.80 in the original TOSCA study (Gadow et al. 2014), the composite CASI score demonstrated an effect size of approximately −2.00 across all weighting methods (10%–55% larger) and eliminated the need for an alpha correction for multiple outcomes. The benefit of a composite score has been shown in other studies of children with ADHD (Conners et al. 2001).

In terms of weighting methods, the largest effect size (−2.04) was obtained using the equal-item weighting method, followed closely by the four other weighting methods (−1.97). Pearson's r correlations between the CGI-S and the weighted CASI composite (range: 0.46–0.47) and CGI-S by SD of CASI change (range: β = 0.53–0.54) provided further evidence for the similarity of results, regardless of the weighting method used.

This study is among the first to demonstrate the utility of a composite score to reflect the clinical reality of ADHD patient populations that emotional dysregulation is an important treatment target. Furthermore, to our knowledge, the composite score developed here is the first to combine symptom severity and impairment into a single measure. A research tool such as the DSM-referenced CASI, with its multiple domains of inquiry and dual focus on symptom severity and impairment, is therefore suitable for measuring the clinical complexity of, and change in, these impairing symptom domains.

The composite CASI score is the planned primary outcome of an RCT currently underway. The Micronutrients for ADHD in Youth (MADDY) study is an 8-week, fully blinded clinical trial recruiting children aged 6–12 years with ADHD and emotional dysregulation (Johnstone et al. 2019) to investigate a broad-spectrum micronutrient supplement (NCT03252522). The MADDY study's primary outcome, designed a priori as a composite score, comprises the five subscales from the CASI-5 (Gadow and Sprafkin 2015), as described in this article. The composite score and weighting methods tested on the TOSCA dataset will be applied to the MADDY data.

In considering which weighting method to use, the equal subscale weighting, as the name suggests, gives equal weight to each of the five subscales and to an overall impairment rating from each of these subscales. It shows a larger mean raw change than three of the five tested weighting methods, and the second largest Cohen's d effect size. The equal item weighting shows a slightly larger mean raw change and the largest Cohen's d of all. Thus, either of these weightings would be appropriate choices. If the goal was theoretical support for equal representation of each domain, the equal subscale would be a good choice. If the mathematically strongest is desired, or if ease of calculation and reader interpretability is preferred, the equal item weighting would be the right choice.

Conceptualizing emotional dysregulation as an important component of clinical presentation in some individuals with ADHD acknowledges that the associated behaviors—irritability, anger, and aggression—are prevalent and impairing symptoms, which are as important to treat as the central diagnostic hallmarks of the disorder. Including symptom-induced impairment as a scorable component, as part of an assessment or outcome measure, gives weight to behavior change that may start with an improvement in functioning before the severity or frequency of the behavior diminishes, or vice versa.

Conclusions

Use of a composite score in randomized clinical trials may prove more sensitive and robust than a single subscale, avoiding statistical limitations associated with multiple comparisons. A composite score also enables the interpretation of research results that may be applied to clinically heterogeneous populations more readily. While the five weighting methods tested on these data produced similar results, the two best appear to be equal item and equal subscale weighting.

Clinical Significance

For researchers and clinicians working with ADHD patient populations, it is important to consider emotional dysregulation as a treatment target and to use assessment tools such as the CASI, with its multiple domains of inquiry, to measure the complexity of and change in this impairing symptom domain. Identifying change in both the severity of a participant's symptoms and the associated impairment may provide a more accurate picture of treatment benefit than symptom change alone, which may aid in reconciling research outcomes with clinical utility.

Disclosures

In accordance with journal policy and their ethical obligations as researchers, the authors are declaring the following potential conflicts of interest:

Dr. Michael G. Aman has received research contracts, consulted with, served on advisory boards, or done investigator training for CogState, Inc.; CogState Clinical Trials, Ltd.; Confluence Pharmaceuticals; J & J Pharmaceuticals; MedAvante-ProPhase; Otsuka Pharmaceutical Development & Commercialization, Inc.; Ovid Therapeutics; ProPhase LLC; Supernus Pharmaceuticals; and Zynerba Pharmaceuticals. He receives royalties from Slosson Educational Publications.

Dr. L. Eugene Arnold has received research funding from Forest, Lilly, Noven, Otsuka, Shire, Supernus, Roche, and YoungLiving (as well as NIH and Autism Speaks); has consulted with Pfizer, Tris Pharma, and Waypoint; and been on advisory boards for Arbor, Ironshore, Otsuka, Pfizer, Roche, Seaside Therapeutics, and Shire.

Dr. Oscar Bukstein has received honoraria from Routledge Press, Wolters Kluwer Health, and Guilford Press.

Dr. Robert L. Findling receives or has received research support, acted as a consultant, and has received honoraria from Acadia, Aevi, Akili, Alcobra, Allergan, Amerex, American Academy of Child and Adolescent Psychiatry, American Psychiatric Press, Arbor, Bracket, Daiichi Sankyo, ePharmaSolutions, Forest, Genentech, Ironshore, KemPharm, Luminopia, Lundbeck, Merck, NIH, Neurim, Noven, Nuvelution, Otsuka, PCORI, Pfizer, Physicians Postgraduate Press, Purinix, Receptor Life Sciences, Roche, Sage, Shire, Sunovion, Supernus Pharmaceuticals, SyneuRx, Teva, TouchPoint, Tris, and Validus.

Dr. Kenneth D. Gadow is shareholder in Checkmate Plus, publisher of the Child and Adolescent Symptom Inventory.

Dr. Barbara Gracious receives or has received research support, research contracts, acted as a consultant, or received honoraria from the American Academy of Child and Adolescent Psychiatry, American Psychiatric Association, AstraZeneca, Brain and Behavior Foundation, Novo Nordisk, the Stanley Medical Research Institute, and Otsuka.

All other authors report no potential conflicts of interest.