Multiple Informant Average Integration of ADHD Symptom Ratings Predictive of Concurrent and Longitudinal Impairment

Abstract

To date, there remains no consensus about the best evidence-based method for integrating multiple informant data in the diagnosis of Attention-Deficit/Hyperactivity Disorder (ADHD). Several approaches exist, including the psychometrically-sound approach of averaging scores, as well as the use of “OR” and “AND” algorithms, which are still commonly used in research. The current study tested these major integration methods in their concurrent and longitudinal prediction of clinician-rated impairment, teacher-rated academic, and parent- and self-rated social skill ratings in children overrecruited for ADHD across a six-year span from childhood to adolescence. The sample included a total of 800 children, 480 with ADHD, ages 6 to 13, who completed a “gold standard” assessment of ADHD and associated impairment. Overall, the “OR,” “AND,” and average integration approaches showed significantly high interrelations with one another (r range from .78-.96) and were all significantly and strongly related to impairment measures concurrently and longitudinally. Multivariate regressions demonstrated that the average integration approach concurrently and longitudinally out predicted the other two approaches. Results demonstrated that the average approach slightly outperformed the other two in its prediction of concurrent and longitudinal clinician-rated impairment, teacher-rated academic skills, and parent- and self-rated child social skills across childhood and adolescence. Evidence-based assessment integration of parent and teacher ratings of ADHD in childhood might best utilize an averaging approach, as it is most related to later impairment ratings, particularly if such findings are replicated by other groups.

The Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), similar to DSM-IV-TR, defines Attention-Deficit/Hyperactivity Disorder (ADHD) with a list of nine inattentive and nine hyperactive-impulsive behavioral symptoms, of which individuals must manifest six in one of the two symptom domains, as well as substantial interference in functioning occurring in two or more settings (e.g., at home and at school or work; American Psychiatric Association [APA], 2013). The DSM-5 strongly encourages clinicians to obtain information from more than one informant covering more than one setting. The text explains that “several inattentive or hyperactive-impulsive symptoms are present in two or more settings (e.g., at home, school, or work; with friends or relative; in other activities; p.36; APA, 2013)” and further states that “confirmation of substantial symptoms across settings typically cannot be done accurately without consulting informants who have seen the individual in those settings (pp.37-38; APA, 2013).” Yet, there remains no standardized approach to integration of these multiple sources of information in making an ADHD diagnosis for research or – perhaps more importantly – clinical purposes. The DSM-5 workgroup did not suggest an optimal approach due to insufficient empirical evidence on the validity of different approaches. A lack of empirical guidance as to how to integrate across multiple informant symptom ratings is a cross-cutting diagnostic issue with ramifications for other disorders (Achenbach, 2011; De Los Reyes & Kazdin, 2005), although ADHD is perhaps most impacted by it due to its diagnostic criteria requiring symptoms to be present in more than one setting.

Several different approaches to integrating multiple informant ratings for ADHD and commonly comorbid disorders have been proposed for modelling multiple informant discrepancies in a research setting. A manifest difference score between different informant reports can be calculated to provide a measure of the magnitude of divergence between reports (Laird & Weems, 2011). In regression analyses, discrepancies in informant reports can be represented using interaction terms predictive of external criteria (De Los Reyes, 2013; Laird & De Los Reyes, 2013; Laird & LaFleur, 2016; Tackett et al., 2013). Further, differential weighting approaches have been advocated, based on either principal component analysis, confirmatory factor analysis, or preference given to the informant who knows the child best; yet, these weighting approaches have historically not outperformed equal ratings despite being more complex and they have been found to be dependent on specific sample weights and/or subjective clinician judgement (De Los Reyes, 2013; Piacentini, Cohen, & Cohen, 1992; van Dulmen & Egeland, 2011).

Yet due to the fact that such approaches focus on discrepancies (vs. cross-situational agreement as specified in the DSM-5; De Los Reyes et al., 2009) and have no ready clinical utility, the most common approach utilized to date for ADHD is the “OR” algorithm, following the approach taken in the DSM-IV field trials (Lahey et al., 1994). The “OR” algorithm specifies that a symptom is present if at least one rater endorses that symptom. After determining whether a symptom is present, the manifest symptoms are then added up to create a sum score. This approach to integration of symptom ratings across multiple informants is simple, easily standardized, and reasonably easy to utilize in clinical settings. Yet, this approach has several problems that may lower the validity of ADHD diagnoses (Solanto & Alvir, 2009; Valo & Tannock, 2010). For example, if one informant has an acquiescence bias and agrees to all symptoms, ratings of the other informant are essentially ignored, which undermines the requirement to incorporate data from multiple informants. Further, as the number of reporters increases, symptom levels become inflated.

Another method for integrating multiple-informant data is through the “AND” algorithm. This algorithm requires that both informants endorse a symptom for it to count toward a diagnosis. This algorithm is less commonly used than the “OR” algorithm because it usually results in low symptom estimates. Further, one reporter with a conservative bias can significantly impact the likelihood of diagnosis, potentially leading to underdiagnosis and under-identification of the disorder. Indeed, in both “OR” and “AND” approaches, there is a striking lack of consensus regarding how to handle more than two informant ratings.

In contrast to the “OR” and “AND” integration approaches, Martel et al. (2015; 2017), has advocated for an averaging approach. This work has found that an average of multiple informant symptom ratings relates more strongly with latent ADHD symptom composites in youth ages 6 to 17 (Martel et al., 2015). In addition, the average algorithm out predicted the “OR” and “AND” algorithms in sensitivity and specificity of prediction of diagnostic team decisions about ADHD diagnosis (Martel et al., 2015). Further, a trifactor model of ADHD suggested that mothers, fathers, and teachers all provided equivalent information in their rating of specific inattention, specific hyperactivity-impulsivity, and general ADHD ratings in a bifactor model of ADHD (Martel et al., 2017), also consistent with an averaging approach to individual symptom ratings.

Although there is a limited body of work examining the clinical utility of different integration models, there appears to be no work investigating the predictive utility and validity of these algorithms. It is well-established that youth with ADHD demonstrate significant life impairments across a number of domains including interpersonal (e.g., parent-child difficulties, peer rejection; Goh, Martel, & Barkley, 2020; Johnston & Chronis-Tuscano, 2014; Pelham & Milich, 1984), academic, (e.g., underachievement; Hinshaw, 1992), and psychological functioning (e.g., comorbid disorders; Hinshaw, 1987). Further, youth with ADHD are also at elevated risk to experience a host of negative outcomes later in life, such as substance use, psychological comorbidity, marital issues, and employment issues (Murphy & Barkley, 1996). Thus, additional work is needed to determine whether these algorithms differ in their prediction of various impairments relevant to youth with ADHD.

Therefore, the goal of the current study was to examine the predictive validity of the “OR,” “AND,” and average integration algorithms, and as compared to a latent factor of ADHD (conceptualized as the gold standard), in relation to concurrent and longitudinal impairment including clinician-rated impairment, teacher-rated academic performance, and child- and parent-rated social skills over a 6-year period between childhood and adolescence. Symptom domains effects were also explored. It was hypothesized that the average integration method would outperform both the “OR” and “AND” algorithms in prediction of concurrent and longitudinal impairment. Determining which method is best in relation to external criteria is of great practical importance for providing structured evidence-based assessment guidance to clinicians.

Methods

Participants

As shown in Table 1, data are reported for 800 children across a six-year period during the important developmental transition from middle childhood into adolescence (6-13 years in Year 1, 9-15 years in Year 3, and 11-18 years in Year 6). Ages across years overlapped to maximize statistical power. At Year 1, participants were between the ages of 6 – 13 (M age = 9.44, SD = 1.56). Of the total sample, 62% (n = 496) were male and 81% (n = 648) were Caucasian. Sixty percent of the participants were diagnosed with ADHD with the remainder not meeting diagnostic criteria, to capture the full range of ADHD symptoms (see Table 1). This was consistent with prior research suggesting ADHD may be best described as falling on a continuum (Haslam et al., 2006; Marcus & Barry, 2011). Comorbid conditions were allowed for those with ADHD and those without. Few children without ADHD exhibited common comorbid conditions (e.g., mood, anxiety, ODD, CD, and Learning Disorders); eight percent met criteria for anxiety, and there were few or fewer falling into other categories). For those with ADHD, there were 5 with mood disorders, 99 with anxiety disorders, 91 with ODD, 8 with CD, and 30 with Learning Disorders. Participants were drawn from the Oregon ADHD Cohort, a well-characterized child cohort for which the community-based recruitment and enrollment procedures and multi-informant assessment procedures for ADHD diagnosis have been published in detail elsewhere (Goh et al., 2020; Musser et al., 2016; Karalunas et al., 2017). Ethics approval was obtained from the Institutional Review Board at Oregon Health & Science University. A parent/legal guardian provided written informed consent and children provided written assent.

Table 1.

Sample Demographic Information Across Waves and Descriptive Statistics Measures at Years 1. 3, and 6

Variable		Wave 1	Wave 2	Wave 3
		N = 836	n = 499	n = 411
Sex (n [%] Girls)		317 (37.9)	188 (37.7)	151 (36.7)
Age (M [SD])		9.6 (1.51)	11.4 (1.48)	15.5 (0.8)
Race (n [%] White)		709 (84.8)	419 (84.0)	348 (84.7)
Family Income a (n [%])
0 – 50,000		204 (24.4)	123 (24.6)	114 (27.7)
50,001 – 100,000		345 (41.3)	210 (42.1)	174 (42.3)
100,001 – 150,000		164 (19.6)	103 (20.6)	80 (19.5)
> 150,001		66 (8.5)	35 (7.0)	25 (6.1)
Estimated Full Scale IQ [M(SD)]		110.7 (13.86)	111.4 (13.99)	112 (13.97)
ADHD Total Sum Score		19.6 (14.32)	17.6 (13.69)	15.9 (12.29)
ADHD (n [%] Positive)		500 (59.8)	302 (60.5)	234 (56.9)
Measure	Informant	Year	Mean	Standard Deviation
ADHD-RS Total Score	Parent	Year 1	20.62	15.10
	Teacher	Year 1	17.23	14.47
Informant Integration Method for Total Symptoms: “OR”		Year 1	8.22	6.64
“AND”		Year 1	3.32	4.20
AVG-continuous		Year 1	2.09	1.49
AVG-dichotomous		Year 1	4.58	5.10
GAF	Clinician	Year 1	73.76	12.66
		Year 3	75.73	11.54
		Year 6	77.96	12.20
SSIS	Parent	Year 6	49.47	65.77
	Child	Year 6	55.59	9.81
	Teacher	Year 6	40.78	10.05
APRS	Teacher	Year 6	70.44	13.69

Note. ADHD-RS = Total Raw Score on ADHD-RS at Year 1. GAF = clinician ratings of Global Assessment of Functioning based on the KSADS at Years 1, 3, and 6. SSIS = Total Raw Score at Year 6. APRS = Total Raw Score at Year 6.

Sample Characterization

Recruitment.

Volunteers were recruited via mass mailings, using commercial mailing lists, to all families with children in the target age range within the geographic radius of 50 miles from the University. The mailing made clear that we were looking for children with possible or definite ADHD, as well as for healthy, typically developing children with no history of learning or attention problems. In response to mailings to parents of all children in the target age range in our catchment area, we received 2,144 inquiries (a response rate of about 1% for non-ADHD participants and about 30% for ADHD participants).

First Screen.

An initial screening phone call served to establish eligibility (below) and interest. Nearly half were ruled out at this stage due to medications, other illnesses (e.g. autism), or lack of interest. Those who were excluded at this stage did not differ reliably from the final sample on sex ratio (p=.11) or non-white race (p=.22), but were marginally lower income (p=.06) and slightly younger (p=.06).

Second screen.

For those remaining (n=1,449), an in-person “diagnostic” visit was then scheduled. Here, a parent completed the Conners' Rating Scales-3^rd Edition short form Strengths and Difficulties Questionnaire long form including the impairment module (SDQ, the ADHD Rating Scale, and a semi-structured clinical interview administered by a Master’s-degree level clinician Kiddie Schedule for Affective Disorders and Schizophrenia (KSADS). Children completed a brief unstructured clinical interview with the same clinician, and with a psychometrician (BA-level staff or volunteer) completed a three-subtest short form of the WISC-IV Vocabulary, Block Design, and Information and the Word Reading and Numerical Operations subtests of the WIAT-II to estimate IQ and academic progress. Interviewers and testers wrote detailed observational notes. Teachers were contacted and completed the Conners’-3, SDQ, and ADHD-RS.

All clinical interviewers were trained to reliability of kappa >.80 for all diagnoses seen at ≥5% base rate in this sample on the KSADS to a master interviewer, and had videotapes viewed by a supervisor and reviewed periodically to prevent procedural drift. Psychometric testers were trained to an accuracy standard prior to beginning work and also had videotapes viewed periodically to prevent drift.

Exclusions.

Children were excluded at baseline if they were taking non-stimulant psychotropic medications. They were also excluded for parent reported: history of non-febrile seizure, head injury with loss of consciousness > 60 seconds, diagnosis of autism spectrum disorder or intellectual disability, or other major medical conditions. After the diagnostic team (Step 2), 103 withdrew due to lack of further interest (e.g., only wanted the diagnostic screen), and 496 were ruled out for the following reasons: excess teacher-parent rating discrepancy (situational problems; 35%), subthreshold symptom count (not control or ADHD, 17%), psychosis, mania, current severe depressive episode, Tourette’s syndrome, or head injury (10%), autism (7%), other health condition (7%), ineligible medication (2%), IQ<80 (n=1), or multiple rule outs. Among the eligible children with ADHD, 35% were prescribed stimulant medications and needed to complete the washout, only slightly lower than rates in community surveys for pre-adolescent children.

“Gold Standard” Diagnostic Assignment using a Best Estimate procedure.

All materials were scored and presented to a clinical diagnostic team comprising board certified child psychiatrist with over 25 years of experience and a licensed child neuropsychologist with over 10 years of experience. Blind to one another’s ratings and to the subsequent genetic or cognitive test scores, they formed a diagnostic opinion based on all available information. Their agreement rate for all diagnoses discussed in this paper was satisfactory (ADHD, kappa=.88; ADHD subtype, k>.80, all other disorders with at least 5% base rate, k>.68). Disagreements were conferenced and consensus reached. Cases where consensus was not readily achieved for ADHD diagnosis were excluded.

Using a best estimate procedure, DSM-IV diagnoses were made independently by each clinician. To count symptoms, the clinicians used the following rule: If both parent and teacher ratings exceeded a t-score of 60 on at least one ADHD scale and both rated at least 3 symptoms as “often” or “very often” on the ADHD rating scale (or for parents, were counted present on the K-SADS), the “or” algorithm could be employed. When either informant fell below this mark, and clinicians judged that this was not explained by successful medication treatment during the school day, then the case was rejected as failing to meet the DSM requirement of substantial symptoms present in more than one setting. In addition, it was required that all other DSM criteria were met, including (a) impairment (identified on the KSADS by the clinician as well as on the SDQ Impairment section), (b) onset prior to age 7 (current at the time we began enrollment), (c) sustained impairing symptoms > 1 year, and (d) symptoms of ADHD were not better accounted for by comorbid conditions, trauma history, or other confounds. Both current and lifetime diagnoses were assigned; for the present report, all children in the ADHD group met current and lifetime diagnosis for ADHD.

Medication Washout

All cognitive measures were administered in a fixed order after a medication washout of stimulant medications of at least seven half-lives. Other psychiatric medications were exclusionary. Among the eligible children with ADHD, 35% were prescribed stimulant medications and needed to complete the washout prior to cognitive testing. Parents and teachers were asked to rate symptoms when the child was not on any medication.

Longitudinal Retention

Participants included 836 children in Year 1of whom n=610 were selected for long term follow up study. They were followed annually from baseline to year 8. In the current paper, we report data on Year 1, 3, and 5-8 (condensed) for network stability, using Years 5-8 for long term prediction. With regard to retention, of those 610, resource limitations mandated a planning missing design from among those youth such that the target N was 535 at Year 3 and to be determined for Years 5-8. In actuality, we saw 527 children in Year 3, and 409 so far children at Years 5-8, which is still ongoing data collection.

Measures

ADHD Symptoms

Parent- and teacher-report version of the ADHD-Rating Scale (ADHD-RS; DuPaul et al., 1998; Puig-Antich & Ryan, 1986) were used to assess ADHD symptomatology. Parents (Year 1: α = .93; Year 2: α = .94; Year 3: α = .94) and teachers (Year 1: α = .97; Year 2: α = .97; Year 3: α = .96) responded to all 18 items on a 0 (rarely or never) to 3 (always or very often) scale. Items on the ADHD-RS are consistent with DSM-5 criteria, and thus this measure was viewed as an appropriate assessment of ADHD across years.

Informant Integration Methods.

Given that multiple informants often disagree on the exact number and severity of symptoms, although they may moderately agree on overall symptom domain ratings, the following methods describe different ways to resolve the discrepancies at the level of total, inattentive, and hyperactive-impulsive ADHD symptom domains.

“OR” Algorithm.

Using the ADHD-RS (DuPaul et al., 1998), the OR method specifies that a symptom is present if either the parent OR the teacher endorses that respective symptom. Each symptom is only counted (i.e., scored as a 1 vs. 0) once even if both raters endorse it by rating it as a 2 or 3. This results in a score from 0 to 9 for the 9 items of the inattentive domain and a score from 0 to 9 for the 9 items of the hyperactive-impulsive symptom domain, which totals the 18 ADHD symptoms.

“AND”Algorithm.

The AND method in comparison counts each symptom as present only if both the parent and teacher agree that a symptom is present, based on the same criteria described above.

Average-Continuous.

The average method averages across informant ratings of symptoms for inattentive and hyperactive-impulsive symptom domains even if raters disagree in the ratings of a specific symptom. Resulting averaged scores range 0 to 3 for inattentive and hyperactive-impulsive symptom domains. Total ADHD symptom counts ranged from 0 to 6 and were the sum of the averaged inattentive and hyperactive-impulsive symptom domains.

Average-Dichotomized.

This method averages the parent and teacher ratings on each item, then dichotomized each symptom as present or not present using the same criteria as the “OR” and “AND” algorithms (i.e., 0-below 2 is absent [coded as 0] and 2-3 is present [coded as 1]), then sums the total number of symptom present in inattentive, hyperactive-impulsive, and total symptoms. Across each individual symptom, rates of disagreement on individual symptoms (i.e., one rater rating as a 0 and the other rating as a 3) were relatively rare, ranging from 1% to 5%.

ADHD Latent Factor.

All parent- and teacher-rated symptoms were subjected to a principle axis factor analysis and the resulting latent factor of ADHD was saved as a gold standard comparison.

Impairment

Clinician-rated impairment was obtained via clinician ratings of Global Assessment of Functioning (GAF) based on the KSADS. As described above, such ratings were reliable.

Academic Performance

The Academic Performance Rating Scale (APRS; DuPaul, Rapport, & Perriello, 1991) was used to assess teacher judgments of student academic performance and behavioral conduct in educational situations (Year 6: α = .64). The APRS consists of 19 items that load onto three scales: Academic Success, Impulse Control, and Academic Productivity.

Social Skills

The Social Skills Improvement System (SSIS) Rating Scale (Gresham & Elliott, 2008) was used to evaluate social skills, problem behaviors, and academic competence. The SSIS consists of 84 items and has four subscales: Communication, Engagement, Bullying, and Autism Spectrum. Three versions of the SSIS exist: a parent-report, teacher-report, and student/self-report version. For the current study, parents (Year 6: α = .89) and students (Year 6: α = .94) rated each social skill and problem behavior on a 4-point scale of not true, a little true, a lot true, and very true.

Results

Classification Table Diagnostic Accuracy

As shown in Table 2, when compared to the “gold standard” diagnostic team ratings of ADHD diagnosis, the average-continuous method generally outperformed the “OR,” “AND,” and average-dichotomous algorithm methods in regard to sensitivity, specificity, positive predictive power, and negative predictive power. The “OR” algorithm also performed generally well.

Table 2.

Sensitivity and specificity of informant integration methods

	‘AND’	‘OR’	AVG-Dichotomized	AVG-Continuous
True Positives (n = 483)	144	419	200	466
True Negatives (n = 282)	282	279	282	278
False Positives	0	3	0	4
False Negatives	339	64	283	17
Sensitivity (%)	29.8	86.7	41.4	96.5
Specificity (%)	100	98.9	100	98.6
PPV (%)	100	99.3	100	99.1
NPV (%)	45.4	81.3	81.3	94.2

Note. PPV = positive predictive validity, NPV = negative predictive validity

Correlations Among Different Computation Methods for Total ADHD Symptoms

Bivariate correlations were conducted across “OR,” “AND,” average-continuous, and average-dichotomized informant integration methods. Correlations were uniformly high and significant, as shown in Table 3, ranging from .78-.96. The lowest correlation was observed between the “OR” and “AND” approaches (r = .78), and the highest correlation was observed between the “OR” and average-continuous approach (r = .96). The average-continuous and average-dichotomized approaches were correlated at r = .93. Further, these methods were all highly and significantly correlated with the ADHD latent factor with correlations ranging from .88 (the “AND” approach) to .99 (the average-continuous approach). The “OR” approach was correlated with the latent factor at .96, and the average-dichotomized approach was correlated with the latent factor at .93 (also shown in Table 3). Due to the high overlap between the average-continuous and average-dichotomized approaches to ADHD symptom integration, to minimize collinearity, the average-dichotomized approach was the focus of the remainder of analyses; it was chosen as the focus due to the similarity of its computation and range compared to the “OR” and “AND” approaches. Overall, all integration approaches were highly correlated with one another and with the ADHD latent factor.

Table 3.

Correlations across “OR”, “AND, ” and Average Informant Integration Methods for Total ADHD Symptoms

Integration Method	1	2	3	4	5
1. Total ADHD sx Average-dichotomous	-
2. Total ADHD sx Average-continuous	.93***	-
3. Total ADHD sx “AND”	.94***	.88***	-
4. Total ADHD sx “OR”	.87***	.96***	.78***	-
5. Latent ADHD Factor	.93***	.999***	.88***	.96***	-

Note.

^*p <.05

^**p < .01

^***p ≤ .001

Associations Between Different Integration Methods to Total ADHD Symptoms and Concurrent and Longitudinal Clinician-Rated Impairment

As shown in Table 4, the “OR,” “AND,” and average approaches to total ADHD symptoms all showed significant and strong associations with clinician-rated impairment concurrently (all at Year 1) and longitudinally (two and five years later, at Years 3 and 6). The average approach had perhaps slightly stronger associations with clinician-rated impairment concurrently (average r = −.61 vs. “AND” r = −.51, “OR” r = −.59). This same general pattern held in subsequent follow-up years.

Table 4.

Correlations of “OR, ” “AND” and Average Informant Integration Methods to Total ADHD Symptoms With Clinician-rated Impairment at Year 1, 3, and 6

Integration Method	Year 1	Year 3	Year 6
“OR”	−.59***	−.51***	−.49***
“AND”	−.51***	−.46***	−.44***
Average-Dichotomous	−.61***	−.53***	−.51***

Note.

^*p <.05

^**p < .01

^***p ≤ .001

Multivariate regression analyses, conducted with “OR,” “AND,” and average approaches to total ADHD symptoms entered as independent variables predicting clinician-rated impairment at Year 1, 3, and 6 indicated that, once shared variance among integration approaches was controlled, the average approach significantly out predicted the other approaches (B range −.59 to −.88, all p<.01; most other approaches were ns); see Table 5.

Table 5.

Regression Coefficients of Informant Integration Method for Total ADHD Symptoms on Clinician-rated Impairment for Years 1, 3, 6

Variable	Unstandardized Coefficients		Standardized Coefficients	t	p
	B	SE	Beta (β)
Year 1
Constant	85.84	0.78		109.56	.000
“OR”	0.49	0.20	.16	2.37	.018
“AND”	0.25	0.22	.13	1.13	.258
AVG	−7.52	1.30	−.88	−5.78	.000
Year 3
Constant	84.42	1.02		83.07	.000
“OR”	0.71	0.28	.04	0.25	.802
“AND”	0.71	0.27	.02	0.27	.790
AVG	−4.74	1.66	−.59	−2.86	.004
Year 6
Constant	86.89	1.25		69.69	.000
“OR”	0.20	0.36	.10	0.55	.584
“AND”	0.24	0.34	.08	0.71	.476
AVG	−5.69	2.04	−.68	−2.79	.006

Note. AVG = Average-Dichotomous Integration Method.

Associations Between Different Integration Methods to total ADHD Symptoms and Concurrent and Longitudinal Academic and Social Skills

As shown in Table 6, bivariate correlations conducted between “OR,” “AND,” and average approach to informant integration for total ADHD symptoms and other longitudinal outcomes of academic and social skills indicated similar results as those of clinician-rated impairment above. More specifically, the average approach was just slightly more related to teacher-rated academic performance and child and parent-rated child social skills at Year 6 compared to the “OR” and “AND” approaches. One exception was that the “AND” and average approaches exhibited identical relationships with parent-rated social skills. Regression analyses (not shown) indicated that the average approach out-predicted the other approaches in relation to teacher-rated academic performance, but none of the predictors were significant for the other outcomes.

Table 6.

Bivariate Correlations of Informant Integration Method for Total ADHD Symptoms With Other Outcomes at Year 6.

Integration Method	Teacher-Rated Academic Performance	Child Report of Child Social Skills	Parent Report of Child Social Skills
“OR”	−.51***	−.15**	−.13*
“AND”	−.46***	−.15**	−.14**
Average-Dichotomous	−.54***	−.16**	−.14**

Note.

^*p <.05

^**p < .01

^***p ≤ .001

Inattentive and Hyperactive-Impulsive Symptom Domains

As shown in Table 7, “OR,” “AND,” and average approaches to multiple informant integration of inattentive versus hyperactive-impulsive symptom domain indicated that regardless of integration method, both inattentive and hyperactive-impulsive symptoms were significantly and strongly related to all outcomes, generally speaking, based on correlations.

Table 7.

Correlations of Informant Integration Methods for Inattentive Symptoms and Hyperactive/Impulsive Symptoms With All Outcome Measures at Year 6.

Outcome	AVG Integration Method		“OR” Integration Method		“AND” Integration Method
	IA	H/I	IA	H/I	IA	H/I
Clinician	−.49***	−.46***	−.46***	−.45***	−.40***	−.36***
Teacher	−.56***	−.45***	−.52***	−.44***	−.46***	−.32***
Parent	−.16**	−.10	−.15**	−.10	−.15**	−.10
Child	−.18***	−.11*	−.18***	−.11*	−.17***	−.07

Note. AVG = Average-Dichotomous Integration Method. IA = Inattention symptoms; H/I = Hyperactivity/Impulsivity symptoms; Clinician = Clinician-rated Impairment; Teacher = Teacher-rated Academic Performance; Parent = Parent-rated Child Social Skills; Child = Child-rated Child Social Skills.

^*p <.05

^**p < .01

^***p ≤ .001

Discussion

This study was the first to examine how common algorithms used to integrate multiple-informant data (i.e., “OR,” “AND,” and average) differed in their ability to predict impairment associated with ADHD over a 6-year longitudinal period. To date, little work has addressed how to best integrate data from multiple informants in the assessment of ADHD (but see review by Martel et al., 2017). Further, no work has evaluated how these different integration methods differ in predicting impairment relevant to ADHD, posing a significant gap in the literature. Therefore, determining which method is best in relation to external criteria is of great practical importance for providing structured evidenced-based assessment guidance to clinicians. Results of the current study suggested that, although all integration methods (i.e., “OR,” “AND,” and average) were highly interrelated with one another and with a latent ADHD factor score, the averaging approach appeared to be slightly better in relation to a “gold standard” diagnostic team approach to ADHD diagnosis and in terms of its prediction of clinician-related impairment over a 6-year period, as well as teacher-ratings of academic functioning and parent- and self-ratings of child social skills.

The average approach appeared to outperform the “OR” and “AND” algorithm approaches in relation to a “gold standard” diagnostic team approach to ADHD diagnosis, although the “OR” algorithm performed fairly well too. The interrelations among all integration methods, as well as their interrelations with the ADHD latent factor, were extremely high. This was particularly true for the average and “OR” algorithm approach. One possible reason for the relatively high interrelations may be the presence of an overarching ADHD latent factor, which suggests that often informant symptom ratings will tend to “hang together.”

However, the fact that average and “OR” algorithm seemed to be most highly related to one another and the ADHD latent factor (compared to the “AND” algorithm) may be due to these approaches flexibility in allowing informants to endorse different symptoms (rather than having to agree on the same symptom as with the “AND” integration method) and allow differential interpretation and endorsement of specific symptom items across setting (De Los Reyes, 2013; De Los Reyes & Kazdin, 2005; De Los Reyes, Thomas, Goodman, & Kundey, 2013; De Los Reyes et al., 2015). Further, average and “OR” approaches may more readily allow for interindividual variability in how people express their symptoms (Nigg, Goldsmith, & Sachek, 2004; Sonuga-Barke, 2005; Sonuga-Barke, Bitsakou, & Thompson, 2010). Interestingly, these two approaches also performed slightly better in the prediction of different forms of impairment, suggesting some real-world validity. Yet, the different integration methods were so highly correlated as to suggest that it might not matter greatly which integration method is chosen as long as it is used consistently. Importantly, such continuous approaches are perhaps an important advancement over prior dichotomized approaches that eliminate important information.

In multivariate regressions, where shared variance was partialled out, the averaging integration approach outperformed the “AND” and “OR” integration approaches in almost all cases. Again, the high interrelations among the three integration approaches may limit the real-world utility of the prediction of such small amounts of remaining variance. However, it is also true that averaging scores across symptoms may be easier and less error-prone than counting across “OR” or “AND” counting methods. Further, the better psychometric properties of averages (vs. counting measures) only increases as the number of raters increases so the superiority of the average approach would likely only improve as there are additional parent and teacher ratings available (Martel et al., 2015).

All integration approaches were more strongly related to academic (vs. social) outcomes, which is perhaps unsurprising given ADHD’s strong relationship with academic problems and learning disorders (Shanahan et al., 2006; Willcutt et al., 2012). However, significant associations were seen for social problems in line with the social problems often seen in children with ADHD (Humphreys, Galán, Tottenham, & Lee, 2016; Kofler et al., 2011). Relatedly, inattention (vs. hyperactivity-impulsivity) demonstrated slightly stronger relations with all outcomes. This may have been due, at least in part, to the older age of the sample, as prior studies have suggested that older children may struggle particularly with inattention whereas younger children may struggle more with hyperactivity-impulsivity (Lahey, Pelham, Loney, Lee, & Willcutt, 2005; Olson, Bates, Sandy, & Schilling, 2002).

Further examination of how such results can be best applied at an individual level is an important future direction. Current results suggest the utility of different cut scores using different algorithms, and these cut-scores need to be replicated in other types of samples, perhaps particularly clinical samples with large numbers of comorbid cases. Individual level statistics would be helpful for examining how to integrate multiple informant reports for assessment of other commonly comorbid disorders. For example, although multiple informant convergence is currently stressed in diagnosing ADHD, multiple informant discrepancies may be more important for diagnosing other commonly comorbid disruptive behavior disorders (Laird et al., 2011; 2013; 2016). Examination of how well algorithms distinguish those with ADHD from those with other disorders is an important next step and future direction. Further, consideration of the best operationalization of ADHD (i.e., cross-situational convergence of ADHD ratings) and the role of informant discrepancies should continue to be considered (see De Los Reyes et al., 2005; 2013).

This paper is a first step in work validating the utility of different multiple informant approaches to symptom integration in diagnosis of ADHD. Other external validity outcomes measures should be examined in other samples at different ages and with more sophisticated analytic techniques. Yet, the current study is a critical first step in beginning to operationalize how to integrate multiple informant reports when assessing for the presence and severity of ADHD symptoms to increase reliability and consistency across clinicians. The current report suggests that all currently available integration approaches might work fine, but that the averaging approach might predict longitudinal outcomes such as academic performance slightly better. Yet, it is of critical importance for this work to be replicated by other groups in other samples (e.g., clinical samples), in other age ranges, and in prediction of other important outcomes (e.g., career success, social support, legal outcomes).

Public Significance Statement:

An average approach to integrating parent and teacher ratings of ADHD slightly outperforms other, more complicated integration approaches in prediction of later clinician-rated impairment, teacher-rated academic skills, and parent- and self-rated social skills. Therefore, average integration of ADHD symptom ratings may be the best and easiest integration approach for use in clinical practice.