Assessing Treatment Outcomes in Attention-Deficit/Hyperactivity Disorder: A Narrative Review

Abstract

Objective: To review measures used to assess treatment response in patients with attention-deficit/hyperactivity disorder (ADHD) across the life span.

Data Sources: Keyword searches of English-language articles in the PubMed database up to and including the May 4, 2011, index date were performed with the search strings (1) (attention deficit disorder with hyperactivity [MeSH] OR ADHD) AND (outcome assessment [MeSH] OR adaptation of life skills OR executive function [MeSH]) and (2) (attention deficit disorder with hyperactivity [MeSH] OR ADHD) AND (function OR functioning OR quality of life [MeSH]).

Study Selection: Articles found through this search were then selected based on relevance to the topic area; no specific quality criteria were applied.

Data Extraction: Narrative review.

Results: The vast majority of studies assessing ADHD treatments have measured treatment response using ADHD symptom measures. Additional domains relevant for assessing treatment response among children and adults with ADHD include functional impairment, quality of life, adaptive life skills, and executive function. Validated rating scales exist for assessing these additional domains, but there has been minimal research evaluating the sensitivity of these instruments for detecting treatment response in pediatric and adult samples.

Conclusions: Assessment of treatment outcomes in ADHD should move beyond symptom assessment to incorporate measures of functioning, quality of life, adaptive skills, and executive function, especially when assessing long-term treatment response. The authors recommend a potential battery and schedule of measures that could be used to more comprehensively assess treatment response in patients with ADHD.

Clinical Points

• ▪ Attention-deficit/hyperactivity disorder (ADHD) impacts multiple domains, beyond the characteristic symptoms of hyperactivity, impulsivity, and inattention.

• ▪ A meaningful assessment of treatment effectiveness requires measures of functionality, quality of life, adaptive life skills, and executive function, as well as reductions of ADHD symptoms.

• ▪ Validated rating instruments are available to assess the effects of treatment in all of these domains.

An estimated 9.5% of children aged 4 to 17 years¹ and 4.4% of adults² in the United States meet diagnostic criteria for attention-deficit/hyperactivity disorder (ADHD). Pharmacologic and behavioral therapy can be effective treatments for ADHD,^3–6 as reflected in ADHD treatment guidelines.^7–10 However, assessment of treatment response in patients with ADHD has generally been limited to narrowly focused measures of ADHD symptoms as defined by the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR),¹¹ with relatively little attention to other relevant outcome domains.^4,5,12

Patients with ADHD are most often referred for treatment because of dysfunction in familial, social, emotional, academic, and occupational roles rather than because of ADHD symptoms.¹³ Children are often referred for diagnosis and treatment because of poor academic performance, difficulty making friends, or low self-esteem (functional impairments) rather than because they cannot remain seated or they lose things or interrupt others (ADHD symptoms). Because functional impairments are a primary impetus for seeking treatment, they may be the most appropriate targets for assessing treatment response.¹⁴ Although behavioral symptoms show moderate correlation with functional impairments, assessment of symptom improvement alone may not adequately reflect changes in functional status.¹⁵ There is a strong need to broaden outcome assessments to encompass a multidimensional array of outcomes in patients with ADHD.^12,14

We recognize 5 core conceptual areas that are integral to assessing treatment outcomes in ADHD: (1) DSM-IV-TR ADHD behavioral symptoms; (2) functional impairment in familial, social, emotional, academic, and occupational areas; (3) quality of life (QOL) or the burden of illness on the daily lives of patients and their families; (4) adaptive life skills (the development of strategies and skills to cope with problem behaviors); and (5) executive function (working memory, goal-directed activity, planning, organization, prioritizing, self-monitoring, and self-regulation).^16,17 These overlapping but distinct concepts each contribute to the real-life outcome of the patient. Although some of these concepts are poorly differentiated in the literature and current measurement approaches, we believe that each should be considered when assessing ADHD outcomes. The objective of this review is to discuss approaches for measuring ADHD treatment outcomes in these 5 core conceptual areas.

METHOD

PubMed searches were performed in the days leading up to and including May 4, 2011. The following search strings were used:

(1) (attention deficit disorder with hyperactivity [MeSH] OR ADHD) AND (outcome assessment [MeSH] OR adaptation of life skills OR executive function [MeSH]);

(2) (attention deficit disorder with hyperactivity [MeSH] OR ADHD) AND (function OR functioning OR quality of life [MeSH]).

Search fields were limited to the title/abstract (string 1) or title (string 2); both searches were limited to humans and English-language articles. Additional references were identified using the “related citations” function of PubMed and by reviewing the reference lists of obtained articles.

Our intent in this narrative review is to clarify multiple concepts relevant to the assessment of ADHD. Research in these areas varies in quality and, in some cases, is largely underdeveloped. Therefore, articles were selected based on their relevance to the topic area and were not systematically evaluated based on their methodological quality. Table 1 provides a summary of the scales described in this review. Their frequency of citation in the ADHD literature, based on a PubMed search of titles and abstracts for each scale by name, is provided to place their use in historical context. However, frequency of citation should not be construed as an index of a scale’s quality.

Table 1.

Rating Scales Available for Assessing Various Outcome Domains in ADHD

Rating Scale	Domain	Brief Descriptiona,b	Available Psychometric Assessmentsc
Early Childhood Attention Deficit Disorders Evaluation Scale (ECADDES)23	ADHD (children)	Ages 2–7 y	Test-retest reliability, internal consistency, interrater reliability
		Inattentive and hyperactive/impulsive subscales
		Parent (50 items) and teacher (56 items) reports
		Available at www.excellenceforchildandyouth.ca/about-learning-organizations/measure-profile?id=109
		ADHD references: rarely cited
ADHD Rating Scale–IV (ADHD RS-IV)24,58	ADHD (children)	18 items assessing frequency of DSM-IV symptoms	Reliability, validity
		Inattentive and hyperactive/impulsive subscales
		Parent and teacher reports
		Available at www.healthchoiceaz.com/forms/Child%20ADHD%20Rating%20Screener.pdf
		ADHD references: widely cited
ADHD Symptoms Rating Scale (ADHD-SRS)25,36	ADHD (children)	56 items based on DSM-IV symptoms	Test-retest reliability, internal consistency, interrater reliability, validity
		Inattentive and hyperactive/impulsive subscales
		Parent and teacher reports
		ADHD references: widely cited
Conners Rating Scales–Revised (CRS-R)21,22,26	ADHD (children)	Includes symptoms not specific to ADHD	Test-retest reliability, internal consistency, normative data
		7 subscales, including DSM-IV inattentive, hyperactive/impulsive, and total subscales
		Parent, teacher, and adolescent self-reports
		Full-length (parent, 80 items; teacher, 59 items) and abbreviated (27 items; 28 items) formats
		ADHD references: rarely cited
Swanson, Nolan, and Pelham–IV Teacher and Parent Rating Scale (SNAP-IV)21,27,28,32	ADHD (children)	Measures DSM-IV and non–DSM-IV ADHD symptoms and items for other DSM-IV disorders	Reliability, internal consistency, validity, normative data
		Includes inattention and hyperactivity/impulsivity subscales
		90 items in full-length format
		Parent and teacher reports
		Available at www.adhd.net/snap-iv-form.pdf
		ADHD references: widely cited
Strengths and Weaknesses of ADHD Symptoms and Normal Behavior Scale (SWAN)28,31,33	ADHD (children)	Similar to SNAP-IV, but items are reworded and Likert scale is extended to incorporate strengths and weaknesses in each area	Test-retest reliability, internal consistency, validity
		30 items
		Parent and teacher reports
		Available at www.adhd.net/SNAP_SWAN.pdf
		ADHD references: commonly cited
Swanson, Kotkin, Agler, M-Flynn, and Pelham rating scale (SKAMP)22,34,35	ADHD (children)	Internal consistency, test-retest reliability, validity, normative data
		Included in full version of SNAP-IV
		Separate subscale for evaluating classroom impairment related to attention (eg, initiating assignments) and deportment (eg, following school rules)
		10 items
		Teacher report
		ADHD references: commonly cited
Vanderbilt ADHD Diagnostic Rating Scales (VARS)21,29,30	ADHD (children)	Assesses DSM-IV symptoms as well as anxiety and depression and school functioning	Internal consistency, interrater reliability, validity
	Functional impairment	Parent (VADPRS29: 55 items) and teacher (VADTRS30: 43 items) report
	Comorbid anxiety, depression, tics, oppositional defiant disorder (ADHD-specific)	Includes separate subscale for performance (teacher version includes academic and classroom behavioral performance)
		Includes both initial assessment and follow-up forms that evaluate side effects
		VADPRS available at www.childrenshospital.vanderbilt.org/uploads/documents/DIAGNOSTIC_PARENT_RATING_SCALE(1).pdf
		VADTRS available at www.brightfutures.org/mentalhealth/pdf/professionals/bridges/adhd.pdf
		ADHD references: commonly cited
		Currently incorporated into the American Academy of Pediatrics ADHD Toolkit
Adult ADHD Self-Report Scale-v1.1 (ASRS)38,39	ADHD (adult)	Frequency of DSM-IV-TR symptoms	Test-retest reliability, internal consistency, validity
		18 items (screener version has 6 items)
		Self-report
		Available at http://webdoc.nyumc.org/nyumc/files/psych/attachments/psych_adhd_checklist.pdf
		ADHD references: widely cited
Barkley Adult ADHD Rating Scale–IV (BAARS-IV)49	ADHD (adult)	Current DSM-IV symptoms	Reliability, validity
		Self- and other report
		Long version and quick screen
		ADHD references: rarely cited
Conners Adult ADHD Rating Scales (CAARS)40,43–45	ADHD (adult)	DSM-IV symptom severity	Interrater reliability, internal consistency, test-retest reliability, discriminatory validity
		Full-length (66 items) and abbreviated (18 items) formats
		Self-report and observer report
		ADHD references: widely cited
Current Symptoms Scale (CSS)46,47	ADHD (adult)	Behavior in the past 6 mo, including DSM-IV symptoms and symptom interference with function	Reliability, discriminatory validity
		78 items plus employment and social history
		Self-report or observer report
		Self-report form available at www.mnadhd.com/Current_Symptoms_Scale_self.pdf
		ADHD references: rarely cited
Wender Utah Rating Scale48	ADHD (adult)	61 items	Internal reliability, validity
		Patient report regarding problems at ages 6–10 y
		A subscale of 25 items can be used to identify whether the adult had ADHD as a child
		Available at http://www.venturafamilymed.org/Documents/Wender_Utah%20Rating%20Scale.pdf
		ADHD references: widely cited
Columbia Impairment Scale67	Functional impairment (broadband)	13-item rating of children and adolescents	Test-retest reliability, validity, internal consistency
		Assesses 4 areas of function: interpersonal relations, psychopathology, function at work or school, use of leisure time
		Items rated on a 4-point Likert scale
		Does not measure specific functional domains
		Available at www.dhs.state.il.us/OneNetLibrary/27896/documents/By_Division/MentalHealth/Columbia/CIS-Y%20-youth%20web%20system%20version%20w%20instructions_1.pdf
		ADHD references: not cited
Academic Performance Rating Scale (APRS)70,71	Functional impairment (broadband; child)	19-item scale for rating academic performance over the past week	Test-retest reliability, internal consistency, divergent validity, normative data
		Each item scored on a 5-point Likert scale
		Available at www.excellenceforchildandyouth.ca/sites/default/files/meas_attach/Academic_Performance_Rating_Scale_(APRS).pdf
		ADHD references: commonly cited
Endicott Work Productivity Scale68	Functional impairment (broadband)	25-item self-report	Reliability, validity
		Assesses absenteeism, presenteeism
		ADHD references: rarely cited
Social Skills Improvement System (SSIS)72 (authors’ revision of Social Skills Rating Scale [SSRS]59,69)	Functional impairment (broadband; child)	3 domains: social skills (7 subscales), problem behaviors (5 subscales), academic competence (5 subscales)	Interrater reliability, validity (convergent and discriminant), normative data
		Items scored on 4-point rating scales by teachers and parents and by student
		ADHD references: SSIS (rarely cited); SSRS (commonly cited)
Impairment Rating Scale (IRS)73	Functional impairment (ADHD-specific)	Versions for parent (7 domains of impairment) and for teacher (6 domains)	Stability, interrater reliability, validity
		Ratings on visual analog scale (no problem to extreme problem) but scored by dividing line into segments to form a 7-point Likert scale)
		ADHD references: rarely cited
Weiss Functional Impairment Rating Scale74–76	Functional impairment (ADHD-specific; child and adult)	Self-report: 69-item survey in 7 areas (family, work, school, life skills, self-concept, social, risk)	Test-retest reliability, internal consistency, convergent and discriminant validity, factor analysis, domain validation, minimal important difference, receiver operating characteristic cutoff from normal population
		Parent report: 50-item survey in 6 areas (family, school, life skills, child’s self-concept, social activities, risky activities)
		Items scored on 4-point Likert scale
		Certified translation into 16 languages
		Available at www.caddra.ca/cms4/pdfs/caddraGuidelines2011WFIRS_S.pdf ; www.caddra.ca/cms4/pdfs/caddraGuidelines2011WFIRS_P.pdf
		ADHD references: rarely cited
Strengths and Difficulties Questionnaire37,77	Functional impairment (ADHD-specific; child)	25-item survey for parents or teachers of children age 4–16, or self-report in children age 11–16	Interrater reliability, internal consistency, test-retest stability
		5 areas (emotional symptoms, conduct problems, hyperactivity/inattention, peer relationship problems, prosocial behavior)
		Includes an impact factor (rates the impact of symptoms on the patient)
		Available at www.sdqinfo.com
		ADHD references: commonly cited
Child Health and Illness Profile (CHIP)91–93	QOL (broadband)	Versions for self- and parent report cited	Stability, internal consistency, validity
		For children (CHIP-CE91,92: ages 6–11), domains are satisfaction, comfort, resilience, risk avoidance
		For adolescents (CHIP-AE93: ages 11–18), domains are satisfaction, discomfort, risk avoidance, resilience, achievement
		Items scored on 5-point Likert scale
		Normal mean = 50, normal range 44–56 (≤ 43, poor health; ≥57, excellent health)
		Availability www.childhealthprofile.org
		ADHD references: commonly
Child Health Questionnaire90,95,96	QOL (broadband)	28- and 50-item versions for parents of children ages 5–18; 87-item self-report version for children ≥ 10	Internal consistency, validity
		Measures 14 physical and psychosocial concepts
		Scoring on 4- to 6-point scales
		ADHD references: commonly cited
Pediatric Quality-of-Life Inventory (PedsQL)97,98	QOL (broadband)	15 core items and 8 add-on modules for specific symptoms and treatment domains reflecting physical, emotional, social, and school functioning	Internal consistency, validity
		Items scored on 5-point Likert scale
		Available at www.pedsql.org
		ADHD references: rarely cited
Short-Form-3699,100	QOL (broadband)	36-item survey of physical and mental health and function (4 subscales each)	Reliability, validity
		Items scored on 5-point response scales
		Normal mean = 50, SD = 10
		ADHD references: rarely cited
ADHD Impact Module (AIM)102–105	QOL (ADHD-specific)	8-item child scale (eg, schoolwork), 10-item home scale (eg, effect of ADHD on home); 9 descriptive items on treatment status (AIM-C102,103)	Reliability, validity
		Adult self (AIM-A104,105)
		Items scored on 5-point Likert scale
		AIM-C available at www.healthact.com/pdf/surveys/aim-c.pdf
		AIM-A available at www.healthact.com/pdf/surveys/aim-a.pdf
		ADHD references: rarely cited
Adult ADHD QOL Scale (AAQOL)106	QOL (ADHD-specific)	29-item survey in 4 domains (life productivity, psychological health, relationships, and life outlook)	Reliability, validity, internal consistency
		Items scored on a 5-point Likert scale
		ADHD references: rarely cited
Vineland Adaptive Behavior Scales (VABS)118,119	Adaptive life skills	Semistructured interview of parents or caregivers to nonhandicapped children (0–19 years)	Reliability, validity
		Assesses communication, daily living skills, socialization, and motor function
		For adaptive behavior composite score, mean = 100, SD = 15
		ADHD references: rarely cited
Kidcope121	Adaptive life skills	Questionnaire for children and adolescents	Reliability, validity, test-retest reliability
		Assesses 10 coping strategies in terms of frequency of use and efficacy of use
		Frequency and efficacy scored, respectively, on 4-point and 5-point Likert scales
		ADHD references: not cited
Life Participation Scale—Child version122	QOL (ADHD-specific)	24-item survey analyzed as 2 factors (17-item Self-control and 7-item Agreeable subscales)	Internal consistency, validity
		Items scored on a 4-point Likert scale
		Available with publication122
		ADHD references: rarely cited
Adaptive Behavior Assessment System, second edition (ABAS-II)123,124	Adaptive life skills	Assessment of 10 adaptive skills (as given in DSM-IV) in 3 domains asking what the child does rather than what they can do	Reliability, validity
		Available in 5 forms for different age ranges and respondents (193–241 items across these forms)
		For norm-referenced skill scores, mean = 10, SD = 3; for general adaptive composite score, mean = 100, SD = 15 (also given with 95% CI and percentile ranking)
		ADHD references: not cited
Instrumental Activities of Daily Living (IADL) Scale125	Adaptive life skills	Assesses 8 items, with 3–4 behaviors included in each item; only 5 items assessed in men who cannot perform housework	Interrater reliability, validity
		Behaviors are scored with either a 0 or 1
		Scored by individual administering the questionnaire
		Most often used in geriatric populations
		Available at www.abramsoncenter.org/pri/documents/iadl.pdf
		ADHD references: not cited
Behavior Rating Inventory of Executive Function (BRIEF)135,137	Executive function	86-item questionnaire for parents and teachers of children ages 5–18 (adult version also available)	Test-retest reliability, interrater reliability, validity, normative data
		Evaluates 8 domains of executive functioning to derive a Behavioral Regulation Index and Metacognition Index
		Items scored on 3-point Likert scale
		Global Executive Composite T score is the sum of the 2 Index scores
		ADHD references: commonly cited
Brown Attention-Deficit Disorder Scales (BADDS)139–141	Executive function	40 items in 5 domains of ADHD-related effects on executive function (subscale scores in activation, attention, effort, affect, memory)	Test-retest reliability, internal consistency, validity
		Includes aspects of executive function and emotional self-regulation beyond DSM-IV criteria
		Items scored on 4-point Likert scale
		Total score with T scores available for each domain
		4 versions by age ranges and respondent
		ADHD references: commonly cited
Barkley Deficits in Executive Functioning Scale142	Executive function	For adults, available in self-report and other report forms, long and short versions	Test-retest reliability, internal consistency, interrater reliability, validity
		Items scored on 4-point Likert scale
		ADHD references: not cited
Children’s Organizational Skills Scales (COSS)143,144	Executive function	Assesses organization of time, materials, and actions at home and school	Test-retest reliability, internal consistency, discriminant validity
		Includes responses from parents (66 items), teachers (42 items), and a self-report (63 items)
		Includes 3 scales: task planning, organized actions, and memory and materials management
		Uses a 4-point Likert scale
		ADHD references: rarely cited

^aWidely cited (> 50 publications); commonly cited (10–50 publications); rarely cited (< 10 publications); based on PubMed search of published titles and abstracts using the scale name AND (attention-deficit/hyperactivity disorder OR ADHD) as search terms.

^bIf publicly available at no cost, the Web site is provided.

^cListing of tests performed; not intended as endorsement of the quality of the scale’s properties.

Abbreviations: ADHD = attention-deficit/hyperactivity disorder; CI = confidence interval, DSM-IV = Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition; DSM-IV-TR = Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision, QOL = quality of life.

SYMPTOMS

The core ADHD symptoms observed in children, as reflected in the DSM-IV-TR¹¹ criteria, are inattention, hyperactivity, and impulsivity. However, symptom manifestation differs considerably across the life span. Although hyperactivity and impulsivity may improve with increasing age, symptoms of inattention are more persistent, and the presentation of symptoms in all 3 domains shifts with changing developmental challenges.^18,19 The DSM-5 diagnostic criteria are expected to better reflect symptom manifestation of ADHD in adults.^18,20

Measurement Approaches

Reviews by Collett et al²¹ and Madaan et al²² provide detailed descriptions of ADHD rating scales for children. Differences among scales include length, scope (eg, broad band vs narrow band [ADHD-specific]), rater (eg, parent or teacher), and psychometric validation. Differences across scales relate mainly to minor differences in wording or anchor points and whether symptoms are assessed by frequency versus severity. Essentially, all scales measure the 18 symptoms of DSM-IV–rated ADHD, with most using 4-point Likert scales (Table 1).^23–31 Sensitivity and test-retest reliability are particularly important for treatment monitoring.

Although ADHD rating scales are generally reliable,¹⁴ the adequacy of normative and/or psychometric data are less well established²² for the Swanson, Nolan, and Pelham–IV Teacher and Parent Rating Scale (SNAP-IV)^27,28,32 and the Strengths and Weaknesses of ADHD Symptoms and Normal Behavior (SWAN).^28,31,33 A short rating scale, such as the Swanson, Kotkin, Agler, M-Flynn, and Pelham rating scale (SKAMP),^34,35 may be more convenient for the clinician than a longer scale, such as the Early Childhood Attention Deficit Disorders Evaluation Scale²³ or the ADHD Symptoms Rating Scale.^25,36 However, the SKAMP was designed as an observational tool for the laboratory or classroom rather than for the clinical setting. The 18-item ADHD Rating Scale–IV (ADHD RS-IV)²⁴ and the abbreviated 27-item Conners Rating Scales–Revised²⁶ are good options for symptom-based outcome assessments based in part on good psychometric properties.^21,22 Clinicians in primary care may be more likely to use scales in the public domain, such as the Vanderbilt ADHD Diagnostic Rating Scales,^29,30 the SNAP, the SWAN, or the Strengths and Difficulties Questionnaire.³⁷

Several adult ADHD rating scales are also available (Table 1), including the Adult ADHD Self-Report,^38,39 the Conners Adult ADHD Rating Scales,^40–45 and the Current Symptoms Scale.^46,47 Assessment of adults requires demonstration of a childhood history of ADHD in addition to current symptoms. The Wender Utah Rating Scale, which has established validity, provides a simple and reliable way of confirming a childhood diagnosis when collateral information, report cards, or the patient’s recollection of early development is insufficient.⁴⁸ In most cases, scales are modified from those originally developed for children, using descriptive language appropriate for adult patients. One of the few adult ADHD symptom scales empirically derived using an adult population is the Barkley Adult ADHD Rating Scale– IV.⁴⁹ This instrument has established reliability and validity. It includes a quick screen, a battery of scales developed to assess current symptoms, and informant report of childhood and/or current symptoms; it also assesses impairment in different domains.

The accuracy of a clinical assessment can be improved when symptom information is acquired from more than 1 source (eg, from parents and teachers of children with ADHD).⁴² Teachers may be better able to report daily behavior because they see the child when medication is at its peak and they observe the child in a structured environment in which medication effects (eg, ability to sit still, ability to concentrate for long periods) are most likely to be apparent. Furthermore, teacher reports, based on comparison and experience with peers in attention-demanding settings, may be more sensitive to ADHD behaviors that are less salient in a home environment. Both teacher and parent rating scales can be useful to obtain detailed knowledge of social and academic performance, adaptive skills, differential diagnosis, and the child’s strengths and key disabilities. Similarly, in adults, rating of symptoms by an observer (eg, spouse) in addition to a self-report rating can contribute useful information.⁵⁰

Treatment Effects

A change in ADHD symptoms has been the standard measurement of treatment response and demonstration of efficacy relative to placebo. Treatment effects on ADHD symptom scales usually have moderate-to-large effect sizes.^4–6 A meta-analysis of 32 studies that evaluated the effects of stimulant and nonstimulant medications in children across 20 different measures reported a large overall effect size (Cohen d = 0.78) for ADHD symptom improvement.⁴ A separate meta-analysis of 18 studies in adult ADHD that incorporated 18 different symptom outcome measures demonstrated statistically significant symptom reduction with large effect sizes for short- and long-acting stimulants (Cohen d = 0.96 and 0.73, respectively) and moderate effect sizes for nonstimulants (Cohen d = 0.39).⁵

Positive effects of behavioral therapy on ADHD symptoms are well established in children, with effect sizes (Cohen d) ranging from 0.39 to 3.70 depending on study design and rater (eg, parent or teacher).³ There are now 6 randomized controlled trials of cognitive behavioral therapy (CBT) and other psychological treatments in treated and untreated adults, all demonstrating medium-to-large effect sizes.^51–56

FUNCTIONAL IMPAIRMENT

People with ADHD exhibit impairment in behavioral, academic, and social functioning.^57–59 Children with ADHD as young as 3 years demonstrate impoverished social skills and more problem behaviors in home and school environments compared with children without ADHD.^58,59 In a longitudinal study of children from birth to a median age of 18.4 years, those with ADHD had significantly lower reading achievement scores and approximately 3-fold higher rates of grade retention and dropout compared with controls.⁵⁷ In the Multimodal Treatment Study of Children with ADHD, those with ADHD had poorer social skills and were less socially accepted than children without ADHD and had inaccurate social perceptions of themselves.⁶⁰

Studies in adults demonstrate the chronicity of these impairments. Blase et al⁶¹ reported that, among 3,400 university undergraduates, those with self-reported ADHD had significantly lower grade point averages, poorer emotional stability, and greater academic and social concerns than students who had never been diagnosed with ADHD. In a community sample of 500 adults, those with a self-reported ADHD diagnosis were significantly less likely to have graduated from high school or college or to be currently employed and significantly more likely to report relationship problems compared with those without an ADHD diagnosis.⁶² Among married adults, those with ADHD report poorer marital adjustment and family functioning than those without ADHD.⁶³

Measurement Approaches

Multidimensional rating scales assess multiple domains of impairment, whereas domain-specific scales assess a single domain of impairment. Narrowband scales are specific for ADHD-related impairment, whereas broadband scales assess impairments across a range of psychopathologies (Table 1).

Broadband scales^{15,59,63–70} that have been used in children and adults include the multidimensional Columbia Impairment Scale⁶⁷ and the domain-specific Academic Performance Rating Scale (APRS),^70,71 Endicott Work Productivity Scale,⁶⁸ and the Social Skills Improvement System,⁷² which is an update of the Social Skills Rating Scale^59,69 (SSRS). According to Pelham et al,¹⁴ multidimensional measures are shorter and less costly yet correlate well with more extensive domain-specific scales. Some scales include a single question on impairment, thus providing a global impairment rating. Given the considerable variability in domains of functional impairment across individuals (eg, a child may be impaired academically but not socially), multidimensional scales are preferred over those with only a global rating.¹⁴ A multidimensional rating scale also indicates appropriate concomitant treatments (eg, tutoring for a child with specific academic impairment or marital therapy for an adult with specific problems in spousal communication).

Compared with these broadband functional impairment rating scales, narrowband rating scales are relatively brief, because they focus only on areas of impairment noted to be most problematic in patients with ADHD. The Impairment Rating Scale (IRS)⁷³ is a brief multidimensional instrument for children with ADHD. Both the parent version, assessing 7 domains (relationship with peers, relationship with siblings, relationship with parents, academic progress, self-esteem, influence on family functioning, and overall impairment), and the teacher version, assessing 6 domains (relationship with peers, relationship with teacher, academic progress, self-esteem, influence on classroom functioning, and overall impairment), have acceptable to excellent stability, evidence of validity, and reliability.⁷³ The Vanderbilt ADHD Diagnostic Teacher Rating Scale (VADTRS)³⁰ and Vanderbilt ADHD Diagnostic Parent Rating Scale (VADPRS)²⁹ are primarily ADHD symptom scales but also include performance subscales to assess multidimensional impairment in areas such as reading, writing, mathematics, and relationships. The Vanderbilt scales have established concurrent validity and internal consistency.²¹

The Weiss Functional Impairment Rating Scale^74–76 is a clinician-friendly tool completed by a parent or adolescent/adult that scores distinct domains (school [learning and behavior] or work, family, social, leisure, self-concept, and risky activities) on a 4-point Likert scale. The scale, translated into 9 languages, has excellent internal consistency, intercorrelation between domains, validation by factor analyses, test-retest validity, sensitivity to change, and receiver operating characteristics to determine cutoff scores and normative data. It has also been analyzed to determine reasonable change/minimal important differences (unpublished data, M.D.W.), so it can be used as a primary outcome in clinical trials. The scale is in the public domain (available at www.caddra.ca)⁷⁴ and, like the Strengths and Difficulties Questionnaire (www.sdqinfo.com),⁷⁷ is free and designed to facilitate clinical as well as research use.

Treatment Effects

A 2008 meta-analysis of randomized controlled trials in children with ADHD reported significant effects of behavioral therapy and methylphenidate on social and academic functioning.⁷⁸ Additional randomized controlled trials have broadened these findings, demonstrating significant improvements in social, emotional, academic, and before-school functioning in children receiving ADHD pharmacotherapy.^65,79,80 However, a case series of 785 children with ADHD treated by community physicians generally using pharmacotherapy demonstrated large improvements in ADHD symptoms (VADTRS and VAPRS) but only limited improvement in functional impairment (teacher rating of writing and assignment completion).⁸¹ These findings suggest that global functional improvement very likely requires multimodal treatment to address ADHD-related impairments. Consistent with this conclusion, combined treatment with atomoxetine and behavioral therapy produced significantly greater improvements in some aspects of functioning, including parent-rated problem behaviors (on the SSRS), academic progress (on the IRS), and impulse control (on the APRS), than atomoxetine treatment alone.⁶⁵ Data from a randomized trial of atomoxetine also suggest that functional improvements in those who respond to treatment may be maintained with long-term psychosocial therapy.⁸⁰

Data on functional outcomes in adults are sparse. A 2008 meta-analysis of pharmacotherapy in adults with ADHD found no studies evaluating academic, workplace, or social functioning.⁸² However, randomized trials by Adler et al⁶⁴ and Retz et al⁶⁶ suggest that pharmacotherapy improves family, social, and workplace functioning in adults with ADHD.

QUALITY OF LIFE

Studies using parent reports of QOL uniformly demonstrate significantly lower ratings across psychosocial domains (eg, behavior, self-esteem, mental health, and emotional functioning) in children and adolescents with ADHD versus individuals without ADHD, whereas differences in physical domains have not generally been found.^83–85 Although self-report may be more relevant than an observer’s report of the impact of ADHD on QOL, children with ADHD have been noted to have a positive illusory bias,⁸⁶ raising questions about their reliability in reporting QOL. The few studies that have incorporated self-report offer less consistent evidence for reduced QOL in children and adolescents with ADHD.^83,87 Available data suggest that adults with ADHD have significantly lower self-ratings of QOL compared with those without ADHD across global, social, and occupational domains.^88,89

Measurement Approaches

Most QOL measures are multidimensional, assessing a range of physical and psychosocial health outcomes (Table 1). Numerous multidimensional measures have been used to assess QOL in children with ADHD.⁸³ Because multidimensional measures often include items that are not relevant to ADHD (eg, questions about pain interference with daily activities),⁹⁰ those with separate ratings for physical and psychosocial domains may be preferable.

The Child Health and Illness Profile (CHIP-CE), a multidimensional QOL measure for children (6–11 years)^91,92 and adolescents (11–17 years),⁹³ assesses health in 5 domains: satisfaction, comfort, resilience, risk avoidance, and achievement. CHIP-CE can be completed as self- or parent-report. Both scales can generate separate t scores for each domain and have good psychometric characteristics.^91,93 The CHIP-CE items are closely linked to functional impairment associated with ADHD, with less emphasis than other scales on irrelevant aspects such as pain or mobility. The CHIP-CE has also been validated in ADHD in a large European multinational study.⁹⁴

The Child Health Questionnaire (CHQ)^90,95 is another commonly used multidimensional measure with self- (for ages ≥ 10 years) and parent-report versions. The parent version is available as a 50- or 28-item form encompassing 14 concepts of physical and psychosocial functioning that can be assessed as separate summary measures. The CHQ has been psychometrically validated in children and adolescents with ADHD.⁹⁶

The Pediatric Quality-of-Life Inventory (PedsQL)^97,98 was developed for pediatric cancer patients but can be used across pediatric chronic health conditions to assess health-related QOL in terms of physical, psychological, and social functioning. It comprises patient self-report forms for children (8–12 years) and adolescents (13–18 years) and parent proxy-report forms. Psychometric characteristics suggest good reliability and validity.⁹⁷

For adults, the Short Form-36 General Health Survey (SF-36)⁹⁹ is a broadband tool providing a multidimensional QOL assessment (physical functioning, bodily pain, social functioning, general mental health, physical and emotional role limitations, vitality, and general health perceptions) in 5 to 10 minutes. Internal consistency, test-retest reliability, and construct validity have been established.¹⁰⁰

Data from Weiss et al¹⁰¹ suggest that ADHD-specific QOL measures may be more sensitive than multidimensional measures. ADHD-specific rating scales include the ADHD Impact Module (AIM), which is available in a child version completed by the parent (AIM-C)^102,103 and an adult self-report version (AIM-A).^104,105 The AIM-C includes subscales evaluating QOL at home and the impact of ADHD, as well as 9 items assessing treatment status, history, and demographic information. The AIM-A has 4 global QOL items, 5 economic impact items, and 5 subscales (living with ADHD; general well-being; work, home, and school performance and daily functioning; relationships and communication; and impact of symptoms). Psychometric data from limited clinical samples indicate acceptable internal consistency, validity, reliability, and sensitivity.^103,105

In adults, the Adult ADHD QOL Measure (AAQOL) assesses health-related QOL during the past 2 weeks.¹⁰⁶ The 29-item measure provides an overall score and 4 subscale scores (life productivity, life outlook, relationships, and psychological health).¹⁰⁶ Psychometric data show good reliability, validity, and sensitivity.^106,107

Treatment Effects

Based on studies using different measures, there is good evidence that ADHD pharmacotherapy can improve QOL in children and adolescents with ADHD.^80,83,85,108 Manos et al¹⁰⁸ demonstrated significant improvement in AIM-C QOL scores with very large effect sizes (Cohen d ≥ 1.5) in children treated with transdermal methylphenidate for 8 weeks. Data also suggest that QOL improvements are maintained with prolonged treatment (eg, 40 weeks to 2 years).^80,85 Studies using the AAQOL and AIM-A have also reported significant improvement in QOL in adults with ADHD receiving pharmacotherapy.^64,88,109 In a large community-based sample of adult patients with ADHD, Weiss et al¹⁰¹ demonstrated that treatment with long-acting mixed amphetamine salts simultaneously improved symptoms (measured by ADHD RS-IV) and QOL (AIM-A and SF-36). The effect of psychosocial treatment on QOL is an area of needed research.

ADAPTIVE LIFE SKILLS

Adaptive functioning (“activities of daily living”) refers to the ability to interact with society and to care for oneself. It represents one aspect of overall functioning that targets the individual’s ability to meet the demands of his or her environment and to manage routine tasks of self-care. Relatively few studies have measured adaptive life skills in ADHD. Despite the fact that adaptive life skills are outcomes that might be expected to be responsive to medication, studies to date have suggested that this is an area that is often severely impaired and treatment resistant. Russell A. Barkley, PhD, author of ADHD and the Nature of Self-Control,¹¹⁰ has written, “ADHD is not a disorder of knowing what to do but one of doing what one knows.”^111(p8) This is particularly obvious in activities of daily living; for example, a child may know how to shower and brush his teeth but fail to carry through these activities on his own initiative.

Stein et al¹¹² demonstrated that children with ADHD have significantly impaired adaptive functioning across several domains (eg, daily living, socialization, and communication) compared with normative data. A similar discrepancy between average intelligence and impairment of adaptive functioning was reported by Roizen et al.¹¹³ Stein et al¹¹² suggest that children with ADHD fail to perform routine tasks because of inattention, poor self-control, and motivational deficits. Others suggest that people with ADHD develop maladaptive and counterproductive coping strategies (eg, blaming, avoidance, and resignation) owing to high stress levels and inadequate social support structures.^114,115

Adults may be particularly vulnerable to adaptive dysfunction as a result of many years of living with ADHD and the repeated reinforcement of negative thoughts and beliefs.^116,117 As a result, maladaptive coping strategies, such as brinkmanship and procrastination, may emerge.¹¹⁷

Measurement Approaches

Tools for measuring adaptive functioning are limited (Table 1). The Vineland Adaptive Behavior Scales (VABS)^118,119 can be used to assess adaptive functioning in children and young adults (≤ 19 years) across communication, daily living skills, socialization, and motor domains; it is administered as an interview with the parent or primary caregiver.¹¹⁹ The VABS has good psychometric characteristics¹¹⁹ and has been used to demonstrate adaptive dysfunction in ADHD.^112,113

Other assessments include indirect evaluation of self-esteem and self-efficacy^55,120 and the use of a multidimensional coping scale such as the Kidcope,¹²¹ which asks adolescents to rate the frequency of use of 10 coping strategies. The Kidcope has been used to evaluate outcomes of these strategies in hyperactive girls.^115,121 Psychometric data suggest adequate reliability and concurrent validity.¹²¹ Another ADHD-specific measure for children with ADHD is the 24-item Life Participation Scale, which assesses changes in adaptive functioning related to ADHD treatment in home, school, and social environments.¹²² Psychometric evaluation indicates adequate to good internal consistency, responsiveness, convergent validity, divergent validity, and discriminant validity.

The Adaptive Behavior Assessment System (ABAS) is a reliable and well-normed scale completed as a parent or teacher report that provides standard scores for communication, functional academics, self-directed activity, leisure, social skills, community use, home living, school living, health and safety, and self-care.^123,124 The ABAS has been widely used as an indicator of the need for services. The Instrumental Activities of Daily Living (IADL) Scale is an 8-item, clinician-administered questionnaire typically used in geriatric populations; interrater reliability is good.¹²⁵

Treatment Effects

Further research is needed to evaluate treatment effects on adaptive functioning. However, there is some limited research addressing the effects of behavioral therapy on this outcome. Both CBT and ADHD coaching encourage the development of coping strategies to replace ineffective behaviors resulting from negative thoughts and beliefs.^114,116

EXECUTIVE FUNCTION

Children and adolescents with ADHD demonstrate deficits in executive functioning (eg, memory, executive attention, planning, task switching, and response inhibition) across multiple neuropsychological tasks.^126–129 A study in adolescents reported a nearly 3-fold higher frequency of such deficits in individuals with versus without ADHD (52% vs 18%).¹²⁷ In adulthood, the daily demands on executive function tend to increase owing to increasing responsibilities and diminishing parental support. As such, the negative impact of deficits in cognitive and executive function in adults with ADHD is not unexpected.^130,131

Measurement Approaches

Traditionally, executive function has been assessed using neuropsychological tests that target discrete tasks considered representative of executive functioning. Tests suggested to have particular utility in ADHD¹²⁸ are the Continuous Performance Test (sustained and selective attention and impulsivity), Stroop task (reaction time and cognitive flexibility), Trail Making test (visual attention and task switching), verbal fluency test (short-term memory), Wechsler Adult Intelligence Scale (general cognitive function, working memory, and processing speed), Stop-Signal test (inhibitory control), and Paced Auditory Serial Addition test (working memory, sustained and divided attention, and processing speed).^132–134

Neuropsychological tests require considerable training to administer, so simpler and faster options are desirable in the clinical setting (Table 1). The Behavior Rating Inventory of Executive Function (BRIEF) for children and adolescents aged 5–18 years (86 items)¹³⁵ and BRIEF-A for adults aged 18–90 years (75 items)¹³⁶ evaluate multiple domains of executive functioning (inhibition, shift, emotional control, self-monitor [adults only], initiation, working memory, planning/organizing, organizing materials, and task monitoring). The first 3 (4 for adults) and last 5 domains are used to derive a Behavioral Regulation Index and Metacognition Index, respectively, which can be summed to obtain a Global Executive Composite score. The BRIEF includes separate questionnaires for parents and for teachers; the BRIEF-A includes a self-report and informant report. The BRIEF and BRIEF-A have good psychometric characteristics, with normative data available for children and adults.^135,136 Because children with ADHD were included in the BRIEF validation, the manual includes a section on clinical utility in ADHD.^135,137 There is some overlap between ADHD symptom scales and the BRIEF.¹³⁸

The Brown Attention-Deficit Disorder Scales (BADDS),¹³⁹ developed based on clinical interviews, have separate rating scales for individuals aged 3–7, 8–12, 12–18, and ≥ 18 years.¹⁴⁰ A self-report is available for ages 8–12 years, and the version for ages ≥ 12 years incorporates a self-report and observer report. Adolescent and adult scales comprise 40 items spanning 5 clusters of executive functioning (activation, focus, effort, emotion, and memory); the child and adolescent scales include 1 additional cluster for self-regulation. Psychometric characteristics are good across these age groups.¹⁴¹

The Barkley Deficits in Executive Functioning Scale (BDEFS) evaluates multiple dimensions of adult executive functioning in daily life, including time management, organization and problem solving, self-restraint, self-motivation, and emotional self-regulation.¹⁴² The BDEFS, available as a self-report and other report and in long and short form, has high internal consistency and test-retest reliability, good interobserver agreement, and demonstrated validity. Unlike the BRIEF, which has adolescent and adult versions, the BDEFS measures executive function exclusively in adults. The BDEFS is easily obtained at minimal cost.

The Children’s Organizational Skills Scales (COSS)^143,144 assess how children organize their time, materials, and actions to accomplish tasks at home and school; it is a multi-informant assessment that includes responses from parents and teachers and a self-report.

Neuropsychological testing reveals that some but not all children with ADHD have enduring executive dysfunction.¹⁴⁵ The prevalence of executive dysfunction is much higher with narrowly defined rating scales than with broadly defined tests^18,146 and probably has greater ecological validity with narrowly defined scales. This is particularly true since many of the most critical aspects of executive function, such as carrying through with long-term goals, require assessment over time that is impossible in a cross-sectional evaluation on a single battery of psychological tests. Using self-ratings of executive functioning from the Adult ADHD Clinical Diagnostic Scale, Kessler et al¹⁸ identified executive function deficits in 78% of patients who met full childhood and adult ADHD criteria. Executive function and mood regulation are salient impairments in ADHD.^147,148

Treatment Effects

Stimulants can significantly improve some aspects of executive function. In children, improved executive function has been demonstrated using neuropsychological tests¹⁴⁹ and the BRIEF.^79,150 There have been attempts to systematically determine whether methylphenidate affects the working memory component of executive function. In an elegant series of experiments, Bedard et al^151–154 have demonstrated that methylphenidate can improve visual spatial working memory and listening.

In adults with ADHD, data from an open-label trial of lisdexamfetamine dimesylate¹⁵⁵ and pooled data from 2 randomized controlled trials of triple-bead mixed amphetamine salt⁸⁸ demonstrate improvement across all 5 BADDS clusters. Biederman et al¹⁵⁶ reported that young adults with ADHD currently taking stimulants had significantly better sustained attention and verbal learning scores on neuropsychological assessments than did those not currently taking stimulants. There were no significant differences in aggregate scores or scores for working memory, interference control, set shifting, visuospatial organization, and processing speed. Cognitive-behavioral therapy treatments that specifically target executive dysfunction, such as time management and organization training for school, or the CBT for adults with ADHD programs,^157–159 have all demonstrated moderate-to-large effect sizes in improving executive function.

DISCUSSION

The literature suggests that patients with ADHD exhibit impaired functioning, QOL, adaptive skills, and executive function that often persist throughout their lives. These areas of dysfunction are the problems that lead patients to seek treatment. Although ADHD symptoms are a core aspect of diagnosis, focusing on symptom improvement as a treatment outcome is insufficient, because symptoms represent only part of outcome. When initiating a new treatment (eg, titrating medication), it is appropriate to collect symptom ratings to assess response, because ADHD symptom ratings are most sensitive to treatment response in the short term. However, once treatment is stabilized, it becomes necessary to assess additional domains to ensure that meaningful improvement is realized in patients with ADHD.

During initial titration, as well as during maintenance of treatment, it is critical that ratings be collected from multiple raters (eg, parent and teacher for a child; self and spouse for an adult), because the level of agreement among raters may be low. For example, in a report by Lavigne and colleagues,¹⁶⁰ it was noted that agreement between parent and teacher ratings for ADHD symptoms was too low for clinicians to rely only on parent reports. This may be especially important when assessing multiple domains of functioning (eg, academic impairment), for which input from other raters (eg, teachers) provides a broader and often more objective perspective regarding improvement and degree of impairment than is typically found in parent ratings or in adult self-ratings. Ratings indicating residual deficits in functionality, QOL, adaptive life skills, or executive functioning suggest the need for escalation of current treatment or, more likely, for additional treatment modalities to remediate the full breadth of ADHD-related impairment.

We have demonstrated that there are now psychometrically sound measures of adaptive skills, executive function, functioning, and QOL in ADHD. Thus, we believe that it is currently feasible to broaden outcome assessments beyond symptom rating scales. In an attempt to provide guidance to mental health professionals treating patients with ADHD, we have derived a sample battery of measures that could be used to assess the full range of outcomes discussed in this article (Table 2). We have selected assessment batteries for children and adults using measures that are readily accessible, economical, comprehensive, and sensitive to treatment effects. Moreover, to be parsimonious, we have preferentially selected measures that assess multiple domains in a single instrument as opposed to selecting different instruments for individual domains. We also have included time estimates (55 minutes for the child test-battery; 40 minutes for the adult battery) to demonstrate that the suggested battery should not place an undue burden on patients or their families, especially because such assessment of treatment response need only occur 2 to 4 times per year.

Table 2.

Suggested Battery for Assessing a Full Range of Treatment Outcomes in Children and Adults With ADHD

Treatment Outcome	Children (time to complete)	Adults (time to complete)
ADHD symptoms	NICHQ Vanderbilt Assessment Follow-Up162,163 parenta and teacherb reports (5 min)	Adult ADHD Self-Report Scale38,39 (5 min)
Functional impairment	NICHQ Vanderbilt Assessment Follow-Up162,163 parenta and teacherb reports (5 min)	Weiss Functional Impairment Rating Scale74–76 (5 min)
Quality of life	Child Health and Illness Profile91–93 (20 min)	Short Form-36 General Health Survey99 (20 min)
Adaptive skills	Adaptive Behavior Assessment System,123,124 parent and teacher report (15 min)	No good scale for measuring adaptive life skills in adults with ADHD is currently available; the scales have been developed for geriatric and stroke victims and focus on ability to do tasks rather than functioning in day-to-day activities
Executive functioning	Behavior Rating Inventory of Executive Function,135 parent and teacher report (10 min)	Barkley Deficits in Executive Functioning Scale, (ADHD-specific)142 (10 min) or the Behavior Rating Inventory of Executive Function–Adult version,135 broadband (10 min)

Abbreviation: ADHD = attention-deficit/hyperactivity disorder, NICHQ = National Initiative for Children’s Healthcare Quality.

^aAvailable at http://www.nichq.org/toolkits_publications/complete_adhd/05VanFollowUp%20Parent%20Infor.pdf.

^bAvailable at http://www.nichq.org/toolkits_publications/complete_adhd/06VanAssessFollowUpTeachInfor.pdf.

We also propose that a broader range of measures be included in ADHD treatment outcome research. The vast majority of ADHD treatment outcome studies have examined pharmacologic treatments, and most pharmacologic studies have focused on ADHD symptoms as the primary outcome. Studies like the Multimodal Treatment Study of Children with ADHD suggest that, although medication may be superior to psychosocial treatment for improving ADHD symptoms, psychosocial treatment alone or in combination with medication may better target other areas of impairment (eg, parent-child relationship and academic functioning).¹⁶¹ We believe that adding other measures of outcomes can elucidate areas in which pharmacologic treatments may be ineffective and identify other treatment modalities (eg, psychosocial treatment) that may be necessary to address areas of impairment in patients with ADHD.

Additional research is needed to characterize interrelationships among these concepts. This will also require measures that avoid redundancy and confusion between distinct outcomes. Further research on functioning, life skills, executive function, mood regulation, and QOL will provide us with a better understanding of the patient’s life experience of ADHD, which moves beyond whether or not they still meet DSM-based diagnostic criteria. Given the uncertain reliability of patient report (eg, positive illusory bias), this research should also improve our understanding of differences in perspective.

This review is limited by the absence of a quality assessment of the studies cited and the absence of studies that review the interrelationships between different outcomes. We have demonstrated that our field has grown beyond core symptoms to examine not just the disorder but the patient’s overall well-being.

Drug names: atomoxetine (Strattera), lisdexamfetamine (Vyvanse), methylphenidate (Focalin, Daytrana, and others), mixed amphetamine salts extended release (Adderall XR).

Potential conflicts of interest: Dr Epstein has received research funds from Eli Lilly. Dr Weiss has received research funds, honoraria and consulting fees from Janssen, Eli Lilly, Shire, and Purdue.

Funding/support: Shire Development LLC, Wayne, Pennsylvania, provided funding to Complete Healthcare Communications, Inc (Chadds Ford, Pennsylvania) for support in writing and editing this manuscript.

Role of sponsor: Although Shire Development LLC was involved in the topic concept and fact checking of information, the content of this article, the ultimate interpretation, and the decision to submit it for publication in The Primary Care Companion for CNS Disorders was made by the authors independently.