Developing the Symptoms and Functional Impairment Rating Scale: A Multi-Dimensional ADHD Scale

Abstract

Objective

This research developed a practical, multi-dimensional attention deficit hyperactivity disorder (ADHD) rating scale (i.e., the Symptoms and Functional Impairment Rating Scale, SFIRS) for Chinese children, aged 6–12 years, with ADHD.

Methods

The structural validity, criterion validity, internal consistency, and test-retest reliability of the scale were evaluated. Item screening was conducted with 412 ADHD patients and 322 developmentally typical controls.

Results

The scale includes 44 items, divided among Hyperactivity-Impulsivity, Self-Control, Inattention, Self-Management, Academic Performance, and Social Interaction. The six-factor model showed good data fit, with each factor significantly correlated with its corresponding criterion (r=0.690–0.841). The Cronbach's α of the full scale was 0.976. Total score test-retest reliability was r=0.816 (p<0.01).

Conclusion

The SFIRS thus demonstrated good reliability and validity and may be used to assess ADHD among children aged 6–12 years in China.

INTRODUCTION

Attention deficit hyperactivity disorder (ADHD) is among the most common mental disorders affecting school-age children around the world.¹ ADHD diagnosis is primarily based upon presentation of characteristic symptoms on two or more occasions, and the impairment of social and academic functions. Functional impairment is important, but is currently poorly defined. The DSM-5 describes it as poor performance in school (i.e., poor grades); being teased, ignored, or rejected by peers; and disturbed family relationships.² Functional impairment is a major treatment target and negatively affects ADHD prognosis. Additionally, it is significantly correlated with clinical referral, choice of treatment, and utilization of medical services. However, it is usually inconsistent with symptom severity, and, accordingly, requires separate evaluation.³ The tools available for this purpose are typically time-consuming and have not been standardized within a Chinese population. The Weiss Functional Impairment Rating Scale (WFIRS) is the only available multi-dimensional, ADHD-specific, functional impairment assessment scale.⁴ The WFIRS examines a wide range of functional impairment symptoms, with many of the items included assessing the severest forms of impairment. Therefore, this scale typically exhibits a significant floor effect, resulting in a limited ability for clinical application.⁵ The Children's Global Assessment Scale (CGAS) is also commonly used to assess ADHD symptoms. However, the results of this scale can easily lead to the confusion of clinical symptoms with impairment. Furthermore, the one-dimensional structure of this measure cannot examine performance in different functional areas.⁶

Children with ADHD tend to exhibit executive function deficits, causing impairment in function in daily life,⁷ leading to negative effects on academic and occupational functions in adulthood.⁸ Chinese elementary schools tend to impose a strict classroom environment and considerable academic pressure on students, therefore requiring high-quality executive function. In this context, executive function deficits often drive parents to take their children to hospitals. However, executive function deficit evaluation within this context is often limited to neuropsychological tests.

Extensive research indicates that the primary impairment of ADHD is in executive function.⁹,¹⁰,¹¹,¹² Accordingly, several scales have been developed to examine executive function behavior deficits in daily life, such as the Brown Attention Deficit Disorder Scale (BADDS), the Behavior Rating Inventory of Executive Function (BRIEF), and the Current Behavior Scale (CBS). Among these scales, only the BRIEF has been translated into Chinese; however, it is not ADHD specific and is relatively time-consuming to use. Barkley et al.¹² found that six executive function items, combined with three DSM items, best distinguished between adult clinical controls, community controls, and adult ADHD patients; yet, no research has examined the characteristics of executive function deficits among school-aged Chinese children with ADHD in daily life and contribution of these deficits to an ADHD diagnosis.

Some school-age ADHD assessment scales are commonly used in China; however, those with established norms and broadly tested reliability and validity are mainly “broadband” symptom screening scales, such as the Conners Parent Symptom Questionnaire (PSQ)¹³ and the Strength and Difficulty Questionnaire (SDQ).¹⁴ Availability of ADHD-specific assessment tools and scales examining executive function deficits and functional impairment is poor, and even when an ADHD-specific measure is available, cross-cultural adaptability is often problematic. For instance, some of the WFIRS scale items are not applicable within the Chinese cultural context. This would include questions regarding the use of illegal drugs and problems keeping regular appointments with doctor/dentist. Within the Chinese culture, there are very few drug problems in elementary schools. As such, some parents feel offended by this item, and therefore, stop attending the evaluation. Furthermore, keeping medical appointments is not a routine in China; therefore, this item is not applicable.

China contains only around 500 child psychiatrists (as reported by ChinaNews, http://finance.chinanews.com/jk/2014/08-07/6467509.shtml). Accordingly, pediatricians are often the first health care professionals to assess children with ADHD. Pediatric psychiatrists typically spend thirty minutes or less examining children with possible ADHD. Patients generally do not make appointments, largely eliminating the possibility of prior examination, and many patients do not return for follow-up examinations. In the West, it is a common practice to combine a range of tools to examine the patient's symptoms, executive function, academic performance, social adaptation, and, possibly, family function; however, this is economically unrealistic in China. ADHD examination is therefore often limited to the examination of symptoms. To address this, the current study aimed to screen for the most common ADHD-specific executive function deficits and impairment traits, and to develop a simple and practical multi-dimensional rating scale that meets the primary demand for general outpatient service in China. Developing such a scale will help Chinese psychiatrists implement standardized multi-dimensional evaluations of ADHD.

In this context, this research aimed to 1) develop a simple, practical, multi-dimensional ADHD rating scale, targeting Chinese children with ADHD, aged 6–12 years, which examines the most common ADHD-specific symptoms, executive function deficits, and functional impairment and 2) modify the preliminary version of this scale and establish its psychometric properties.

METHODS

Preliminary research

Our scale, developed in Mandarin, was named the Symptoms and Functional Impairment Rating Scale for ADHD (SFIRS). It was designed to examine parents or caregivers of children with ADHD. The target population of The SFIRS is Chinese children, aged 6–12 years, with ADHD or in the ADHD diagnosis process. We chose this age group because most discovery and diagnosis of ADHD occurs in elementary school,¹⁵ and because the learning and living environment of this age group is relatively consistent, leading to considerable consistency in ADHD manifestation. Following guidelines of the DSM-5, this scale requires raters to recall the target child behaviors and difficulties from the previous six months. The SFIRS is principally intended for use in clinical assessment of ADHD severity; therefore, responses use a 5-point Likert scale (0=never, 1=occasionally, 2=sometimes, 3=often, 4=always). The items are all positively scored. Higher scores indicate more serious symptoms or more severe impairment.

A pilot study involving the development and selection of the item pool was performed from February 2013 to June 2014. Content validity was established through literature reviews, patient interviews, pilot testing, and Delphi consultation.

The SFIRS was theoretically divided into the two subscales of “Clinical symptoms” and “Functional Impairment.” “Clinical Symptoms” include “Hyperactivity-Impulsivity,” “Inattention,” and “Executive Function Deficits.” Based on the executive function theories of Brown, Barkley and others,⁹,¹⁰,¹² items were designed to examine the deficits of working memory, planning, time management, self-monitoring, and emotional control. Following the DSM-5, “Functional Impairment” was divided into two factors of “School performance” and “Social interaction.” This framework was intended to overcome limitations affecting existing clinical assessment scales and to combine ADHD's core symptoms, executive function deficit presentations, and functional impairment, into a single, relatively brief scale.

The original item pool was generated by translating and rewriting the symptom criteria of the DSM-5, with some symptoms divided or concretized, to facilitate understanding among caregivers. We also searched the literature published in China and abroad using keywords such as ADHD, rating scale, executive function, functional impairment, and evaluation, and referred to Barkley et al., Barkley, Brown, McCloskey, Chandler.¹⁵,¹⁶,¹⁷,¹⁸,¹⁹ Regarding existing ADHD scales, we considered the PSQ,¹³ CBCL,²⁰ SDQ,¹⁴,²¹ Vanderbilt rating scale,²² BRIEF,²³ WFIRS,⁴ and IRS.²⁴ Each of these sources informed the original item pool.

Unstructured interviews were conducted with 12 ADHD outpatients and their parents at the Child & Adolescent Clinic of Shanghai Mental Health Center. These interviews examined descriptions of ADHD symptoms and functional impairments provided by interviewees, and further informed the development and modification of SFIRS items. The following questions were asked:

• 1) What kind of difficulties and problems does your child have right now?

• 2) Can you describe these difficulties?

• 3) What approaches do you use to help your child?

• 4) What consequences have these difficulties had?

• 5) How do these difficulties and their consequences affect your daily life?

Interviewees' answers were recorded in writing, and the item developers evaluated descriptions not included in the item pool for addition to the item pool. Parents' expressions were preferentially used to modify existing items to adapt the items to parents' language habits, thereby facilitating item comprehension.

We then invited another eight parents of ADHD outpatients to evaluate the preliminary item pool by verbally describing their opinion of each item (e.g., regarding the clarity, apparent meaning, and accuracy of each item as a description of their child). We further modified the existing items following the results of this interview.

Four psychometric experts approved the overall feasibility and rationality of each item using Delphi consultation. We asked these experts to judge if each item should be retained in the item pool according to the theoretical assumptions of the scale. Items supported by at least three experts were retained.

We then performed pilot testing, using 139 items to examine 78 and 81 children with and without ADHD, respectively. Item selection was performed using t-tests, Pearson's correlation analysis, Cronbach's α, and principal component analysis (PCA).

All mean scores of each item differed significantly between the two groups (p<0.001). We listed 48 items for deletion using the cutoff criterion of t<9. In both groups, the mean score of each item was significantly correlated with the mean total score of each group (p<0.01). Only two items had correlation coefficients <0.4. We listed 35 items for deletion using a cutoff score of r<0.6. Some of these items were the same as those listed in the previous step. Deletion of two items led to an increase in Cronbach's α; therefore, these items were also listed for deletion. Six items were listed owing to a factor load of <0.45; some of these were the same as those listed in the previous steps. Ten items were listed because their mean score was <1.5 or because they showed high skewness, and some of these were the same as those listed in the previous steps. Finally, we calculated correlations between each item. If the correlation between two items was >0.6 and those two items were similar in content (as judged by the developer), one of these items was listed for deletion. Fourteen items were listed at this step.

In total, we deleted 71 items following pilot testing. We then conducted a second Delphi analysis with a further six experts, examining a preliminary version of the SFIRS, including 68 items retained following the pilot study. This analysis asked the experts to retain, delete, or modify each item. Experts rated the importance of retained items using a 5-point scale (1=unimportant, 5=very important) and made specific suggestions regarding items identified for modification. Items were listed for deletion if one or more expert(s) identified them for deletion or if they received two or more importance ratings of <3.

We subsequently modified the identified items following the experts' suggestions and established the preliminary SFIRS with 59 items. The above method has been submitted for publication as a PHD thesis in Chinese.

This research reports modification of the preliminary SFIRS and establishment of its psychometric properties.

Participants

We collected data from 412 ADHD outpatients (Shanghai=162, Nanjing=47, Fuzhou=44, Tianjin=47, Zhengzhou=53, Shenyang=42, and Hefei=17) from September to December 2014. All participants with ADHD met DSM-5 diagnostic criteria, were aged 6–12 years, and had an IQ ≥80 (as measured by the Wechsler Intelligence Scale for Children-IV, WISC-IV²⁵). Developmentally typical children (n=322) were recruited from a primary school in Shanghai's Fengxian District, and were selected using a random number table method. Age and gender distribution of the two groups were matched. We excluded children with major physical diseases, bipolar disorder, psychotic disorder, and other severe psychiatric disorders. Children in the control group with parent-reported diagnosis or Conners Parents Symptom Questionnaire hyperactivity index greater than x+2SD were excluded. Each participant's caregiver voluntarily provided written consent to participate in the study prior to study commencement. Ethical approval for this study was provided by the Ethical Review Board at Shanghai Mental Health Center (2015-01).

Measures

The Swanson, Nolan, and Pelham Questionnaire (SNAP-IV)

The SNAP-IV²⁶,²⁷ was developed using the DSM-IV criteria for ADHD and Oppositional Defiant Disorder (ODD).²⁸ It includes subscales of Inattention, Hyperactivity/Impulsivity, and ODD. The target child's teacher or parents respond to 18 question items using a 4-point Likert scale (0=not at all, 3=very much). The Chinese version of SNAP-IV has good internal consistency (α=0.95) and test-retest reliability (ICC=0.68), and its sensitivity and specificity for ADHD diagnosis are 0.87 and 0.79, respectively.²⁹

Conners Parent Symptom Questionnaire (PSQ)

The PSQ³⁰,³¹ examines behavioral disorder symptoms. The target child's caregivers respond to 48 question items using a 4-point Likert scale (0=Not true at all, Never, Seldom; 3=Very much true, Very often, and Very frequently). The PSQ includes factors for Conduct Problems, Learning Problems, Psychosomatic, Impulsive Hyperactivity, and Anxiety. A Hyperactivity Index is also calculated by averaging ratings given to 10 items. The Chinese version of the PSQ scale has been widely used in China since the 1990s, and normative national data was established in 2001 and 2005.¹³,³² This scale has a Cronbach's α of 0.932 and test-retest reliability of 0.594.¹³

The Weiss Functional Impairment Rating Scales-Parent (WFIRS-P)

The WFIRS-P⁴ contains 50 items organized into the functional domains of Family, Child's Self-Concept, Learning & School, Social Activities, Life Skills, and Risky Activities. Caregivers of children and teens, aged 6–16 years, respond to items using a 4-point Likert scale (0=Never or not at all, 3=Very often or very much). In the domains of the WFIRS-P, two or more item scores ≥2, one item score of 3, or a mean score >1.5 indicates impairment. Qian, Du, Qu, and Wang translated this scale into Mandarin.⁵ In this study, the test-retest reliability was 0.61–0.87 and the Cronbach's α of domains were 0.70–0.92 in a Chinese sample.

Behavior Rating Inventory of Executive Function (BRIEF)

The BRIEF-Parent Version²³ examines executive function deficits in daily life among children and teens aged 6–16 years. This scale contains 86 items divided among the factors of Monitor, Organization of materials, Plan/organize, Working memory, Initiate, Emotional control, Shift, and Inhibit. Responses to question items use a 3-point Likert scale (N=never, S=sometimes, O=often). Scores on the former five and the latter three factors are summed to give indexes of Meta-Cognition and Behavioral Regulation, respectively. Qian translated this scale into Mandarin,³³ with a test-retest reliability and Cronbach's α of 0.61–0.89 and 0.74–0.96, respectively.

Statistics

Statistical and factor analysis were carried out using SPSS 17.0 (IBM Corp., Armonk, NY, USA) and AMOS 17.0 (IBM Corp.), respectively.

Item analysis

Skewness and kurtosis of item scores were calculated to evaluate the normality of the score distribution, with smaller scores considered to indicate greater normality. Skewness and kurtosis scores were considered acceptable between −1 and 1. Scores <−1 and>1 were considered to indicate floor and ceiling effects, respectively. Differences in mean item scores between groups were compared using t-tests. Pearson's correlation coefficient, Cronbach's α, and PCA were used to examine item-scale homogeneity.

Reliability

Raw factor scores and total subscale scores were calculated. Cronbach's α and the Spearman-Brown split-half coefficient were used to examine internal consistency. Reliability was considered acceptable or good if α>0.8 or 0.9, respectively.³⁴

Test-retest reliability represents the consistency of the score of subjects at different time points. Ninety-three control group participants were randomly selected after stratification by grade and sex of the whole control group. These individuals were re-tested after six weeks test scores were compared using paired t-tests to examine test-retest reliability.

Validity

Content validity

The development process of the SFIRS was designed to yield maximal content validity, with the theoretical background of the scale supported by extensive evidence and its items drawn from published literature and clinical interviews and revised by experts in the field.

Construct validity

To examine the construct validity of the SFIRS, we randomly separated participants into two groups, each of which was used for exploratory and confirmatory factor analysis (EFA and CFA, respectively).

A separate sample of participants (203 with ADHD and 158 controls) was used to test the hypothetical model obtained via EFA. Fit indices and their corresponding criteria were as follows: χ²/df<5, comparative fit index (CFI) >0.95, root mean square error of approximation (RMSEA) <0.06.³⁵

Convergent and divergent validity

Bonferroni-corrected bivariate correlation coefficients were calculated between participant SFIRS scores and the scores of other measures, to test the convergent validity of the SFIRS. These included the WFIRS (Functional Impairment), the SNAP-IV's subscales and the PSQ's Hyperactivity Index (ADHD symptoms), and the BRIEF (executive deficit presentations). Anxiety and psychosomatic symptoms differ from ADHD symptoms and functional impairment. Accordingly, the divergent validity of the SFIRS was examined by calculating correlations between participant scores on the SFIRS and scores on the PSQ non-ADHD subscales, including the Psychosomatic and Anxiety subscales.

RESULTS

Demographic characteristics

Participants were 344 male and 68 female children with ADHD, and 273 male and 49 female developmentally typical children. The ADHD group was aged 8.67±1.68 years (range: 6.05–12.98); the control group was aged 8.76±1.42 years (range: 6.22–11.38). The groups did not differ significantly regarding age (t=0.799, p=0.424) or gender (χ²=0.224, p=0.685).

Thirty-two and 60 ADHD cases were comorbid with tic disorder and ODD, respectively. Regarding medication, 275 of the children with ADHD were not taking medication, 50 were taking methylphenidate, and 61 were taking atomoxetine.

Item analysis

Two items showed skewness (i.e., <−1) in the ADHD group, however, no other items showed floor or ceiling effects. In the ADHD group, total score skewness and kurtosis were −0.439 and 0.207, respectively. These values indicate a generally normal score distribution.

All mean scores of items differed significantly between the groups (p<0.001), indicating that all items were able to distinguish between children with and without ADHD. Regarding the ability of the SFIRS to indicate relative symptom severity, we compared scores on items 1–37 between participants who scored in the highest and lowest 27% on the SNAP-IV. All mean scores of items differed significantly between these two subgroups; however, four items yielded t-scores <3 and were therefore deleted. Regarding the ability of the SFIRS to indicate relative functional impairment, we compared scores on items 38–59 between participants who scored in the highest and lowest 27% on the WFIRS. Two items yielded p-values>0.05 and were therefore deleted. Therefore 53 items were retained following this analysis.

All items were adequately correlated with their respective subscale's total score (r>0.4; criterion chosen by the author; p<0.01). All items were more strongly correlated with their own subscale than with any other subscale. The Cronbach's α of the 53 retained items was 0.980 and could not be increased by the deletion of any item.

Construct validity

The 53 retained items were entered into EFA with PCA and orthogonal rotation. Items without a factor load >0.45 or inconsistent with factor content were deleted; 44 items were subsequently retained.

EFA yielded six factors with a variation contribution of 70.48% (KMO=0.969, Bartlett Test of Sphericity p=0.000). The extracted factors were consistent with the theoretical hypothesis.

Table 1 presents factor load on 44 items retained after EFA.

Table 1. Exploratory factor analysis^*.

^*203 and 158 participants in the ADHD and control groups, respectively. H-I: hyperactivity-impulsivity, S-C: self-control, INA: inattention, S-M: self-management, SP: school performance, SI: social interaction

Factor one included eight items mainly describing hyperactivity-impulsivity symptoms; accordingly, it was named “Hyperactivity-impulsivity.”

Factor two included four items mainly describing emotion control difficulties and delay aversion; it was named “Self-control.”

Factor three included nine items mainly describing inattention symptoms; it was named “Inattention.”

Factor four included seven items mainly describing difficulties with working memory and planning; it was named “Self-management.”

Factor five included seven items mainly describing academic and school performance impairment; it was named “School performance.”

Factor six included nine items mainly describing social interaction impairment; it was named “Social interaction.”

Figures 1 and 2 present CFA results describing the data fit of the extracted model. In both groups, χ²/df<1, RMSEA <0.001, CFI >0.999, indicating good data fit in both groups.

Figure 1. Goodness-of-fit for the six-factor model (ADHD group).χ²=2.3 (p=0.522); df=3; RMSEA=0.000; CFI=1.000. df: degree of freedom, RMSEA: Root Mean Square Error of Approximation, CFI: comparative fit index.

Figure 2. Goodness-of-fit for the six-factor model (control group). χ²=2.5 (p=0.473); df=3; RMSEA=0.000; CFI=1.000. df: degree of freedom, RMSEA: Root Mean Square Error of Approximation, CFI: comparative fit index.

The hypothetical model of the scale proposes that “Clinical Symptoms” describes ADHD symptoms and “Functional Impairment” describes the consequences of those symptoms on the patient's daily life. The author attempted to differentiate between “Clinical Symptoms” and “Functional Impairment” in the factor analysis. The score of each factor was calculated, and another round of EFA was performed on the six resulting variables, using PCA and orthogonal factor rotation with maximum variance. This analysis yielded only one common factor explaining 75.03% of total variance. This result suggests that the six extracted factors cannot be further differentiated.

In the course of CFA modeling, the author attempted to form two distinct dimensions using the four “Clinical Symptoms” factors and the two “Functional Impairment” factors, but found that the standardized coefficient was >1 between the two dimensions, indicating poor data fit. In contrast, the single-dimension model, using all six extracted factors, showed good data fit in both groups.

To further verify this result, in the absence of other criteria, the author examined the scores of other scales used in this research, including the SNAP-IV (ADHD symptoms), BRIEF (executive function deficit), and the WFIRS (functional impairment), using the same EFA method. The results were consistent with those obtained from SFIRS, further supporting that the three components do not constitute independent dimensions (Figure 3).

Figure 3. The three-dimension model. SNAP-IV: Swanson, Nolan, and Pelham Questionnaire, BRIEF: Behavior Rating Inventory of Executive Function, WFIRS: Weiss Functional Impairment Rating Scales.

In contrast, combining these components into a common dimension yielded a model with good data fit (Figure 4).

Figure 4. The single-dimension model. SNAP-IV: Swanson, Nolan, and Pelham Questionnaire, HYP/IMP: Hyperactivity/Impulsivity, INA: inattention, BRIEF: Behavior Rating Inventory of Executive Function, BRI: Behavioral Regulation Index, MI: Meta-Cognition Index, WFIRS: Weiss Functional Impairment Rating Scales.

Consequently, despite consistency with prevailing theoretical assumptions, the author retained the SFIRS as a six-factor model, with one dimension respecting the CFA result.

Reliability

Cronbach's α for the “Self-control” subscale was 0.822, and was 0.901–0.976 for the other subscales and the total scale. Values of the Spearman-Brown split-half coefficient were 0.719–0.924 (Table 2).

Table 2. Internal consistency^*.

^*412 and 322 children in the ADHD and control groups, respectively. H-I: hyperactivity-impulsivity, S-C: self-control, INA: inattention, S-M: self-management, SP: school performance, SI: social interaction, TS: total score

The test-retest reliability (at baseline and six weeks) of subscales was 0.562–0.779 (p<0.01) (Table 3).

Table 3. Factor scores at baseline and six-week follow-up^*.

^*93 children in the control group, ^**significantly correlated at p<0.01. H-I: hyperactivity-impulsivity, S-C: self-control, INA: inattention, S-M: self-management, SP: school performance, SI: social interaction, TS: total score

Convergent validity and divergent validity

The subscale and total scores of the SFIRS were all significantly correlated with SNAP-IV's inattention and hyperactivity-impulsivity factors, except the “School performance” factor and the Hyperactivity-Impulsivity subscale among ADHD group members (p=0.183; significance for bivariate correlations set at p<0.05/12=0.004, after Bonferroni correction) (Supplementary Table 1 in the online-only Data Supplement). Correlation coefficients were 0.216–0.725 and 0.493–0.773 in the ADHD and control groups, respectively. None of the SFIRS factors were significantly correlated with PSQ Psychosomatic and Anxious factors among ADHD group members (p<0.05/12=0.004); however, all factors, except School performance (p=0.032) were moderately correlated with PSQ Hyperactivity index (r=0.410–0.595, p<0.05/6=0.008), supporting the hypothesis that the SFIRS measures ADHD-specific clinical manifestations. SFIRS factors were significantly correlated with many of the subscales from the BRIEF (p<0.05/48; 0.001) and WFIRS (p<0.05/36; 0.001). For example, the correlation between “Self-control” factor and the “Inhibit” subscale of the BRIEF was 0.631, and between “Self-management” factor and “Working memory” subscale of BRIEF was 0.638 in the ADHD group. The correlation between the “School performance” factor and “School-learning and school-behavior” subscale of WFIRS was 0.620 in the ADHD group. The correlation between the “Social interaction” factor and “Social activities” subscale of WFIRS was 0.738 in the ADHD group.

All six of the SFIRS factors were weakly or moderately correlated in the ADHD group (r=0.231–0.662, p<0.01) (Table 4).

Table 4. Correlation between factors^*.

^*412 and 322 children in the ADHD and control groups, respectively, ^**significantly correlated at p<0.01. H-I: hyperactivity-impulsivity, S-C: self-control, INA: inattention, S-M: self-management, SP: school performance, SI: social interaction, TS: total score

DISCUSSION

Our study verified that the content of SFIRS reflects the characteristics of 6–12 year old children with ADHD in elementary schools, that the time of evaluation is relatively short, and that administration is easily accepted by parents and grandparents (as the common caregivers of children in China). The number of females with ADHD taking part in our study is much less than males. This is in accordance with the findings that ADHD is generally 2–3 times more prevalent in boys than girls around the world, with boys 1.5–4.4 times more likely to receive outpatient services than girls.³⁶

In factor analysis, the factors of Hyperactivity-impulsivity, Inattention, School performance, and Social interaction were consistent with the assumption. However, items examining executive function deficit were separated into two factors, specifically “Self-control” and “Self-management.” According to the dual-pathway model proposed by Sonuga-Barke,³⁷,³⁸ the executive functions that reflect cognitive processes with little emotional arousal are defined as cold executive functions. Conversely, executive functions with emotional arousal are often referred to as hot executive functions, which activate brain areas of emotional control and reward systems. The study of Gioia¹⁰,²³ with the BRIEF also showed the existence of an emotional control factor apart from other executive function factors in a real-world situation. The four items of the “Self-control” factor address delay-aversion and emotion regulation difficulties, supporting Sonuga-Barke's “hot” executive function deficit theory. Similarly, the “Self-management” factor appears to address deficits in “cold” executive function, mainly describing working memory and planning deficits.³⁷,³⁸ Accordingly, the author retained these factors in the SFIRS.

The 6 factors of the SFIRS had low to moderate correlation with each other in the ADHD group. Nevertheless, we failed to differentiate “Clinical Symptoms” and “Functional Impairment” in the factor analysis. The SFIRS provides ADHD-specific assessment of functional impairment, which is closely related to ADHD symptom severity. The high commonality we found of these factors supports research presented by Barkley et al.¹⁵ Gordon found a highest correlation coefficient of 0.65 between ADHD symptom severity and single-dimension functional impairment; however, this value has been considerably smaller in other studies that used other measurement methods and samples.³⁹ Nonetheless, Barkley et al.¹⁵ found correlation coefficients of 0.43–0.88, and mostly >0.70, between symptom severity and a synthesized multi-dimensional index of functional impairment. Moreover, in contrast with other studies combining different tools, the SFIRS uses the same language style and methods to evaluate both symptoms and function. This result suggests that, although clinical symptoms and functional impairment are conceptually distinct, they probably reflect a common empirical dimension (i.e., “ADHD clinical manifestations” or “behavior regulation”). However, our findings with the SNAP-IV, BRIEF, and WFIRS showed that these theoretically distinct scales could only form one dimension as well. Whether this result is due to language issue or the shortcoming of assessment tools is still unknown, with further study undoubtedly needed.

Values of Cronbach's α ≥0.8 are generally considered ideal. The Cronbach's α of each factor and the full scale were all greater than 0.8; indeed, the Cronbach's α of the full scale was 0.976, exceeding that of the PSQ (0.932),¹³ the WFIRS-P (0.70–0.92),⁵ and approaching that of the SNAP-IV (0.88–0.95).²⁹

Test-retest reliability is generally considered ideal if the obtained correlation coefficient is >0.7. The “Inattention,” “Self-management,” and “School performance” subscales yielded correlation coefficients >0.7 (p<0.01). The Self-Control and Hyperactivity-Impulsivity subscale values were 0.562 and 0.663, respectively. These results may reflect the fact that many items in the latter two subscales examine impulse control and emotional stability, which are inherently concerned with instability, and is therefore more likely to vary depending on environment, mood, and pressure. A similar consideration may apply to the “Social interaction” subscale (r=0.612), as peer relationships may vary within six weeks among children without ADHD (i.e., depending on the relatively rapid acquisition or development of communication skills). This result resembles parallel findings regarding the PSQ (r=0.193–0.782),¹³ SDQ (a parent-report scale; r=0.4340–0.7870),¹⁴ SNAP-IV (r=0.24–0.76),²⁹ and WFIRS-P (r=0.61–0.87).⁵ Furthermore, the test-retest reliability for individuals with ADHD can also be studied in our future research to complete the reliability profile of SFIRS.

So far, our study has verified the validity and reliability of SFIRS. The six factors were able to give us systematized information about the severity of children's symptom, executive function deficits, and functional impairment of ADHD. We are also testing SFIRS on a larger sample, to verify its diagnostic sensitivity and specificity, the ability to differentiate different subtypes of ADHD, the response to drug treatment, and the possibility to predict prognosis.

This study has several limitations. Even though we roughly tested the readability in the preliminary study, a formal assessment of the requisite reading level is still needed to determine if the scale can be understood by the majority of our target users around the country.

The content of SFIRS has involved behaviors at home, school, and community settings. However, we found this scale more appropriate for caregivers of children with ADHD to complete, since they receive frequent feedback from teachers about children's performance at school, but teachers usually know little about their students in home and community settings. It is possible to modify this scale for the use of teachers by limiting the setting at school. The authors also plan to test inter-rater reliability in future research.

Because of time constraints, a nationally representative sample was not recruited. Participants with ADHD were recruited from seven cities in east, north, central, and northeast China, and the control group was recruited from a primary school in the Shanghai Fengxian district. The authors therefore plan to test the SFIRS using a sample that includes participants from other areas of China in order to yield more robust and nationally normative data.

This research tested the SFIRS on children with ADHD, but not on children with other types of mental health problems. In future research, the authors plan to test if the SFIRS can differentiate children with ADHD from those with tic, mood, and other disorders.

The overall score of the SFIRS may have been made more informative if representative weights were assigned to factor scores. The authors therefore plan to identify the relative weight of the SFIRS factor scores in a nationally representative sample, in order to make the overall score of the SFIRS more clinically relevant and informative. The authors are also considering examining items to evaluate response bias, negativity, and inconsistency. Additionally, since the items are all positively scored, reverse-scored items may be added in order to protect the validity of the scale.

In summary, the SFIRS may be reliably and validly used to assess ADHD symptoms, deficits in executive function, and functional impairment among Chinese children aged 6–12 years. Additionally, the SFIRS is suitable for further study, and may be clinically applied in order to help meet the demand for general ADHD outpatient treatment and psychological counseling in China.

Supplementary Materials

The online-only Data Supplement is available with this article at https://doi.org/10.4306/pi.2018.15.1.13.

Supplementary Table 1

Correlation between SFIRS and SNAP-IV