A psychometric evaluation of the interRAI Child and Youth Mental Health instruments (ChYMH) anxiety scale in children with and without developmental disabilities

Abstract

Background

With 10 to 20% of Canadian children suffering with mental illness, the importance of early identification and accurate assessment systems is clear. Unfortunately, many do not receive the mental health treatment necessary and wait-times for assessment can span up to a year. In response, the interRAI suite of assessments were designed to comprehensively assess early signs of mental health impairments in children from birth to 18 years.

Methods

This study assesses the psychometric properties of the Anxiety Scale and addresses the identification of anxiety within children diagnosed with intellectual and developmental disabilities (IDD); a commonly underrepresented sample in mental health psychometric studies. Data was collected from children aged 4–18 years in three different samples.

Results

Results indicated reliable internal consistency and factor structure, as well as moderate-to-strong convergent validity.

Conclusions

We conclude that the Anxiety Scale exhibits psychometric qualities which demonstrate its clinical utility for use within a child sample, as well as in children with IDD. The findings provide support to a larger body of research which show consistent psychometric rigour of the interRAI measures.

Background

With 10 to 20% of Canadian children and youth (hereafter both referred to as children) currently suffering from mental illness [1], the importance of early clinical identification is of paramount importance [2]. Specifically, results from Canada’s National Longitudinal Survey of Children and Youth (NLSCY) indicate the prevalence of anxiety problems as ranging from 2 to 12%. Most adult psychiatric disorders originate in childhood and show stability, persistence, and long-term aversive outcomes in adulthood [2–4]. Specifically, there is large-scale longitudinal evidence which suggests that early childhood diagnosis (as early as 3 years old) can predict later mental health diagnoses by five-fold, in addition to homotypic and heterotypic continuity [5].

Acknowledging the chronicity and devastating economic impact of long-term mental health issues [6], it is important to have an accurate assessment system of childhood psychiatric symptoms. Unfortunately, less than 75% of children receive the mental health treatment they need and wait-times for assessment at service centres range from 6 months to 1 year [7–9]. In addition, assessment of co-morbid psychiatric disorders is an exceptionally complicated problem in children with intellectual disability, who represent a non-trivial (1–3%) portion of the population [10]. A study by the Office of National Statistics in Great Britain found that children with intellectual disabilities are six-times more likely than the general population to have one or more co-morbid psychiatric disorders (e.g., anxiety, attention-deficit hyperactivity disorder, depressive disorders, conduct disorders) [11], as well as experience somatic complaints, aggressive behaviour, anxiety, and other internalizing and externalizing problems [12]. Approximately 30–50% of children with intellectual disability have a mental health disorder, compared to 8–18% for those without [13]. Further complicating the challenges for children and their families is a dearth of appropriate assessment tools for this unique population. Many assessments have a narrow focus, which may omit areas of need that could be captured through comprehensive assessment [14, 15]. Further, there is inconsistent use of assessment instruments across organizations, with many using tools that have not undergone systematic psychometric evaluation [16]. Consequently, clients may be triaged based on multiple instruments that lack well-established reliability and validity [17].

The primary difficulty when assessing children with co-morbid mental health issues and/or developmental disabilities is the variability inherent in their physical, emotional, and intellectual development [18], particularly with respect to anxiety. Anxiety-related disorders are often severe and chronic, frequently stemming from early childhood and persisting into later stages of life [19–21]. Diagnosis is frequently based on an abnormal pattern of presentation of symptoms (e.g., headache, sweating, and excessive worry) and there is a lack of consensus on its etiology [22]. However, it is the most common mental health disorder among children [23] and therefore warrants research into efficiency of its assessment. A child’s anxiety can vary with the environmental context; as a result, behaviours can be misunderstood or difficult to interpret [24]. Specifically, as children experience more complex social milieus, they often become aware of their differences, resulting in increasingly high levels of anxiety and emotional stress [25]. These issues contribute to frustration when attempting to access appropriate services, particularly for children with complex service needs [26, 27].

The need for effective assessment systems which provide a comprehensive and scientifically rigorous approach to mental health assessment led to the development of an assessment system incorporating a lifespan approach by interRAI, a non-profit, international organization network of 100 members from over 35 countries [28]. A unique feature of the interRAI assessment instruments is their integrated use along the service continuum (e.g., homecare, palliative care, emergency department) and intellectual level; they are designed to monitor symptoms and treatment outcomes across a wide range of age groups and vulnerable populations [29]. The instruments have strong psychometric properties and criterion validity in adult and geriatric samples [30, 31], children/youth samples [32–36], and across different cultures [37, 38].

One of the newest assessment suites available in the interRAI family of assessments are the child and youth instruments, designed to comprehensively assess early signs of mental health impairments in children from birth to 18 years [39–42]. Assessments are conducted through a variety of sources, including: communication with the primary caregiver, observation of the child, communication with healthcare providers, and review of medical records. The interRAI Child and Youth Mental Health Instrument (ChYMH) and the interRAI Child and Youth Mental Health and Developmental Disability (ChYMH-DD) instruments are comprised of over 400 items that assess a range of variables relevant to the physical and mental health of children [39, 40]. They assess pediatric mental health in both a dimensional and holistic framework, encouraging clinicians to understand both the individual and environmental context of the child. Specifically, information collected includes the child’s relationship with family (e.g., parenting), home environment, stress and trauma (e.g., abuse), childcare services, medications, physical health problems, treatment service utilization (e.g., self-care skills training), neuropsychological development (e.g., motor skills), communication abilities (e.g., ability to understand, ability to be understood), as well as socioemotional and behavioural skills (e.g., control of anger). In addition, the instruments also have collaborative action plans (CAPs) embedded in the instrument to provide real-time, evidence-informed recommendations for care-planning, particularly around areas of risk [39, 40, 43, 44].

Rationale for current study

The suite of interRAI Child and Youth assessments are widely used in Ontario, Canada and are now being implemented across multiple countries (e.g., United States, Netherlands, Finland). Given the importance of early identification of anxiety-related problems in children at varying levels of functioning, the present study aims to explore the factorial structure, reliability, and validity of the Anxiety Scale embedded within the interRAI Child and Youth suite among three samples: children referred for mental health services from high risk schools, children receiving mental health services and children receiving mental health services who have developmental/intellectual disabilities. This scale is embedded within a variety of instruments within the interRAI child and youth suite of instruments; therefore, it is important that the individual components are performing as expected to facilitate appropriate screening and triaging of children with mental health concerns. Moreover, the present study addresses the identification of anxiety within a subgroup of children with diagnosed developmental/intellectual disability, addressing a key piece of the literature where there is lacking available information on the prevalence of co-morbid developmental/intellectual and mental health concerns.

Methods

Data collection

We examined data from three independent samples of children, each of whom completed one of the three following instruments: the interRAI Child and Youth Mental Health assessment (ChYMH) [39], the interRAI Child and Youth Mental Health and Developmental Disability assessment (ChYMH-DD) [40], or the interRAI Child and Youth Mental Health-Screener assessment (ChYMH-Screener) [41], which is a revised, shorter screening instrument developed based on the ChYMH. Assessors who had at least 2 years of experience working with children with mental health issues received a 2.5-day intensive training on administering the assessments and completed all assessments. In a validation effort, two well-established external criterion measures were also administered to a subset of the children included in this study. The Child Behaviour Checklist (CBCL) [45] was completed for a subsample of children in the ChYMH sample, and the Behaviour Assessment System for Children, Third Edition (BASC-3) [46] was completed with children in the ChYMH-Screener sample. Only a subset of children in the study were administered these validation measures, and therefore, only this subset data was available for analysis.

Data collection for the ChYMH sample

Over 6000 clinically referred children between the ages of 4 and 18 years were assessed using the ChYMH assessment tool. Data from these children were collected from 46 mental health agencies across Ontario, Canada. Primary caregivers for 57 children independently completed the CBCL assessments within 3 days of the initial ChYMH assessments. These 57 families were selected because data was available on both the CBCL and ChYMH assessment instruments within the allocated 3-day timeframe as part of a previous validation study.

Data collection for the ChYMH-DD sample

A total of 657 children between the ages of 4 to 18 years were assessed using the ChYMH-DD assessment tool at participating mental health agencies. The ChYMH-DD assessments were completed by trained assessors at four agencies that provided mental health services to children with developmental delays or intellectual disabilities in the province of Ontario, Canada.

Data collection for the ChYMH-screener sample

A total of 79 children aged 4 to 14 years were assessed using the ChYMH-Screener. These children were recruited from various high-risk schools in a southwest city of Ontario, Canada. The ChYMH-Screener assessments were administered by trained assessors. The BASC-3 [46] was also administered to this sample, and was independently completed by the child’s primary caregiver within 3 days of the initial ChYMH-Screener assessment. Both BASC-3 and ChYMH data was only available for 79 participants within the allocated 3-day timeframe as part of a previous validation study.

Ethics approval

Data collections for the ChYMH (REB # 106415), ChYMH-DD (REB 106415), and ChYMH-Screener (REB # 106741) samples were all approved by the University of Western Ontario research ethics board.

Measures

interRAI anxiety subscale

The interRAI Anxiety Scale consists of six items and aims to capture the frequency of several anxiety symptoms of the child. Such symptoms include: have anxious complaints or concerns, unrealistic fears, obsessive thoughts, intrusive thoughts or flashbacks, episodes of panic, and nightmares. The items are scored as 0 (“Not present”), 1 (“Present but not exhibited in last 3 days”), 2 (“Exhibited on 1-2 of last 3 days”), and 3 (“Exhibited daily in last 3 days, 1-2 episodes”), or 4 (“Exhibited daily in last 3 days, 3 or more episodes or continuously”). Higher scores on the Anxiety Scale therefore indicate higher levels of anxiety. This scale is embedded in all of the assessment instruments utilized in this study (the ChYMH, the ChYMH-DD, and the ChYMH-Screener).

CBCL-internalizing

The CBCL provides evaluations of both internalizing and externalizing symptoms for a child. It is comprised of 120 items which are rated on a three-point scale (0 = not true; 1 = somewhat or sometimes true; 2 = very often true). The internalizing scale captures syndromes such as Withdrawn/depressed, Anxious/depressed, and Somatic Complaints, whereas the externalizing scale of the CBCL captures syndromes such as Rule-breaking and Aggressive behaviours. The CBCL is widely used as a measure of children’s behavioural and emotional problems and has strong empirical support for its reliability and validity [47]. In the current study, the internalizing scale of the CBCL was used as an established external criterion, against which the convergent validity of the Anxiety Scale was tested.

DSM-IV provisional diagnosis-anxiety disorder

As children in the ChYMH-DD sample were not administered any validity measures, their diagnostic data on anxiety disorders were used as the convergent criterion measure. The ChYMH-DD includes a subsection, diagnostic and other health information, which collects a child’s information on 12 provisional including an anxiety disorder as per the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR) [48]. These diagnoses were determined by a psychiatrist, psychologist, or attending physician. All applicable diagnoses were ranked for the importance as a contributing factor to a child’s reason for service. In the current study, the ranking data were recoded into either “Presence” or “Absence” of a DSM-IV Anxiety Disorder diagnosis.

BASC-3 anxiety

Another widely used assessment for the behavioural and emotional disorders experienced among children is the Behaviour Assessment System for Children (BASC). The BASC allows the child’s information to be collected from multiple sources using different components such as the Teacher Rating Scale, Parent Rating Scale, Self-Report of Personality, Structured Development History, and Student Observation System. The BASC has good reliability indices for its different components [49, 50], and good validity as a measure of children’s behaviours [51, 52]. The Parent Rating Scale from the third edition of the BASC was used in this current study (BASC-3), and children’s standardized scores on the BASC-3 Anxiety Scale were compared to their Anxiety Scale scores on the interRAI child and youth instruments. The BASC-3 Anxiety Scale is comprised of 13 items assessing nervousness, generalized fears, and worries that are typically irrational. Three of the items are rated on a dichotomous scale (0 – True; 2 – False). An example item includes: “I can never seem to relax”. The other ten items are rated on a 4-point Likert Scale ranging from 0 (Never) to 3 (Almost Always). Example items here include: “I get so nervous I can’t breathe” and “I worry when I go to bed at night.” The score ranges from 0 to 36.

Statistical analyses

Only children with valid responses on all the Anxiety Scale items in the samples were included in the analyses for the current study. The ChYMH and ChYMH-DD samples were utilized to test the scale reliability. To test the convergent validity, the Anxiety Scale was tested against the CBCL internalizing scores, the DSM-IV anxiety disorder diagnosis, and the BASC-3 anxiety t-scores, utilizing the ChYMH subsample, the ChYMH-DD sample, and the ChYMH-Screener sample, respectively.

Scale reliability

To assess the inter-item reliability of the Anxiety Scale, polychoric correlations were calculated for the scale items, together with Cronbach’s alpha based on polychoric correlations. Polychoric correlations provide estimations of the association between two variables that are continuously distributed but measured categorically or ordinally [53]. Polychoric correlations have been found to provide more accurate estimates of pair-wise correlations and factor loadings when the continuous variables are measured as categorical data, when compared to Pearson’s product-moment, Spearman’s rho, and Kendall’s tau-b correlations [54]. The inter-item correlations and Cronbach’s alpha that are calculated using polychoric correlations can be interpreted in the same way as those based on Pearson’s correlations [54]. In addition, we conducted a robust unrestricted factor analysis [55, 56] using the diagonally weighted least squares (DWSL) method, which has been shown to provide relatively accurate estimates with ordinal data [57]. The factor analysis was also based on the polychoric correlation matrix of the items. Five thousand bootstrap samples were used to estimate the asymptotic covariance matrix. The FACTOR software, Version 10.4.01 [58] was used to carry out all the previously described analyses based on polychoric correlations.

Convergent validity

Three sets of analyses were used to establish the convergent validity of the Anxiety Scale. First, the ChYMH subsample was divided into two groups using their CBCL internalizing scores: the ones with a t-score above 70 were categorized as falling into the clinical range, whereas those with a t-score below 70 were considered as subclinical [45]. The cut-off score of 70 corresponds to any score above the 97th percentile and identified as within the clinical range. Those below a score of 70 fall into either the normal or borderline categories. A score within the clinical range for the higher-order internalizing domain incorporates anxiety as one of the major features. These children’s scores on the Anxiety Scale were then used in a receiver operating characteristic (ROC) curve analysis [59] to predict their CBCL internalizing group membership.

Second, the convergent validity of the Anxiety Scale among children with developmental delays and intellectual disabilities was tested against their DSM-IV anxiety disorder diagnosis. Children’s scores on the Anxiety Scale were used in a ROC analysis to predict whether they had an anxiety disorder diagnosis in the ChYMH-DD sample.

Finally, using the ChYMH-Screener sample, children’s scores on the Anxiety scale were correlated with their standardized scores on the BASC-3 Anxiety scale. All the validation analyses were carried out using SPSS version 23.0.

Results

Sample characteristics

ChYMH sample

A total of 6086 children (59.8% male), aged between 4 and 18 years (M = 11.98, SD = 3.60), from the ChYMH sample had responses on all the Anxiety Scale items. See Table 1 for the demographic information of children included in the analyses from the ChYMH sample. The total scores on the Anxiety Scale range from 0 to 24, with a mean of 5.18 (SD = 4.84), skewness = 1.02 (SE = .03), kurtosis = 0.56 (SE = .06). See Table 2 for means and standard deviations of children’s scores on the Anxiety Scale by gender and age groups for all the samples.

Table 1.

Demographic information for the children included in the analyses from the ChYMH sample

	N (% of all the children included in the analyses)
Gender
Male	3640 (59.8%)
Female	2446 (40.2%)
Patient Type
Inpatient	508 (8.3%)
Outpatient	5578 (91.7%)
	N (%)	Presence (%)	Absence (%)
DSM-IV Provisional Diagnosis
Reactive Attachment Disorder	4534 (74.5%)	135 (2.2%)	4399 (72.3%)
Attention Deficit Hyperactivity Disorder	4800 (78.9%)	2524 (41.5%)	2276 (37.4%)
Disruptive Behaviour Disorder	4660 (76.6%)	1276 (21.0%)	3384 (55.6%)
Learning or Communication Disorder	4634 (76.1%)	1204 (19.8%)	3430 (56.4%)
Autism Spectrum Disorder	4537 (74.5%)	559 (9.2%)	3978 (65.4%)
Substance-related disorders	4619 (75.9%)	155 (2.5%)	4464 (73.3%)
Schizophrenia and other psychotic disorders	4639 (76.2%)	36 (0.6%)	46.3 (75.6%)
Mood disorders	4460 (73.3%)	921 (15.1%)	3539 (58.1%)
Anxiety disorders	4534 (74.5%)	2088 (34.3%)	2446 (40.2%)
Eating disorder	4624 (76.0%)	99 (1.6%)	4525 (74.4%)
Sleep disorders	4588 (75.4%)	165 (2.7%)	4423 (72.7%)
Adjustment disorders	4554 (74.8%)	177 (2.9%)	4377 (71.9%)

Information regarding DSM-IV Provisional Diagnosis was not available for children who had not been seen by a Psychiatrist prior to the assessment

Table 2.

Means and Standard Deviations of Total Scores on the Anxiety Scale by Age Group and Gender

Age Group	Gender	Samples Mean (Standard Deviation)
		ChYMH	ChYMH-DD	ChYMH-screener
Age 7 and Under	Male	4.29 (4.41)	3.57 (3.77)	4.60 (3.68)
	Female	4.97 (4.70)	3.04 (3.12)	2.47 (2.97)
Age 8 to 11	Male	5.49 (4.88)	4.81 (4.09)	5.29 (5.05)
	Female	5.37 (5.08)	5.87 (4.72)	3.78 (4.37)
Age 12 and Above	Male	4.50 (4.42)	5.65 (4.42)	3.80 (3.63)
	Female	5.93 (5.15)	5.70 (4.66)	3.63 (4.23)

ChYMH-DD sample

The ChYMH-DD sample consisted of 657 children (73.2% male), aged between 4 and 18 years (M = 11.88, SD = 3.78). See Table 3 for the demographic information of children included in the analyses from the ChYMH-DD sample. Among these children, the total scores on the Anxiety Scale range from 0 to 22, with a mean of 5.16 (SD = 4.36), skewness of 0.92 (SE = .10), and kurtosis of 0.61 (SE = .19).

Table 3.

Demographic information for the children included in the analyses from the ChYMH-DD sample

	N (% of all the children included in the analyses)
Gender
Male	481 (73.2%)
Female	176 (26.81%)
Patient Type
Inpatient	103 (15.7%)
Outpatient	554 (84.3%)
	N (%)	Presence (%)	Absence (%)
DSM-IV Provisional Diagnosis
Reactive Attachment Disorder	560 (85.2%)	12 (2.1%)	548 (83.4%)
Attention Deficit Hyperactivity Disorder	541 (82.3%)	219 (33.3%)	322 (49.0%)
Disruptive Behaviour Disorder	548 (83.4%)	92 (14.0%)	456 (69.4%)
Learning or Communication Disorder	558 (84.9%)	292 (44.4%)	266 (40.5%)
Autism Spectrum Disorder	569 (86.6%)	297 (45.2%)	272 (41.4%)
Substance-related disorders	564 (85.8%)	15 (2.3%)	549 (83.6%)
Schizophrenia and other psychotic disorders	556 (84.6%)	8 (1.2%)	548 (83.4%)
Mood disorders	549 (83.6%)	23 (3.5%)	526 (80.1%)
Anxiety disorders	528 (80.4%)	116 (17.7%)	412 (62.7%)
Eating disorder	561 (85.4%)	7 (1.1%)	554 (84.3%)
Sleep disorders	554 (84.3%)	23 (3.5%)	531 (80.8%)
Adjustment disorders	561 (85.4%)	9 (1.4%)	552 (84.0%)

Information regarding DSM-IV Provisional Diagnosis was not available for children who had not been seen by a Psychiatrist prior to the assessment

ChYMH-screener sample

The ChYMH-screener sample consisted of 79 children (51.9% Male) aged from 4 to 14 years (M = 8.33, SD = 2.58). The screener sample is not represented in the demographic data tables as it does not include any DSM items and therefore, such information for this sample is unavailable.1

Scale reliability

The reliability of the Anxiety Scale was examined using both the ChYMH sample and the ChYMH-DD sample. Both samples generated comparable results that indicate good reliability (as per guidelines [54]) of the scale (Table 3). Polychoric correlations among the items ranged from .284 to .546 in the ChYMH sample and from .208 to .448 in the ChYMH-DD sample with the exception of the item reflective of nightmares where the correlation was .140. Correlations were weaker for children with developmental/intellectual disabilities in comparison to those without such disabilities. Despite this, the correlations were all positive and none of the correlations have a 95% confidence interval that contains 0, indicating significant correlations among all the items in both samples. See Table 4 for the polychoric correlations among the Anxiety Scale items for the ChYMH and the ChYMH-DD samples.

Table 4.

Polychoric correlations (95% CI) among the Anxiety Scale items

	Polychoric Correlation (95% CI)
	1.	2.	3.	4.	5.	6.
ChYMH Sample
1. Anxious complaints/ concerns	–
2. Unrealistic fears	.546 (.519–.575)	–
3. Obsessive thoughts	.461 (.413–.478)	.479 (.458–.515)	–
4. Intrusive thoughts or flashbacks	.297 (.267–.340)	.388 (.351–.421)	.466 (.433–.501)	–
5. Episodes of panic	.424 (.402–.459)	.496 (.461–519)	.454 (.420–.483)	.459 (.428–.491)	–
6. Nightmares	.293 (.269–.334)	.398 (.364–.452)	.284 (.237–.307)	.435 (.404–.470)	.345 (.308–.380)	–
ChYMH-DD Sample
1. Anxious complaints/ concerns	–
2. Unrealistic fears	.448 (.352–.532)	–
3. Obsessive thoughts	.351 (.256–.446)	.263 (.156–.363)	–
4. Intrusive thoughts or flashbacks	.208 (.097–.301)	.356 (.250–.459)	.325 (.212–.420)	–
5. Episodes of panic	.292 (.198–.384)	.453 (.362–.543)	.293 (.182–.381)	.432 (.315–.523)	–
6. Nightmares	.148 (.043–.256)	.304 (.196–.399)	.140 (.037–.246)	.440 (.335–.552)	.255 (.156–.365)	–

An unrestricted factor analysis was then conducted to examine the number of latent variables that underlie the scale items. Polychoric correlations among the six items were used as opposed to Pearson’s correlations to account for the ordinal nature of the data [60]. The Kaiser-Meyer-Olkin (KMO) test was 0.80 in the ChYMH sample and 0.71 in the ChYMH-DD sample, suggesting that the scale items are suitable for unrestricted factor analysis. Using the unrestricted factor analysis based on the polychoric correlations among the scale items, only one factor was extracted from the data in both samples; all items loaded on the first unrotated factor, with the factor loadings ranging from 0.52 to 0.74 in the ChYMH sample and from 0.44 to 0.69 in the ChYMH-DD sample (Table 5). The first factor explained 51.45% of the total variance in the ChYMH sample and 43.18% of the total variance in the ChYMH-DD sample. The standardized Cronbach’s alpha for the Anxiety Scale, also calculated based on polychoric correlations, was 0.81 in the ChYMH sample and 0.73 in the ChYMH-DD sample. See Table 6 for the descriptive statistics for all the items of the Anxiety scale in both samples.

Table 5.

Unrotated factor loadings and communalities for the Anxiety Scale items

Item	ChYMH Sample	ChYMH- DD Sample
	Factor Loading	Communality	Factor Loading	Communality
1. Anxious complaints/ concerns	.643	.414	.525	.276
2. Unrealistic fears	.743	.551	.690	.477
3. Obsessive thoughts	.673	.453	.476	.227
4. Intrusive thoughts or flashbacks	.624	.389	.616	.379
5. Episodes of panic	.678	.459	.634	.401
6. Nightmares	.521	.271	.443	.196

Table 6.

Descriptive statistics for the Anxiety Scale items

	Mean	95% CI	Variance	Skewness	Kurtosis	Item-Total Correlation
ChYMH Sample
1. Anxious complaints/ concerns	1.452	1.40–1.50	2.415	0.536	−1.276	.739
2. Unrealistic fears	0.974	0.93–1.02	1.839	1.151	−0.071	.693
3. Obsessive thoughts	0.970	0.92–1.02	2.009	1.154	−0.197	.672
4. Intrusive thoughts or flashbacks	.462	0.43–0.49	0.879	2.253	4.512	.494
5. Episodes of panic	.744	0.71–0.78	1.140	1.499	1.542	.622
6. Nightmares	.582	0.55–0.61	0.880	1.841	3.094	.493
ChYMH-DD Sample
1. Anxious complaints/ concerns	1.583	1.42–1.75	2.735	0.385	−1.524	.693
2. Unrealistic fears	0.837	0.71–0.97	1.679	1.398	0.631	.594
3. Obsessive thoughts	1.351	1.18–1.52	2.774	0.666	−1.296	.667
4. Intrusive thoughts or flashbacks	0.266	0.20–0.34	0.494	3.166	10.773	.415
5. Episodes of panic	0.742	0.63–0.86	1.294	1.534	1.382	.560
6. Nightmares	0.389	0.31–0.47	0.592	2.389	6.091	.303

Convergent validity

CBCL internalizing clinical group membership

Among the 57 children in the ChYMH sample who were assessed with the CBCL, 27 had an internalizing t-score of 70 or above (within the clinical range), 29 had a t-score below 70 (not in the clinical range), and the score was missing for the remaining one child. The children’s scores on the Anxiety Scale were used to predict whether their CBCL internalizing t-score was within the clinical range or not. The Area Under the Curve (AUC) for the ROC analysis was significant, AUC = .709 (SE = .068), p = .007, Asymptotic 95% CI = .575–.843 (Fig. 1). Pythagorean’s Index was used to determine appropriate cut-off points and provided a range for both sensitivity and specificity [61, 62]. Here, Pythagorean’s Index suggested a cut-off point of 5.0 for the Anxiety Scale, which corresponded to a sensitivity ranging from 70.4 to 81.5% and a specificity ranging from 44.8 to 62.1% in predicting a CBCL internalizing score that falls within the clinical range.2 The bivariate correlation between the children’s scores on the Anxiety Scale and their CBCL internalizing t-scores was r = .444, p < .001.

Fig. 1.

ROC curve for anxiety scale score for the prediction of CBCL internalizing score. The children’s scores on the Anxiety Scale were used to predict whether their CBCL internalizing t-score was within the clinical range or not. AUC = .709 (SE = .068), p = .007; Asymptotic 95% CI = .575–.843

DSM-IV anxiety disorders diagnoses

The convergent validity was examined against the DSM-IV Anxiety Disorders diagnoses in the ChYMH-DD sample. The ROC analysis generated an AUC of .714 (SE = .025), p < .001, Asymptotic 95% CI = .665–.763 (Fig. 2). An optimal cut-off score of 5.0 was determined according to Pythagorean’s Index, which.

Fig. 2.

ROC curve for anxiety scale score for the prediction of DSM-IV Anxiety Disorder diagnosis. The children’s scores on the Anxiety Scale were used to predict clinical diagnoses of anxiety. AUC = .714 (SE = .025), p < .001; Asymptotic 95% CI = .665–.763

corresponded to a sensitivity ranging from 63.8 to 68.1% and a specificity ranging from 60.3 to 68.9% in predicting the clinical diagnoses in anxiety disorders.

BASC-3 anxiety scores

A correlation was calculated between the total scores on the Anxiety Scale and the BASC-3 Anxiety t-scores using the ChYMH-Screener sample. Among the 72 children who had data available for both scales, the correlation was positive and significant with a large effect size, r = .517, p < .001.

Discussion

Anxiety is one of the most prevalent psychiatric conditions in childhood [23] and the accuracy and efficiency of its assessment is of the upmost importance. Evidence suggests that early identification and assessment are critical to improving the lives of individuals with chronic anxiety and other internalizing symptomologies. The present study therefore aimed to establish the validity of the Anxiety Scale on the interRAI child and youth assessment instruments, including the ChYMH, ChYMH-DD, and ChYMH-Screener. The Anxiety Scale is being utilized in clinical settings and is embedded within the larger assessments to provide a comprehensive, holistic evaluation of children’s mental and physical health, as well as behavioural and emotional functioning. Establishing this scale’s psychometric properties provides further supporting evidence to a larger collection of literature regarding the ChYMH assessment qualities [32–36]. Moreover, given the challenges of assessing co-morbid psychiatric conditions in children with intellectual and developmental disabilities, inclusion of a unique sample with developmental disability was a crucial step towards effective assessment of anxiety in this vulnerable population.

To begin this examination, we assessed scale reliability in each sample using inter-item reliability. Polychoric correlations between the Anxiety Scale items were significant, both with the ChYMH and ChYMH-DD samples. A factor analysis further demonstrated that all of the items load onto a single factor in each sample, implying that not only are the items capturing the intended construct, but they are also functioning as a cohesive, integrated scale.

It should be noted that nightmares were more weakly associated with other items on the Anxiety Scale. While several studies have demonstrated the association between nightmares and anxiety [63, 64], other studies have failed to find such an association [65, 66]. This may account for the weaker correlation and may be further complicated by the fact that children with intellectual disabilities may also struggle with communicating and verbalizing their nightmares. However, given that nightmares have been associated with pathological symptoms of trait anxiety [67–69], as well as internalizing disorders in childhood [70], this item was included within the scale. Items (e.g., anxious complaints or concerns reflected through worry, unrealistic fears, obsessive and intrusive thoughts, as well as episodes of panic) were developed and included in this scale based on key symptoms identified in the literature related to anxiety disorders as well as based on expert clinical consultation. Despite the fact that the "nightmares" item correlates less strongly with other items on the scale, the Anxiety Scale demonstrates adequate internal consistency overall and is a reliable component of the ChYMH instruments.

Next, the convergent validity of the Anxiety Scale was tested using CBCL internalizing scores, DSM-IV Anxiety Disorders diagnosis, and BASC-3 Anxiety t-scores for each of the three samples, respectively. In the ChYMH subsample, a ROC analysis found that children’s scores on the Anxiety Scale significantly predicted whether their CBCL internalizing scores fell within the clinical range (t-scores> = 70) or not. The AUC value of .709 suggested that a child randomly selected from those who display clinical levels of internalizing behaviours, according to the CBCL, will score higher 70.9% of the time than a child randomly selected from those whose CBCL internalizing score does not fall within the clinical range. This finding indicates a moderate accuracy of the Anxiety Scale score in predicting the CBCL internalizing clinical group membership [71]. It should be noted that the specificity at predicting the clinical group membership was lower than sensitivity for the ChYMH sample. Specifically, sensitivity ranged from 70.4–81.5% suggesting that a cut-point of 5 would be sensitive enough to capture the majority of the cases. However, specificity, or the ability to identify children who truly have anxiety was lower than sensitivity. Given that sensitivity and specificity are inversely proportional to one another, more false positives than false negatives would result when utilizing this cut-point. An important consideration is that sensitivity and specificity may have been negatively impacted by the fact that the CBCL internalizing scale represented a broad band domain that incorporates withdrawn/depressed, anxious/depressed, and somatic complaints rather than a specific scale that only represents the construct of anxiety.

Meanwhile, in the ChYMH-DD sample, children’s scores on the Anxiety Scale also significantly predicted whether they were diagnosed with DSM-IV Anxiety Disorders or not. The AUC value of .714 indicated that, for this specific population, a child randomly chosen from those who have a DSM-IV Anxiety Disorders diagnosis will score higher on the Anxiety Scale 71.4% of the time than a child randomly chosen from those who do not have an Anxiety Disorders diagnosis. Similar to the other results, specificity was lower than sensitivity. Previous research has indicated that it can be much more difficult to diagnose anxiety in these children due to their limited language and communication abilities. A variety of diagnostic challenges have been noted in the literature [72] when diagnosing children with developmental disabilities, which is further complicated by the fact that they may not display age-typical symptoms of anxiety, and often struggle to articulate their internal state [73]. Consequently, a gold standard criterion may require a combination of tests (e.g., standardized parent, teacher, child report as well as observation in concert with a diagnosis of anxiety), administered all within a strict time period to provide a more accurate model for diagnostic decision making.

Further, the CBCL findings are well supported by the BASC-3 results, which also demonstrated a positive and significant correlation between the BASC-3 anxiety scale and the Anxiety Scale on the interRAI ChYMH. The ChYMH-Screener sample showed a large, positive correlation with a large effect size between their scores on the Anxiety Scale and the BASC-3 anxiety t-scores, providing further evidence in support of this scale’s convergent validity when tested against a well-established measure. This observed convergent validity for both the ChYMH and ChYMH-DD samples also speak to the appropriateness and clinical utility of this scale in applied settings, as it shows consistent screening ability in both sub-populations.

With respect to symptom capturing, the interRAI Anxiety scale has a few advantages over other measures of pediatric anxiety such as that captured by the CBCL and the BASC. For example, the CBCL is completed by parents, whereas the Anxiety Scale on the interRAI ChYMH is completed by the clinician, using multi-informant data. This disagreement between the various sources might also have contributed to the moderate, rather than high, correlation between children’s CBCL internalizing scores and Anxiety Scale scores. Previous research on assessment of child anxiety symptoms has found a discordance between parents’ and children’s own reports and stressed the need for a multi-informant approach in assessing anxiety among children [74–76]. Moreover, anxiety is difficult to assess because, in many cases, it is quite covert, and its underlying symptoms can be rather subjective (e.g., negative thoughts). The ChYMH uses all available information including reports of the child, parent, teacher, assessor, as well as collateral information; this allows integration of multiple observations of anxious behaviour. It also assesses the frequency, intensity, and severity of the symptoms. While many anxiety scales rely on the child’s self-reported symptoms, this can be impacted significantly by their cognitive and emotional development, which in turn, impacts the quality of responses [77]. Therefore, utilizing a clinician-based assessment (like the interRAI system) reduces the social desirability effect, where anxious children may feel embarrassed or uncomfortable reporting their feelings or thoughts [78]. It also combines the most comprehensive information from a variety of sources to get a well-rounded, accurate picture of anxious symptomology. In addition, measures such as the CBCL combine a variety of symptoms such as social withdrawal, preference for being alone, not speaking in social situations, and gaze avoidance; common features of Autism Spectrum Disorder (ASD). Children with developmental disabilities, such as ASD, might show elevated social anxiety, especially when facing increasingly complex social situations starting from adolescence [24]. Anxiety measures that include a subscale assessing social skills (e.g., eye gaze) might therefore overestimate the actual anxiety symptoms of children with developmental disabilities. In comparison, the Anxiety Scale is embedded into the interRAI ChYMH and ChYMH-DD assessment tools, which employ multi-modal methods of assessment to help differentiate core ASD symptoms from comorbid anxiety disorders, an approach that is viewed as more comprehensive given that clinician ratings have clinically important information that is not available through parent or self-report measures alone that is unique to prognosis [79].

Overall, the Anxiety Scale was examined in three different samples (developmental, mental health, and school/community sample) with results suggesting that it has generalizability across a variety of children with diverse care needs. Additionally, unlike other scales, a combination of self-report and interview data, observed behaviour/arousal, and multiple informant data have been reported to be the best approach to identify anxiety-based symptomatology in children with developmental disabilities, especially ASD [80]. The interRAI Child and Youth suite of instruments utilizes this comprehensive, multi-informant approach to evaluate anxiety and related symptoms in young, vulnerable populations. However, it is important to note that this scale is not intended to replace other comprehensive anxiety measures, and while it has clinical utility in the context of the ChYMH assessments, is not intended to be used as a standalone measure for anxiety disorder identification. This scale while reliable, has a limited number of items, and should be utilized as a part of the ChYMH’s comprehensive assessment of functioning.

Limitations

There are a few limitations of this study that should be acknowledged. First, with respect to anxiety diagnoses, high co-morbidities contribute to the challenging nature of formulating diagnoses of children with internalizing disorders, as other co-morbid disorders might be more difficult to differentiate from one another [81]. This difficulty is further complicated by communication issues that are often present in children with intellectual disabilities (e.g., expressive and receptive language weaknesses).

With respect to the assessment process, it is possible that an assessor may be faced with disparate information between the various sources, and therefore, must determine the best response based on the conflicting reports. This can be particularly difficult as it involves determining how to weigh the various reports, understanding that behaviour may change across different contexts, and then integrate all the clinical information into one final score that is the most accurate and representative of the child’s functioning and behaviour.

While examining the utilization of the Anxiety Scale across three diverse samples (high-risk school sample, mental health, and intellectual disabilities) was a strength, all validation data (e.g., actual diagnosis, CBCL, BASC) was not available across all groups. This was a result of the fact that the services provided for children who participated in the study varied across service sectors. As a result, several children within the database may have been unable to obtain access to diagnosticians or other health care practitioner with the ability to communicate the controlled act of a diagnosis. Consequently, several children may have met the diagnostic criteria for an anxiety disorder but did not receive the diagnosis due to the limited access to specialized services (e.g., access to a psychiatrist, psychologist). This, in turn, would have significantly reduced our validation sample. Therefore, we were unable to compare and contrast the same validation measures across all samples. In addition, the sample sizes for the CBCL and BASC-3 samples were small, and therefore the conclusions drawn from this data must be considered carefully. And as previously mentioned, the CBCL is a global internalizing scale, and in the future, a validated scale specifically measuring anxiety only, (rather than utilizing a global internalizing scale), should be utilized to further validate the scale.

Finally, generalizability is limited to children and youth attending school in a high-risk area or for those children who have been referred for mental health services. At this point in time, generalization to the broader community beyond these samples are not possible.

Summary

This study examines the psychometric properties of the Anxiety Scale among children. The Anxiety Scale is a 6-item scale assessing the symptoms of anxiety and is embedded in the ChYMH, ChYMH-DD, and ChYMH-Screener of the interRAI child and youth assessment tools. Despite the limitations noted above and the potential areas for future research, this study broadened the psychometric support for the interRAI ChYMH to a wider population of children and youth, including those with developmental and intellectual disabilities. The study results suggest that the Anxiety Scale is a valid scale within the instruments, exhibiting clinical utility. These findings provide additional support to a larger body of research surrounding the interRAI measures, which show consistent psychometric rigour [32–36].

Importantly, this study also includes a sample of children with developmental/intellectual disability. Intellectual disability is considered a lifetime condition and can be extremely challenging for the child and caregivers who support them [30]. Children with these delays often experience increased anxiety as they get older and begin to understand and perceive the disparities in social and cognitive abilities, compared to other age-mates [82, 83]. The ChYMH-DD is a comprehensive and, importantly, context-specific assessment designed explicitly for children with developmental/intellectual disabilities. Therefore, the observed validity of scales is essential to ensuring that children with these unique needs are appropriately assessed.

Conclusions

Results suggest that the Anxiety Scale is a valid scale, which exhibits clinical utility in children with and without developmental disabilities. This finding contributes to the existing literature which supports the use of interRAI ChYMH assessments for the evaluation of children’s mental and physical health concerns. In addition, unlike other assessment systems, interRAI instruments can be utilized across service sectors as well as longitudinally as a child ages due to identical core items across the family of interRAI instruments. This data can also be compared to other diagnostic information or professional insight, used often to provide corroborating evidence in multiple service sectors across the lifespan. Consequently, many countries are now utilizing the interRAI assessment system as a health information system to develop case-mix systems for more appropriate allocation of resources (e.g., determining eligibility for support services and assisted-living programs) [36].

In summary, the development of the interRAI instruments allows opportunities for assessment and outcome measurement across developmental stages [16, 31, 32]. This research findings reported herein provides empirical support to the larger body of literature that illustrates the utility of the interRAI suite of instruments for a variety of health-related issues, including anxiety [35]. The use of these assessments over several developmental stages enables clinicians and mental health service providers to facilitate more efficient assessment, care planning, and prioritization, while also providing a framework for fostering optimal outcomes for vulnerable children, youth and families.

Abbreviations

ASD

Autism Spectrum Disorder

AUC

Area Under the Curve BASC-3: Behaviour Assessment System for Children, 3rd Edition

IDD

Intellectual and developmental disabilities (IDD)

ChYMH

InterRAI Child and Youth Mental Health Instrument

ChYMH-DD

InterRAI Child and Youth Mental Health and Developmental Disability

ChYMH-Screener

InterRAI Child and Youth Mental Health-Screener assessment

CAPS

Collaborative action plans

CBCL

Child Behaviour Checklist

DSM-IV-TR

Diagnostic and Statistical Manual of Mental Disorders, 4th Edition, Text Revision

DWSL

Diagonally weighted least squares

KMO

Kaiser-Meyer-Olkin

ROC

Receiver operating characteristic

Authors’ contributions

SLS was the P.I. on this study. She designed the study protocol and oversaw data collection. She developed the scales, contributed to the development of the manuscript, reviewed study results, gave guidance to junior authors, and provided revised drafts and editorial review throughout the manuscript. SEB conducted the literature review and drafted initial sections of the manuscript including the methods, results, and discussion. She revised written work and provided editorial review throughout the manuscript and was responsible for formatting the manuscript for submission. YL undertook all statistical analyses and was also responsible for written sections of the results and discussion. She assisted with literature review, and provided editorial review and written revisions throughout the manuscript. HPD assisted with literature review and provided written sections for the introduction and discussion. He provided formatting support by reviewing all citations and references. He provided editorial revisions throughout the manuscript. All authors contributed to and have approved the final manuscript.

Funding

Initial funding was provided by the London Community foundation. Funders had no direct involvement with the study; rather, funding provided was utilized by the research team to support development, data collection, and subsequent analyses and interpretation of the data. Their initial donation allowed for the development and evaluation of the ChYMH & ChYMH-DD assessment tools. In addition, the London Community Foundation Vitality Grant supported the development and evaluation of the ChYMH-S instrument.

Availability of data and materials

Due to the highly sensitive and confidential nature of the data, as well as the ethical requirements required for use of such data at the present institution (i.e., data collected on secure server, VPN protected, password protected data in secure room with no access to internet or USB ports, etc.), data will not be made freely available. Moreover, participating mental health agencies required that data not be made freely accessible, to protect the anonymity of participants.

Ethics approval and consent to participate

Data collection for the ChYMH (REB # 106415), ChYMH-DD (REB 106415), and ChYMH-Screener (REB # 106741) samples were all approved by the University of Western Ontario research ethics board.

Informed, written consent was given by guardians of children in the present study. The interRAI assessment is done as part of standard practice. Consent to receive treatment includes consent to be assessed, in this case with the interRAI instrument. Deidentified data for secondary analysis was used. Access to those data for research purposes is completed through interRAI licensing agreements with users and researchers who implement our system. However, we do not have permission to transfer those data to third parties, including depositing them in public repositories.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.