Incremental validity of the Child Behavior Checklist (CBCL) and the Strengths and Difficulties Questionnaire (SDQ) in Vietnam

Abstract

Objective

Valid but efficient psychiatric assessment is essential for mental health development in Asian low and middle-income countries. This study’s objective was to assess the validity of Vietnamese versions of the Child Behavior Checklist (CBCL), and the Strengths and Difficulties Questionnaire (SDQ) in Vietnam.

Methods

Measures were completed by a community sample of 1314 parents of children 6–16 years old from 10 Vietnamese provinces, and by parents of 208 children recruited from 3 psychiatric facilities in Hanoi.

Results

Internal consistency was in the fair to excellent range for all CBCL scales (.76–.96) and for the SDQ Total Problems scale (.81); SDQ subscale internal consistency was in the poor to fair range (.31–.73). All CBCL and SDQ scales and most individual items significantly discriminated between referred and non-referred children, with referred children scoring in the more pathological direction; the CBCL had significantly larger referral effect sizes than the SDQ for all four pairs of comparable scales. At the item level, the largest referral status effect for the CBCL were #73 (Sexual Problems), #84 (Strange Behavior), and #91 (Talks about suicide), and for the SDQ they were #10 (Constantly fidgeting), #15 (Easily Distracted) and # 25 (Good Attention Span-reverse scored). Five CBCL (#2 Drinks alcohol; #99 Uses tobacco, #32 Has to be perfect; #53 Overeats; #56A Aches and pains) and one SDQ items (#23 Gets along better with adults than children) did not discriminate referral status, suggesting the influence of cultural values on clinical referrability (e.g., that Vietnamese parents may not see use of tobacco as an issue of concern, or related to health).

Conclusions

There is good support for the reliability and validity of the Vietnamese version of the CBCL, and for the SDQ Total Problems scale. Overall, the CBCL appears to be the stronger measure psychometrically, particularly if in-depth assessment is needed.

1. INTRODUCTION

Asia is the world’s largest continent, with a dense population and a relatively high proportion (20–30%) of children and adolescents (World Bank, 2015). The general prevalence of mental health problems and disorders in youth in Asia has been reported to be in the range of 10–20% (Srinath et al., 2010), indicating a need for valid but efficient (e.g., inexpensive) psychiatric assessment tools, particularly for Asian low and middle-income countries (LMIC). In Vietnam, for example, the prevalence of significant mental health problems among children and adolescents has been estimated at 12%, similar to those reported in HIC countries (Weiss et al., 2014). Yet like most LMIC in Asia, Vietnam has highly limited mental health infrastructure, particularly in regards to children and adolescents (Weiss et al., 2012). In their review of the treatment gap for child mental health problems, Patel et al., (2013) identified low level of detection of child mental health disorders due to a lack of appropriate screening and diagnostic tools as one of three main factors underlying the psychiatric treatment gap.

Over the last two decades, the Child Behavior Checklist (CBCL, Achenbach, 2009) and the Strengths and Difficulties Questionnaire (SDQ, Goodman et al., 1998) have become the two most widely used instruments in the world for assessing the mental health functioning of children and adolescents (Rescorla et al., 2012; Stone et al., 2010; Warnick et al., 2008). The CBCL was developed by Achenbach as a dimensional rating scale of childhood psychopathology (Achenbach, 2009). It has been translated into more than 60 languages and validated in 31 countries with multicultural comparisons of scores, including Asian countries such as Thailand, Taiwan, Singapore and others (Ang et al., 2012; Ivanova et al., 2007; Rescorla et al., 2007). Similarly, the SDQ has been translated into 40 languages and has normative data by age and gender for 10 countries (see http://www.sdqinfo.com). Both instruments have been studied in community and clinical samples. In contrasting the two instruments, the SDQ has advantages that it is relatively short (25 items), and is available for use without charge. The advantage of the CBCL is that it provides a more in-depth assessment (e.g., 8–9 syndrome scales vs. 4 syndrome scales for the SDQ), including more severe psychiatric symptoms, (e.g., compulsions, hallucinations). The CBCL produces three primary scales (total problems, externalizing problems, and internalizing problems) and eight syndrome subscales (anxious/depressed, withdrawn/depressed, somatic complaints, social problems, thought problems, attention problems, rule-breaking behavior, and aggressive behavior). Because of these characteristics, the CBCL may be more suited for situations that require a detailed assessment across a more comprehensive range of problems.

Although both the CBCL and SDQ have been validated across a number of countries, there have been few studies validating the CBCL and/or SDQ in Asian LMIC. Among 31 countries in the Rescorla et al., (2007) study, six were from Asia including two from Asian upper middle income countries (China, Thailand). Our own literature review in English language journals identified six validation studies of the CBCL and/or SDQ in Asian LMIC: One on the CBCL in India (Senaratna et al., 2008), four on the SDQ, in Sri Lanka (Lukumar et al., 2008), Pakistan (Samad et al., 2005), Bangladesh (Mullick and Goodman, 2001) and Indonesia (Stevanovic et al., 2015), and one comparing the CBCL and SDQ in Pakistan (Syed et al., 2009). However, none of these studies examined the measure’s construct validity by comparing a clinical versus community sample, which is generally seen as one of the strongest approaches for measure validation (e.g., Newton & Shaw, 2014). The purpose of the present study then was to assess the validity of the two measures as well as their incremental validity relative to each other, in the Asian LMIC of Vietnam.

2. METHOD

2.1 Samples

We assessed the construct validity of these two instruments by comparing clinical (i.e., admitted to a psychiatric facility) vs. community samples, based on the fact that individuals who have been referred and accepted for mental health services have been judged by a psychiatric assessment as having higher levels of mental health symptoms than non-referred (i.e., community) samples (Newton & Shaw, 2014). The clinical sample consisted of 208 inpatient and outpatient children age 6–16 years old who had been referred to the psychiatric unit of one of two national multi-field hospitals (the National Children’s Hospital; Bach Mai General Hospital) or one psychiatric hospital (Mai Huong Hospital). These three hospitals are in Hanoi, the capital of Vietnam. All children admitted to the psychiatric units were eligible to participate with the exclusion of children who had previously received psychiatric treatment, because such treatment might have influenced their CBCL and SDQ scores. A nurse uninvolved in the child’s treatment described the study to the parents and children including its voluntary nature, answered questions about the study, and then obtained signed consent from parents and verbal assent from children interested in participating. The participation rate was 97% (i.e., 6 families declined to participate). Questionnaires were completed by parents at the Psychology Service of the psychiatric facilities. Data were collected over 6 months, from February to August 2013. The population/community based sample consisted of 1314 children and adolescents from 6 to 16 years of age collected in 2012 from 10 provinces in Vietnam as part of a national epidemiological study of Vietnamese child mental health (Dang et al., 2015; Weiss et al., 2014). The study was conducted by Vietnam National University and approved by their US FWA IRB (00018223). The sample was nationally representative of Vietnam in relation to geographical character (i.e. costal vs. inland), urban/rural and socio-economic status. A random sample of 60 neighborhoods across Vietnam was used in the study, with a random sample of families within in each neighborhood obtained using public population lists in Vietnam.

Families were contacted in their homes by a project research assistant who explained the study including its voluntary nature, answered questions about the study, and then obtained signed consent from parents and verbal assent from children interested in participating. An in-home interview was scheduled for several days later for families interested in participating. The participation rate was 99% (i.e., 6 families declined to participate). At the end of the data collection, the research assistant scanned the questionnaires to determine whether any suicidal ideation had been indicated on the YSR or CBCL; if it was, the parent was informed of this and a treatment referral was made. More generally, all families were given a contact sheet that contained information regarding mental health resources locally available, and it was suggested to them that if they had any concerns regarding their child’s mental health they should contact the facility. The interviewer also offered to assist families in this if they wanted. The study was approved by the Vietnam National University’s US FWA IRB (00018223).

2.2 Measures

Child Behavior Checklist (CBCL) is a multiaxial empirically-based set of measures that includes parent-, self- and teacher-report versions for assessing social competence and emotional/behavioral problems in children (Achenbach, 1991). The parent-report CBCL is comprised of 118 items assessing the child’s emotional, behavioral and social problems over the past six months. The items produce eight original empirically-derived syndrome scales: Social Withdrawal, Somatic Complaints, Anxiety/Depression, Social Problems, Thought Problems, Attention Problems, Rule- Breaking Behavior, and Aggressive Behavior. It also produced two broadband scales: Externalizing Problems and Internalizing Problems. The original CBCL was translated and adapted from English into Vietnamese by a Vietnamese clinical psychologist and a Vietnamese psychiatrist both fluent in English, and an American clinical psychologist who is fluent in Vietnamese. The Vietnamese version of the CBCL then was sent to the authors of CBCL for the back translation. Those items that were changed during the procedure were discussed within the Vietnamese team and American team to produce the most appropriate expression in Vietnamese.

Strengths and Difficulties Questionnaire

The SDQ (Goodman, 1998) assesses emotional and behavioral disorders problems among children aged 3–17. The SDQ can be completed by parents and teachers, and a self-report version is available for children aged 11–17. The SDQ consists of 25 items relating to social, emotional, and behavioral functioning across five subscales: Conduct Problems, Inattention- Hyperactivity, Emotional Symptoms, Peer Problems, and Prosocial Behavior. The Vietnamese version of the SDQ-P (e.g., Graham and Jordan, 2011) was available as a free download from www.sdqinfo.com.

2.3 Statistical Analysis

Internal consistency of the various scales was estimated using Cronbach’s alpha. The main analyses, assessing the discriminative validity of the CBCL and SDQ scales, used a general linear models analysis (SAS 9.4 Proc GLM) with demographic characteristics (e.g., child age) as covariates, Referral Status (clinical vs. community sample) as a fixed effect, the independent variable; the various CBCL and SDQ scales served as the dependent variables, in separate analyses. The differential discriminant validity of the CBCL vs. the SDQ was assessed using this same model, but including Measure (CBCL vs. SDQ) as a within-subject repeated measure, and its interaction with Referral Status. The discriminant validity of the individual CBCL and SDQ items also was assessed, with CBCL and SDQ items collapsed to a 0/1 (absence/presence) binary variable, which was analyzed using a binary logit model (SAS 9.4 Proc GLIMMIX).

3. RESULTS

3.1 Preliminary analyses

We first tested whether the clinical and community samples differed in regards to child age, and child sex. The two groups differed significantly on child age (F[1,1525]=22.00, p<.0001), with the mean age of the clinical sample 12.28 years and the mean age of the non-clinical sample 11.15 years. The two groups also differed significantly on child sex (X²_[1]=20.47) with the clinical sample 67% male and the non-clinical sample 50% male. Consequently, in all following analyses child age and child sex were included as covariates. Cronbach alpha coefficients were computed to estimate the internal consistency. Cronbach’s alphas for the 3 CBCL broadband scales were in the good to excellent range, from .89 to .96 (see Table 1a), which is comparable or slightly higher to the mean alphas (.83 to .93) reported for these same scales averaged across 31 societies (Rescorla et al., 2007). Alpha for the 8 CBCL narrowband scales were in the fair to good range (from .76 to .88; see Table 1a), somewhat higher than mean alphas (.58 to .84) for the scales averaged across 31 societies (Rescorla et al., 2007). The SDQ alpha Cronbach coefficients ranged from the poor to good range (.31 to .81), with alphas for the Total Problems Scale in the good range, but in the poor to fair range for the Prosocial Behaviors, Emotional Problems and Hyperactivity/Inattention scales. The Peer Problems scale produced the lowest alpha (.31). In general, these results are consistent with the existing literature on the reliability of the SDQ’s scales across countries (Kovacs, 2011). The Peer Problems scale typically has displayed the lowest alpha, especially among non-Western countries (Du et al., 2008; Kovacs and Sharp, 2014; Lai et al., 2010; Mieloo et al., 2014).

Table 1a.

Internal Consistency and Discriminant Validity of CBCL Scales

#CBCL scale	Alpha	F[1,1517]	ω2
Total	.96	969.19****	.38
Internalizing	.89	583.16****	.27
Externalizing	.91	642.11****	.29
Anxious - Depressed	.77	408.13****	.21
Withdrawn - Depressed	.80	538.84****	.25
Somatic Complaints	.79	255.08****	.14
Social Problems	.76	593.91****	.28
Thought Problems	.81	1038.31****	.40
ADHD	.84	825.75****	.34
Aggressive Behavior	.88	639.52****	.29
Rule-breaking Behavior	.77	384.78****	.19

Notes

^****p< .0001.

F test is for the Group (clinical vs. non-clinical) effect on syndrome level, controlling for child age and child sex. In all instances, mean clinical group > mean non-clinical group. ω² is semi-partial omega-square.

3.2. Discriminant validity

We tested the discriminant validity of the 11 CBCL and 5 SDQ scales in a series of general linear model analyses, with Referral Status (clinical vs. non-clinical) as the independent variable, and the CBCL scales as the dependent variables in 11 separate analyses. Child age and child sex were included as covariates. As Table 1a shows, the clinical and non-clinical samples differed significantly on all 11 CBCL scales. Effect sizes were large, with effects ranging from R²=.14 (Somatic Complaints) to R²=.40 (Thought Problems); in all cases the level of the CBCL syndrome was higher in the clinical sample. In regards to the SDQ scales, the two groups differed also significantly on the Total Problems scale and the four subscales with the SDQ scales were higher in the clinical sample (table 1b).

Table 1b.

Internal Consistency and Discriminant Validity of SDQ Scales

#SDQ scale	Alpha	F[1,1517]	ω2
Total Problems	.81	500.51****	.24
Hyperactivity/Attention	.67	523.22****	.24
Behavioral Problems	.58	356.59****	.19
Emotional Problems	.68	127.66****	.08
Peer Problems	.31	125.83****	.08
Prosocial Behavior	.73	227.14****	.13

Notes

^****p< .0001.

F test is for the Group (clinical vs. non-clinical) effect on syndrome level, controlling for child age and child sex. In all instances, mean clinical group > mean non-clinical group. ω² = semi-partial omega-square.

We also tested the discriminant validity of the individual CBCL items, using a binary logit model. Group (clinical vs. non-clinical) again was the independent variable, with child age and child sex covariates. As other researchers often have done (e.g., Bilenberg, 1999), we collapsed each CBCL item to the reported presence or absence of the item (i.e., for each item, “2” was recoded as “1”) for the dependent variable. Of the 118 individual CBCL items, all but 5 significantly discriminated between the clinical and non-clinical samples (see Supplemental Materials-Discriminant validity of CBCL items). The five items with the largest odds ratios (i.e., with the largest clinical/non-clinical difference) were #73 Sexual Problems (OR=29.85), #84 Strange Behavior (OR=28.82), #91 Talks about suicide (OR=25.94), #18 Harms self (OR=22.64), and #82 Steals outside home (OR=22.48). Five items did not discriminate between the clinical and non-clinical samples: #2 Drinks Alcohol without Parental Permission, #99 Uses Tobacco, #32 Has to be Perfect, #53 Overeats, and #56A Aches and Pains. In regards to the SDQ, all but one item significantly discriminated between the clinical and non-clinical samples (Supplemental Materials-Discriminant validity of SDQ items). The three items with the largest odd ratios were #10 Constantly fidgeting (OR=11.5), #15 Easily Distracted, and # 25 Good Attention Span (reverse scored). It is interesting to note that although the Peer Problems scale had very weak internal consistency, the items showing the largest clinical/community discrimination were from this scale. The only SDQ item that did not differentiate between two groups was #23 Gets Along Better with Adults than Children.

3.3 Differential discriminant validity

In our final analyses, we evaluated the differential discriminant validity of the CBCL vs. the SDQ. For these analyses, we used a general linear models repeated measures analysis, with Referral Status as the independent variable, Measure (CBCL vs. SDQ) as the within-subject repeated measure, and child age and child sex as covariates. The effect of interest was the Referral Status X Measure interaction, which assessed the extent to which the CBCL vs. SDQ showed differential discriminant validity (i.e., differences between the clinical vs. non-clinical samples varied as a function of the CBCL vs. SDQ). Because the CBCL and SDQ syndrome scales have different metrics, they were equated by standardizing within each scale. There are four comparable sets of scales within CBCL/SDQ (see Table 2). For all four pairs, the discriminant validity of the CBCL was significantly larger than that for the SDQ, with the difference in R² ranging from .10 (CBCL scale: Externalizing; SDQ scale: Behavioral) to .19 (CBCL scale: Internalizing; SDQ scale: Emotional).

Table 2.

Differential validity of CBCL vs. SDQ scales

CBCL scale	SDQ scale	F[1,1516]	ω2(CBCL); ω2(SDQ)
ADHD	Hyperactivity/Attention	19.33****	.34; .24
Social Problems	Peer Problems	93.79****	.28; .08
Internalizing	Emotional	113.01****	.27; .08
Externalizing	Behavioral	22.70****	.29; .19
Total	Total	43.28****	.38; .24

Notes

^****p< .0001.

F test is for the Group (clinical vs. non-clinical) X Measure (#CBCL vs. SDQ) interaction, controlling for child age and child sex. ω² = semi-partial omega-square.

4. DISCUSSION

The need for valid but efficient instruments to assess children mental health in Asian LMIC such as Vietnam is great. Without strong instruments to accurately evaluate and assess children’s mental health, it will be difficult or impossible to provide appropriate and efficient treatment services to meet patients’ needs. The present study is among the first to validate and compare the CBCL and the SDQ in an Asian LMIC. Its goal was to test the reliability, validity, and incremental validity of the Vietnamese versions of the CBCL and SDQ. Alphas for CBCL broadband Total Problems, Internalizing Problems and Externalizing Problems scales were excellent, while those for its syndromes scales were adequate or higher, consistent with findings reported in Rescorla et al.’s multicultural comparisons. In contrast, with the exception of the Total Problems scale, alphas for the SDQ scales were generally in the fair range or worse, down to .31 for the Peer Problems scale. This is similar to reports regarding the SDQ from other non-Western countries and Asian countries such as China (Lai et al., 2010). In both our study and the Lai et al.,’s (2010) study in China, the SDQ Peer Problems subscale had the weakest reliability; one possible explanation is possible inconsistency in the factor structure across countries (e.g., five factors in Euro-American samples, vs. three factors in LIMC) (Du et al., 2008; Stevanovic et al., 2015).

Our results indicate that both the CBCL and SDQ distinguish between referred and non-referred children, with referred children scoring significantly higher (in the more pathological direction) than non-referred children for all 11 CBCL scales and all 5 SDQ scales, with large effect sizes, a strong indicator of validity for the scales in Vietnam. These findings are consistent with those from Western countries (Novik, 1999; Rescorla et al., 2007; Schmeck et al., 2001) as well as in Asian countries (Ang et al., 2012; Leung et al., 2006).

At the item level, the largest referral status effects for the CBCL were for items #73 (Sexual Problems), #82 (Steals outside home), #84 (Strange Behavior), #18 (Harms Self), and #91 (Talks or Thinks of Suicide). These results are different from those of parents in Singapore for whom attention problems and school-related items were the strongest reasons for mental health referral (Ang et al., 2012). This suggests that Vietnamese parents may be more aware of and concerned with symptoms that are noticeable and upsetting to others. The results also are different from those of parents in a number of Western countries for which item #103 Unhappy, Sad or Depressed has been the best discriminator (Bilenberg, 1999; Schmeck et al., 2001; Rescorla et al., 2007).

It is interesting that the CBCL Sexual Problems item yielded the largest referral/non-referral odds ratio (i.e., effect size), suggesting that this is an area of particular concern to Vietnamese parents. This possibility that sexual problems are among the biggest concerns of Vietnamese parents in regards to their children’s development and health is supported by other research (Hoang-Minh et al., 2009). As with many other Asian countries, sex and sexuality is not a topic that Vietnamese parents, or even teachers or other professionals, are comfortable discussing, particularly with their children. Sex education is relatively limited in Vietnam, and frank discussions in the media are limited. Trinh et al. (2009) found out that Vietnamese parents were apprehensive and embarrassed when attempting to discuss sexual issues with their children due to their lack of knowledge and communication skills. This discomfort may be one reason why issues related to child sexuality are so closely linked to mental health referral.

At the other end of the scale, five CBCL items did not discriminate the two groups, including items #2 (Uses Alcohol without Parental Permission) and #99 (Uses Tobacco), which indicate that they are not the reasons for which parents refer their children for mental health services. Drinking alcohol and smoking tobacco in many Asian countries are considered a normal social activity and socially acceptable, even among young people (Jordan et al., 2013). Recent research in Vietnam indicates, for instance, that the prevalence of alcohol and tobacco consumption actually has increased among adolescents with the age of use initiation decreasing to early adolescence (Thoa et al., 2013). These items do discriminate between referred and non-referred samples among high-income Western countries (e.g., Achenbach & Rescorla, 2001). This lack of concern regarding the use of tobacco and alcohol by children suggests that public health attempts to reduce usage of these significantly harmful and addictive substances may be particularly challenging but particularly important in Vietnam and similar countries.

In regards to the SDQ, the only item that did not discriminate the two groups was #23, Gets along better with adults than children, although this item is highly discriminative in Western countries (Achenbach et al., 2008). Vietnam as well as most other developing countries are still agriculture-based, with children expected to participate in family labor in farming, managing livestock, etc. Within this context, a child who “gets along better with adults” may be seen as more mature (i.e., more focused on adults than on other children), which would be viewed positively, as more able to support the family’s work.

The primary limitation of the study is that although it utilized a nationally representative community sample, the referred sample was not nationally representative. The referred sample did come from three different psychiatric facilities, but they all were from the same city and it is possible that results might have varied somewhat if a broader geographic range of psychiatric facilities had been included. However, this limitation is true for most studies that have compared community and clinically-referred samples and results do suggest that the Vietnamese versions of the CBCL and SDQ are valid.

Overall, in our Vietnamese sample the CBCL performed better than the SDQ. The CBCL had stronger reliability as well as the validity. This result contrasts with what has been found in Western countries where the SDQ often has shown similar or even stronger reliability and validity than the CBCL (Goodman and Scott, 1999; Klasen et al., 2000; Kovacs and Sharp, 2014). Our recommendations are that if psychiatrists or other mental health professionals in Vietnam are interested in a comprehensive, detailed assessment of child mental health functioning, the CBCL may be the better instrument for this purpose. On the other hand, if a rapid, relatively simple and broad screening is desired, then the SDQ with its Total Problems scale may be adequate for this purpose.

HIGHLIGHTS.

• Valid but relatively inexpensive psychiatric assessment is essential for mental health development in Asian low and middle-income countries.

• This study assessed the validity of Vietnamese versions of the Child Behavior Checklist (CBCL), and the Strengths and Difficulties Questionnaire (SDQ) in Vietnam.

• There is good support for reliability and validity of the Vietnamese version of the CBCL and SDQ.

• The CBCL appears to be the stronger measure psychometrically, particularly if in-depth assessment is desired.