First evaluation of a population-based screen to detect emotional-behavior disorders in orphaned children in Sub-Saharan Africa

Abstract

Due to the HIV/AIDS pandemic which has left 12 million children orphaned in Sub-Saharan Africa, children are at increased risk for mental health problems. Currently, no validity data exist for any screening measure of emotional-behavior disorders in pre-adolescent children in Sub-Saharan Africa. The aims of the current study were to evaluate the construct validity of the caregiver-, teacher-, and self-report versions of the one-page Strengths and Difficulties Questionnaire (SDQ) in 466 orphans in South Africa between the ages of 7 and 11 (M_age = 9.23 years, SD = 1.33, 51.93% female) and to provide, for the first time, clinical cut-offs for this population. Findings demonstrated support for the caregiver SDQ, but not the teacher and selfreport versions. We provide clinical cut-offs, but caution their use before further research is conducted. There remains a critical need for further psychometric studies of the SDQ in the developing world.

INTRODUCTION

Currently, 12 million children in Sub-Saharan Africa and 1.9 million children in South Africa are orphaned by HIV/AIDS (1). Orphan numbers for South Africa are predicted to reach 2.3 million in 2020 (2) and it is estimated that by 2015 approximately 15% of all children under the age of 15 will be orphaned (3). The global number of children orphaned by AIDS (accounted for mostly by developing countries) is estimated to be 25 million by the year 2020 (4).While HIV/AIDS is the most prominent cause of orphanhood globally, and in South Africa, other causes of orphanhood include desertion, non-HIV-related illness and road accidents.

The impact of orphanhood on children in the developing world is well documented (5–8). Over the last 10 years, several cross-sectional studies in South Africa (9, 10) and other developing countries (11–16) have demonstrated high rates of psychopathology, especially internalizing disorders, in children orphaned by HIV/AIDS. Longitudinal studies confirm the persisting negative impact of orphanhood in South Africa (17). Against this background, the early detection and management of emergent emotional-behavior disorders in this population is crucial to preventative efforts. Undetected and untreated emotional-behavior disorders incur great cost for both the child and society (18). Therefore, a focus on pre-adolescent children for the early detection of emotional-behavior disorders is warranted (19). Yet, the evaluation of the psychometric properties of psychiatric measures in Sub-Saharan Africa, and South Africa in particular, is in its infancy.

There are several problems with using unvalidated measures in countries in the developing world. First, it ignores the well-known fact that psychiatric disorders express themselves differently in different cultures (20). Second, the use of unvalidated measures hinders the establishment of clinical cut-offs (6). It also makes it hard to interpret the findings of studies and to compare studies across different countries and cultures (6). In addition, when an instrument developed in one culture is applied to another culture without validation, it is likely that a “category fallacy” of incorrectly categorizing distress as pathology may occur (21).

A viable candidate population screen for the early detection of emotional-behavior disorders in pre-adolescent orphans in the developing world is the Strengths and Difficulties Questionnaire (SDQ; 22, 23). The SDQ was developed in the United Kingdom as a population-based screen for the early detection of children at risk for emotional-behavior disorders. The SDQ is ideal for use in community or primary care settings in developing countries for several reasons. First, it is shorter than established instruments—with only 25 questions taking approximately 5 minutes to complete. It is free to use, with versions for download available on the internet (http://www.sdqinfo.com). Electronic scoring assistance is provided through the internet and free reports on children’s functioning can be downloaded.

Second, it can be adapted for use in other countries. Construct validity studies have been reported for seven countries. For instance, the original study in the United Kingdom demonstrated excellent construct validity (24). Third, criterion validity studies have demonstrated the SDQ’s validity in detecting emotional-behavior disorders in the community. The SDQ can be used to derive three psychopathology-specific subscale scores (hyperactivity, conduct problems, and internalizing problems [anxiety/depression]), as well as a peer problems subscale. The total difficulties score of the SDQ provides the most meaningful index of emotional-behavior disorders in the pre-adolescent age range. For instance, multi-informant total SDQ scores have identified children with emotional-behavior disorders with a specificity of 94.6% (95% CI = 94.1–95.1) and a sensitivity of 63.3% (CI = 59.7–66.9%; 24). Similarly promising findings have been reported in criterion validity studies for the SDQ in Australia (25), the Nordic countries (26), Dhaka (27), Southern European countries (28), the Netherlands (29), and Germany (30). Moreover, in most areas, the SDQ has been shown to function as well as the established Rutter questionnaires and it has some advantages over the Child Behavior Checklist (31), which is more time-consuming and costly to administer (22).

Apart from its practical feasibility, its validity in pre-adolescent samples as well as in developing countries, and its criterion and predictive validity, another advantage of the SDQ is that it can be completed by multiple informants. In adolescent samples, the use of self-report measures as a quick-and-easy way (compared to individually administered semi-structured diagnostic interviews) to screen for psychiatric disorder is a viable option, and several studies in the South African context have demonstrated feasibility of self-report measures in general (e.g. 32–34). However, younger samples require multiple informant ratings and should not rely on self-report only (24). The SDQ’s caregiver and teacher forms are therefore frequently used in several countries to screen pre-adolescent children. However, as yet, no measure for screening for emotional-behavior disorders in pre-adolescent children has been validated for Sub-Saharan Africa.

Taken together, the SDQ shows tremendous promise as a population screen to detect early emotional-behavior disorders in children orphaned by HIV/AIDS. Indeed, the SDQ was included in the longitudinal Orphan and Resilience Study in adolescents (5), despite the fact that it is yet to be validated in the South African context. Against this background, the overarching aim of the current study was to evaluate the construct validity of the caregiver-, teacher-, and self-report forms of the SDQ in a large sample of 7–11 year old orphans in South Africa. The AERA, APA, NCME (35) Standards for Educational and Psychological Testing, suggested a framework for organizing evidence to evaluate proposed interpretations of test scores. Five categories of evidence are proposed, each varying in their importance according to how test scores are used, the consequences of interpretive errors, the consequences of not using an objective test, the population to which it is applied, and so on. Users of tests then weigh the evidence presented, while being knowledgeable of background literature, and determine their own evaluation of the evidence in support of the test. The five areas suggested are evidence based on test content (i.e., themes, wording, and format of the items, questions, guidelines for administration and scoring, and the like); evidence based on response processes (i.e., the fit between the latent constructs of the test and the detailed nature of performance by the examinee and conduct of the examiner); evidence based on internal structure (i.e., the degree to which the relationships among the component parts of the test conform to the hypothesized constructs); evidence based on relations to other (external) variables (i.e., the relationships between test scores and variables external to the test including developmental variables and scores on other tests of similar and dissimilar constructs); and, evidence based on consequences of testing (i.e., the intended and unintended outcomes of the use or application of a test). The current study focuses on evidence in support of the SDQ based on relations to other (external) variables. These variables included interview-based diagnoses. In addition, we examined agreement between caregiver-, teacher- and self-report ratings on the SDQ, and we determined clinical cut-offs for the SDQ against interview-based diagnostic measures.

The study was conducted in the Free State, which is located in the heart of South Africa (see Figure 1) and is one of the areas in South Africa affected most by HIV, with a prevalence rate of 14.9% (36). The Free State is the third largest province in the country and covers 10.6% of the country’s surface area. It is also the third most urbanized province, with the Mangaung Local Municipality in the Motheo District being the most densely populated. Of the 752,906 people living in Mangaung, 618,408 (82%) are Black (mostly Sesotho), 32,071 are Coloured; 1,257 are Indian/Asian; and 101,170 are White. Note that the South African term, “Coloured” is widely used and accepted to categorize individuals of mixed race and does not carry the same connotations as in the United States. Currently, 31% of children in the Motheo district are orphaned (37). In evaluating the psychometric properties of the SDQ, we made use of the NIMH Diagnostic Interview Schedule for Children (CDISC-IV; 38) as an interview-based criterion measure of emotional-behavior disorders because the CDISC-IV was developed to be a highly structured interview-based diagnostic measure that does not require clinical decision-making for use by laypersons in low-resource settings where clinically trained research staff are not available. The CDISC-IV was also selected due to prior work establishing the cultural validity of the CDISC-IV in the Sesotho population of South Africa (39).

Figure 1.

Map of distribution of Sesotho speakers in South Africa: darker shading indicates denser population of Sesotho speakers in the Free State Province (67

In summary, the evaluation of the construct validity of the SDQ for use as a population-based screen in orphaned and vulnerable populations in the developing world answers the call to scale up mental health services in low and middle income countries (40).

METHODS

Participants

466 orphans between the ages of 7 and 11 were recruited for this study. The average age in this sample was 9.23 years (SD = 1.33) and 51.93% of the sample was female. A verbal autopsy measure (41) was used to verify orphan status (by HIV/AIDS vs. other causes). Verbal autopsy is a method of obtaining information about a deceased person by asking questions of family and others who can describe the mode of death and circumstances preceding death. This information is then reviewed by a physician who determines the cause of death post-mortem. This method is often used in developing countries in which postmortem pathologic examination is not feasible. Caregivers included orphans’ surviving parent (2.7%), grandfathers (1.4%), grandmothers (39.2%), extended family related by blood (22.2%), extended family not related by blood (34.4%). The sample was characterized by high rates of indices of poverty. For example, fifty-seven percent (57%) of the sample did not have adequate clothing and 60% of the children did not have adequate school uniforms. Almost half of the children received three meals per day (49.1%). Just more than half of the households had incomes of which more than 75% emanated from grants. The mean average household monthly expenditure was R1253 (approximately $156;min. R0 – max. R6467). The households within which the children resided spent a mean 15.6% (min. 1% – max. 63%) of their total expenditure on clothing, while mean food expenditure came to 62.3% of the total household expenditure (min. 15.0% – max. 100%).

Measures

Strengths and Difficulties Questionnaire

The SDQ (22) is a 25-item screening measure of emotional and behavioral disorders designed for children aged 3–17. It has been translated into over 60 languages and is available as a free download from (www.sdqinfo.com). Both informant and youth versions of the SDQ utilize a three-point Likert scale including ‘not true’ (0), ‘somewhat true’ (1), and ‘certainly true’ (2). The following five subscales can be derived from the measure and are each comprised of five items: Conduct Problems, Inattention-Hyperactivity, Emotional Symptoms, Peer Problems, and Prosocial Behavior. A Total Difficulties score can also be derived by summing across the four problem behavior scales (i.e., Emotional Symptoms, Conduct Problems, Inattention-Hyperactivity, and Peer Problems). Items comprising the Emotional Symptoms scale include unhappy mood, fearfulness, headaches or stomachaches, clinginess, and worries. The Conduct Problems scale includes items pertaining to temper tantrums, obedience, lying or cheating, stealing, and fighting. The Inattention-Hyperactivity scale includes items pertaining to restlessness, fidgeting or squirming, distraction, concentration problems, impulsiveness, and task completion. The Peer Problems scale includes items regarding preference for solitary play, friendships, being liked or bullied, and preference for adults. The Prosocial scale includes consideration of others’ feelings, sharing, displays of kindness, and willingness to help others.

The caregiver, youth, and teacher reports of the SDQ were completed for the present study using a Sesotho version for elementary school-age children (7–11 years of age). The SDQ was adapted and translated in accordance with published guidelines for translation of instruments in cross-cultural research (42–45). Thus, the SDQ was translated independently into Sesotho by two native Sesotho speakers, who then met and agreed upon a version, which a third native speaker then back-translated into English. Lastly, all translators met and agreed on the final version.

The SDQ has demonstrated acceptable reliability across a variety of settings. In a very large sample from the United Kingdom, internal consistency for the Total difficulties subscale was estimated at 0.82 for the parent form, 0.87 for the teacher form, and 0.80 for the youth form (23). Internal consistency for the subscales (depending upon respondent) was as follows: Emotional Symptoms 0.66–0.78, Conduct Problems 0.60–0.74, Hyperactivity-Inattention 0.67–0.88, Peer Problems 0.41–0.70, and Prosocial Behavior 0.65–0.84. These estimates mirror internal consistency coefficients reported with American samples (parent report), ranging from 0.63–0.82 for the core problem scales, with weaker reliability for the Peer Problems and Prosocial Behavior scales (0.46–0.62 for Peer Problems and 0.71–0.77 for Prosocial Behavior; 46, 47).

Computerized Diagnostic Interview Schedule for Children – 4^th Edition (CDISC-IV)

The Diagnostic Interview Schedule for Children (CDISC-IV) was chosen as criterion measure in the current study and was administered to caregivers by fieldworkers. The CDISC-IV was chosen firstly because it adheres strictly to DSM-IV diagnostic criteria. The measure underwent development in 1979 by the National Institute of Mental Health as a highly structured diagnostic instrument designed for use by nonclinicians and can be administered via paper-pencil or computer (48). The parent report was designed for parents/caregivers of children aged 6–17. Most of the items can be answered with ‘yes’ ‘no’ and ‘sometimes’ or ‘somewhat’. The interview is administered following computerized prompts that the interviewer reads out loud. The caregiver’s answer is then inputted in the program and the program presents the next appropriate prompt. The interview therefore does not require clinical decision making on the part of the interviewer and inter-rater reliability is not calculated.

The interviews were always administered in private with the interviewer and caregiver facing one another and the computer monitor within viewing distance of the interviewer. The interview is divided into a series of 24 modules pertaining to separate diagnoses that can be selectively administered based on the clinical or research question (see (38) for a summary of the measure’s development and reliability/validity findings). For the purposes of the current study, only modules of the CDISC-IV that pertain to emotional-behavior disorders in children were selected. The Anxiety Disorders module included generalized anxiety disorder (GAD), panic disorder (PD), posttraumatic stress disorder (PTSD), and separation anxiety disorder (SAD). The Mood Disorders module included major depressive disorder (MDD) and dysthymia. The Disruptive Behavior Disorders module included oppositional defiant disorder (ODD) and conduct disorder (CD). Attention-deficit hyperactivity disorder (ADHD) was considered independently.

Beyond its general validity and reliability data in the developed world, the CDISC-IV was chosen as criterion measure because it was designed for use by trained laypersons. South Africa is a developing country where there is an estimated rate of four psychologists per 100,000 residents compared with 26.4 psychologists per 100,000 in the USA (49). A recent study estimated an average of 0.3 psychiatrists per 100,000 in South Africa (50). There is therefore a dramatic lack of clinically trained and skilled professionals and certainly none that are available to participate in research studies. The CDISC-IV is therefore ideal for use in studies where an interview-based diagnosis is needed, because it is highly structured and designed for use by trained laypersons. It therefore shows great promise for use in developing countries such as South Africa, where there is alack of clinically trained and skilled professionals at the primary care level.

In a prior study (39) the CDISC-IV was investigated for its sociocultural appropriateness for use with Sesotho families in South Africa. Qualitative methodology of expert review and contextualized content analyses were used. Results suggested that the CDISC-IV was appropriate for use in the Sesotho context, but also showed that several cultural specific themes in the administration of the CDISC-IV were important to consider in adapting it for further use. These included the structure of the CDISC-IV, its computerized nature, Americanisms, problems in interpretation due to the adversity children live under, language problems, the effect of rural settings and education level, and cultural norms regarding psychiatric symptoms, gender, the experience of time, the expression of emotion, and family structure. In the current study, during the training of community mental health workers who administered the CDISC-IV, emphasis was placed on these cultural themes. Training involved four days, including teaching on the philosophical underpinnings of the CDISC-IV, its development, its installation, and its administration. Each of the relevant modules was discussed and a demonstration of the use of the CDISC-IV was given. Each of the fieldworkers received their own computerized copy of the CDISC-IV as well as the CDISC-IV manual.

It should be underscored again that the CDISC is highly structured, leaving no room for clinical interpretation, hence making it appropriate for use by laypersons.

Procedures

The study was approved by the IRB Boards of the University of Houston, the University of the Free State, and the University of Stellenbosch in 2010. Fieldworkers were also trained on confidentiality and informed consent prior to the study. Orphans were identified through NGO partners who keep registries of orphans in the Free State Province. This method of recruitment ensured that both school-attenders and non-attenders could be identified. Once a child was identified from the registers, the caregiver of the child was contacted in person by study staff. Caregivers and children consented and assented, respectively, in person. If more than one caregiver was available to complete measures, caregivers were asked to nominate which caregiver knew the child best.

The study was carried out door to door. Caregivers and children were interviewed separately from one another at the research participants’ homes. Despite the fact that our research team had an office in the township of Manguang, research participants did not have transport of child care options, justifying home-based assessments. Efforts were made to find a quiet room in which to conduct the interviews in the house.

A challenge in the current study, as with many studies in Sub-Saharan African and other resource constrained developing countries, was high levels of illiteracy. Following guidelines used in studies with young children (see e.g. (51)), which we have successfully applied in prior research with younger children (19, 52), study staff assisted research participants by reading questions and response options to participants, taking care not to influence their answers and ensuring confidentiality of response. The alternative is to not make use of standardized assessment tools, or to ignore the voices of illiterate populations – neither of which are viable or ethical options for advancing our understanding of the psychosocial consequences of HIV/AIDS in the developing world. Supervision from the first author was available via Skype conferencing and face-to-face visits throughout the study.

Data analytic strategy

We based our data analytic strategy on the principle that the SDQ was created to reflect DSM-IV criteria for several common disorders in childhood. Therefore, the subscales were matched to detect the following CDISC-IV diagnostic groups: the Emotional Symptoms scale matched to detect anxiety disorders (i.e., Separation Anxiety, Panic, GAD, or PTSD) and affective disorders (i.e., Major Depressive Disorder and Dysthymic Disorder), the Conduct Problems scale matched to detect Oppositional-Defiant Disorder (ODD) and Conduct Disorder (CD), and the Inattention-Hyperactivity scaled matched to detect ADHD. Bipolar Disorder was not included in the affective disorders because the SDQ does not include items pertaining to manic symptoms. The Total Problems score was used to detect any of the preceding diagnostic groups. The presence of eating disorders, psychosis, substance abuse/dependence, learning disorders, developmental disabilities, etc. was not included in the determination of Axis I disorders, as the SDQ was not designed to detect these difficulties. We did not investigate the Peer Problems or the Prosocial Subscales as these are not DSM-IV-based scales that can map onto the CDISC-IV.

Another important consideration in conducting the Receiver Operating Characteristic (ROC) analyses should also be emphasized. We conducted ROC analyses on SDQ subscale scores disregarding levels of internal consistency, as there are many factors that contribute to specificity and sensitivity beyond internal consistency of a measure. However, we revisit this issue in the Discussion. Two-dimensional ROC curves were graphed from scores of the test measure and criterion measure for each of the subscale scores. The scores were then used to derive the sensitivity rate along the Y axis and the false positive rate (1–specificity) along the X axis. The area under the curve (AUC) is the most commonly used index of measuring accuracy of diagnostic classification and represents the ability of a test to correctly classify individuals with and without a given condition. An AUC of 1.0 represents ideal discrimination (100% sensitivity and 100% specificity), and an AUC of 0.50 represents chance, where no discrimination exists. Graphically, the closer the ROC curve is to the upper left corner, the better the performance of the instrument. It is widely accepted that AUCs of 0.90–1.0 are considered excellent, 0.80–0.90 are good, 0.70–0.80 are fair/moderate, 0.60–0.70 are poor, and below 0.60 is no better than chance (53). Rice and Harris (54) compared AUC values with Cohen’s d and determined that .71 corresponds to a d of .80, which is considered a large effect size by Cohen (55). For the present study, a benchmark of .70 was set for AUC values as the minimum value necessary for meaningful detection ability.

RESULTS

Preliminary analyses and rates of psychopathology

Table I summarizes the descriptive statistics on rates of psychopathology using the clinical fieldworker interview-based CDISC-IV. 58% of children met criteria for any Axis I diagnosis, 31.7% met criteria for an anxiety disorder, 5.6% met criteria for an affective disorder, 18.9% met criteria for ODD or CD and 40.6% met criteria for ADHD. Note that (1) percentages do not add up to 100 because frequency of endorsement of diagnosis does not take into account the fact that the same child may meet criteria for two types of disorders; (2) percentages are high because positive cases refer to cases in which either full or intermediate criteria on the CDISCIV were met given relatively low frequency of endorsement of full-blown disorder that would have precluded ROC analyses due to small cell sizes. We therefore also report the percentages of full criteria met in the third column of Table I, which showed much lower percentages.

Table I.

Frequency Distribution of Diagnoses Determined by the CDISC-IV

Diagnostic Group	CDISC-IV
	Positive	Intermediate	Negative Cases	Missing	Group Total
Any Axis I	73(15.8%)	202(43.6%)	188(40.6%)	3	463
Anxiety	29(6.2%)	118(25.4%)	317(68.3%)	2	464
Affective	9(1.9%)	17(3.7%)	439(94.4%)	1	465
Disruptive Behavior	19(4.1%)	69(14.8%)	378(81.1%)	0	466
ADHD	35(7.5%)	154(33.0%)	277(59.4%)	0	466

Note. Anxiety = Separation Anxiety, Panic, GAD, or PTSD. Affective = MDD or Dysthymia. Disruptive Behavior = Conduct Problems or Oppositional Defiant Disorder. Any Axis I = Anxiety, Affective, Disruptive Behavior, or ADHD. Positive cases refer to cases in which full criteria on the CDISC-IV were met. Intermediate cases refer to cases in which diagnostic criteria are not met, but symptoms and/or impairment are present. Negative cases are those in which “minimal symptoms” are present. The intermediate and positive groups were combined for analyses in light of very small cell sizes corresponding to positive diagnoses only. Diagnostic groups are not mutually exclusive, meaning that participants can be assigned to more than one group at a time.

Table II lists the means and SDs for the caregiver and self-report SDQ scales, suggesting for caregiver- and teacher-report a clear pattern of means that indicate high levels of psychopathology in the current sample. In contrast, means for self-report psychopathology on the SDQ were relatively low.

Table II.

Summary Statistics for SDQ-Caregiver (SDQ-C), SDQ-Youth (SDQ-Y), and SDQ-Teacher (SDQ-T) report

	Current Study
Subscale	Mean	SD	α	Range	Min	Max
	SDQ-Caregiver report (n = 466)
Total Difficulties*	14.59	6.18	.72	0–40	1	35
Emotional Symptoms Conduct Problems	4.12	2.56	.60	0–10	0	10
Conduct Problems	3.19	2.41	.66	0–10	0	10
Inattn.-Hyperactivity	4.27	2.22	.30	0–10	0	10
	SDQ-Youth self-report (n = 465–466)
Total Difficulties*	12.31	5.16	.62	0–40	2	27
Emotional Symptoms	4.35	2.41	.50	0–10	0	10
Conduct Problems	1.65	1.63	.34	0–10	0	7
Inattn.-Hyperactivity	3.07	1.97	.26	0–10	0	10
	SDQ-Teacher report (n = 462–466)
Total Difficulties*	12.92	7.43	.84	0–40	0	33
Emotional Symptoms	3.73	2.86	.77	0–10	0	10
Conduct Problems	2.20	2.36	.70	0–10	0	10
Inattn.-Hyperactivity	4.17	2.62	.67	0–10	0	10

Note.

^*The Total Difficulties subscale includes all items except those on the Prosocial Behavior Scale. The Peer Problems and Prosocial Behavior Scales have been excluded from this paper since the focus is on emotional-behavior disorders.

Taken together, dimensional results suggest high levels of psychopathology among orphans when considering clinical fieldworker diagnosis and caregiver report, but not when considering youth self-report.

Internal Consistency

Reliability statistics for the test measures are also presented in Table II. Internal consistency (Cronbach’s alpha) varied widely with generally poorer consistency for youth report (.26–.62) compared with caregiver report (.30–.72) and teacher report (.67–.84). The consistency of the Inattention-Hyperactivity scale in parent report was substantially lower than other subscales (.30 compared with .60–.72).

Agreement between sources

In order to assess agreement between sources, intraclass correlations (ICC; using a two-way mixed model and consistency definition) were computed for caregiver-, teacher- and youth self-report SDQ scores. The intraclass correlations between caregiver and self-report were as follows: ICC = .46 (p< .001) Total Difficulties, ICC = .32 (p < .001) Inattention-Hyperactivity, ICC = .43 (p < .001) Emotional Symptoms, and ICC = .36 (p < .001) Conduct Problems.

The intraclass correlations between caregiver and teacher were as follows: ICC = .30 (p < .001) Total Difficulties, ICC = .27 (p < .001) Inattention-Hyperactivity, ICC = .05 (p = .28) Emotional Symptoms, and ICC = .37 (p < .001) Conduct Problems.

The intraclass correlations between self-report and teacher were as follows: ICC = .07 (p = .22) Total Difficulties, ICC = .13 (p = .06) Inattention-Hyperactivity, ICC = -.09 (p = .82) Emotional Symptoms, and ICC = .09 (p = .16) Conduct Problems.

Construct validity

The results of the ROC analyses are presented in Table III. Fair/moderate to excellent AUC values (ranging from lowest .73 to highest .82) were evident for the caregiver report of the SDQ. In contrast, values for the child self-report were no better than chance (<.60) in predicting any CDISC Axis I disorder, anxiety disorder and ADHD. Self-report SDQ was also poor (.60–.70) in predicting CDISC affective and conduct disorders. We reran the ROC analyses splitting the sample by age (7/8 year olds vs. 9–11 year olds) to investigate whether poor AUC values were a function of age (which in turn, may relate to illiteracy). Results showed no differences between 7/8 year olds in AUC values compared to 9–11 year old children. Regarding teacher-reported SDQ, all SDQ scores were no better than chance (<.60) in predicting CDISC diagnosis. Graphical representations of the ROC analysis are presented in Figure 2 clearly demonstrating the superior performance of the caregiver SDQ compared to teacher- and self-report SDQ.

Table III.

Results of ROC Analysis for Criterion Validity of SDQ predicting CDISC-IV

CDISC-IV	SDQ-Caregiver	AUC	SE	Sensitivity	Specificity	Current Cut-Off	UK Cut-off
Any Axis I	Total Difficulties	.76***	.02	.70	.73	13.5	17
Anxiety	Emotional Symptoms	.73***	.03	.66	.72	4.5	5
Affective	Emotional Symptoms	.82***	.04	.72	.74	5.5	5
Disruptive Behavior	Conduct Problems	.80***	.03	.72	.71	3.5	4
ADHD	Inattn.-Hyperactivity	.73***	.02	.66	.68	4.5	7
CDISC-IV	SDQ-Youth	AUC	SE	Sensitivity	Specificity	Current Cut-Off	UK Cut-off
Any Axis I	Total Difficulties	.58**	.03	.58	.51	11.5	20
Anxiety	Emotional Symptoms	.56	.03	.50	.58	4.5	7
Affective	Emotional Symptoms	.65*	.06	.60	.56	4.5	7
Disruptive Behavior	Conduct Problems	.64***	.03	.69	.51	1.5	5
ADHD	Inattn.-Hyperactivity	.57*	.03	.64	.47	2.5	7
CDISC-IV	SDQ-Teacher	AUC	SE	Sensitivity	Specificity	Current Cut-Off	UK Cut-off
Any Axis I	Total Difficulties	.56*	.03	.58	.56	11.5	16
Anxiety	Emotional Symptoms	.54	.03	.50	.54	3.5	6
Affective	Emotional Symptoms	.50	.07	.52	.53	3.5	6
Disruptive Behavior	Conduct Problems	.57	.03	.60	.52	1.5	4
ADHD	Inattn.-Hyperactivity	.60***	.03	.54	.62	4.5	7

Note. Anxiety = Separation Anxiety, Panic, GAD, or PTSD. Affective = MDD or Dysthymia. Disruptive Behavior = Conduct Problems or Oppositional Defiant Disorder. Any Axis I = Anxiety, Affective, Disruptive Behavior, or ADHD. AUC = Area under the curve. SE = Standard error.

^*p<.05,

^**p<.01,

^***p <.001.

Figure 2.

ROCCurves Predicting CDISC-IV Diagnosis from SDQ

Note. Anxiety = Separation Anxiety, Panic, GAD, or PTSD. Affective = MDD or Dysthymia. Disruptive Behavior = Conduct Problems or Oppositional Defiant Disorder. Any Axis I = Anxiety, Affective, Disruptive Behavior, or ADHD.

The second to last column of Table III displays the optimal cut-off scores (and relevant specificity and sensitivity) for using the SDQ to predict CDISC-IV diagnoses in this sample with the last column providing a comparison of UK norms. Caregiver-report cut-offs were in line with UK cut-offs for emotional symptoms and conduct problems, but reduced for ADHD and consequently the total problems scale. Self-report cut-offs were significantly lower for the orphan sample compared to the UK cut-off scores. Teacher-reports were similarly reduced.

DISCUSSION

An urgent need for the development of psychometric assessment tools to assess mental health in HIV/AIDS affected children in South Africa has been identified (56). The necessity for validated screening measures that adequately assess emotional and behavioral problems in children affected by HIV/ADIS is not specific to South Africa, but is relevant to the rest of Sub-Saharan Africa as well and also extends to the rest of the developing world where the impact of HIV/AIDS on children’s mental health has been dramatic (17, 57). In this study, we aimed to address this need by evaluating the construct validity of the SDQ in a subsection of the population most in need of screening, namely children made vulnerable through their orphan status. The death of a parent during childhood has a profound and lifelong impact on a child’s psychosocial wellbeing and children in low-resource settings may face additional psychological and social challenges (57). Consistent with the notion that orphans face psychological challenges, descriptive analyses revealed the vulnerability of these children (in terms of mental health) with about 15% of children meeting criteria for full psychiatric disorder, and nearly half meeting criteria for intermediate (sub-threshold) levels of psychopathology. In addition, high means on the dimensionally-scored SDQ for both caregiver- and teacher-report were reported.

The examination of psychometric properties of the SDQ revealed that, according to the current psychometric evaluation, only the SDQ caregiver report can be used with any certainty at this point in time. Internal consistency and ROC results were poor for all self-report indices of psychopathology. While the internal consistency of the teacher report was acceptable, especially for the total score, the results of the ROC analyses suggested poor construct validity.

In contrast, the caregiver SDQ performed better, with good sensitivity and specificity, and acceptable internal consistency (ranging from .60–.72) for all scores except the ADHD subscale. Reynolds and Livingston (58) note there are multiple factors for considering a reliability coefficient such as the construct being measured, time available for testing, how the scores are to be used, amongst others, so that reliability coefficients as low as 0.60 are acceptable. We therefore argue that the range reported here is acceptable given that this study measures emotional-behavior symptoms, is conducted in a resource-constrained environment under time pressure and that the purpose of the SDQ is an initial screening to be followed up by additional broad measures and clinical observations. The low internal consistency (even by these standards) for the caregiver-reported ADHD subscale of the SDQ suggests potential problems with this subscale when used with caregivers. Example items here included for instance item 2 (restless, overactive, cannot stay still for long) and item 21 (thinks things out before acting). The fact that the ADHD subscale evidenced low internal consistency across all three sources may point to cultural differences regarding views of the pathological (or not) nature of hyperactivity between African and Western culture so that hyperactivity is more tolerated in African culture where it is not in Western culture.

Inter-rater reliability suggested discrepancy between sources of report. Discrepancy is the rule and not the exception in child psychiatric research (59). In the current study, greatest agreement was evident for self- and caregiver report. Teachers and caregivers showed significant agreement for all scales except the emotional problems scale. Poor agreement was found for teacher and self-report. When then taking into account the results of the ROC analyses, where caregiver-report outperformed both teacher and self-report, a picture emerges to suggest that caregiver-SDQ is the most valid of the three. This notion is further supported by comparable clinical cut-off scores for the caregiver SDQ when compared with UK cut-offs (except for the ADHD subscale).

Taken together, the current study offered a unique opportunity to provide clinical cut-off scores and to investigate, for the first time, the validity of the SDQ across multiple sources against an interview-based measure in a large sample of elementary school-age orphans in a developing Sub-Saharan African country. We demonstrated evidence for the construct validity of the SDQ caregiver report. At this stage validity cannot be confirmed for the teacher- or self-report versions. This conclusion is based on cut-off scores and means that clearly deviate from established scores in other countries, in addition to unacceptable reliability coefficients and construct validity. It is the case that the developers of the SDQ recommend against the use of the self-report version of the SDQ in children younger than 11 years of age (see sdqinfo.com). However, validity for the self-report version of the SDQ has been demonstrated in 7–11 year old children in many countries (60, 61, 24, 27, 25, 62, 19), mitigating against a conclusion that lack of validity of the self-report SDQ is a developmental issue. This is further supported by comparing ROC results for younger (8 and 9 year olds) versus older (9–11 year olds) children, which showed comparable findings. Rather, we argue that the low validity may have something to do with the extreme vulnerability of affected children in South Africa. Children in the South African townships like Mangaung who, especially if orphaned, receive limited cognitive and socio-emotional stimulation to stay on track developmentally (63). The task demands of completing the SDQ (even with help from fieldworkers in reading some of the questions) may have been too large.

Many limitations to the current study can be addressed in future research. Most notably, future research should consider the inclusion of Sesotho clinician diagnosis as the criterion measure. While the low rates of trained Sesotho clinical psychologists or psychiatrists were noted in the introduction, non-structured clinician diagnosis is unconstrained by a structured diagnostic instrument like the CDISC-IV which, even though deemed culturally relevant (39), remains associated with Western assessment characteristics that decouples the instrument from its African context. However, given the extreme lack of trained Sesotho clinicians, conducting a study with clinician report may be challenging. Second, test-retest reliability was not investigated and should be a focus of future research. Third, the fact that assessments were conducted in research subjects’ home and that researchers had to assist children and caregivers complete the measures in some cases due to high rates of illiteracy may have led to social desirability biases. However, given the high rates of emotional-behavior problems demonstrated in our results we consider this less likely. Also, these are the realities of conducting research in the developing world where rigorous scientific methods need to be adapted to real-word resource constrained settings. Further, we caution against the use of the clinical cut-off scores reported for teacher-and self-report versions of the SDQ. ROC analyses were conducted on the self-report for consistency, but without strong internal consistency the findings of a ROC analyses cannot be interpreted confidently. Finally, in the introduction to this paper we noted that five categories of evidence can be evaluated when establishing the psychometric properties of a measure, including test content, evidence based on response processes, internal structure, relations to other (external) variables and evidence based on consequences of testing. In this study we examined only one of these categories and it is important that further work also examine other categories in relation to the SDQ. Of particular interest in this regard, may be the use of qualitative methodology to examine the relevance of the translated questions for Sesotho children. While adults were consulted in the translation process, children were not and it may be that improvements to the self-report version of the SDQ can be made through child input.

Notwithstanding these limitations, this study makes an important contribution to global mental health in that it provides the first validity for a one-page population-based screen for emotional behavior disorders for resource constrained countries significantly affected by the HIV/AIDS pandemic. Most mental disorders begin in youth, yet screening and treatment for children and adolescents with emotional-behavior disorders is nearly non-existent in most developing countries (64). While epidemiological studies have often used the Achenbach scales (see (65–66) for reviews), they are more time consuming compared to the SDQ and have not typically been used in Sub-Saharan Africa. We suggest the SDQ as a viable alternative for population-based screening and have provided preliminary support for the validity of the SDQ caregiver form.