Measuring symptoms of psychopathology in Zambian orphans and vulnerable children: scale validation and psychometric evaluation

Abstract

There is a paucity of validated mental health measures for assessing psychosocial wellbeing among HIV-affected youth. We sought to explore the psychometric properties and validity of the Achenbach Youth Self Report (YSR) and Child PTSD Symptom Scale (CPSS) among orphans and vulnerable children (OVC) living in Lusaka, Zambia. These scales were administered to 210 OVC aged 13–17 via Audio Computer-Assisted Self-Interview (ACASI). Confirmatory factor analysis was used to assess scale structure, Cronbach’s alpha for internal consistency, and correlations between scales related to mental or psychosocial health for construct validity. A known-groups validation was conducted using local identifications of youth with and without significant psychosocial problems, and test-retest reliability was assessed. Scales exhibited good internal reliability (α>.80), adequate criterion validity (area under the curve>.70), and moderate test-retest reliability (.62-.68). Findings support the utility of these symptom scales for identifying OVC experiencing significant psychosocial problems in Zambia.

Introduction

An estimated 1.5 million children are living with HIV in sub-Saharan Africa and 10.9 million have lost one or both parents to HIV/AIDS (UNAIDS, 2015). In this region, 10–36% of children are living in households with at least one HIV-infected adult, and as a result, are more likely to face adversities associated with parental illness such as stigma and discrimination (Short & Goldberg, 2015). In Zambia, a country with a generalized HIV epidemic that ranks in the top 10 in terms of prevalence globally, an estimated 94,000 children are living with HIV and 450,000 have been orphaned due to HIV (UNAIDS, 2016). In addition, 13 percent of children aged 18 and under in Zambia have experienced the death of at least one parent (UNICEF, 2012). Orphans and vulnerable children (OVC) have been shown to be more likely to be malnourished, food insecure, homeless, living in poverty, and to not be attending school (Andrews, Skinner, & Zuma, 2006; Bryant & Beard, 2016; Cluver, Gardner, & Operario, 2009; DeSilva et al., 2012; Goldberg & Short, 2016).

Among the general population of Zambian youth, exposure to trauma and violence is common. A school-based survey in Zambia found that nearly 1 in 2 adolescents (the majority of whom were aged 13–15) had been in a physical fight in the past year and approximately 1 in 3 had been forced to have sex in their lifetime (Brown et al., 2009). Further, more than 60% of Zambian students in this survey reported having experienced bullying (Siziya et al., 2012). Given that quantitative and qualitative research indicates that HIV-affected children may be exposed to a wide-range of traumas, including death and sickness of close family members, bullying, and physical or sexual abuse (Cluver, Bowes, & Gardner, 2010; Cluver & Orkin, 2009), OVC in Zambia are a particularly important population in which to consider the impact of stressful live events and exposure to trauma. This is supported by a prior qualitative study among women living with HIV and their children in Lusaka, who reported domestic violence and sexual abuse as high priority problems (Murray et al., 2006)

Accordingly, evidence from multiple sub-Saharan African settings indicates that OVC can experience a litany of negative mental health outcomes (Atwine, Cantor-Graae, & Bajunirwe, 2005; Betancourt et al., 2014; Betancourt, Meyers-Ohki, Charrow, & Hansen, 2013; Bryant & Beard, 2016; Doku & Minnis, 2016; Familiar et al., 2014; Narh Doku, 2016). For example, AIDS-orphans in South Africa were more likely than non-orphans or non-HIV-orphans to experience depression and anxiety, and their psychological health worsened relative to these groups over time (Cluver, Orkin, Gardner, & Boyes, 2012). More than a quarter of Zambian adolescents with HIV were found to score above the established cut-off for clinical significance on the Center for Epidemiologic Studies Depression Scale in a clinic-based study (Okawa et al., 2018). In addition to increased levels of depression and anxiety, Rwandan children living with or affected by HIV (i.e. had a parent living with HIV/AIDS or a parent die due to AIDS-related complications) were found to be more likely to exhibit conduct problems and functional impairment as compared to HIV-unaffected children (Betancourt et al., 2014). Posttraumatic stress has also been found to be more common among both HIV/AIDS orphans and children living with an HIV-infected parent, as compared to non-orphaned or non-affected peers in the region (Cluver et al., 2009; Sherr et al., 2014).

In response, a variety of intervention strategies have been developed to improve the psychosocial well-being of OVC ranging from microfinance, livelihood and cash transfer programs to mentorship, social support and psychotherapy (Kilburn, Thirumurthy, Halpern, Pettifor, & Handa, 2016; Kumakech, Cantor-Graae, Maling, & Bajunirwe, 2009; Murray et al., 2015; Schenk, Michaelis, Sapiano, Brown, & Weiss, 2010; Wang, Ssewamala, & Han, 2014). However, the impact of these services on child mental health has rarely been rigorously assessed (Betancourt et al., 2013; King, De Silva, Stein, & Patel, 2009; Visser, Zungu, & Ndala-Magoro, 2015). A critical barrier to the rigorous evaluation of mental health and psychosocial interventions is the availability of adapted and validated instruments for measuring OVC mental and behavioral health. Development of instruments that are feasible and appropriate for use in diverse contexts would facilitate comparison of results across studies, but there is currently little consistency in measurement methods used for studies with this population in sub-Saharan Africa (Betancourt et al., 2013). A recent systematic review identified only 17 criterion validation studies of mental health measures among any child or adolescent population in a low- or middle-income country (LMIC), 12 of which were specific to measures of depressive disorder (Ali, Ryan, & Silva, 2016). In Zambia, there has been some limited research on valid mental health asssessment among adult populations (Chipimo & Fylkesnes 2010, Chishinga et al., 2011, Mbewe et al. 2013), but only one study among children (Murray et al., 2011). This study found the PTSD-RI to be valid for measuring traumatic stress symptoms among a sample with a high prevalence of experience of child sexual abuse in Lusaka (Murray et al., 2011).

Two challenges to conducting validation studies in LMIC are a lack of psychiatrists whose diagnosis can be used as a “gold standard” and reporting bias on sensitive behaviors and topics when instruments are administered face-to-face by lay interviewers from the community. To address this first challenge, our research group has developed an alternative method of validity testing that relies on local identification of cases and non-cases based on multiple raters determined by the community to be knowledgeable on mental health. This known-groups approach to the assessment of cross-cultural validity has been described in detail (Applied Mental Health Research Group, 2013b) and successfully used with youth in Zambia previously (Murray et al., 2011). Regarding reporting bias, a randomized controlled trial of youth in neighboring Zimbabwe demonstrated substantial variation in the observed prevalence of a locally defined affective disorder (41.5–63.6%) depending on the method used to administer the instrument, with the lowest prevalence identified in interviewer administered surveys and the highest with surveys administered via Audio Computer-Assisted Self-Interview (ACASI) (Langhaug, Cheung, Pascoe, Hayes, & Cowan, 2009). Substantial differences were observed in the report of 12 out of 14 symptoms included on the scale, with twice as many youth reporting feeling like they wanted to commit suicide when responding via ACASI as compared to an interviewer (12% vs. 5%) (Langhaug et al., 2009).

Despite demonstrated need, fewer than 20% of OVC in Zambia receive any external support (UNICEF, 2012) and only 10% of OVC across sub-Saharan Africa are estimated to access social or psychological support (Lee et al., 2014). Given this large psychosocial support and mental health treatment gap, the identification of appropriate and accurate instruments for assessing interventions is critical to the development and incorporation of effective mental health interventions for OVC into existing services to improve access (Marais et al., 2014). We therefore sought to explore the factor structure, psychometric proprieties, and validity of two commonly used measures of youth mental health, Achenbach’s Youth Self Report (YSR) and the Child PTSD Symptom Scale (CPSS), as well as a locally developed measure of functional impairment, administered via Audio Computer-Assisted Self-Interview (ACASI) and using an alternative criterion validity approach. We selected the Achenbach Youth Self Report (YSR) because it has been used in multiple sub-Saharan African settings to assess a wide range of youth mental health-related behaviors and symptoms, including among children living with HIV, HIV-orphans, and children of HIV-infected parents (Chi & Li, 2012; L. Cluver, Gardner, & Operario, 2007; Govender, Reardon, Quinlan, & George, 2014; Ndetei et al., 2015; Nduwimana et al., 2016; Okello et al., 2015). Only a few studies have examined the factor structure of the YSR in sub-Saharan Africa (Geibel et al., 2016; Hall et al., 2014; Harder et al., 2014), and its validity has only been thoroughly examined in one sub-Saharan African setting: Ethiopia (Hall et al., 2014; Geibel et al., 2016). Support was found for the validity of the internalizing, but not externalizing, subscale of the YSR by Hall et al. (2014), and Geibel et al. (2016) found strong support for only the anxious depression and social problems sub-scale among girls and attention problems among boys. We selected the CPSS as it is one of the only freely available measures of posttraumatic stress symptoms in children that has been found to be valid after rigorous study in a LMIC setting (Kohrt et al., 2011).

Methods

Sample

This validity study was a part of a larger randomized control trial (ClinicalTrials.gov identifier: NCT02054780) evaluating the effectiveness of an evidence-based psychotherapy (Trauma-Focused Cognitive Behavioral Therapy) for reducing HIV-risk behaviors and mental health problems in OVC living in Lusaka, Zambia. In cooperation with the Archdiocese of Lusaka, which oversees a network of home-based care workers (HBCWs) providing supportive services to OVC, adolescents proficient in Nyanja or English were recruited from three communities within Lusaka. HBCWs are commonly used throughout Zambia to provide care to families within their own communities. For this trial, HBCWs reviewed their existing client lists to identify youth between the ages of 13 and 17 whom they judged to be at high risk of acquiring HIV due to their behaviors (i.e. an HIV-risk case) or who exhibited few or no HIV-risk behaviors (i.e. an HIV non-case). They reviewed these same lists to identify youth of the same age thought to be experiencing significant psychosocial problems (i.e. a psychosocial case) or who exhibited no psychosocial problems (i.e. a psychosocial non-case). HBCWs contacted youth that they judged to fall into at least one of these four categories using a recruitment script. If interested in joining the study, the adolescent and a caregiver of their choosing were invited to complete an eligibility screening at a pre-specified time and location.

The eligibility screening consisted of asking the HBCW, the adolescent, and his or her caregiver separately whether they considered the adolescent to fall into any of the four eligibility categories described above, based on the presence or absence of psychosocial and HIV-risk behaviors (see Kane et al. 2017). Each participant was asked to respond ‘yes’ or ‘no’ to the following four questions (when asking adolescents to respond, the language was modified from “is/does the adolescent” to “are/do you”):

1. Is the adolescent at high risk of HIV because of his/her behaviors?

2. Is the adolescent at little or no risk of HIV because of his/her behaviors?

3. Does the adolescent have significant psychosocial problems?

4. Does the adolescent have no significant psychosocial problems?

To be eligible for one of the two non-case categories, all three parties had to agree that the adolescent did not exhibit psychosocial problems or HIV-risk behaviors (i.e., responses of “yes” by each of the three reporters to question 2 to be a HIV-risk non-case and responses of “yes” by each of the three reporters to question 4 to be a psychosocial problem non-case). To be considered an HIV-risk case, the adolescent alone and/or the caregiver plus the HBCW had to report that the adolescent was at high risk of HIV because of his or her behaviors (responses of “yes” to question 1). To be considered as a psychosocial case, the adolescent plus either the HBCW or the caregiver had to report that the adolescent had significant psychosocial problems (responses of “yes” to question 3). Eligible adolescents could belong to more than one category, but could not be considered both a case and a non-case for the same set of problems. We have included the concordance eligibility grid that was used in the field as supplemental material (Figure S1).

Once screened as eligible, all adolescent participants provided written informed consent and their caregivers provided permission. The Johns Hopkins Bloomberg School of Public Health’s Institutional Review Board and the University of Zambia’s Ethical Review Board both reviewed and approved the study.

Measures

Adolescents completed the assessment on laptops with Audio Computer-Assisted Self-Interview (ACASI) software (Tufts University School of Medicine, 2014), which allows participants to read each question on their own while listening to a recording of each question read aloud using headphones. ACASI was used due to the sensitive nature of the assessment questions (e.g. sexual behavior), as it has been found to reduce (but not eliminate) underreporting on a variety of sensitive topics in sub-Saharan African settings (Adebajo et al., 2014; Beauclair et al., 2013; Kelly et al., 2014; Langhaug et al., 2011; Waruru, Nduati, & Tylleskär, 2005). Study assessors introduced ACASI to adolescents and remained in the vicinity during the assessment to assist as needed. In rare instances where an adolescent had no or very low literacy or computer competency, the participant would listen to the audio recording of the question and response options and indicate to the assessor which option to choose. The assessor would input the selection for the participant and control navigation (Kane et al. 2016). Participants could choose to complete the ACASI assessment measures in either Nyanja (a common language spoken in Lusaka) or English. All measures were translated from English to Nyanja using a professional translation service in Lusaka. This process involved two translators who were both fluent in English and Nyanja. Following initial translation into Nyanja by the first translator, a second individual performed a back translation to English to ensure the Nyanja version retained semantic equivalency. Discrepancies between the translations were resolved through discussion between the two translators. We then conducted pilot group testing on the final translated version with adolescents fluent in both English and Nyanja to further check semantic equivalency.

The full battery of instruments included in ACASI covered mental health, HIV risk behaviors, substance use, and OVC well-being (Kane, Murray, Bass, Johnson, & Bolton, 2016). The present analysis focuses on two mental health measures: the Achenbach Youth Self Report (YSR) and the Child PTSD Symptom Scale (CPSS).

Achenbach YSR

The 119-item YSR is a widely-used measure of mental health and behavioral problems among youth aged 11 to 18 years (Achenbach & Rescorla, 2001; Achenbach, 1991). Originally developed in high-resource country-based clinical samples and revised in 2001, 95 of the YSR items make up eight empirically derived syndrome subscales: anxious depression, withdrawn depression, somatic complaints, social problems, thought problems, attention problems, rule breaking behavior, and aggressive behavior. In addition, the anxious depression, withdrawn depression and somatic subscales combine to measure internalizing problems, while the rule breaking and aggressive behavior subscales together measure externalizing problems. Study participants indicated how true the item was of themselves over the past four weeks using a Likert scale of 0 “not true”, 1 “somewhat or sometimes true”, or 2 “very true or often true.”

CPSS

The 17-item CPSS was designed to measure DSM-IV defined posttraumatic stress disorder among children and adolescents aged 8–19 years (Foa, Johnson, Feeny, & Treadwell, 2001). Items are derived from all three PTSD symptom clusters: re-experiencing, avoidance and numbing, and hyperarousal. Originally developed in a sample of US-based child earthquake survivors (Foa et al., 2001), the CPSS has also demonstrated strong psychometric properties among youth affected by other traumas, such as sexual violence or injury (Gillihan, Aderka, Conklin, Capaldi, & Foa, 2013; Nixon et al., 2013). Study participants indicated how often they experienced each symptom in the last two weeks: 0 “never”, 1 “once in a while”, 2 “more than half the time”, or 3 “almost always.” The CPSS also includes a 14-item inventory of exposure to potentially traumatic events (e.g. natural disasters, community or interfamily violence, and accidents), scored by summing the number of events an individual indicated having experienced. Seven additional items ask participants to indicate whether any of the symptoms of trauma interfered with tasks or important aspects of daily life (e.g. prayers, schoolwork, or happiness). This CPSS specific impairment scale was scored by summing the number of items to which participants responded “yes.”

Functional impairment

Following methods detailed elsewhere (Applied Mental Health Research Group, 2013a; Bolton & Tang, 2002), a locally specific 22-item index for assessing impairment in completing tasks of daily living was developed using qualitative descriptions of common activities of children or adolescents. Study participants indicated how much difficulty they experienced in doing each task or activity in the past four weeks on a Likert scale of 0 “none”, 1 “little”, 2 “moderate amount”, 3 “a lot”, or 4 “often cannot do.” Participants also had the option to choose not applicable for each item on the functional impairment measure. The full measure is available as supplemental material (Table S1).

Additional psychosocial well-being measures

The study instrument also included 10 items on support received by adolescents from their parent(s) or caregiver(s). For these items, youth rated how often their caregiver or parent engaged in a variety of behaviors with them (e.g. “helps me make decisions” or “politely points out my mistakes”) and asked how often the youth “feel(s) loved by my parents or guardians” and “get(s) along well with my parents or guardians.” Youth responded to each item using a Likert scale of 0 “never or almost never” to 3 “always or nearly always.” In addition, adolescents responded to the 36-item Orphan and Vulnerable Children wellbeing tool developed by Catholic Relief Services for evaluation of support services delivered to OVC aged 13–18 years. This self-report measure was piloted in several low-income countries including Zambia, revised through factor analysis, and found to exhibit both a strong correlation with a previously validated measure of hope and strong inter-item reliability (Senefeld, Strasser, & Campbell, 2009). The OVC well-being tool covers ten domains: food and nutrition, shelter, protection, family, health, spirituality, mental health, education, economic opportunities, and community cohesion. Participants rate how often each item is true of themselves on a scale of 0 “none of the time,” 1 “some of the time,” or 2 “all of the time” (Senefeld, Strasser, & Campbell, 2009).

Analysis

Summary statistics for participant demographics were generated for the total sample and compared by psychosocial problem case status using independent sample student’s t-tests and chi-squared tests. Scale scores were calculated for the YSR and CPSS total scales and subscales by summing responses across items for each participant per standard practice (Achenbach & Rescorla, 2001; Achenbach, 1991; Foa et al., 2001). Functional impairment scores were generated by averaging participant responses for all items, with a higher average score indicating greater impairment. For these scales, mean imputation was used to account for missing item responses due to a low level of missingness (<3% for any item).

Factor structure

To assess whether the 8-syndrome or 2-syndrome YSR subscale structure was more appropriate in our sample, we conducted two confirmatory factor analyses (CFA) with a polychoric correlation structure and a weighted least squares estimator. In both CFAs, each item was assessed as loading onto only one factor and correlations were allowed between all factors. The chi-square goodness of fit test, comparative fit index (CFI) (Bentler, 1990), Tucker-Lewis index (TLI) (Marsh, Balla, & McDonald, 1988), and the root mean square error of approximation (RMSEA) (Hu & Bentler, 1999) were used to compare the overall and relative fit of these non-nested models with different numbers of factors. A CFI and TLI over 0.9 was considered indicative of adequate fit (Little, 2013). An RMSEA of 0.05 or under was considered excellent fit, 0.06 to 0.08 adequate or mediocre fit, and over 0.1 poor fit (Browne & Cudeck, 1993; Little, 2013). In addition, we re-ran the CFAs using a maximum likelihood estimator to produce BIC values that could be used to directly compare the fit of these non-nested models. Using the same approach, we assessed whether a one-factor solution or a three-factor solution comprised of the diagnostic symptom cluster subscales was a better fit for the CPSS measure. Items were considered to meaningfully load on a factor if the loading was >0.4. All confirmatory factor analyses were conducted in Mplus Version 7 (Muthen & Muthen, 2011)

Internal consistency and test-retest reliability

Cronbach’s alpha statistics were calculated for the mental health and functioning measures to assess internal consistency. Correlations among scale items were also examined, with items having an item-rest or item-test correlation of under 0.30 considered for removal, as well as any item whose exclusion would improve the Cronbach’s alpha of the scale by 0.001 or more. To assess test-retest reliability, a random sub-sample of 33 adolescents were re-administered the same assessment via ACASI five to seven days after their initial assessments. Spearman rank correlation coefficients were calculated between the original and follow-up score on all scales.

Validity

Adolescent, caregiver, and HBCW agreement on whether adolescents did or did not exhibit significant psychosocial problems as described above was used as a local criterion for caseness, as no appropriate gold standard exists in Zambia for diagnosing mental health problems (Applied Mental Health Research Group, 2013b; P Bolton, 2001). To assess scale validity using this local “known groups” criterion, mean score on the mental health scales were compared between psychosocial cases and non-cases using the nonparametric Mann-Whitney U test (due to the non-normal distribution of scores) (Hart, 2001). A p-value of less than 0.05 was considered statistically significant. Using locally determined caseness, receiving operator characteristics (ROC) curves were also plotted, and the area under the curve was calculated (AUC) to assess how well the mental health scales could differentiate cases from non-cases. Cut-off scores were chosen for diagnoses using the Liu method, which maximizes the product of sensitivity and specificity (Liu, 2012). We also calculated Youden’s index (J) at this optimal cut point as an indicator of overall diagnostic effectiveness (Youden, 1950). This statistic ranges from 0 to 1 with higher values indicating better diagnostic performance.

To assess convergent construct validity, Spearman rank correlations (due to non-normal scale distributions) were calculated between all mental health (CPSS and YSR) scale and sub-scale scores, functioning score, and other psychosocial constructs. As an assessment of convergent validity, we hypothesized that greater mental health symptom severity and frequency would be associated with lower caregiver support, worse overall wellbeing, exposure to more types of traumatic events, and greater functional impairment. As a demonstration of discriminant validity, we hypothesized that the relationship between YSR scores would be stronger than that between YSR and posttraumatic stress and that traumatic event exposure would be more strongly correlated with posttraumatic stress scores. All construct and internal consistency analyses were conducted using Stata 14 (StataCorp, 2015).

Results

Participants

A total of 349 adolescents were screened by HBCWs, of which 219 were found to be eligible. Nine eligible individuals who consented to be in the study were not able to complete the assessment the same day. Though invited by the study team, none of these individuals returned to complete the assessment. Of the 210 adolescents identified as eligible for the validity study who completed the assessment, 55 met the definition of a non-psychosocial case, 110 met the definition of a psychosocial case, and 45 met neither criteria but were enrolled in the study because they met criteria as either a HIV-risk case or HIV non-case. Data from the 45 participants who did not meet the definition of either a psychosocial case or non-case were used in the factor, internal consistency, and construct validity analyses, but not the assessment of criterion validity. Among the 110 individuals who qualified as a psychosocial case, 58 (53%) also met criteria as an HIV-risk case. For 65% of psychosocial cases, all three reporters (the adolescent, the caregiver, and the HBCW) agreed that the adolescent had significant psychosocial problems; in 16% of cases, only the adolescent and caregiver agreed, and in 18%, only the adolescent and HBCW.

Table 1 provides a characterization of the study sample in total and separately by case status. Among the total sample, 44% of participants were male and the average age was approximately 15 years. Psychosocial cases were approximately half a year older on average than non-cases (15.08 vs. 14.62). Overall, the adolescents were ethnically diverse with individuals identifying as Bemba (30%) or Ngoni (25%) most commonly. Just over three-quarters of the adolescents were in school, and non-cases were more likely than cases to be enrolled (85% vs. 71%). Cases were more likely to have either their biological mother or father as a primary caregiver than non-cases (69% vs. 51%). Approximately one-third of adolescents in the sample were double orphans (both parents had died), 9% were HIV-infected, and 22% reported a disability. The average number of traumatic experiences reported by psychosocial cases was 4.55 compared to 2.45 among non-cases (p<0.001). Among all participants, the most common types of traumatic events experienced were seeing a dead body that was not at a funeral (47.1%); seeing someone being beaten up, shot at, hurt badly, killed or almost killed (42.4%; and, seeing a family member being hit, punched or kicked very hard at home (36.7%).

Table 1.

Demographics of total sample and by psychosocial caseness^a

	Non-psychosocial case (n=55)	Psychosocial case (n=110)	Totalb (n=210)
Age, mean (sd)*	14.62 (1.46)	15.08 (1.40)	14.88 (1.45)
Male, no (%)	22 (40.00)	49 (44.55)	92 (43.81)
Ethnicity
Ngoni	12 (21.82)	30 (27.27)	58 (27.62)
Bemba	18 (32.73)	31 (28.18)	60 (28.57)
Other	25 (45.45)	49 (44.55)	92 (43.81)
Currently in school*	47(85.45)	78 (70.91)	158 (75.24)
Biological parent is primary caretaker*	38 (69.09)	56 (50.91)	114 (54.29)
Orphan status
Single	29 (52.72)	55 (50.00)	101 (48.09)
Double	14 (25.45)	40 (19.05)	69 (32.86)
HIV-positive	3 (5.56)	13 (11.93)	18 (8.65)
Disability	10 (18.18)	30 (27.27)	47 (22.38)
Traumatic experiences*	2.45 (2.39)	4.55 (3.62)	3.77 (3.34)

^aSignificance is for test of difference by case-status using independent sample student's t-tests and chi-squared tests.

^bIncludes 45 individuals who did not meet the criteria to be considered either a psychosocial case or non-case, but were included in the study by meeting the criteria of one of the two HIV-risk behavior categories.

^*p<0.05

Factor Analysis

CPSS

CFA model fit indices are presented in Table 2. The Χ² p-value was significant for both models (3-factor: 0.004; 1-factor: <0.001), indicating worse fit in comparison to the saturated model. However, the RMSEA value fell in the adequate range for the 1-factor model (RMSEA 0.052, 05% CI: 0.037, 0.066) and excellent range for the 3-factor solution (0.043, 95% CI: 0.025, 0.058). CFI and TLI fit statistics fell above the 0.9 threshold of adequate fit for both models (3-factor: CFI=0.988, TLI=0.986; 1-factor: CFI=0.982, TLI=0.979). As the 3-factor model also had a lower BIC value than the 1-factor model (7133.0 vs. 7149.6), we choose to proceed with the 3-factor model in subsequent analyses, but present a total CPSS score in the following as well, as the 1-factor model overall exhibited adequate fit.

Table 2.

Confirmatory factor analysis fit statistics for child mental health measures (n=210)

Scale	Model	RMSEA (95% CI)	X2 p-value (saturated model)	CFI	TLI	BIC
Youth Self Report	2 factor	0.035 (0.030, 0.039)	<0.001	0.941	0.939	20628.6
Child PTSD Symptom Scale	3 factor	0.043 (0.025, 0.058)	0.004	0.988	0.986	7133.0
	1 factor	0.052 (0.037, 0.066)	<0.001	0.982	0.979	7149.6

The factors in the 3-factor model were highly correlated (range: 0.874–0.951; all p-values<0.001). In the 1-factor model, loadings ranged from 0.65 to 0.85 in magnitude with a mean of 0.75 (standard deviation (sd)=0.05). In the 3-factor model, all items loaded on their respective factors above 0.4 (all p-values<0.001). The average item loading was 0.78 (sd=0.07) for the re-experiencing factor, 0.77 (sd=0.04) for the avoidance and numbing factor, and 0.77 (sd=0.06) for the hyperarousal factor (full results available online in Table S2).

YSR

As can be seen in Table 2, the two-factor model exhibited good fit with an RMSEA estimate under 0.05 (0.035, 95% CI: 0.030, 0.039), and a CFI and TLI exceeding 0.9 (0.941 and 0.939, respectively). The The Χ² p-value was significant (<0.001) however, indicating worse fit in comparison to the saturated model.

The 8-factor model produced a non-positive definite psi correlation matrix and was not able to be interpreted. Given the large number of parameters being estimated in the 8-factor model with only 210 data points and high Spearman correlations observed between the eight subscales (range: 0.67, 0.83), we only proceeded with the 2-factor model. Item loadings in the 2-factor model ranged from 0.309 to 0.780 for the internalizing factor and from 0.348 to 0.836 for the externalizing factor (full results available online in Table S3). The mean item loading for the internalizing factor was 0.67 (SD=0.10) and the median was 0.68. For the externalizing factor, the median and mean item loading was 0.71 (SD=0.10). All item loadings were significant at a p-value of less than 0.001. Although significant, the item “feeling that one needed to be perfect” (0.309) did not load strongly on the internalizing factor and “trying to get attention” (0.348) did not load strongly on the externalizing factor. The internalizing and externalizing factors had a 0.887 correlation (p-value <0.001). Given the high correlation between the internalizing and externalizing factors on the YSR in the 2-factor model, we also fit a 1-factor model as a sensitivity analysis. The fit indices for the 1-factor model (RMSEA=0.038, 95% CI 0.033, 0.042; CLI=0.930; TFI=0.927; BIC=20771.2) were not as strong as the indices of the 2-factor model, but overall were adequate.

Internal Consistency and Test-Retest Reliability

Cronbach’s alphas indicated good to excellent internal reliability for all mental health scales and subscales (>0.80) (Table 3). All YSR items had an item-test and item-rest correlation of greater or equal to 0.30 with other subscale items except for feeling a need to be perfect for the internalizing subscale (item-rest correlation=0.329; item-test correlation=0.274; alpha improvement=0.001); and, trying to get attention for the externalizing subscale (item-rest correlation=0.280; item-test correlation=0.223; alpha improvement=0.002). All CPSS items had item-test and item-rest correlations over 0.3. Based on these findings as well as the CFA results, feeling a need to be perfect was dropped from the YSR internalizing subscale and trying to get attention from the externalizing subscale for all criterion and subsequent construct validity analyses. On these revised scales, correlations between test and re-test scores ranged from 0.62–0.68 across the symptom scales (Table 3), with the strongest test-retest reliability observed for the CPSS total and hyperarousal subscale.

Table 3.

Comparison of mental health symptom score by local case category and significant problem thresholds (n=165)¹

	Youth Self Report (YSR)		Child PTSD Symptom Scale (CPSS)
	Externalizing	Internalizing	Total	Re-experiencing	Avoidance and numbing	Hyperarousal
Range of possible scores	0–62	0–60	0–51	0–15	0–21	0–15
Score among cases, mean (SD)	18.84 (14.63)	23.76 (13.81)	18.57 (12.92)	5.56 (4.26)	7.53 (5.68)	5.48 (4.03)
Score among non-cases, mean (SD)	9.60 (9.84)	12.78 (11.11)	9.41 (9.66)	2.67 (3.27)	3.85 (4.69)	2.88 (2.97)
Difference in score by case status2	9.24*	10.98*	9.16*	2.89*	3.68*	2.60*
AUC (95% CI)	0.72 (0.64, 0.80)	0.74 (0.66, 0.82)	0.73 (0.64, 0.81)	–	–	–
Liu estimated cut-off	7.5	13.5	11.5	–	–	–
Sensitivity	0.79	0.77	0.68	–	–	–
Specificity	0.56	0.62	0.69	–	–	–
Youden’s index (J)	0.35	0.39	0.37	–	–	–
Cronbach’s alpha	0.94	0.93	0.93	0.84	0.86	0.82
Test-retest correlation3	0.66	0.62	0.68	0.63	0.68	0.64

¹Cronbach’s alpha conducted with full sample of 210 adolescents

²Significance for Wald Ranksum Test of difference in means between cases and non-cases

³Test-retest assessed in a randomly selected sub-sample of 33 adolescents

^*<0.001

Cronbach’s alpha for the functional impairment measure was 0.94; dropping two items on the functioning scale (difficulty “going to school” and “eating well”) would each have improved the alpha (by 0.001). The correlation between test and retest functioning scale scores was 0.78.

Criterion and Construct Validity

Cases had a statistically significant greater mean on all mental health scales and subscales than non-cases, including a 49% greater externalizing score (cases: mean=18.8, sd=14.6; non-cases: mean=9.6, sd=9.8), a 46% greater internalizing score (cases: mean=23.8, sd=13.8; non-cases: mean=12.8, sd=11.1), and a 49% greater posttraumatic stress score (cases: mean=18.6, sd=12.9; non-cases: mean=9.4, sd=9.7) (Table 3). The AUC was above 0.7 for all scales and the lower 95% confidence limit was above 0.5, indicating better prediction of case status than chance alone. Cut-off scores that optimize and balance sensitivity and specificity are also presented in Table 3. Sensitivity ranged from 68% for the posttraumatic stress scale to 79% for the externalizing scale, and specificity from 56% for externalizing problems to 69% for posttraumatic stress symptoms.

Mental health scale scores were all substantially and statistically significantly correlated with one another, with the greatest correlation between the internalizing and externalizing problem scales (rho=0.811) and the weakest between the externalizing and CPSS hyperarousal subscale (rho=0.438). Table 4 also presents the pairwise correlation coefficients between each mental health symptom scale and variables in the nomological network. None of the mental health scales were meaningfully or significantly correlated with the caregiver support scale (internalizing rho=0.02; externalizing rho=0.05; posttraumatic stress subscales rho range: −0.013–0.049). Number of types of traumatic experiences correlated most highly with the CPSS re-experiencing core (rho=0.562), though impairment from posttraumatic stress symptoms was slightly more strongly correlated with the YSR internalizing subscale than the CPSS subscales. All scales were inversely and significantly correlated with the OVC well-being scale, though at a level below rho=0.4.

Table 4.

Spearman correlations for mental health and functional impairment scales (n=210)

	Youth Self Report (YSR)		Child PTSD Symptom Scale (CPSS)
	Internalizing	Externalizing	Re-experiencing	Avoidance and numbing	Hyperarousal	Functional Impairment
YSR Externalizing	0.811***	--	--	--	--	--
CPSS Re-experiencing	0.616***	0.495**	--	--	--	--
CPSS Avoidance and numbing	0.589***	0.480***	0.799***	--	--	--
CPSS Hyperarousal	0.546***	0.438***	0.702***	0.745***	--	--
Functional Impairment	0.401***	0.375***	0.363***	0.375***	0.383***	--
Caregiver Support	0.023	0.054	−0.013	0.015	0.049	−0.010
Traumatic Experiences	0.527***	0.487***	0.562***	0.553***	0.477***	0.293***
Trauma Impairment	0.445***	0.419***	0.425***	0.411***	0.321***	0.248**
OVC Well Being	−0.237**	−0.163*	−0.275**	−0.300***	−0.299***	−0.124

^***<0.001

^**<0.01

^*<0.05

All mental health scales were significantly and similarly correlated with functioning, with internalizing symptoms exhibiting the strongest relationship to functional impairment (rho=0.401). Functional impairment was less strongly related to other variables in the nomological network compared to the mental health scales.

Discussion

We explored the validity and psychometric properties of two scales of child mental health symptoms, the Achenbach’s Youth Self Report (YSR) and the Child PTSD Symptom Scale (CPSS), administered using ACASI to OVC in Zambia. In a known-groups validation using local definitions of caseness, youth experiencing psychosocial problems scored roughly twice as high on these scales than non-affected youth. All mental health scales and subscales also exhibited strong internal consistency and were highly correlated with each other. These finding support the validity of these symptom scales for measuring significant psychosocial problems of local relevance and importance. This finding is in accordance with the assessment of validity of Zambian adaptations of adult measures of depression and general psychological distress (Chishinga et al., 2011; Chipimo & Fylkesnes, 2010) and posttraumatic stress in children (Murray et al., 2011) that were originally developed for use in high income settings.

One concern for the use of these scales is that test-retest reliability for all measures was adequate but not strong, which should be assessed in additional larger samples with variation in length of time between test and retest to explore variability in symptom expression and measurement psychometrics. In addition, low specificity of identification of problematic symptoms and behaviors, particularly for externalizing behaviors measured by the YSR and the functional impairment scale, suggest these scales may be more appropriate for use in identifying change in symptoms over time rather than screening. The YSR may also not be a feasible option for screening in clinical settings in Zambia given its length. If the instruments were to be used for screening, selecting a higher threshold that is more specific (at the expense of sensitivity) than the optimal cut points presented here, and thus targeting the most severe cases for identification, may be warranted in settings like Zambia where the need for services far outweighs supply. This is especially true for children and adolescents; a prior situational analysis found only 10 outpatient facilities for mental health treatment, none of which focused on children and adolescents specifically. Further, only 11% of in-patient hospital beds were designated for children and adolescents (Kleintjes, Lund & Fisher, 2010).

The YSR is a widely-used scale that has been translated into more than 70 languages and dialects (see http://www.aseba.org/ordering/translations.html). Given the diverse set of psychosocial problems reported by OVC in Zambia and elsewhere (Murray et al., 2006; Sherr et al., 2014), a strength of the YSR for use in this population is the wide range of mental health problems and behaviors that it covers. Internal consistencies for the broader internalizing and externalizing scales were strong in our study and on par with the original 2001 Achenbach assessment that found an alpha of 0.9 for internalizing and externalizing scales, as well as what has been found in other sub-Saharan African settings (Geibel et al., 2016; Hall et al., 2014; Harder et al., 2014). In a validation study in Ethiopia that also used locally identified cases and non-cases, support was found for the internalizing but not the externalizing scale, possibly due to social desirability bias in reporting behaviors. It is possible that use of ACASI in this study is a reason for the difference in our finding and a potential strength of our approach. This would be consistent with the findings of a recent systematic review that use of ACASI was associated with higher reporting of sexual behaviors among youth in LMIC than in face-to-face interviews (Langhaug, Sherr, & Cowan, 2010). Although rates of literacy and computer competency were likely lower among this OVC population compared to populations in high income countries, we also found the use of ACASI to be an acceptable and feasible measurement modality (Kane et al., 2016).

While the 8-factor syndrome structure of the YSR has been evaluated among 30,000 participants from 23 countries, most of these countries were classified as high-income and only one was in sub-Saharan Africa (Ethiopia) (Ivanova et al., 2007). Since publication of the 23-country analysis, the factor structure of the YSR has also been evaluated in Kenya (Harder et al., 2014), where the 8-factor model fit well and loadings were in the same range as the 23-country study. While we were unable to assess the 8-factor structure in our data, we did find that a 2-factor YSR structure (one for internalizing and one for externalizing symptoms) exhibited good fit in this population, with strong item loadings. Median item loadings in the two-factor model we tested (0.68 and 0.71 for internalizing and externalizing respectively) were higher than the range observed in both Kenya and the 23-country study (0.56 and 0.53–0.67, respectively). Items that did not load well included feeling a need to be perfect (0.309) on the internalizing factor and trying to get attention (0.348) on the externalizing factor. Low loadings of these items were also observed in Kenya (Harder et al., 2014). However, several items that did not perform well in the Kenyan sample did load well in the two-factor solution we assessed (e.g. prefers older kids, fears going to school). In addition, several items with similarities to questions that exhibited differences in conceptualization among an adult Zambian population performed well in our child and adolescent sample (Chipimo & Fylkenses, 2010). This included having headaches, having an upset stomach, feeling tense, and feeling tired. This may be due to different framing of items from what was used in the previous study with adults. For instance, the measure we used asked participants if they experienced an upset stomach specifically when thinking about a traumatic event, or having felt tired without a reason rather than just feeling fatigued in general.

The CPSS subscales and total scale exhibited excellent internal reliability in our sample, similar to what has been found in other studies from LMIC (Kohrt et al., 2011; Tol et al., 2008, 2012; Ventevogel, Komproe, Jordans, Feo, & De Jong, 2014). Studies of the factor structure of this scale are mixed as to whether it is best characterized as a measure of one general PTSD construct or of three distinct symptom clusters (Meyer, Gold, Beas, Young, & Kassam-Adams, 2014; Stewart, Ebesutani, Drescher, & Young, 2015). In our study, both models showed adequate fit, though the 3-factor model performed marginally better. In a prior latent class analysis of posttraumatic stress symptoms among children in Zambia, groupings based on symptoms were found to be distinguished by severity rather than type (Familiar et al., 2014). Taken together, the subscales of the CPSS perform well in Zambia and their characterization is empirically supported; however, there may not be a distinct advantage in using a 3-factor over a 1-factor model in practice.

The CPSS cut-off we found for separating cases characterized by significant symptoms from non-cases is similar to what was identified during the scale’s development in the US among child survivors of an earthquake (Foa et al., 2001) and among refugees in Ethiopia (Hall et al., 2014). It is substantially lower, however, than what has been found among conflict-affected populations in Nepal and Burundi (Kohrt et al., 2011; Ventevogel et al., 2014). Ventevogel et al. suggest the higher cut-off they found in Burundi could be due to reports of general, non-specific distress arising from exposure to armed conflict. While a variety of traumatic experiences have been observed among our study population, Zambia (like the US) does not have a recent history of wide-scale political violence, which may be one reason for the difference in observed optimal cut-off score. A second possibility is a difference in the criterion used: both in our study and the Ethiopia study, local identifications of youth with problematic symptoms were used. In Burundi and Nepal, a structured clinical interview was used to diagnose youth.

Limitations

For this study, we did not assess participants’ primary language. We only assessed whether respondents preferred responding in either English or Nyanja. We were therefore not able to perform analyses to see if our findings were sensitive to respondents’ primary language status. Due to limited sample size, we also did not assess for measurement variance by language of response. Our analysis of test-retest reliability may have been influenced by instability associated with the small sample size of participants. To assess the effects of social desirability, it would be useful to compare the reliability and validity of each scale, particularly the YSR externalizing behavior subscale, administered via ACASI versus in-person interview. Without accompanying cognitive interviewing, we are also limited in our ability to understand the reasons that some items performed poorly. Reasons for similar findings on some low-performance items across LMIC for the YSR is an important area of further research. Discriminant validity analyses and testing of factor structure were performed in the same dataset due to a limited sample size; however, validity findings were similar when we repeated the analyses using the 8-subscales of the YSR or 3-subscales of the CPSS rather than the two and one-factor structure, respectively (findings available upon request). In addition, due to the recruitment strategy used, the OVC in our study were engaged with the healthcare system and may not be representative of other OVCs.

Conclusions

We found support for the validity of instruments measuring multiple components of psychosocial distress (posttraumatic stress, internalizing, and externalizing symptoms) administered via ACASI to adolescents orphaned or otherwise made vulnerable by HIV/AIDS. Given the intense public health interest in this population, there is a need for accurate tools to evaluate the need for, and impact of, mental health and psychosocial services targeting this group. Given a lack of such tools in LMIC (Ali et al., 2016), their development and testing is an important programmatic and research priority. We also studied a locally-developed functional impairment scale and found it to be valid for the same population. The development of valid measures of functional impairment is essential to helping service providers ensure that programs have tangible effects on the daily lives of HIV-affected adolescents. Future research should focus on the relative performance of these scales for OVC of different ages, genders, and HIV-status. Given the limited assessment of ACASI for administering mental health measures among youth in sub-Saharan Africa, future research should also focus on the feasibility of this approach for wide-scale screening.

Supplementary Material

Supplementary Material

Figure S1. Concordance grid for eligibility screening

Table S1. Items contained on the locally-developed scale of functional impairment

Table S2 Three-factor confirmatory factor analysis results for the Child PTSD Symptom Scale (n=210)

Table S3. Two-factor confirmatory factor analysis results for the Youth Self Report (n=210)