A Brief Assessment of Learning for Orphaned and Abandoned Children in Low and Middle Income Countries

Abstract

Assessment of children’s learning and performance in low and middle income countries has been critiqued as lacking a gold standard, an appropriate norm reference group, and demonstrated applicability of assessment tasks to the context. This study was designed to examine the performance of three nonverbal and one adapted verbal measure of children’s problem solving, memory, motivation, and attention across five culturally diverse sites. The goal was to evaluate the tests as indicators of individual differences affected by life events and care circumstances for vulnerable children. We conclude that the measures can be successfully employed with fidelity in non-standard settings in LMICs, and are associated with child age and educational experience across the settings. The tests can be useful in evaluating variability in vulnerable child outcomes.

Introduction

Since the onset of the HIV/AIDS pandemic, program developers, policy makers, and researchers have focused on the large number of orphaned and abandoned children in high HIV prevalence and low and middle income countries (LMIC). The ill effects of being an orphaned or abandoned child (OAC) in a LMIC are reported to include fewer years of schooling, poorer school performance, poorer households, less adequate housing, increased malnutrition, higher morbidity and mortality, poorer access to medical care, and a higher prevalence of depression [1–10]. Many investigators describe the challenges faced by families and communities in providing food, shelter, health care, and education for increasing numbers of OAC while the number of potential caregivers is diminishing due to early mortality from HIV/AIDS and other causes [11–15]. The challenges may be notably different from those faced by the growing number of children in LMIC who are separated from one or both parents for economic reasons, including labor migration within and across nations. While the latter experience some of the detrimental effects of parental separation and the challenges of variable kinship care, they may also benefit from more household income and the maintenance of an ongoing relationship, albeit more limited, with distant caregivers [16–19].

Child development is supported and challenged by multiple inter-dependent pathways, including the context in which development occurs [20–24]. Child physical, social-emotional, and mental health are the results of a complex and dynamic system of risk and protective factors in health (e.g., cerebral malaria, HIV/AIDS, preventive health care), factors in the care environment (e.g., food insecurity, maltreatment or exploitation, consistency of the primary caregiver) as well as the presence or absence of resiliency (e.g., temperament, intelligence) [24–26]. The effects of multiple negative influences on development are rarely as apparent as with children who are living in dire poverty, who are orphaned or abandoned by parents, who may experience changes in primary caregivers multiple times, and who are also vulnerable to the stigma of HIV/AIDS [27].

One recurring difficulty for evaluators and investigators lies in identifying feasible and valid assessment of child risk factors and outcomes while, of necessity, using assessment strategies developed in other contexts. This is especially true in the assessment of the child’s learning ability and performance. Any assessment of learning is called upon to be methodologically relevant in the local context as well as valid in measuring the child’s learning strengths and difficulties relative to his/her age peers. There are no gold standards for assessing child learning across LMIC; that is, there are no tests developed in LMIC that have been shown to have the same psychometric properties across contexts. When used in large studies that examine multiple domains, the assessment of learning also needs to be relatively brief.

In previous studies of child intelligence and learning in LMIC, cognitive development after disease is a prevalent theme, with research exploring the effects of HIV/AIDS [28–31] and malaria [32–40] on the abilities and achievement of children in African countries. Investigators in these studies often use instruments developed in high income countries; and, for the most part, the strategies have been successful in demonstrating a hypothesized compromise in cognitive development due to disease effects on the central nervous system. Investigators frequently note that (1) a country-specific norm reference group and/or a pre-disease baseline might have resulted in more informative results and that (2) disease mediated developmental processes are modified also by the environmental context and quality of care [34, 35].

Child learning in high risk circumstances has also been studied with regard to psychosocial factors, such as potentially traumatic events [41] and having parents who died of, or are sick with, HIV/AIDS [10, 42]. These investigators also discuss the difficulties of assessment in these settings, such as the lack of a norm reference group specific to the setting and the poor applicability of assessment tasks or items for the context. The obvious challenge is the lack of a gold standard assessment shown to be a valid test of learning and performance across LMIC.

As noted, tests of cognitive development used for these studies are often tools developed and standardized elsewhere that require translation and, in most instances, local adaptation as well as translation. Direct translation approaches have used the Bayley Scales of Infant Development (BSID) [43] and the Kaufman Assessment Battery for Children (KABC) [44] and, less frequently, the Griffiths Scales [45] and the McCarthy Scales [46]. Alternatively, several studies involved the development of local norms for use with translated and adapted test strategies. Examples of the latter include use of the KABC in Uganda [30, 37] and the Bayley Scales of Infant Development in Kenya [47].

Prior research notwithstanding, the use of tests of cognitive abilities in settings different from those in which the test was developed and standardized is still widely criticized. Debate continues about if and how to adjust existing tests, whether existing tests should be re-standardized in the new setting, and whether entirely new tests for a specific context should be created. Concerns are especially pronounced around assessments of cognitive skill and learning that rely on verbal knowledge, reasoning, and expression. In addition to the obvious language differences, skills that are germane in Western and more wealthy nations are sometimes thought to be relatively culture dependent in their emphasis on knowledge content and problem solving strategies with which children in LMIC may have less experience or need [48–51].

To summarize, approaches to testing child learning in LMIC generally fall into one of three categories: (1) use of tools developed and validated in higher income settings, translated for local use, (2) translation of existing tools with adaptation to adjust for local language and customs with the expectation that culturally adjusted measures may capture previously unexamined effects [20, 52], and (3) strategies created in and for the specific context to be studied in an attempt to address factors in learning presumed more salient in that context [47, 51, 53] The latter approach, although it may prove more valid among children in a specific environment, makes findings about child development and functioning across various cultural contexts difficult, and, as a result, does not contribute to understanding of child needs across LMIC that share the similar burden of raising children without their biological parents.

Among the choices for using translations of existing tools, the KABC is a tool that, due to its structure and theoretical base, is seen as relatively more applicable across diverse contexts [44]. The KABC was first developed in the U.S. in 1983 as a measure of intellectual functioning and academic performance and has been used in a number of studies in different cultures [30, 37, 40, 48, 49]. Comparisons of the KABC and the Wechsler Intelligence Scale for Children (WISC DT) indicated that the KABC is the preferred cognitive assessment across varied cultural settings [54]. Jansen and Greenop [55] explored factor loading within the KABC longitudinally, attempting to establish its stability over time; overall support of the KABC’s two-factor structure of sequential and simultaneous processing was found. Although adding important support to the use of the KABC in diverse settings and LMIC, one cannot conclude that the KABC and KABC-II revision are “culture free”. Rather, the KABC may lend itself to being used in more diverse settings rather than other assessments of learning ability and achievement that have higher verbal demands.

The California Verbal Learning Test-Children’s Version (CVLT-C) [56] is a measure of children’s executive functioning that is widely used in neuropsychological assessment as an indicator of strengths and deficits related to brain injury, attention, language abilities, and information processing and recall. In the CVLT-C, children complete five short-term learning trials of a list of items commonly encountered while shopping, followed by the presentation of a different short-term learning list and delayed recall and recognition for the original list. This measure was selected for adaptation for the current study because of its (1) known utility in western settings, (2) its relative ease of administration and limited requirements for complex verbal responses, and (3) the aim of assessing child variability in aspects of executive function, such as attention and motivation.

In response to the need for assessment tools that (a) capture learning and performance in children in LMIC, (b) are sensitive to health and psychosocial factors that influence development, (c) are sufficiently brief to be incorporated into ongoing care and/or program evaluation, and (d) can be used in the context of a more general evaluation of health and living circumstances, our research team employed and examined the usefulness of brief assessments of learning and cognitive performance in a large study of orphaned and abandoned children in five LMIC [57]. This report describes the performance of three of the Nonverbal Index subtests from the KABC-II and an adaptation of the CVLT-C for children across five culturally diverse sites, with the objective of seeing if these test strategies are valid indicators of child individual differences that could be affected by events, such as parent death, other trauma, and other events and processes relevant to the child’s subsequent care and education. This report describes the psychometric findings from these tests across child age, education, gender, and site in order to answer questions about the usefulness of these tools across settings in LMIC.

Methods

Research Design

In a study of the developmental well being of children orphaned and abandoned in six areas of five LMIC (Positive Outcomes for Orphans, POFO) [57], we sought to make use of existing assessment tools to evaluate a series of research questions about the relevance of a child’s living circumstances to overall well-being.

Positive Outcomes for Orphans Study

This report uses cross-sectional baseline data for children age six to 12 years from the longitudinal POFO study to assess the utility and validity of four brief assessments of learning and performance. The POFO study employed a two-stage random sampling survey methodology in six sites across five LMIC to identify a representative sample of community-living and institutionally based OAC ages 6–12. The data collection was conducted between May 2006 and February 2008 and involved periodic on site or digital video review by a senior psychologist of the interview and assessment protocol to ward against drift in administration approaches.

The six sites in the POFO study included two sites in India (Hyderabad and Nagaland) and one each for Tanzania (Moshi), Kenya (Bungoma), Ethiopia (Addis Ababa), and Cambodia (Battambang). The sampling was systematic and random in all six sites, and in five of the sites the sampling represented the entire geographical region. In one site, Hyderabad, the random sampling represented only slum areas; therefore, the Hyderabad sample is not included in these analyses of tests of learning and achievement. The sampling of children was representative of the population of community based OAC in the five sites included in this study.

All measures were translated into the local language and back translated to verify consistency. Ethical approval was provided by the Duke University Institutional Review Board (IRB), the IRBs of Meahto Phum Ko’mah (Bat-tambang, Cambodia), SaveLives Ethiopia (Addis Ababa, Ethiopia), Sharan (Delhi, India), ACE Africa (Bungoma, Kenya), and Kilimanjaro Christian Medical Centre (Moshi, Tanzania), and regulatory agencies in all participating countries: National Ethics Committee for Health Research (Cambodia), Ministry of Science and Technology (Ethiopia), Indian Council of Medical Research (India), Kenya Medical Research Institute (KEMRI), and the National Institute for Medical Research (Tanzania).

Test Procedures

The scores from three nonverbal subtests from the Second Edition of the Kaufman Assessment Battery for Children, 2nd edition (KABC-II) [44] and from the Market List, an abbreviated adaptation of the California Verbal Learning Test-Children’s Version (CVLT-C) [56] are examined for their efficiency in indicating the child’s level of learning and performance. These particular assessments were selected in an effort to mitigate issues about the cultural dependence and transferability of cognitive and learning measures that were developed and used in more wealthy nations and may involve assumptions about the generalizability of verbal knowledge and problem solving. The efficiency of the tests is examined primarily by the relationship between test results and the child’s chronological age, taking into account educational attainment, gender, and residency in each of the five study sites.

The KABC-II, as previously described, is frequently used with more confidence in testing children from a variety of backgrounds and cultural settings [30, 37,40,44, 48, 49]. For the purpose of our study of orphans and abandoned children, three subtests less dependent on language differences were chosen from the KABC-II Nonverbal Index. The first of these, Hand Movements (HM), involves the child copying the examiner on a series of Hand Movements using fist, palm, or side of the hand. HM provides an indicator of the child’s sequential processing, attention, and short term memory through visual-motor abilities. We found the children in our settings to be highly interested and engaged with HM. The second KABC-II subtest, Triangles (TR), requires the child to assemble plastic and foam forms to match a model constructed by the examiner or shown in a picture. The test evaluates the child’s abilities with visual motor integration and spatial reasoning; and as stimulus forms become more complex, they require more advanced reasoning and problem solving. Pattern Reasoning (PR) is the third KABC-II subtest used. In PR, the child is shown a series of pictures in the form of shapes that vary, for example in terms of shading and spatial rotation, and that form a logical pattern with one picture missing. The child must select the best fit from several picture options. PR requires sustained attention, visual problem solving, and memory.

The fourth task, the Market List (ML), represents an adaptation and abbreviation of the California Verbal Learning Test-Children’s Version (CVLT-C) [57], chosen to measure children’s executive functioning (attention and motivation) as well as verbal learning and memory. Requiring children to encode, store, and retrieve information, the CVLT-C measures multiple aspects of verbal learning and memory recall and is widely used in assessment of memory, learning, and executive functioning (e.g., attention, impulsivity, encoding strategies), all of which represent foundation skills for complex learning typically encountered in a formal educational setting. We selected the CVLT-C and chose to adapt it to the international settings because of our observations in pilot work in East Africa that there was variability in children’s engagement with the tests, suggesting that a tool that reflects motivation and attention as well as intellectual ability per se would be an important addition to the learning tasks on the KABC-II. With the assistance of the interviewers in each of the five sites, new lists were developed to reflect what children would see in their local markets. The categories used in the original CVLT-C (things a child would eat, wear, and play with) were used to maximize consistency with the original test (See Market Lists for Cambodia, India, and the three African countries in Supplementary Fig. 1). The current study uses the mean of the first three repetitions of the first (Monday) list for the analyses.

One local male and female interviewer and a lead investigator from each site were trained on the POFO study protocol and procedures, including the tests of learning and performance. The testing was reviewed in the field and by videotape by the first and second authors, child psychologists at Duke University, on several occasions in each site. The POFO study has weekly internet conferencing attended by the first author for inquiries regarding testing. Interviews were conducted at the child’s residence, so the on-site and video supervision was very important for maintaining test fidelity in non-standard situations.

For the KABC-II subtests, the Nonverbal Index standard score could not be derived as we used only three of its five component subtests. Given that KABC-II norms are based on a United States sample and the importance of avoiding the direct comparison of child learning across countries and cultures, raw scores were used to indicate the child’s performance relative to his/her age, as well as other variables that could affect learning, such as highest level achieved in formal education.

Child age was determined by report from the primary caregiver; when the year only was known, the child’s birthday was assumed to have occurred mid-year; when only a month was known, the birthday was assumed to occur mid month. Education was described by the highest grade or level of schooling completed; the range in this cohort was from 0 (no school or Kindergarten or preschool only) to 8. Zero was coded for Kindergarten, preschool, none, or never for the question about highest schooling completed. Grades, levels, or standard 1, 2, 3, and so forth were coded as 1, 2, 3, and so forth. A series of questions about the child’s living circumstances, health, and development across domains were asked of these data; the results are published elsewhere [57, 58].

Data were analyzed using ST ATA 11.1 Data Analysis and Statistical Software [59]. Analysis of variance (ANOVA) was used to evaluate differences in age, education, and test scores among sites. Linear regression models were used to analyze bivariable and multivariable associations of test scores with age, education, and gender. To test the extent to which age, education, and gender acted the same across sites, we also examined their interactions with site. These specifications included main effects for age, education, and gender, indicator variables for each site, and interactions of sites with age, education, and gender. Correlations among test scores were analyzed using Pearson product moment correlation coefficients.

The primary hypothesis tested is: does child performance on the four selected tests of learning improve systematically and across sites with age and education and, thereby, provide an index of the child’s learning ability and achievement relative to his/her age peers.

Results

Description of Participating Children

The analyses describe the variation on the tests of learning in 1,206 children orphaned and/or abandoned and living in community settings for whom complete data were available on the four measures. The children who were tested using the three KABC-II subtests and the Market List are described in Table 1. More boys than girls were tested, but the gender ratio was not statistically different among sites. Child ages ranged from 6.0 to 12.9 years, with no statistically significant differences among sites. The educational levels achieved by the children was significantly different across sites (F = 23.97; P < 0.0001), with the lowest educational levels sampled in Kenya and the highest in Ethiopia.

Table 1.

Description of participating children

	Cambodia	Ethiopia	Kenya	Nagaland	Tanzania	Total
N	236	247	248	226	249	1206
Gender (N, %)
Male	115 (48.73)	131 (53.04)	129 (52.02)	129 (57.08)	129 (51.81)	633 (52.49)
Female	121 (51.27)	116 (46.96)	119 (47.98)	97 (42.92)	120 (48.19)	573 (47.51)
Age (years)
Mean (SD)	9.56 (1.86)	9.15 (1.73)	9.24 (1.91)	9.17 (1.86)	9.54 (1.87)	9.33 (1.85)
Median	9.5	9.3	9.2	9.1	9.8	9.4
Range	[6, 12.9]	[6.2, 12.7]	[6, 12.7]	[6, 12.9]	[6, 12.8]	[6, 12.9]
Education (highest level completed)
Meana (SD)	3.78 (1.73)	4.09 (1.98)	2.68 (1.62)	3.33 (1.86)	3.89 (1.72)	3.54 (1.85)
Median	3	4	2	3	4	3
Range	[1,8]	[1,8]	[0,8]	[1,8]	[1,8]	[0,8]

^aDifferences among sites for education were significant (F = 23.97; P < 0.0001)

Scores on the Tests of Learning and Performance

The raw scores on each of the four tests for the five sites are summarized in Table 2. Possible scores ranged from 0 to 23 for Hand Movements, 0 to 27 for Triangles, 0 to 36 for Pattern Reasoning, and 0 to 15 for the Market List. Using an analysis of variance for the raw scores by site, the differences among sites are statistically significant for each of the four tests. The KABC-II subtest Triangles scores showed the greatest variability across sites (F = 37.69; R² = 0.11; P < 0.0001) with the highest scores found in Cambodia and the lowest in Tanzania.

Table 2.

Scores on the four tests by site

	Cambodia	Ethiopia	Kenya	Nagaland	Tanzania	Total
KABC-II Hand Movements
Meana (SD)	8.22 (3.29)	8.21 (3.04)	8.23 (3.04)	6.84 (2.65)	7.54 (2.52)	7.82 (2.96)
Median	8	8	8	6	7	7
Range	[0, 18]	[0, 20]	[1, 20]	[0, 14]	[3, 14]	[0, 20]
KABC-II Triangles
Meanb (SD)	14.92 (7.39)	11.74 (3.99)	10.94 (4.92)	13.54 (5.18)	9.70 (4.13)	12.11 (5.55)
Median	15	12	10	13	10	12
Range	[0, 30]	[0, 28]	[1, 29]	[2, 29]	[1, 25]	[0, 30]
KABC-II Pattern Reasoning
Meanc (SD)	6.81 (5.97)	7.22 (5.25)	6.19 (5.82)	6.54 (4.66)	5.44 (5.03)	6.43 (5.40)
Median	6	6	4	5	4	5
Range	[0, 38]	[0, 27]	[0, 31]	[2, 34]	[0, 37]	[0, 38]
Market List
Meand (SD)	8.07 (2.16)	7.73 (1.79)	7.54 (2.03)	6.99 (1.97)	7.58 (1.90)	7.60 (2.00)
Median	8.3	7.7	7.5	7	7.3	7.7
Range	[3.3, 12.7]	[3.7, 13]	[3.3, 14]	[3.3, 12.7]	[3.3, 13]	[3.3, 14]

^aSite differences (ANOVA; F = 10.42; Adj. R² = 0.03; P < 0.001)

^bSite differences (ANOVA; F = 37.69; Adj. R² = 0.11; P < 0.001)

^cSite differences (ANOVA; F = 3.89; Adj. R² = 0.01; P < 0.004)

^dSite differences (ANOVA; F = 8.17; Adj. R² = 0.02; P < 0.001)

The correlations among the four tests are shown in Table 3. The correlations among KABC-II subtests in the current study are generally consistent with those of KABC-II subtest scores in the U.S. standardization sample (range 0.36–0.47) selected in test development to be representative of the U.S. population [44]. Correlations between KABC-II subtests and Market List mean scores were lower, as could be expected between measures with varying emphases on verbal abilities.

Table 3.

Correlations among raw test scores

	KABC-II Hand Movements	KABC-II Triangles	KABC-II Pattern Reasoning
KABC-II Hand Movements	1
KABC-II Triangles	0.44a	1
KABC-II Pattern  Reasoning	0.44a	0.54a	1
Market List	0.34a	0.35a	0.36a

^aPearson product-moment correlation for all comparisons P < 0.0001

Do Child Age, Education, and Gender Predict Learning and Performance Scores Across Sites?

In bivariable regression analyses of test scores with child age (see Table 4, Panel 1), the results indicate that age is significantly and positively associated with raw scores on all four tests (r² 0.11–0.18; all P < 0.001.) For each year of age, the child’s performance increases by approximately 1/2 point on Hand Movements and the Market List and by approximately 1 point on Triangles and Pattern Reasoning. The child’s highest level of education is also associated with the scores on all four tests (r² 0.15–0.29; all P < 0.001.). For each level of education attained by the child, the scores also increase by approximately 1/2 point for Hand Movements and Market list and 1 point for Triangles and Pattern Reasoning. The bivariable analyses also indicate gender differences on the Triangles subtest, with boys scoring higher than girls, (r² = 0.01; P < 0.001) and on the Market List, with girls scoring higher than boys (r² = 0.01; P < 0.05.)

Table 4.

Associations of test scores with age, education, and gender

	Panel 1			Panel 2		Panel 3
	Bivariable associationsa			Multivariable associationsb		F-test: joint significance of interactionsc
	Coeff	CI	R 2	Coeff	CI
KABC-II Hand Movements
Age (years)	0.53	[0.45, 0.61]***	0.11	0.05	[−0.08, 2.18]	0.35
Education	0.61	[0.52, 7.70]***	0.15	0.61	[0.46, 9.75]***	0.58
Gender (F)	−0.27	[−0.60, 4.07]	0.00	−0.23	[−0.52, 8.08]	1.45
Site fixed effects		F = 11.30***	0.03		F = 17.79***	2.54*
KABC-II Triangles
Age (years)	1.16	[0.99, 1.32]***	0.15	0.07	[−0.15, 7.30]	0.26
Education	1.40	[1.24, 1.57]***	0.22	1.43	[1.19, 1.66]***	9.66***
Gender (F)	−1.21	[−1.83, −2.59]***	0.01	−1.04	[−1.55, −1.53]***	2.38*
Site fixed effects		F = 34.19***	0.11		F = 57.92***	7.28***
KABC-II Pattern Reasoning
Age (years)	1.23	[1.08, 1.38]***	0.18	0.03	[−0.17, 1.22]	1.21
Education	1.59	[1.42, 1.76]***	0.29	1.67	[1.42, 1.92]***	6.86***
Gender (F)	−0.35	[−0.95, 3.26]	0.00	−0.21	[−0.72, 6.31]	0.74
Site fixed effects		F = 4.18**	0.01		F = 10.75***	5.29***
Market List
Age (years)	0.38	[0.32, 0.44]***	0.12	0.11	[0.01, 1.29]*	1.05
Education	0.42	[0.36, 0.48]***	0.15	0.35	[0.26, 1.44]***	2.33
Gender (F)	0.29	[0.05, 3.52]*	0.01	0.28	[0.06, 2.49]*	0.14
Site fixed effects		F = 7.67***	0.03		F = 9.79***	2.04

^aBivariable regressions predicting each test result using age, education, gender, and site effects; *P < 0.05; **P < 0.01; ***P < 0.001

^bMultivariable regressions predicting each test result using age, education, gender, and site effects; *P < 0.05; **P < 0.01; ***P < 0.001

^cF-test of the joint significance of site fixed effects and their interactions with age, gender, and education, controlling for variables in Panel 2

In order to understand whether there are site differences in the test scores, we (1) tested site differences in mean scores after adjusting for age, gender, and educational level, and (2) examined whether the slopes for age, gender, and education were similar across sites. In multivariable regression models (See Table 4, Panel 2), educational level remained significantly associated with each outcome {t(HM) = 8.4, P < 0.001; t(TR) = 11.86, P < 0.001; t(PR) = 13.2, P < 0.001; t(ML) = 7.54, P < 0.001}. Chronological age was not associated with any of the scores in the multivariable analyses with the exception of the Market List (t = 2.21; P < 0.05.). Significant gender differences remained for the Triangles subtest {t = −4.00, P < 0.001} and the Market List {t = 2.53; P < 0.05.) In the multivariable models that include all the interactions (see Table 4, Panel 3), there remained site differences in mean scores for Hand Movements (F = 2.54; P < 0.05), Triangles (F = 7.28; P < 0.001), and Pattern Reasoning (F = 5.29; P < 0.001) but not for the Market List. For Triangles and Pattern Reasoning, the association with test scores varied by site (F = 9.66 and F = 6.86, respectively; both P < 0.001). For Triangles, the associations with gender also differed by site (F = 2.38; P < 0.05).

Given the remaining site differences in test scores after adjusted for the predicted factors, we explored post hoc whether the observed effect of education on test scores could be explained by other variation between sites. We tested the relationship between several factors that may differ among sites and influence child performance: whether the child was a single or double orphan, if the guardian was the biological mother, if the guardian worked 40 or more hours/week out of the household, whether the family was engaged in farming, if the household was in a rural or urban setting, a wealth index derived from household characteristics, and whether the guardian was elderly.

Household wealth was assessed by using the World Bank’s Child Needs Assessment (CNA) Toolkit [60] and an asset checklist administered to all participating caregivers at the time of the child assessment. Elements of the asset checklist and the CNA that were comparable to items used by the Demographic Health Survey (DHS) in the construction of the DHS Wealth Index [61] were used to construct a comparable wealth index for each participating household. The age of the guardian was compared to mean life expectancy by gender for each country, and guardians older than the average were considered “elderly” for their setting. The findings from these sensitivity analyses suggested that although all of these factors varied across the five sites, they did not alter the observed significant association between education and performance on the four tests.

Discussion

The performance of orphaned and abandoned children ages 6–12 from five LMIC on the brief tests of learning was evaluated here to assess the extent to which these tests, developed elsewhere and translated, and in once case adapted for local use, provide useful information about child learning and performance. This report is seen as important because of the number of investigations about vulnerable children in LMIC settings that seek to examine cognitive development or learning as child outcomes, often attempting to do so within the constraints of having no or few tests developed for or thought to be valid in that context.

The testing approach used here is a relatively brief one (approximately 20 minutes), a length necessitated by its inclusion in a much longer interview. We found that the testing could be done in a household setting, often with limited materials, e.g., on a bench or floor instead of table and chairs. Children and their caregivers enjoyed the testing; and children appeared eager to do well, and they showed pride in their achievements.

The results suggest that these tests of learning can be done in LMIC in non-standard settings and that they are, as predicted, associated with child age and even more strongly with the child’s formal educational experience despite the diversity among sites. Unlike more wealthy nations that have a very high association between age and grade level in school, the five low and middle income countries in this report have greater variability in the relationship between age and level of formal education; (in these five LMIC, correlations range from 0.6807 (Cambodia) to 0.8202 (Tanzania). Many children attend school inconsistently within and across years and enter school at different ages, depending on family circumstances. The latter is particularly true for OAC who may be less likely to attend school in order to perform domestic chores or work outside the home to help support the family [58].

It is concluded that this type of brief assessment, based on tests derived in countries with higher socioeconomic status, is applicable for understanding child individual differences in contexts in which learning is a function of not only age and innate abilities but also of highly variable opportunities for learning. The results indicate that these subtests generally act the same as in the countries in which the tests were developed; that is, the raw scores increase systematically with child age. Significant interactions were present between educational level and site as test score predictors, indicating that the effect of educational level varies across the five sites, a finding that remained significant after statistically controlling for child site, age, and gender.

Of note, the scores for the subtest Pattern Reasoning were on average lowest in Tanzania where the average educational level was highest. We observed that many children across sites struggled to understand the PR task, despite the many opportunities offered by the administration directions for teaching the task. It is possible that PR taps a type of visual problem solving skill less relevant to the Tanzanian education system, but it is not possible to draw conclusions about these differences from the data. Also, gender differences for the KABC-II subtest Triangles and the Market List remained in the multivariable analyses. The five sites were queried about these findings, and both African and Asian interviewers indicated that the results were likely related to the different roles of boys and girls in their settings e.g., “Boys have more time for playing, innovation, and creativity than girls who help in household chores”, “Girls’ play is more geared toward “cha mama”, that is, imitating the mother and her duties, like cooking, buying things at the market, and running the household.”

The strong and consistent association of years of education to test performance raises additional questions about what specifically is the contribution of formal education to child development and why formal education could be relatively more important to performance than chronological age alone. Of course, it is possible that there is a bias for which children attend school and stay in school longer; that is, it is possible that children seen as brighter are more likely to attend school and to stay in school longer?

It is worth considering whether the controversial question of “what do IQ tests measure?” could be even more relevant for diverse cultures than in relatively more wealthy countries in which education is fairly uniform among all children at specific ages. Given that the child’s experience in formal education in these five countries makes a large contribution to the child’s performance on the tests, is formal education increasing intellectual ability or is there other learning in school that is associated with child performance? In other words, tests drawn or derived from standard IQ tests may not always index static cognitive capacity but also non-cognitive learning skills and behavior that are strongly influenced by educational opportunities. Of course, within any IQ tests there is considerable variability among subtests and items as to the extent that they measure mutable or invariant characteristics.

One hypothesis about these findings comes from studies of preschool programs in the United States in which Heckman and colleagues [62–64] conclude that formal education contributes to non-cognitive skills as well as to cognitive abilities per se and that non-cognitive abilities learned in formal education can contribute even more to successful lives in school and in adulthood. The studies suggest that while cognitive abilities (i.e. IQ) are important to success as an adult, so also are “non-cognitive” skills such as emotional adjustment, physical and mental health, attention, motivation, and self confidence. These skills develop during the life cycle in a social context as noted by others [e.g., 65, 66] but, most importantly, during early formal education. For example, in the U.S., early educational intervention programs do not appear to raise IQ per se; they are found to be associated with adolescents and young adults being employed, having higher salaries, not engaging in health risks like smoking, and exhibiting less criminal behavior [62–64]. The argument is made that innate cognitive ability is only one aspect of child developmental outcomes and that these so-called “non-cognitive” abilities are also developed in the family and in school where the child learns to attend, to be motivated to achieve, and to play by the rules of the setting.

This possible explanation of our findings of the relative importance of formal education to test performance may or may not apply to the social dynamics in diverse LMIC settings, but it is worth considering not only by researchers but also by program and policy makers in determining priorities for funding development and support. If applicable to LMIC settings, it follows that the child who performs on these tests, with items never before encountered in their context, with enthusiasm, motivation, and attention is a child who not only has the intellectual capacity but also who learned these (non-cognitive) skills in school and may be more likely to use these skills to negotiate a more successful adolescence and adulthood.

In summary, the valid assessment of learning in diverse settings is challenged by the complexity of child development, the relevance of cultural differences in child performance, in variable formal education, and in adult priorities for child learning, as well as the lack of tests developed for and in the context to be studied. This report offers a relatively brief approach for testing that is associated with child age and strongly also with variability in exposure to formal education. The investigators discuss the findings in terms of the relevance of educational experiences to the child’s developing cognitive abilities and also, it is speculated, to non-cognitive skills such as attention and motivation that predict not only performance in the present but also successful maturation into adulthood.

Limitations

This study represents a venture toward one approach to the assessment of vulnerable children’s learning and performance in LMIC as part of a larger study, and it is subject to a number of limitations. There exists no gold standard assessment of child learning in these contexts, which limits conclusions about validity of our test protocol but also argues for the need to do and report these kinds of studies. These data are cross-sectional in nature, and we cannot infer causality or exclude other possible influences on scores, such as administration variability or quality of schooling. Variability in scores across the five sites was higher than would be expected in more wealthy contexts, suggesting the large range of other factors that vary among cultural contexts and are likely associated with child abilities and learning skills. Indeed, formal education itself is highly variable in consistency, quality, and attendance within and across the five nations in the study. The conclusion that these tests behave the same in these settings as in the settings in which they were developed does not justify the conclusion that the results necessarily mean the same.

Of note, the KABC-II subtests used require training and periodic quality assurance as well as materials not necessarily available in LMIC. This could be a disadvantage of our approach. On the other hand, the adapted CVLT may provide an easy-to-administer indicator of not only verbal memory but also the child’s motivation and attention, that is, factors that may improve with interventions that address a number of factors, such as nutrition, psychosocial well being, and access to education. The use of the Market List should be subjected to further psychometric exploration in these settings. With forthcoming longitudinal data, the results from the four tests used in the POFO study [57] may prove effective in understanding orphaned and abandoned children even better over time.

Appendix.

The POFO Research Team consists of:

Chris Bernard Agala

Frehiwot Alebachew

Sisay W. Ameya

Robin Briggs

Sopheak Chan

Haimanot Diro

Belaynesh Engidawork

Dafrosa Itemba

Becky Kinoti

Rajeswara Rao Konjarla

Mao Lang

Dean Lewis

Ira Madan

Cyrilla Manya

Restituta Mvungi

Laura K. Murphy McMillan

Imliyanger Pongen

Pelevinuo Rai

John Shao

Neville Selhore

Amani Sizya

Vanroth Vann

Augustine Wasonga

Gopala Krishna Kaza