Predictive validity of a computerized battery for identifying neurocognitive impairments among children living with HIV in Botswana

Abstract

Children living with HIV (HIV+) experience increased risk of neurocognitive deficits, but standardized cognitive testing is limited in low-resource, high-prevalence settings. The Penn Computerized Neurocognitive Battery (PennCNB) was adapted for use in Botswana. This study evaluated the criterion validity of a locally adapted version of the PennCNB among a cohort of HIV+ individuals aged 10–17 years in Botswana. Participants completed the PennCNB and a comprehensive professional consensus assessment consisting of pencil-and-paper psychological assessments, clinical interview, and review of academic performance. Seventy-two participants were classified as cases (i.e., with cognitive impairment; N=48) or controls (i.e., without cognitive impairment; N=24). Sensitivity, specificity, positive predictive value, negative predictive value, and the area under receiver operating characteristic curves were calculated. Discrimination was acceptable, and prediction improved as the threshold for PennCNB impairment was less conservative. This research contributes to the validation of the PennCNB for use among children affected by HIV in Botswana.

Introduction

The human immunodeficiency virus (HIV) is a leading cause of morbidity among children in Sub-Saharan Africa (1). In Botswana, approximately 10,000–15,000 individuals age 0–14 years are living with HIV (HIV+) (2). Due to the availability of free antiretroviral therapy (ART) in Botswana, many children who were perinatally infected with HIV have survived to live relatively healthy lives (3). However, children living with HIV experience increased risk of neurocognitive deficits, specifically in the domains of attention, episodic memory, executive functioning, information processing speed, and psychomotor functioning (4–7). These deficits are multifactorial. It has been hypothesized that some of the increased risk of impairment is due to in utero toxicities (8), such as HIV replication in specific brain cells, ART toxicity, and opportunistic infections, resulting in direct damage to the central nervous system (9–11). Early life nutritional and social deprivation, late access to treatment, and greater HIV disease severity can exacerbate the deficits (4, 12, 13). Studies show that initiation of ART in HIV+ children can cease progression of neurocognitive dysfunction, but ART cannot reverse dysfunction (14, 15). Thus, to support the functional and educational attainment of HIV+ children and adolescents, early detection and treatment of neurocognitive impairments is critical.

Due to the lack of comprehensive neurocognitive assessments in Botswana, the Penn Computerized Neurocognitive Battery (PennCNB) was culturally and linguistically adapted for use in this setting (16). Composed of “neurobehavioral probes” (17) validated by functional neuroimaging (18), the PennCNB measures performance accuracy and response speed for major neurocognitive domains (19, 20). The neurobehavioral probes include adaptations of common “pencil and paper” tests traditionally administered by neuropsychologists (e.g., Trailmaking tests), as well as tests that can be performed more precisely via computer (e.g., finger tapping). Importantly, the PennCNB has the potential to streamline neurocognitive assessment, which presents many advantages for implementation (especially in resource-limited settings), such as increased standardization and reliability, ease of administration, and automated scoring and database generation. The version of the PennCNB adapted for use in Botswana includes 13 tests selected a priori based on the cognitive domains commonly impaired among children with HIV infection and in utero HIV or antiretroviral therapy exposure (6, 7, 16, 21). Through a rigorous cultural and language adaptation process informed by WHO guidelines, the battery was translated into Setswana (local language) and locally appropriate English (16, 22, 23). Variations of the tests composing the adapted PennCNB have been used extensively to identify neurocognitive deficits among pediatric populations (19, 24–26).

Previous research examined the structural validity of the adapted PennCNB among a cohort of HIV+ and HIV-exposed-uninfected (HEU) children in Botswana (27), providing evidence of the convergent and discriminant validity of the tool. However, establishing the validity of the battery on the population level differs from demonstrating the extent to which the PennCNB correctly classifies neurocognitive deficits on an individual level. Thus, understanding the extent to which the adapted PennCNB accurately identifies children with clinically significant neurocognitive impairment and children without clinically significant impairment is a critical next step in the validation of the tool.

Assessing the criterion validity (e.g., agreement with a gold standard assessment) provides an estimate of a battery’s ability to predict outcomes, and these outcomes (criteria) are often established measures that have well-described theoretical links to the trait measured by the scores under scrutiny. However, in Botswana and similar resource-limited settings, a rigorous gold standard for cognitive screening is not available. Therefore, this research aimed to evaluate the predictive validity of the adapted PennCNB in Botswana utilizing the best, most feasible data available and expert consensus to constitute a “gold standard” evaluation.

Methods

This sub-study evaluating criterion validity is part of a larger research project that is described in detail elsewhere (16). All procedures were approved by the Institutional Review Boards at the Health Research and Development Committee within the Ministry of Health and Wellness of Botswana, the Botswana-Baylor Children’s Clinical Centre of Excellence (CoE), and the University of Pennsylvania.

Participants

HIV+ children aged 10–17 years enrolled for care at the CoE, a facility providing pediatric HIV care and treatment in Gaborone, Botswana, were recruited for participation. Patients eligible for inclusion were proficient in English or Setswana and did not have severe physical impairments or profound developmental delays that would prohibit completion of the PennCNB assessment. The majority of participants for this sub-study were randomly selected from the larger study cohort. Children aged 7–9 years were ineligible for this sub-study due to the data required for the professional consensus classifications described below (i.e., the local clinical psychologist does not administer the assessments to children under 10 years due to concerns about validity in this age range). To ensure inclusion of enough participants with neurocognitive impairment, the randomly selected participants were supplemented by purposively recruiting patients enrolled for care at the CoE who were previously identified as having cognitive impairment by the clinic psychologist. “Cases” were defined as individuals with clinically significant neurocognitive impairment that impacted their functional behavior, and “controls” were defined as children without clinically significant neurocognitive impairment. A detailed description of the methodology used to classify each participant is given in the next section. Informed consent and assent were obtained prior to data collection, and demographic data and health information were reported by participants’ caregivers.

Professional Consensus Classification

Due to the limited validated cognitive assessments in Botswana available to serve as a gold standard test, the team completed a multi-step, team-based approach to systematically evaluate the neurocognitive function of each possible participant. First, a local, experienced clinical psychologist completed a comprehensive assessment that included a local standard-of-care cognitive assessment and added supplementary interview components. The local assessment consisted of the following: 1) the Montreal Cognitive Assessment (MoCA) (28), a 30-question cognitive screener that assesses cognitive ability in eight domains (orientation, short-term memory, executive function, language abilities, abstraction, attention, animal naming, and clock-drawing test); 2) the Draw-a-Person Test (29), an assessment that measures cognitive maturity and mental age; 3) an intake interview that elicited information about the child’s current school performance, educational and developmental history, mental health, possible diagnosis of dyslexia, history of school performance for other siblings, and caregiver perceptions of intelligence and comprehension; and 4) a review of school records and history of academic performance. The MoCA is not typically administered to young children, but the local psychologist uses this assessment in practice because it is culturally acceptable and effective as a screening tool for cognitive impairments. Suggested adjustments to the cut-off score for determining impairment in younger and less-educated populations were applied to the MoCA scores (i.e., adding one point for individuals with less than 12 years of education), and impairment was classified by a MoCA score less than 26 (28). In Botswana, primary school starts with Standard 1 (equivalent to 1^st grade in the United States) and ends with Form 5 (equivalent to 12^th grade) (30). Upon review of the data, the clinical psychologist preliminarily classified each potential participant as a case or control and determined impairment in three cognitive domains (executive functioning, episodic memory, and sensorimotor function). Prior to inclusion in the final cohort, the preliminary classifications and corresponding data were discussed by a group of five clinicians with local and international expertise in neuropsychology, psychology, pediatrics, and HIV to ensure consistency of assessments and determine assignment of case/control status and impairment in each of the subdomains. The team was blinded to PennCNB performance when classifying participants. To be considered a “case”, a participant needed to demonstrate evidence of abnormalities on the clinical psychologist’s assessments and some functionally limiting impairment at school and/or home. When the group lacked consensus (i.e., the group could not confidently classify a patient according to the data), individuals were considered “unclassifiable” and excluded from the final cohort (N = 27). For example, children with average grades in school who had impairments on the assessments for the professional consensus classification were not utilized as controls, because controls were meant to represent typically developing children. Likewise, children with average scores on the formal assessments (e.g., MoCA) who were failing at school were considered “unclassifiable”. Thus, classification was conservative with identifying cases/controls, so the final cohort represents “extremes” of impairment/no impairment.

PennCNB Administration

Table 1 lists the 13 tests composing the PennCNB adapted for use in Botswana and the cognitive domains measured by each module. The tests have previously been described in detail (16, 20, 24). Briefly, the battery measured executive functioning (i.e., working memory, cognitive flexibility, sustained attention, and inhibition) (Fractal N-Back Test, Penn Continuous Performance Test, Penn Go/No-Go, and Penn Trailmaking Test Part B), episodic memory (i.e., ability to recall specific events or experiences) (Penn Face Memory Test, Visual Object Learning Test, and Digit Symbol Substitution Test recall), complex cognition (i.e., abstraction, reasoning, and spatial processing) (Penn Conditional Exclusion Test, Penn Line Orientation Test, and Penn Matrix Reasoning Test), and sensorimotor/processing speed (i.e., ability to rapidly coordinate visual motor movements or complete cognitive tasks) (Finger Tapping Test, Digit Symbol Substitution Test, Motor Praxis Test, and Penn Trailmaking Test Part A).

Table 1.

Penn Computerized Neurocognitive Battery Tests Administered

Cognitive Domain	Test	Description
Executive Functioning	Fractal N-Back Test (FNB)	Presents series of fractal images at 1 Hz and requires response either to pre-specified target fractal (0-back), whenever fractal repeats preceding fractal (1-back), or whenever fractal is same as fractal presented 2 images before (2-back).
	Penn Continuous Performance Test (PCPT)	Press space bar when a 7-segment display forms a number (1st half of test) or a letter (2nd half of test) and not press for non-numbers or non-letters.
	Penn Go/No-Go (GNG)	Press space bar when letter “x” appears in upper half of screen with y’s and lower-half x’s as distractors.
	Penn Trailmaking Test, Part B (TMTB)	Connect dots on screen in alternating numerical and alphabetical order.
Episodic Memory	Digit Symbol Substitution Test, recall (DSSTr)	Recall portion of DSST testing which symbols were paired with which numbers.
	Penn Face Memory Test (FMEM)	Presents series of 20 faces and requires selection of previously-presented faces from mixed set of images containing distractors.
	Visual Object Learning Test (VOLT)	Presents series of 3D Euclidean shapes to determine, from mixed targets and distractors, whether shape appeared previously.
Complex Cognition	Penn Conditional Exclusion Test (PCET)	Determine which of 4 shapes does not belong in group.
	Penn Line Orientation Test (PLOT)	2 line segments appear on screen and require rotation of 1 segment until lines become parallel.
	Penn Matrix Reasoning Test (PMAT)	Presents series of geometric shapes and requires selection of shape that completes a pattern.
Sensorimotor/Processing Speed	Finger Tapping Test (CTAP)	Press space bar with index finger as quickly as possible for 10 seconds for 5 trials with alternating hands.
	Digit Symbol Substitution Test (DSST)	9 symbol-digit pairs serve as a reference set and require indication whether digit-symbol pairs presented match reference.
	Motor Praxis Test (MPT)	Use computer mouse to click on a green square that appears and disappears in different places on the screen and gets increasingly small.
	Penn Trailmaking Test, Part A (TMTA)	Connect dots in sequential order.

The PennCNB was administered on a laptop by a trained member of the research team in a private space in the CoE. Individuals facilitating the PennCNB assessments were blinded to the participants’ status as cases or controls. Standard operating procedures for administration were followed, which included reading the PennCNB instructions aloud to each participant and providing practice PennCNB modules to ensure task comprehension before beginning the actual assessment. Participants had the option to complete the Setswana version of the PennCNB or the Botswana-adapted English version of the battery. The tests were administered in a fixed order.

Statistical Analysis

Determination of PennCNB Classification

The PennCNB yields accuracy (total or percent correct) and speed (mean response time) values for each test. Neurocognitive efficiency scores were calculated as the average of the z-standardized accuracy and speed (Response Time multiplied by −1 so that higher is faster) scores (19). To determine impairment/no impairment from the PennCNB data, composite scores were calculated from the efficiency data, which were then age-adjusted and z-standardized. Composite scores were calculated by averaging the age-regressed efficiency data for overall performance (i.e., all of the PennCNB tests) and each of the specific domains (i.e., only the PennCNB tests that measured the specific cognitive domain) for each participant. This approach was used because generating composite scores will be more feasible for the implementation of the PennCNB in resource-limited settings compared to more sophisticated psychometric approaches such as calculating and interpreting factor scores. As mentioned above, the participants recruited for this study were a subset of a larger cohort of HIV+ and HIV-exposed-uninfected (HEU) children (N = 306) at the CoE who had completed the PennCNB assessment (16). To create z-standardized scores, PennCNB scores for the classified cases and controls were compared to the distribution of scores from this larger sample of 306 children. Standard clinical cut-off values for z-scores were used to enable confident determination of whether a participant fell outside of the normal range (i.e., 2 SD, p = 0.05) (31). For the primary analysis, individuals with composite scores −1.5 SD from the global mean or −2 SD from the mean on any of the specific domains (executive functioning, episodic memory, or sensorimotor speed) were considered overall PennCNB-impaired; all other participants were considered overall PennCNB-unimpaired. To classify impairment in the specific cognitive domains, participants with scores −1.5 SD from the mean were considered impaired in the respective domain. To account for the potential enrichment of cognitive deficits in the sample due to the inclusion of only HIV+ and HEU participants, sensitivity analyses that evaluated different thresholds for determining impairment (−1.5 SD, −1.0 SD, −0.5 SD, and 0 SD) were also performed.

Classification Accuracy

To examine the predictive validity of the PennCNB, PennCNB efficiency scores were used to “predict” the professional consensus classifications. The primary analysis assessed overall impairment by evaluating the PennCNB’s ability to distinguish between children with clinically significant impairment and children without clinically significant impairment (i.e., agreement between case/control classifications). A secondary analysis explored the PennCNB’s agreement with the gold standard within specific domains of neurocognitive impairment (executive functioning, episodic memory, and sensorimotor function). PennCNB tests measuring complex cognition were included in the calculation of overall impairment, but this domain was excluded in the secondary analysis due to an inability to confidently classify impairment in complex cognition in the professional consensus assessment. Domains considered unclassifiable according to the professional consensus assessment for an individual child were considered missing for the secondary analyses. Sensitivity (i.e., true positive rate), specificity (i.e., true negative rate), positive predictive value (PPV; proportion of positive calls that are correct), negative predictive value (NPV; proportion of negative calls that are correct), and the area under receiver operating characteristic curves (AUC) were calculated to assess the ability of the PennCNB to identify cases and controls (32–34). The utility of the classification was determined based on standard values, where 1 indicated perfect prediction (AUC of 0.5 = no discrimination, AUC of 0.7–0.8 = acceptable discrimination) (35). Cross-validation procedures were unnecessary because the classification rules (prediction model) were determined a priori (i.e., there was no prediction model to build). All analyses and plots were completed in R (v3.3.3; R Core Team, 2017) and Stata Statistical Software (15.1; College Station, 2017).

Results

Participant Characteristics

Seventy-two participants (48 cases and 24 controls) were enrolled (Table 2). Baseline WHO clinical staging system for HIV/AIDS (i.e., most severe classification before highly active antiretroviral therapy) was higher in the children with neurocognitive impairments. However, at the time of enrollment into the study, all participants were stage 1 on the WHO clinical staging system (i.e., asymptomatic) and had an undetectable viral load (i.e., < 400 copies/mL) except for one control who had a viral load of 1,455 copies/mL. The mean age of cases (13.69 years) and controls (13.33 years) was approximately equal. All individuals identified as Black African. Overall, sex distribution was approximately equal. Cases included more males (58.33%), and controls included more females (62.50%). There were no statistically significant differences in the highest level of education achieved between cases and controls (Pearson’s chi-square = 10.4, p-value = 0.321). Most participants (N=65 (90.28%)) completed the Setswana version of the PennCNB rather than the English version of the tool (cases: 100% Setswana; controls: 71% Setswana).

Table 2.

Participant Characteristics

	Total N (%)	Cases N (%)	Controls N (%)
Total	72	48	24
Mean age (years)	13.57	13.69	13.33
Sex
Female	35 (48.61)	20 (41.67)	15 (62.50)
Male	37 (51.39)	28 (58.33)	9 (37.50)
Education
None	2 (9.72)	2 (4.17)	0
Standard 1	0	0	0
Standard 2	0	0	0
Standard 3	6 (8.33)	6 (12.50)	0
Standard 4	8 (11.11)	5 (10.42)	3 (12.50)
Standard 5	15 (20.83)	8 (16.67)	7 (29.17)
Standard 6	7 (9.72)	4 (8.33)	3 (12.50)
Standard 7	9 (12.50)	5 (10.42)	4 (16.67)
Form 1	7 (9.72)	5 (10.42)	2 (8.33)
Form 2	7 (9.72)	7 (14.58)	0
Form 3	9 (12.50)	5 (10.42)	4 (16.67)
Form 4	2 (2.78)	1 (2.08)	1 (4.17)
Form 5	0	0	0

Prevalence of Neurocognitive Impairment among Participants

Table 3 displays the number of participants classified as having neurocognitive impairment overall and within the specific neurocognitive domains for each of the classification approaches. Based on the professional consensus procedures, 48 participants were considered cases overall, with 31 demonstrating deficits in executive functioning (13 unclassifiable), 40 in episodic memory (6 unclassifiable), and 3 in sensorimotor speed (17 unclassifiable). The mean MoCA score among all participants was 20 (range: 7–30), and the mean score among cases and controls was 13 (range: 7–25) and 27 (range: 25–30), respectively. When applying the −1.5 SD cut-off to the PennCNB scores, results yielded 13 overall PennCNB-impaired and 59 overall PennCNB-unimpaired. As expected, the number of participants determined to have impairment overall increased as the cut-off became less conservative: −1.0 SD (23 PennCNB-impaired, 49 PennCNB-unimpaired), −0.5 SD (30 PennCNB-impaired, 42 PennCNB-unimpaired), and 0 SD (41 PennCNB-impaired, 31 PennCNB-unimpaired). When examining the specific neurocognitive domains, the PennCNB data did not capture all of the deficits in episodic memory as identified through the professional consensus approach (−1.5 SD: 7, −1.0 SD: 18, −0.5 SD: 31, 0 SD: 38), but the number of participants considered to have impairment in executive functioning (−1.5 SD: 11, −1.0 SD: 20, −0.5 SD: 27, 0 SD: 39) or sensorimotor speed (−1.5 SD: 7, −1.0 SD: 14, −0.5 SD: 23, 0 SD: 36) eventually surpassed the professional consensus counts.

Table 3.

Frequency of Participants with Neurocognitive Impairment by Domain and Classification Approach

	Cognitive Domain
Classification Approach	Overall	Executive Functioning	Episodic Memory	Sensorimotor/Processing Speed
Gold Standard*	48	31	40	3
−1.5 SD	13	11	7	7
−1.0 SD	23	20	18	14
−0.5 SD	30	27	31	23
0 SD	41	39	38	36

^*Some participants were considered unclassifiable in the specific neurocognitive domains

Classification Accuracy of Overall Impairment

Table 4 presents the results for the diagnostic accuracy of the PennCNB across the range of cut-off points. The sensitivity increased as the cut-off for determining neurocognitive impairment approached the mean (−1.5 SD: 0.2292, −1.0 SD: 0.4375, −0.5 SD: 0.5833, 0 SD: 0.7292), while the specificity remained constant at 0.9167 until examining the 0 SD cut-off (0.7500). Overall, the probability of identifying impairment among participants with a neurocognitive deficit was high (positive predictive value > 0.846 for all thresholds). Based on the 0.70 threshold, overall predictive utility of the battery became acceptable at the −0.5 SD (AUC = 0.7500) and 0 SD cut-off points (AUC = 0.7396) (Figure 1).

Table 4.

Descriptive Classification Accuracy

	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	AUC
-1.5 SD
Overall impairment	22.92	91.67	84.6	54.6	0.5729
Executive functioning	29.03	96.43	90.0	55.1	0.6273
Episodic memory	15.00	96.15	85.7	42.4	0.5558
Sensorimotor/processing speed	33.33	92.31	20.0	96.0	0.6282
-1.0 SD
Overall impairment	43.75	91.67	91.3	44.9	0.6771
Executive functioning	41.94	89.29	81.3	58.1	0.6561
Episodic memory	37.50	88.46	83.3	47.9	0.6298
Sensorimotor/processing speed	33.33	82.69	10.0	95.6	0.5801
-0.5 SD
Overall impairment	58.33	91.67	93.3	52.4	0.7500
Executive functioning	58.06	85.71	81.8	64.9	0.7189
Episodic memory	57.50	84.62	85.2	56.4	0.7106
Sensorimotor/processing speed	66.67	78.85	15.4	97.6	0.7276
0 SD
Overall impairment	72.92	75.00	85.4	58.1	0.7396
Executive functioning	77.42	71.43	75.0	74.1	0.7442
Episodic memory	65.00	73.08	78.8	57.6	0.6904
Sensorimotor/processing speed	100.00	59.62	12.5	100.0	0.7981

Figure 1.

Comparison of Receiver Operating Characteristic Curves for Overall Impairment

Classification Accuracy of Specific Neurocognitive Domains

The accuracy of the classifications varied across the specific neurocognitive domains (Table 4). Overall, the specificity was high for each of the domains. The positive predictive value for sensorimotor speed was poor (PPV < 0.200 for all thresholds). Similar to the primary analysis, the models demonstrated acceptable discrimination at the −0.5 SD and 0 SD cut-offs for executive functioning and sensorimotor speed, but the prediction for episodic memory was borderline unacceptable (AUC = 0.6904) at the least conservative threshold.

Discussion

Children and adolescents living with HIV experience increased risk of neurocognitive deficits, but standardized neurocognitive screening is not available in low-resource, high-prevalence settings in Sub-Saharan Africa. This study aimed to assess the criterion validity and predictive/classification utility of the PennCNB adapted for use in Botswana against a professional consensus classification consisting of a multi-step, team-based approach utilizing the best available local psychological assessment tools. Results provided insight into the criterion validity of the adapted tool, as well as the range of classification accuracy/utility that can be expected from the tool at various classification cut-offs.

Overall, the adapted PennCNB demonstrated acceptable discrimination, indicating that the battery accurately identified impairment and had reasonable agreement with locally available best practices at the individual child level. These findings provide evidence of criterion validity while encouraging additional inquiry. Further, this evaluation of extreme groups of impairment/no impairment helps inform future investigations. If the adapted PennCNB were not sensitive to differences in this design, then findings would suggest redirecting scarce monetary resources toward alternative screening strategies for this setting. Using norms generated from this HIV+ and HEU cohort, results suggested that −0.5 SD may be an optimal cut-off for interpretation of the PennCNB data. However, since norms for some subtests could be skewed in this cohort, future work will establish norms in HIV-unexposed and uninfected (HUU) children in Botswana to assess differences in classification accuracy. Moreover, normative values established in HUU children will assist with more generalized application of the PennCNB in clinical practice. Sensitivity analyses were performed to assess different cut-offs and account for this potentially enriched sample. When evaluating cut-offs, there are tradeoffs between the sensitivity and specificity of the tool. Determination of the optimal point should aim to maximize both sensitivity and specificity, but decisions need to consider the potential negative outcomes of false positives (e.g., misidentifying children as having neurocognitive deficits, which can result in stigma) and false negatives (e.g., missing children in need of cognitive support).

The low sensitivity and poor prediction at more conservative cut-offs were likely a statistical byproduct of having a sample highly enriched for cognitive impairment, such that no conservative cut-off could work given the number of positive cases. The low sensitivity may also be attributable to unmeasured factors that impact children’s cognitive function. For example, the Botswana-adapted versions of the PennCNB did not include specific linguistic assessments due to the relative difficulty of creating comparable linguistic assessments across languages and cultures. The linguistic assessments on the MoCA are limited, and the professional consensus classification did not attempt to diagnose specific language impairments. Nevertheless, linguistic deficits can impact performance both on the PennCNB and at school. In addition, although no participants enrolled in this study had a known diagnosis of attention-deficit hyperactivity disorder (ADHD), children with undiagnosed ADHD may have influenced the results given the prevalence of the disorder in Sub-Saharan Africa (36) and lack of formal assessment prior to enrollment.

Notably, findings suggested low positive predictive values for sensorimotor speed. Sensitivity and specificity reflect characteristics of the test, but the prevalence of an outcome influences the positive predictive value and negative predictive value (37). This study was powered to provide a probability of 80% of detecting a difference in the AUC of two ROC curves at a critical value of 0.70 using a one-sided test of significance with a p-value equal to 0.05. Although statistically sufficient, this yielded a relative small sample size that limits precision of the estimates. Three participants were classified as having a sensorimotor deficit according to the professional consensus classifications (17 unclassifiable in this domain). In a sample with a greater prevalence of sensorimotor impairment (i.e., population sample of HIV-affected youth), it is possible that the predictive values will improve.

Since a gold standard for neurocognitive evaluation does not exist in this setting, the approach to determining the professional consensus classifications may have resulted in some misclassification of case/control status. Further, the potential for misclassification may have increased for the specific cognitive domains of impairment in the secondary analyses due to the slight differences in domains between the PennCNB and MoCA, as well as the MoCA’s limited assessment of specific domains of cognitive functioning, potentially resulting in lower sensitivity within specific domains. Standardized assessments included all elements of evaluations currently completed by the local psychologist supplemented by qualitative assessments, and expert clinicians helped to monitor fidelity and consistency of the assessments. Nonetheless, the professional consensus assessments were not as rigorous as validated gold standard tools.

Demonstrating acceptable criterion validity of the PennCNB has practical implications for implementation. Since administration of the PennCNB can be performed by individuals with limited training, showing that the battery has acceptable agreement with classification established by a professional evaluator for individual children suggests that the PennCNB may be reasonably relied on to identify children who would benefit from additional support. Although the PennCNB provides advantages such as ease of administration and automated scoring, implementation of the PennCNB in clinical settings in Botswana will require additional efforts, such as training in administration of the tool, establishment of data interpretation guidelines, and possibly inclusion of additional evaluations to examine literacy and linguistic cognitive deficits.

Conclusions

Findings indicated that the PennCNB adapted for use in Botswana demonstrated acceptable criterion validity, suggesting that the battery could be a valuable tool for identifying neurocognitive impairments among children and adolescents in this, and other, resource-limited settings with a scarcity of specialized psychologists. Future research should refine the determination of cut-off points to provide adequate interpretation of the scores in practice. The successful implementation of the PennCNB may facilitate early detection of neurocognitive deficits, which is critical for supporting the functional and educational attainment of youth living with HIV in resource-limited settings such as Botswana.

Availability of data and material:

All data support published conclusions.