Construct Validity Supports Use of a Novel, Tablet-Based Neurocognitive Assessment for Adolescents and Young Adults Affected by Perinatal HIV from Vulnerable Communities in the United States

Abstract

Construct validity of novel tablet-based neurocognitive tests (in the NeuroScreen app) measuring processing speed, working memory, and executive functioning in adolescents and young adults (AYA) living with perinatally-acquired HIV (PHIV) and perinatal HIV-exposure without infection (PHEU) was examined. Sixty-two AYA (33 PHIV, 29 PHEU) were recruited from an ongoing longitudinal study (CASAH) in New York City. Medium to large and statistically significant correlations were found between NeuroScreen and gold standard, paper-and-pencil tests of processing speed, working memory, and executive functioning. Results provide partial support for NeuroScreen as an alternative to cumbersome paper-and-pencil tests for assessing neurocognition among HIV-affected AYA.

Introduction

Neurocognitive impairment (NCI) is common among adolescents and young adults (AYA) living with perinatally-acquired HIV (PHIV) and perinatal HIV-exposure without HIV-infection (PHEU) [1]. AYA with PHIV have lifelong HIV; HIV can enter the central nervous system (CNS) early in infection where it may persist as a viral reservoir, infects and depletes CNS immune cells (e.g., CD4+ lymphocytes, macrophages), can cause HIV encephalopathy, and contributes to CNS abnormalities observable in brain imaging studies, all of which can lead to NCI [2]. Even AYA with PHIV on ART who achieve long term viral suppression can show NCI [3]. Among PHIV and PHEU AYA, socioeconomic status, poor educational opportunities, and early life adversity may contribute to the development (and among PHIV AYA exacerbation) of NCI [1].

Assessing NCI can be challenging across populations, including among AYA with PHIV and PHEU. Neurocognitive testing can be time consuming, and require specialized forms, equipment and highly trained personnel to administer, score and interpret (e.g., psychologists and neuropsychologist) – all of which may preclude assessing NCI in many research and clinical settings [4]. Furthermore, many of the commonly used tests for NCI may have test norms that do not reflect the racial and ethnically diverse communities that most AYA with PHIV and PHEU reside in [1]. Novel neurocognitive testing applications (apps) or digital neuropsychological tests are changing how NCI is assessed and some apps are making NCI assessment more widely available in resource limited settings [4]. However, because they are relatively new, many testing apps lack psychometric evidence demonstrating their construct validity [5].

NeuroScreen [6–7] is a novel, highly automated, relatively brief (25 minutes), easy-to-use by any staff, Android operating system, tablet-based neurocognitive assessment tool that has potential to make assessing NCI among people living with HIV more widely available to clinics and researchers. NeuroScreen includes twelve brief neuropsychological tests that assess the cognitive domains of working memory, processing speed, verbal learning, verbal recall/memory, executive functioning, and motor speed. NeuroScreen has been shown to have high sensitivity and specificity to detect NCI among adults with behaviorally acquired HIV infection in South Africa when administered by lay counselors and compared to a gold-standard, expert administered neuropsychological assessment [7].

This study examined the construct validity (i.e., convergent validity) of the NeuroScreen app tests of processing speed, working memory and executive functioning among African American and Latinx, disadvantaged AYA with PHIV and PHEU in New York City based on established gold-standard, paper-and-pencil tests of those domains. Evaluating the construct validity of NeuroScreen for use with ethnically diverse PHIV and PHEU AYA, we hope to provide an evidence base for use of the app to assess NCI, thus making assessment more widely available and making a substantive impact on addressing health and prevention needs in resource-limited settings where NCIs are common, but often go undetected.

Methods

Participants

Data were collected from participants enrolled in the CASAH study [8]. CASAH is an ongoing longitudinal study investigating the mental health and risk behaviors of youth living with PHIV and PHEU that began in 2003 and currently has eight waves of follow-up (FU) data collection. Full details of the CASAH study have been published [8]. For the current study, sixty-one young adults between the ages of 18–30 with PHIV (n=33) and PHEU (n=28) who completed a FU6 or FU7 study visit were recruited.

Procedures

Participants completed the NeuroScreen app concurrently with either their CASAH FU6 or FU7 study visit (after completing CASAH procedures), or at a separate substudy visit if they had completed their CASAH FU visit more than three months ago. The CASAH protocol for FUs 5, 6 and 7 already contained three paper-and-pencil based neurocognitive tests of processing speed, working memory and executive functioning (see Measures). Participants were either approached by a research assistant about participating in this substudy at their annual CASAH study visit, or by phone. Before completing any procedures, all participants provided their written informed consent. For participants who completed the substudy concurrently with their CASAH FU visit, after completing all CASAH study procedures, they completed NeuroScreen. Participants were compensated with $40 for their time. Participants who came for a separate substudy visit completed the same neurocognitive tests in the CASAH protocol and NeuroScreen. Written informed consent was obtained from participants upon their arrival for this study’s visit. Participants were compensated with $50 for their time and up to $25 for travel expenses.

Measures

Paper-and-pencil neurocognitive tests.

Though CASAH’s primary aims are on mental and behavioral health, beginning in FU5 the study protocol added three neurocognitive tests to provide some data on processing speed, working memory, and executive functioning in an otherwise psychiatric and psychosocial focused study.

Trail Making Test

The Trail Making Test (TMT) is a two-part test measuring processing speed (TMT A) and executive functioning (TMT B) [9]. In TMT A, examinees are asked to connect a series of numbered circles as quickly as possible in numerical order. TMT B assesses task-switching capabilities and dominant response inhibition [10]. Examinees are asked to connect circles with letters and numbers alternating between numbers and letters in numeric and alphabetical order. Completion time (in seconds) is recorded and shorter times indicate better performance.

Digit Span

The Digit Span subtest of the Weschler Adult Intelligence Scale, Fourth Edition [11] assesses verbal working memory. Examinees are first asked to recall increasing spans of digits immediately after they are read by the examiner, and then in reverse order. The total number of correct responses (forwards and backwards) are recorded where higher scores indicate better performance [12].

NeuroScreen tests of processing speed, working memory and executive functioning.

Tests are embedded in a graphical user interface allowing an administrator to enter patient data and administer the tests. After patient data is entered, the administrator is required to read standardized test instructions (only for verbal learning and memory, and number span tests) or play videos that provide audio-visual instructions. Immediately after tests are completed (or when there is an internet connection), results are automatically uploaded to a secure server and web application where data can be managed. Examinees are offered practice trials on selected tests. The app automatically sequences through all the tests in the same order. For this study, we examined the NeuroScreen tests of processing speed (visual discrimination 1 and 2, and number speed, trail making tests 1 and 3), working memory (number span), and executive functioning (trail making test 2).

Processing speed.

Visual discrimination 1.

Examinees must determine if one of two figures match one of five other figures in a row below. The examinee presses the yes button or no button to make the decision on whether one of the symbols matches. After each press of the button, new symbols appear. Response time for each item and response choice (correct or incorrect) are recorded by the app. The trial lasts 45-seconds. For this study, total correct responses (where higher scores indicate better performance) was used for analysis.

Visual discrimination 2.

Examinees must match symbols to numbers. An array of symbols with corresponding numbers is presented at the top of the screen. In the middle of the screen a symbol appears. At the bottom of the screen are number buttons. The examinee must press the number button that corresponds to the symbol that appears in the middle of the screen. Response time for each item and response choice (correct or incorrect) are recorded by the app. The test last 45-seconds. For this study, total correct responses (where higher scores indicate better performance) was used for analysis.

Trail Making Tests.

There are two trail making type tests where examinees must draw a line between circles with numbers in them either in their ordinal sequence (Trails 1), or their ordinal sequence while alternating between blue and yellow colored circles (Trails 2). In the third trail making type test (Trails 3), examinees must trace a line over a dotted line between circles. For all three tests: (a) examinees are instructed to not lift their fingers off the screen and are given audio warnings if they do; (b) examinees are instructed to go back to the last circle they were on if they make a sequencing error and are given an audio warning if they do – examinees cannot continue until they correct the error. For all tests, total completion time, time between each circle, finger lifts off the screen, sequencing errors, and wrong starts (starting on wrong circle) are recorded by the app. The maximum time allowed for each test is 180 seconds – if the test is not completed within the time limit, the test automatically stops. For this study, total completion time (where higher scores indicate worse performance) was used for all three tests in our analyses.

Number speed.

Examinees are presented with a number at the top of the screen. Using a keypad below the numbers, the examinee must enter the numbers as fast as they can. There are five number sequences – each one is longer by one digit. Total completion time across the five trials is the score. Correctly entered numbers turn green in the number sequence. Incorrectly entered numbers turn red in the number sequence and must be corrected. For each trial, the app records completion time, number of errors, and time taken to enter each digit. For this study, total completion time across all five trials was used in analyses.

Working memory.

Number span.

Examinees hear digits one at a time and are asked to repeat them exactly as heard either forwards (Number Span Forwards) or backwards (Number Span Backwards). Staff enter the digits on a keypad on the tablet screen as the examinee repeats them. In each subsequent trial, digits increase by one. In Number Span Forwards, there are a maximum of eight trials. In Number Span Backwards, there are a maximum of seven trials. Each trial has two attempts of the same number span. Correct responses on the first attempt receive one-point, and the examinee advances to the next trial. If the first attempt in any trial is incorrect (0-points), the examinee will have another chance with the same number span. If the second attempt is correct, the examinee receives half a point and advances to the next trial. The test ends when an examinee receives zero points on one trial (0-points for two attempts of the same number span) or is able to attempt the maximum number of trials in Number Span Forwards and Backwards. The app records the exact number sequence by the examinee, as well as the time it took to complete each trial. The app also automatically scores each trial and continues the test or stops it. For this study, total points across all Forward trials were summed and total points across all Backwards trials were summed.

Data Analysis

Univariate statistics were used to examine the participants across demographic factors (age, education, and race/ethnicity). Independent samples t-tests were computed to examine between group performance across all the tests. Chi-square was also calculated to determine the proportion of participants in each group who performed >1 standard deviation below the full sample mean on each test. To evaluate construct validity of NeuroScreen for use with disadvantaged AYA with PHIV and PHEU in New York City, Pearson correlation coefficients (and 95% confidence interval) were computed between raw scores on the paper-and-pencil and NeuroScreen tests.

Results

Median age of participants was 24 years (IQR 22–26); 64% were male, 46% Latinx, and 44% African-American, mean years of education was 13.17 (SD 1.30). There were no differences across these demographic variables between the PHIV and PHEU participants. Among the PHIV participants, at the time of the study visit, all 32 were prescribed at least one HIV medication; the median CD4 count was 475 (IQR:172.50 – 763.50), and the median viral load was 46 copies/mL (IQR: 20 – 5203). The paper-and-pencil and NeuroScreen tests of processing speed, working memory, and executive functioning were all significantly correlated with each other, respectively.

Independent samples t-tests indicated that mean performance across all eight NeuroScreen and four gold-standard tests (Digit Span Forwards and Backwards, and TMT B) did not statistically differ between the PHIV and PHEU groups (Table 2). There was a trend level finding on TMT A where PHIV participants performed more slowly on average (M = 24.61 seconds, SD = 8.21) than PHEU participants (M = 20.46 seconds, SD = 7.66), t(59)=−2.03, p=.05. Proportion of participants performing one standard deviation below the mean did not differ on seven of the NeuroScreen tests and three of the gold-standard tests (Table 3). On the NeuroScreen number span forwards test, a significantly higher proportion of PHIV participants performed in the below one standard deviation range than PHEU participants (21% vs. 0%, respectively; X² (1, N = 61) = 6.7, p = .01. On the gold-standard Digit Span Backwards test, more PHIV participants performed in the below one standard deviation range than PHEU participants (36% vs. 4%, respectively; X² (1, N = 61) = 9.7, p < .01.

Table 2.

Mean Test Performance

	Full Sample		PHIV Participants		PHEU Participants
Test	Mean	SD	Mean	SD	Mean	SD	t	p
TM 1a†	7.42	3.39	7.25	2.739	7.61	4.05	0.41	0.69
TM 3a†	9.03	5.29	8.52	4.395	9.62	6.19	0.80	0.43
VD 1a‡	19.38	3.97	18.72	3.438	20.14	4.44	1.40	0.17
VD 2a‡	42.38	7.85	42.59	8.485	42.14	7.20	−0.22	0.83
NSpeeda†	25.58	7.20	25.30	8.917	25.89	4.66	0.32	0.76
NS (F)b‡	4.87	1.15	4.69	1.306	5.07	0.91	1.30	0.20
NS (B)b‡	3.30	1.27	3.22	1.319	3.39	1.24	0.53	0.60
TM 2c†	15.10	6.98	15.51	8.006	14.62	5.70	−0.49	0.63
TMT Aa†	22.70	8.16	24.61	8.208	20.46	7.66	−2.03	0.05
DS (F)b‡	10.26	2.37	9.76	2.424	10.86	2.19	1.85	0.07
DS (B)b‡	7.95	2.73	7.67	3.038	8.29	2.32	0.88	0.38
TMT Bc†	58.72	34.44	60.64	38.744	56.46	29.11	−0.47	0.64

Note: test of:

^aprocessing speed

^bworking memory

^cexecutive functioning

^†Timed tests (completion time in seconds): higher scores = worse performance

^‡Higher scores = better performance.

NeuroScreen Tests: TM 1, TM 2, and TM 3 = Trail Making 1, 2, and 3; VD 1 and VD 2 = Visual Discrimination 1 and 2; NS (F) and NS (B) = Number Span Forwards and Backwards; NSpeed = Number Speed. Gold-Standard Tests: TMT A and TMT B = Trailmaking Test A and B; DS (F) and DS (B) = Digit Span Forwards and Backwards.

Table 3.

Proportion One Standard Deviation Below the Sample Mean

	Full Sample (N=61)	PHIV Participants (N=33)	PHEU Participants (N=28)
Test	% (N)	% (N)	% (N)	X2	p
TM 1a†	9.8 (6)	3.0 (1)	17.9 (5)	3.76	0.09
TM 3a†	8.2 (5)	6.0 (2)	10.7 (3)	0.44	0.66
VD 1a‡	14.8 (9)	12.1 (4)	17.9 (5)	0.40	0.72
VD 2a‡	11.5 (7)	12.1 (4)	10.7 (3)	0.03	1.00
NSpeeda†	3.3 (2)	3.0 (1)	3.6 (1)	0.01	1.00
NS (F)b‡	11.5 (7)	21.2 (7)	0.0 (0)	6.71	0.01
NS (B)b‡	19.7 (12)	24.2 (8)	14.3 (4)	0.95	0.35
TM 2c†	13.1 (8)	12.1 (4)	14.3 (4)	0.04	1.00
TMT Aa†	13.1 (8)	12.1 (4)	14.3 (4)	0.06	1.00
DS (F)b‡	11.5 (7)	15.2 (5)	7.1 (2)	0.96	0.43
DS (B)b‡	21.3 (13)	36.3 (12)	3.6 (1)	9.71	0.00
TMT Bc†	14.8 (9)	12.1 (4)	14.3 (4)	0.01	1.00

Note: test of:

^aprocessing speed

^bworking memory

^cexecutive functioning

NeuroScreen Tests: TM 1, TM 2, and TM 3 = Trail Making 1, 2, and 3; VD 1 and VD 2 = Visual Discrimination 1 and 2; NS (F) and NS (B) = Number Span Forwards and Backwards; NSpeed = Number Speed. Gold-Standard Tests: TMT A and TMT B = Trailmaking Test A and B; DS (F) and DS (B) = Digit Span Forwards and Backwards.

Processing speed tests.

The NeuroScreen Trail Making Test 1 had a small to medium correlation with TMT A, r(61)=0.28, p < 0.05 such that worse performance on the NeuroScreen test was significantly correlated with worse performance on the analogue paper and pencil test. The NeuroScreen Trail Making Test 3 had a medium correlation with TMT A, r(61)=0.33, p < 0.01, and the NeuroScreen Number Speed test had a large correlation with TMT A, r(61)=0.59, p < 0.001. NeuroScreen Visual Discrimination Tests 1 and 2 had large negative correlations with TMT A, r(61)=−0.53, p < 0.001 and r(61)=−0.63, p < 0.001, respectively, such that better performance on these NeuroScreen tests was significantly correlated with better performance on the paper and pencil test.

Working memory tests.

The NeuroScreen Number Span Forwards test had large correlations with the paper and pencil Digit Span Forwards, r(61)=0.66, p < 0.001 and Digit Span Backwards, r(61)=0.64, p < 0.001 tests such that worse performance on the NeuroScreen test was significantly correlated with worse performance on the paper and pencil tests. Similarly, large correlations were found between the NeuroScreen Number Span Backwards test and the paper and pencil Digit Span Forwards, r(61)=0.50, p < 0.001 and Digit Span Backwards, r(61)=0.70, p < 0.001 measures.

Executive functioning tests.

The NeuroScreen Trail Making Test 2 had a large correlation with TMT B, r(61)=0.63, p < .001 such that worse performance on the NeuroScreen test was significantly correlated with worse performance on the analogue paper and pencil test.

Discussion

The current study aimed to examine correlations (i.e., construct/convergent validity) between a novel set of digital, tablet-based neurocognitive tests of processing speed, working memory, and executive functioning and their traditional, paper-and-pencil test counterparts. Small to large and statistically significant Pearson correlation coefficients were found between the digital, tablet-based tests and the paper-and-pencil based tests of these domains. These results provide psychometric evidence to support the construct validity of the NeuroScreen app tests of working memory, processing speed, and executive functioning for use with AYA with PHIV and PHEU.

Given the need for routine neurocognitive assessments and screenings for people living with HIV and the limited availability of such services in clinical practice, as well as the challenges that traditional paper-and-pencil based tests present for use in routine care in resource limited clinical settings, such as long administration times, extra equipment (e.g., pencils, proprietary forms, stopwatches, etc.), and expert staff to administer and score them, novel methods to assess neurocognitive functioning that are brief, accurate, and can be administered by any level of clinical staff are greatly needed [4]. NeuroScreen has been shown to be acceptable to and reliably used by lay counselors in South Africa to screen for NCI among people living with HIV, as well as having clinically useful sensitivity and specificity to detect NCI there [7]. Results from this study provide partial support for its use as an alternative to cumbersome paper-and-pencil tests for assessing neurocognition among HIV-affected AYA in the US.

It is important to note that this study was conducted on a small sample of predominantly African American and Latinx young adults affected by HIV in a research setting and as such generalizations to other populations cannot be made. While the results provide evidence that the tablet-based tests measure the same constructs as the paper-and-pencil tests, further research needs to examine the factor structure of the NeuroScreen tests as compared to a larger number of paper-and-pencil neurocognitive tests, as well as to examine their criterion validity, that is how well do the NeuroScreen tests detect neurocognitive impairment compared to the gold-standard paper-and-pencil tests. For example, the NeuroScreen Trail Making tests 1 and 2 correlated with all the gold-standard tests, whereas the NeuroScreen Trail Making 3 test only correlated with TMT A and B in the gold-standard battery. This could be due to this test having poor psychometric properties or that it taps into a cognitive domain other than working memory or processing speed. Trail Making 3 is the simplest of the three trail making tests and only requires line tracing; it does not require any sequencing (i.e., processing speed). The correlation with TMT A and B may reflect that all three tests require a motor component. The lack of correlation with Digit Span Forwards and Backwards, which does not involve a motor component, may reflect the lack of higher order cognitive demands in the Trail Making 3 test. Finally, to further enhance NeuroScreen’s ability to detect impairment and measure neurocognitive functions, robust test norms to quantify overall and individual subtest performance will be necessary [1].

Nonetheless, NeuroScreen may provide a useful alternative to neurocognitive assessments for these populations than cumbersome paper-and-pencil based tests. Digital or tablet-based tests have potential to transform neurocognitive assessments. They have potential to make such assessments more widely available to larger segments of society, and to segments of society that typically do not have access to them.

Table 1.

Correlation Matrix

		TMT Aa†	DS (F)b‡	DS (B)sb‡	TMT Bc†	TM 1a†	TM 3a†	VD 1a‡	VD 2a‡	NSpeeda†	NS (F)b‡	NS (B)b‡	TM 2c†
NeuroScreen Tests	TM 1a†	0.28*	−0.21	−0.336**	0.26*	1	0.36**	−0.24§	−0.35**	0.31*	−0.29*	−0.36**	0.23
	TM 3a†	0.33**	−0.19	−0.19	0.27*	0.36**	1	−0.22§	−0.45**	0.36**	−0.14	−0.29*	0.28*
	VD 1a‡	−0.53***	0.42***	0.33**	−0.48***	−0.24§	−0.45**	1	0.61**	−0.50**	0.33**	0.29*	−0.43**
	VD 2a‡	−0.63***	0.37**	0.36**	−0.59***	−0.35**	−0.45**	0.61**	1	−0.70**	0.38**	0.42**	−0.57**
	NSpeeda†	0.59***	−0.32*	−0.29*	0.58***	0.31*	0.36**	−0.50**	−0.70**	1	−0.45**	−0.37**	0.59**
	NS (F)b‡	−0.47***	0.68***	0.64***	−0.48***	−0.29*	−0.14	0.33**	0.38**	−0.45**	1	0.56**	−0.44**
	NS (B)b‡	−0.48***	0.51***	0.70***	−0.48***	−0.36**	−0.30*	0.29*	0.42**	−0.37**	0.56**	1	−0.29*
	TM 2c†	0.47***	−0.28*	−0.27*	0.63***	0.23	0.28*	−0.43**	−0.57**	0.59**	−0.44**	−0.41**	1
Gold-Standard Tests	TMT Aa†	1	−0.49**	−0.33**	0.63**	0.28*	0.33**	−0.53***	−0.63***	0.59***	−0.47***	−0.48***	0.47***
	DS (F)b‡	−0.49**	1	0.51**	−0.42**	−0.21	−0.19	0.42***	0.37**	−0.32*	0.68***	0.51***	−0.28*
	DS (B)b‡	−0.33**	0.51**	1	−0.41**	−0.33**	−0.19	0.33**	0.36**	−0.29*	0.64***	0.70***	−0.27*
	TMT Bc†	0.63**	−0.42**	−0.41**	1	0.26*	0.27*	−0.48***	−0.59***	0.58***	−0.48***	−0.48***	0.63***

Note:

^*p<0.05

^**p<0.01

^***p<0.001

test of:

^aprocessing speed

^bworking memory

^cexecutive functioning

^†Timed tests: higher scores = worse performance

^‡Higher scores = better performance.

NeuroScreen Tests: TM 1, TM 2, and TM 3 = Trail Making 1, 2, and 3; VD 1 and VD 2 = Visual Discrimination 1 and 2; NS (F) and NS (B) = Number Span Forwards and Backwards; NSpeed = Number Speed. Gold-Standard Tests: TMT A and TMT B = Trailmaking Test A and B; DS (F) and DS (B) = Digit Span Forwards and Backwards.