Skip to main content
PMC home page
JMIR mHealth and uHealth logoLink to JMIR mHealth and uHealth
. 2018 Jan 5;6(1):e5. doi: 10.2196/mhealth.9148

A Mobile App to Screen for Neurocognitive Impairment: Preliminary Validation of NeuroScreen Among HIV-Infected South African Adults

Reuben N Robbins 1,✉,#, Hetta Gouse 2,#, Henry G Brown 3,#, Andries Ehlers 3,#, Travis M Scott 4,#, Cheng-Shiun Leu 1,#, Robert H Remien 1,#, Claude A Mellins 1,#, John A Joska 2,#
Editor: Gunther Eysenbach
Reviewed by: Jasmine Buttolph
PMCID: PMC5775487  PMID: 29305338

Abstract

Background

Neurocognitive impairment (NCI) is one of the most common complications of HIV infection, and has serious medical and functional consequences. South Africa has 7 million people living with HIV (PLHIV) with up to three-quarters of antiretroviral therapy (ART)-naïve individuals having NCI. South Africa’s health system struggles to meet the care needs of its millions of PLHIV; screening for NCI is typically neglected due to limited clinical staff trained to administer, score, and interpret neuropsychological tests, as well as long test batteries and limited screening tools for South African populations. Without accurate, clinically useful, and relatively brief NCI screening tests that can be administered by all levels of clinical staff, critical opportunities to provide psychoeducation, behavioral planning, additional ART adherence support, and adjuvant therapies for NCI (when they become available) are missed. To address these challenges and gap in care, we developed an mHealth app screening tool, NeuroScreen, to detect NCI that can be administered by all levels of clinical staff, including lay health workers.

Objective

The purpose of this study was to examine sensitivity and specificity of an adapted version of NeuroScreen to detect NCI (as determined by a gold standard neuropsychological test battery administered by a trained research psychometrist) among HIV-infected South Africans when administered by a lay health worker.

Methods

A total of 102 HIV-infected black South African adults who had initiated ART at least 12 months prior were administered NeuroScreen and a gold standard neuropsychological test battery in the participants’ choice of language (ie, English or isiXhosa). Three composite z scores were calculated for NeuroScreen: (1) sum of all individual test scores, (2) sum of all individual test scores and error scores from four tests, and (3) sum of four tests (abbreviated version). Global deficit scores were calculated for the gold standard battery where a score of 0.5 or greater indicated the presence of NCI.

Results

The mean age of participants was 33.31 (SD 7.46) years, most (59.8%, 61/102) had at least 12 years of education, and 81.4% (83/102) of the sample was female. Gold standard test battery results indicated that 26.5% (27/102) of the sample had NCI. Sensitivity and specificity of age-, education-, and sex-adjusted NeuroScreen scores were 81.48% and 74.67% for composite score 1, 81.48% and 81.33% for composite score 2, and 92.59% and 70.67% for composite score 3, respectively.

Conclusions

NeuroScreen, a highly automated, easy-to-use, tablet-based screening test to detect NCI among English- and isiXhosa-speaking South African HIV patients demonstrated robust sensitivity and specificity to detect NCI when administered by lay health workers. NeuroScreen could help make screening for NCI more feasible. However, additional research is needed with larger samples and normative test performance data are needed.

Keywords: HIV, neurocognitive, impairment, lay health workers, resource-limited settings, South Africa, tablet, app, neuropsychology

Introduction

Neurocognitive impairment (NCI) is one of the most common sequelae and comorbid conditions of HIV infection and has significant medical, functional, and public health consequences. Prevalence estimates of NCI among South Africa’s 7 million people living with HIV (PLHIV) [1] range from 23% to 76% depending on whether the sample is antiretroviral therapy (ART) experienced or naïve, and do not differ between men and women [2,3]. These rates are similar to estimates in high-income countries [4-8]. Neurocognitive impairment in HIV, known as HIV-associated neurocognitive disorder (HAND) when HIV is determined to be the etiology, typically causes impairments in mental processing speed, learning, memory, attention and concentration, executive functions, and motor speed [6,9,10]. It ranges in severity from asymptomatic and mild forms to a severe dementia-type form [11]. Having even asymptomatic or mild NCI has been associated with increased risk for developing more severe impairment and mortality [12-16]. Neurocognitive impairment has also been associated with worse ART adherence (jeopardizing positive health outcomes), employment difficulties, impaired instrumental activities of daily living (eg, planning, driving, finance management), worse overall quality of life, and the need for more social services [12-23]. In addition, NCIs are also associated with poor decision making and greater HIV transmission risk behaviors (eg, unprotected sex) [17-19].

Screening for NCI in HIV is essential to good comprehensive care and treatment strategies [8,20,21]. As a first step in the clinical decision-making process, screening can enable providers to determine who is most likely to have NCI, detect early signs of NCI, allocate limited resources more effectively (eg, comprehensive neuropsychological testing, neurologic services, and social worker intervention), determine when to initiate and adjust ART regimens, track and monitor neurocognitive function, and educate patients about the impact of NCI and ways to minimize it—all which can improve health outcomes [22,23]. The impact of NCI on ART adherence can be minimized through comprehensive behavioral planning that incorporates reminder strategies and social support. Detecting NCI in highly infectious PLHIV (ie, those with detectable viral load) may help in tailoring transmission prevention strategies. Furthermore, if and when adjuvant pharmacotherapies or behavioral interventions become available for NCI in HIV, screening for NCI will be the first step to linking patients with these treatments. However, screening for NCI among PLHIV is not in widespread practice, and faces unique challenges in low- and middle-income countries and resource-limited settings. In South Africa, there are few locally developed screening tests for NCI in the local languages of those most affected by HIV, and a lack of expert personnel to administer them [13,24-28], missing critical opportunities to detect NCI and intervene for the millions of PLHIV there.

A variety of neurocognitive screening tests have been developed and evaluated to detect HAND, including computer-based and paper-and-pencil tests. Recent reviews of the most commonly used screening tests for HAND, mostly from high-income countries, indicated a wide range of sensitivity (<55%-90%) and specificity (70%-90%) depending on the test, the specific population, and type of NCI detected (ie, mild or HIV-associated dementia) [29]. Studies examining NCI screening tests used specifically with South African PLHIV found similarly wide variation in sensitivity and specificity depending on the test or combination of tests used [30,31]. None of these studies evaluated the performance of the screening tests when administered by nonspecialists or highly trained personnel. A screening test that can be administered by all levels of clinical staff is essential for scale-up in low- to middle-income countries and other resource-limited settings.

In South Africa, like many other low- and middle-income countries and resource-limited settings, lay health workers are utilized in clinical settings to provide essential HIV-related services (eg, AIDS education, HIV testing and counseling, ART adherence counseling, and preliminary management of mild mental disorders) that may not be otherwise available due to provider shortages [28,32-34]. Through task shifting, lay health workers work alongside nurses, physicians, pharmacists, and social workers. Lay health workers are ideally situated to provide a number of services to PLHIV, but certain routine tasks are still too complex, such as neurocognitive testing and screening [27].

To address this gap in HIV care, we developed NeuroScreen [35], an mHealth app for tablets designed to be used by all levels of health care personnel, including lay health workers, to screen for NCI. This novel software app for the Android operating system uses the touchscreen capabilities of tablet devices to highly automate neuropsychological testing. The neurocognitive screening test battery is embedded in a graphical user interface that automates test administration and allows for easy data management and reporting. All tests in NeuroScreen are automatically timed and scored—no hand scoring, score converting, or simultaneous and synchronized use of stopwatches is required. Tests that would normally require a pen or pencil use the touchscreen instead. To ensure that each administration is consistent, administrators are forced to sequence through all the standardized instructions. Furthermore, eight of the ten tests have audio-visual instructions, which are useful for low-literacy populations. Because NeuroScreen runs on a tablet and does not require an Internet connection, it is ultraportable and allows screenings to be administered in almost any location, such as remote or rural clinics or fast-paced and busy urban clinics requiring flexible use of examination rooms. Although computerized neuropsychological testing is becoming more common (but not so in resource-limited settings), mobile operating systems and devices are only just starting to be more commonly used as test delivery platforms.

In our previous work [35], we pilot tested a large-format mobile phone version of NeuroScreen in a small sample (N=44) of older (mean age 53.4 years) HIV-positive adults in the United States, most of whom (75%) had NCI as defined by a gold standard neuropsychological test battery. We also assessed acceptability of NeuroScreen among participants and 10 HIV providers. Evidence for construct validity of individual NeuroScreen tests as compared to the gold standard neuropsychological test battery was established. NeuroScreen also showed high sensitivity and moderate specificity to detect NCI at 93.94% and 63.64%, respectively. Both participants and providers indicated high acceptability for NeuroScreen. Providers remarked that NeuroScreen has potential to detect other disease-related NCIs.

The purpose of this study was to evaluate the ability of the lay health worker-administered NeuroScreen to detect NCI, as defined by a gold standard neuropsychological test battery. We examined the sensitivity and specificity, as well as the negative predictive value (NPV) and positive predictive value (PPV) of a South African-adapted NeuroScreen (for use with tablets and with the isiXhosa language) to detect NCI.

Methods

This cross-sectional study adapted NeuroScreen from a large mobile phone screen format to 7-inch tablet format. Specifically, the Google Nexus 7 tablet was used for this study. Both the user interface and tests were updated to take advantage of the larger screen. NeuroScreen was also adapted for isiXhosa-speaking South Africans (the predominant Bantu language spoken in the Western Cape region of South Africa, where this study was conducted). Language translations underwent forward and backward translation, as well as vetting by the bilingual (English and isiXhosa) study staff. IsiXhosa translations were optimized to follow colloquial language conventions because it is the more predominantly used and understood form of isiXhosa in the communities where the study took place. Audio files were recorded with a native isiXhosa speaker for the audio-visual instructions. After all updates were made, the software engineers conducted quality assurance testing.

Sample and Recruitment

A total of 102 HIV-positive black South African adults, aged 18 to 56 years, were recruited following their participation in a larger randomized controlled trial (RCT) of a multimedia, laptop-based, lay health worker-delivered ART-readiness intervention for ART initiators (known as Masivukeni or “Let’s Wake Up”) conducted in Cape Town, South Africa (see [36-38]). Inclusion criteria were (1) HIV-positive, (2) age 18 years or older, (3) willing to be contacted by study staff for participation in other research studies, (4) isiXhosa or English speaking, (5) capable of consent, (6) willing to complete 2 to 3 hours of neuropsychological testing, and (7) willing to allow access to medical records by study staff. Exclusion criteria were (1) not meeting one of the preceding criteria, and (2) presence of a current psychotic disorder, significant current suicidal ideation, and severe cognitive impairment precluding ability to give informed consent or participate based on clinical judgment of providers. All participants enrolled in the larger RCT were eligible to participate in this study. Ethics approval was obtained by the Human Research Ethics Committee at the University of Cape Town and the New York State Psychiatric Institute Institutional Review Board.

Procedure

When the RCT participants completed their final study visit, the study nurse informed them about an additional study and requirements for participation. If a participant was interested and eligible, written informed consent was obtained. Then a trained lay health worker (two at each of the study clinics) administered NeuroScreen. A trained neuropsychology technician (one at each of the two study clinics) then administered the gold standard neuropsychological battery (approximately 3 hours). For participants who were unable to complete all study requirements on that visit, the neuropsychology technician scheduled another visit within 7 days to complete the gold standard neuropsychological battery. Neuropsychology technicians were blind to NeuroScreen results. Participants received the equivalent of US $40 for completion of all study procedures.

Measures

Demographic and Neuromedical Data

Demographic and medical history data were available from the larger RCT. The psychometrist conducted an additional neuromedical questionnaire prior to administering the neuropsychological battery.

NeuroScreen

NeuroScreen [35] is comprised of 10 brief neuropsychological tests to briefly assess individuals across six neuropsychological domains most affected by HIV: verbal learning (two trials, five words) and memory (5-minute delayed recall); processing speed via two trail making test sequencing paradigms, two visual discrimination tasks, and a number input task; attention/concentration via a number span forward and backward task; executive functioning via an alternating trail making test sequencing paradigm; and motor functioning via a finger-tapping task (see Multimedia Appendix 1 for a complete description of each test). The tests are embedded in a graphical user interface that allows the administrator to enter patient data, administer tests, generate instant raw results, and save raw results to a secure and password-protected website and/or an internal storage card. After entering patient data, the administrator is required to read standardized test instructions or play videos that provide audio-visual instructions, is prompted at appropriate points to offer practice trials on selected tests, and is prompted to move on to the next test, thus sequencing through all the tests. NeuroScreen can be administered in English or isiXhosa. Once all data were transferred to the principal investigator’s secure and encrypted hard drive, all data were wiped from the device.

Total raw scores (eg, completion times, total correct) and error scores (eg, sequencing errors in the trail making tests and incorrect number inputs in the number input test) are systematically captured by the app. For this study, all raw scores were converted to z scores using the entire sample and timed tests were reverse scored so that higher scores indicated better performance. Three composite z scores were calculated: (1) sum of all individual test scores, (2) sum of all individual test scores and total errors from the trail making and number speed tests, and (3) sum of four tests (visual discrimination 1 and 2, trail making 1, and number span total).

Neuropsychological Evaluation

This paper-and-pencil neuropsychological battery assessed individuals across all neurocognitive domains using tests that are particularly sensitive to HIV-related NCI: learning and memory (Hopkins Verbal Learning Test-Revised [HVLT-R] [39] and the Brief Visuospatial Memory Test-Revised [BVMT-R] [40]); executive functioning (Color Trails Test 2 [41] and the Wisconsin Card Sorting Test [WCST] [42]); attention/concentration (Wechsler Memory Scales, Third Edition [WMS-III] Spatial Span [43] and Wechsler Adult Intelligence Scales, Third Edition [WAIS-III] Digit Span [44]); processing speed (Trail Making Test, Part A [45], Color Trails Test 1 [41], and WAIS-III Digit Symbol Coding and Symbol Search [44]); language (semantic fluency of animals, fruits, and vegetables [46]), and motor functioning (successive finger tapping and grooved pegboard [47]). This particular paper-and-pencil battery has been used in many HIV studies in the United States [7,26,48] and has been adapted for use with isiXhosa speakers in South Africa [2,3]. Data produced by this battery include (1) raw test scores, (2) T scores, (3) individual test deficit scores, and (5) global deficit scores (GDS).

A GDS was calculated for each participant and used to identify those with and without NCI. The GDS summarizes neuropsychological battery results by converting individual test T scores to a deficit score from 0 (no impairment) to 5 (severe impairment) [49]. Deficit scores are averaged across all tests to create the GDS. The GDS considers number and severity of impairments, assigning less weight to unimpaired performance and overcoming the disadvantage of averaging absolute performance, which gives equal weight to unimpaired and impaired scores [50]. The GDS method detects mild NCI (GDS≥0.5) across varying impairment patterns in different neurocognitive domains [49,51]. We set the threshold of GDS≥0.5 as indicative of NCI.

Statistical Analysis

Univariate analyses were conducted to examine participant characteristics. To evaluate the ability of the lay health worker-administered NeuroScreen to detect NCI and evaluate its sensitivity and specificity, a logistic regression and receiver operator characteristic curve were used. First, the logistic regression model was used to establish a prediction model for NCI using the NeuroScreen score and age, gender, and education in the form of: log[P(X)/ {1- P(X) }]= β0+β1X1+β2X2+β3X3+β4X4, where P(X ) is the probability of having NCI given X=(X1,..., X4)=(patient’s age, sex, years of education, and NeuroScreen total score). After obtaining the estimated regression coefficients, the predicted probability of having NCI was calculated. After calculating the predicted probability for each patient, a receiver operating characteristic (ROC) analysis using the predicted probability and the gold standard NCI determination was calculated to evaluate the area under the curve (AUC) and to generate an optimal age-, education-, and sex-adjusted NeuroScreen score cutoff point to distinguish participants with NCI. The cutoff point was maximized by using the Youden index, which is equal to the sensitivity+specificity–1. Both PPV and NPV were computed for the optimized NeuroScreen cutoff score. All analyses were conducted using IBM SPSS version 23 (IBM Corp, Armonk, NY, USA).

Results

Sample Characteristics

As Table 1 shows, the sample was predominantly female with a mean age of 33.31 (SD 7.46) years and most (59.8%, 61/102) had at least 12 years of education. Although 30 participants reported being held back for at least one grade during their schooling, only nine participants reported receiving special classes to assist them with learning difficulties (the South African education system does not routinely assess for and diagnose learning disorders). Four participants reported having had a loss of consciousness greater than 15 minutes (for head injury and other medical issues). One participant reported having a stroke that resulted in numbness in the right arm, three participants reported having had one seizure, and one participant reported being diagnosed with epilepsy at age 10. Four participants reported taking medication for high blood pressure.

Table 1.

Sample characteristics (N=102).

Characteristic Participants Min Max
Age (years), mean (SDa) 33.31 (7.46) 19 56
Gender (female), n (%) 83 (81) — —
Education (years completed), mean (SD) 11.25 (1.99) 3 14
Traumatic brain injury with loss of consciousness >15 minutes, n (%) 4 (4) — —
Likely learning difficulty, n (%) 9 (9) — —
Most recent CD4 cell count, mean (SD)b 501.31 (287.41) 47 1654
Most recent viral load undetectable, n (%)c 81 (91) — —
Open in a new tab

aSD: standard deviation.

bCD4: cluster of differentiation 4. CD4 cell count available for 88 participants.

cViral load data available for 81 participants.

NeuroScreen Performance

Table 2 displays raw mean scores for all NeuroScreen tests, as well as the three composite total z scores. On average, participants were able to learn 8.45 (SD 1.51) words across two learning trials (same five words per trial) and recall 3.39 (SD 1.20) words after a 5-minute delay. The mean total number span backward and forward score was 5.87 (SD 1.29) with a maximum possible score of 17. The mean total correct responses on Visual Discrimination 1 was 11.28 (maximum=61) and 25.97 on Visual Discrimination 2 (maximum=150). The mean completion time on Number Input Speed was 45.02 (SD 18.08) seconds. Mean completion times for Trails Test 1 was 18.93 (SD 18.17) seconds, 31.95 (SD 21.21) seconds for Trails Test 2, and 18.81 (SD 16.32) seconds for Trails Test 3. The mean total finger taps for both the dominant and nondominant hand was 454.73 (SD 60.03) taps across five trials.

Table 2.

NeuroScreen performance (raw).


Test		 Mean (SDa)
Min
Max
Finger tapping total (both hands) 454.73 (60.03) 190.00 581.00
Visual discrimination 1 total correct 11.28 (3.67) 4.00 19.00
Visual discrimination 2 total correct 25.97 (7.97) 1.00 48.00
Number span total (forward and backward) 5.87 (1.29) 3.50 9.50
Verbal learning total correct 8.45 (1.51) 3.00 10.00
Delayed verbal recall total correct 3.39 (1.20) 0.00 5.00
Trail making 1 completion time (seconds)b –18.93 (18.17) –120.00 –6.06
Trail making 2 completion time (seconds)b –31.95 (21.21) –120.00 –11.17
Trail making 3 completion time (seconds)b –18.81 (16.32) –120.00 –5.05
Number speed completion time (seconds)b –45.02 (18.08) –117.75 –23.60
Full battery completion time (minutes) 23.88 (6.21) 9.00 52.00
Open in a new tab

aSD: standard deviation.

bIndicates reverse scored (slower time=worse performance).

Gold Standard HIV Neuropsychological Battery Performance

Results from the full neuropsychological battery (Table 3) indicated that the sample had a mean global T score of 48.01 (SD 4.79). Table 3 also displays performance (adjusted T scores) across the individual tests. The mean GDS was 0.36 (SD 0.40) and 26.5% (27/102) had NCI using a GDS score of 0.5 or greater to indicate impairment.

Table 3.

Gold standard neuropsychological test battery performance (adjusted).

Test Mean (SDa) Min Max
Global T 48.01 (4.79) 34.00 57.63
Motor functioning tests



Successive finger tapping (dominant hand) 46.51 (11.29) –1.31 64.05

Successive finger tapping (nondominant hand) 44.98 (13.45) –43.87 61.66

Grooved pegboard (dominant hand) 47.41 (7.96) 12.67 60.05

Grooved pegboard (nondominant hand) 49.26 (3.09) 34.09 53.27
Hopkins Verbal Learning Test-Revised



Total trials 1-3 45.97 (8.33) 24.34 62.67

Delay recall total 44.74 (9.86) 22.64 67.13
Brief Visuospatial Memory Test-Revised



Total Trials 1-3 48.40 (9.80) 28.09 73.75

Delay Total Recall 49.58 (11.48) 29.03 72.20
Wechsler Adult Intelligence Scales, Third Edition (WAIS-III)



Digit symbol coding total 46.73 (10.08) 25.45 77.15

Symbol search total 47.24 (8.95) 27.19 64.76

Wechsler Memory Scales, Third Edition spatial span total 50.47 (9.65) 30.06 73.85
Processing speed tests



Trail making test, part A 43.82 (11.99) –7.48 72.04

Color trails test 1 46.72 (9.98) 12.22 64.02

Color trails test 2 48.32 (9.73) 14.83 66.80

WAIS-III Digit span total 49.33 (1.21) 46.70 52.09
Wisconsin Card Sorting Test



Perseverative errors 50.74 (12.89) 3.52 63.76

Trials to first sort 48.10 (11.32) 29.66 57.98

Failures to maintain set 50.59 (4.79) 31.88 53.94
Language tests



Animal fluency total 49.41 (8.70) 27.72 70.48

Fruit and vegetable fluency total 51.83 (8.75) 33.26 72.21
Open in a new tab

aSD: standard deviation.

Sensitivity and Specificity

NeuroScreen Total Score 1 (Sum of All Tests)

Using the logistic model with the first NeuroScreen total score adjusted for age, education, and sex to predict the gold standard NCI in the ROC analysis, the AUC was 0.86 (95% CI 0.78-0.94; see Figure 1). The Youden index NeuroScreen predicted NCI cut-score of 0.21 maximized sensitivity at 81.48% (95% CI 61.92%-93.70%) and specificity at 74.67% (95% CI 63.30%-84.01%). The PPV was 53.66% and the NPV was 91.80%. Using this cut-score yielded 19 false positives and 5 false negatives. The mean completion time for all the tests was 23.88 (SD 6.21) minutes.

Figure 1.

Figure 1

Open in a new tab

Receiver operating characteristic curve for NeuroScreen total score 1.

NeuroScreen Total Score 2 (Sum of All Tests and Available Error Scores)

An ROC analysis using the logistic model with the second NeuroScreen total score to predict NCI had an AUC of 0.86 (95% CI 0.78-0.94; see Figure 2). The Youden index maximal sensitivity was 81.48% (95% CI 61.92%-93.70%) and specificity was 81.33% (95% CI 70.67%-89.40%). The PPV was 61.11% and the NPV was 92.42%. Using this cut-score yielded 14 false positives and 5 false negatives. The mean completion time was the same as the preceding.

Figure 2.

Figure 2

Open in a new tab

Receiver operating characteristic curve for NeuroScreen total score 2.

NeuroScreen Total Score 3 (Sum of Four Tests)

Using the logistic model with the third NeuroScreen total score with age, education, and sex to predict the gold standard NCI in the ROC analysis, the AUC was 0.87 (95% CI 0.80-0.94; see Figure 3). The Youden index NeuroScreen predicted NCI cut-score of 0.18 maximized sensitivity at 92.59% (95% CI 75.71%-99.09%) and specificity at 70.67% (95% CI 59.02%-80.62%). The PPV was 53.19% and the NPV was 96.36%. Using this cut-score yielded 22 false positives and 2 false negatives. The estimated completion time of these four tests was approximately 12 minutes.

Figure 3.

Figure 3

Open in a new tab

Receiver operating characteristic curve for NeuroScreen total score 3 (abbreviated version).

Discussion

In a sample of 102 HIV-positive predominantly female South Africans on ART for at least 1 year, gold standard neuropsychological test results indicated 26% of the sample had at least mild NCI. This rate of NCI is consistent with other research from South Africa [3], but lower than NCI rates found in the United States [48] among ART-experienced PLHIV. NeuroScreen—when administered by a lay health worker—had robust test characteristics to detect gold standard-defined NCI ranging from 81% to 93% sensitivity and 71% to 81% specificity, depending on the combination of NeuroScreen test scores used. Our study provides preliminary evidence for the validity of NeuroScreen to detect NCI among English- and isXhosa-speaking South African adults living with HIV. NeuroScreen shows promise as an easy-to-use, brief NCI screening test that can be administered by lay health workers, and likely most levels of health care staff.

Deciding which NeuroScreen total score to use must be weighed by each provider and/or health care system. Given estimated rates of NCI that exist among PLHIV in South Africa (23%-76% [2,3]), the extra burden screening for NCI places on an already overburdened and resource-limited health care system must be weighed against the consequences misdetection might have on patients and the health care system. Ideally, anyone who screens positive for NCI would be referred for confirmatory testing with a comprehensive neuropsychological assessment. However, doing so is simply not feasible in many settings, such as South Africa. For resource-limited settings, such as South Africa, it may make sense to use the shortest version of NeuroScreen (12 minutes to administer with 92.59% sensitivity and 70.67% specificity) even though the rate of false positives would be higher than using the full NeuroScreen with error scores (25-minute administration time with 81.48% sensitivity and 81.33% specificity). The consequences of a positive screen (eg, extra treatment planning considerations and additional ART adherence support) might outweigh the consequences of missing someone who truly has NCI (eg, poor adherence/health outcomes).

Although the full version with error scores is approximately twice as long to administer than the abbreviated version, its specificity is much higher. As a potential substitute for a full neuropsychological assessment (which is not feasible and highly unlikely to occur in this setting), 25 minutes of a lay health worker’s time is much less resource intensive than 3 to 5 hours of a neuropsychologist’s, psychologist’s, and/or neuropsychology technician’s time to administer a neuropsychological test battery and score it. Although we do not believe a short battery of tests such as NeuroScreen should replace gold standard neuropsychological assessments, it may help provide clinics in resource-limited settings that do not have access to the gold standard measures and procedures with a viable alternative. Having a screening test that can be administered by any staff, lay health worker included, can help clinics identify those PLHIV at highest risk for having NCI and make important treatment recommendations and referrals to address it.

Compared to other computerized and paper-and-pencil screening tests for NCI in HIV, NeuroScreen performed similarly to and better than some tests (see [29]). Compared to screening tests specifically evaluated in South Africa [30,31], NeuroScreen also yielded robust sensitivity and specificity across all three total scores. Moreover, these robust performance characteristics were achieved by NeuroScreen when administered by a lay health worker. NeuroScreen, to our knowledge, is the only computerized, mHealth neurocognitive screening test tablet app for NCI detection in South Africa for English and isiXhosa speakers. Furthermore, it is the only such test designed to be administered by all levels of clinical staff with minimal training and supervision.

It is important to note that there was discrepancy between NeuroScreen and the gold standard neuropsychological battery in defining who had NCI. Overall, NeuroScreen indicated that more participants had NCI than did the gold standard battery. The discrepancy could be explained in part by NeuroScreen’s tests being more difficult than the gold standard tests (ie, floor effects on NeuroScreen tests) or practice effects. We did not randomize test sequence administration—all participants were administered NeuroScreen first then the gold standard tests. Some participants’ performances on the gold standard tests could have benefited from first taking the NeuroScreen tests. If the benefit was big enough, performance on the gold standard tests could have appeared within the normal range (ie, not impaired). Additionally, the discrepancy could also be due to human factors involved in the gold standard battery administration, such as time keeping and recording errors, or subtle biases in test scoring (although our neuropsychology technicians received intensive training and ongoing supervision). There could also be issues regarding language—more participants opted to take NeuroScreen in English (n=27) than the gold standard battery (n=6). Further research is needed to fully understand this discrepancy.

It is important to note this study’s limitations. First, we had a small sample of PLHIV in South Africa to evaluate NeuroScreen’s sensitivity and specificity that was mostly female. Second, we did not formally assess language fluency, for either English or isiXhosa. Participants were asked for their language preference by the lay health worker. Similarly, the psychometrist discussed with the participant in which language they would like to take the gold standard battery. Third, the NCI detected in this study may or may not be a result of HIV—numerous factors can cause and/or contribute to the development of NCIs, such as low education, head injuries, and other medical factors (many of which were observed in this sample). Fourth, normative performance data have not been established for NeuroScreen among isiXhosa-speaking South Africans, or other South African language groups, making generalization of performance on it inappropriate. Finally, the gold standard battery had numerous neuropsychological tests assessing neurocognitive domains that NeuroScreen did not assess (verbal fluency and perseveration), although the tests in NeuroScreen were specifically chosen to assess those neurocognitive domains most typically affected by HIV. Furthermore, NeuroScreen is not meant to be a substitute for a thorough neuropsychological assessment.

Despite these limitations, we believe NeuroScreen has potential to offer busy health clinics and research studies with a brief, easy-to-use solution to screen for NCI among PLHIV, and among patients with other brain-involving diseases and disorders. With a tool such as NeuroScreen, better referrals, tracking, and integration with electronic medical records could be achieved. However, more research is needed to validate NeuroScreen as a screening tool for NCI. A larger sample, statistically powered to establish internal and external validity indicators, is essential for NeuroScreen scale-up.

Computerized neurocognitive testing and screening are transforming clinical practice for neuropsychological assessments. Mobile technology offers a powerful platform that is ultraportable and can be easy to use. This study provides evidence that our app, NeuroScreen, has clinically useful psychometric properties to detect NCI when administered by lay health workers. Taking advantage of mobile platforms and automating many components of the neurocognitive testing process may help to make testing more accurate, efficient, affordable, and accessible to those who need testing, especially in resource-limited settings.

Acknowledgments

This research was supported by grants from the National Institute of Child Health and Human Development (R21-HD084197; principal investigator: RNR), National Institute of Nursing Research (R21-NR015009; principal investigator: RNR), National Institute of Mental Health (R01-MH09557; principal investigator: RHR), National Institute of Mental Health to the HIV Center for Clinical and Behavioral Studies at New York State Psychiatric Institute and Columbia University (P30-MH43520; principal investigator: RHR), as well as the Columbia University Global Mental Health Scholars Program (director: Kathy Pike, PhD). The authors also wish to acknowledge the hard work of our research staff—Michelle Henry, Veronica Jonah, Anna Dreyer, Yoliswa Mtingeni, Thandeka Mbonambi, Zodwa Mbakaza, Tandiwe Mngxuma, Ziyanda Ncusane, and Zodwa Makuluma—and all our study participants. We would like to thank the City of Cape Town Department of Health for their support.

Abbreviations

ART

antiretroviral therapy

AUC

area under the curve

BVMT-R

Brief Visuospatial Memory Test-Revised

GDS

global deficit score

HAND

HIV-associated neurocognitive disorder

HVLT-R

Hopkins Verbal Learning Test-Revised

NCI

neurocognitive impairment

NPV

negative predictive value

PLHIV

people living with HIV

PPV

positive predictive value

RCT

randomized controlled trial

ROC

receiver operator characteristic

WAIS-III

Wechsler Adult Intelligence Scales, Third Edition

WCST

Wisconsin Card Sorting Test

WMS-III

Wechsler Memory Scales, Third Edition

Multimedia Appendix 1

NeuroScreen test descriptions.

mhealth_v6i1e5_app1.pdf (169.7KB, pdf)

Footnotes

Conflicts of Interest: None declared.

References

  • 1.UNAIDS. 2017. Oct 10, [2017-10-10]. AIDSInfo http://aidsinfo.unaids.org/
  • 2.Joska JA, Westgarth-Taylor J, Myer L, Hoare J, Thomas KG, Combrinck M, Paul RH, Stein DJ, Flisher AJ. Characterization of HIV-associated neurocognitive disorders among individuals starting antiretroviral therapy in South Africa. AIDS Behav. 2011 Aug;15(6):1197–1203. doi: 10.1007/s10461-010-9744-6. [DOI] [PubMed] [Google Scholar]
  • 3.Joska JA, Westgarth-Taylor J, Hoare J, Thomas KG, Paul R, Myer L, Stein DJ. Neuropsychological outcomes in adults commencing highly active anti-retroviral treatment in South Africa: a prospective study. BMC Infect Dis. 2012 Feb 15;12:39. doi: 10.1186/1471-2334-12-39. https://bmcinfectdis.biomedcentral.com/articles/10.1186/1471-2334-12-39. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Becker JT, Lopez OL, Dew MA, Aizenstein HJ. Prevalence of cognitive disorders differs as a function of age in HIV virus infection. AIDS. 2004 Jan 01;18 Suppl 1:S11–S18. [PubMed] [Google Scholar]
  • 5.Bonnet F, Amieva H, Marquant F, Bernard C, Bruyand M, Dauchy FA, Mercié P, Greib C, Richert L, Neau D, Catheline G, Dehail P, Dabis F, Morlat P, Dartigues JF, Chêne G, S CO3 Aquitaine Cohort Cognitive disorders in HIV-infected patients: are they HIV-related? AIDS. 2013 Jan 28;27(3):391–400. doi: 10.1097/QAD.0b013e32835b1019. [DOI] [PubMed] [Google Scholar]
  • 6.Grant I, Sacktor N, McArthur J. HIV neurocognitive disorders. In: Gendelman HE, Grant I, Everall IP, Fox HS, Gelbard HA, Lipton SA, Swindells S, editors. The Neurology of AIDS. Oxford: Oxford University Press; 2005. pp. 357–373. [Google Scholar]
  • 7.Heaton RK, Grant I, Butters N, White DA, Kirson D, Atkinson JH, McCutchan JA, Taylor MJ, Kelly MD, Ellis RJ. The HNRC 500--neuropsychology of HIV infection at different disease stages. HIV Neurobehavioral Research Center. J Int Neuropsychol Soc. 1995 May;1(3):231–251. doi: 10.1017/s1355617700000230. [DOI] [PubMed] [Google Scholar]
  • 8.Simioni S, Cavassini M, Annoni J, Rimbault AA, Bourquin I, Schiffer V, Calmy A, Chave J, Giacobini E, Hirschel B, Du Pasquier RA. Cognitive dysfunction in HIV patients despite long-standing suppression of viremia. AIDS. 2010 Jun 01;24(9):1243–1250. doi: 10.1097/QAD.0b013e3283354a7b. [DOI] [PubMed] [Google Scholar]
  • 9.Grant I. Neurocognitive disturbances in HIV. Int Rev Psychiatry. 2008 Feb;20(1):33–47. doi: 10.1080/09540260701877894. [DOI] [PubMed] [Google Scholar]
  • 10.Heaton RK, Franklin DR, Ellis RJ, McCutchan JA, Letendre SL, Leblanc S, Corkran SH, Duarte NA, Clifford DB, Woods SP, Collier AC, Marra CM, Morgello S, Mindt MR, Taylor MJ, Marcotte TD, Atkinson JH, Wolfson T, Gelman BB, McArthur JC, Simpson DM, Abramson I, Gamst A, Fennema-Notestine C, Jernigan TL, Wong J, Grant I, CHARTER Group. HNRC Group HIV-associated neurocognitive disorders before and during the era of combination antiretroviral therapy: differences in rates, nature, and predictors. J Neurovirol. 2011 Feb;17(1):3–16. doi: 10.1007/s13365-010-0006-1. http://europepmc.org/abstract/MED/21174240. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Antinori A, Arendt G, Becker JT, Brew BJ, Byrd DA, Cherner M, Clifford DB, Cinque P, Epstein LG, Goodkin K, Gisslen M, Grant I, Heaton RK, Joseph J, Marder K, Marra CM, McArthur JC, Nunn M, Price RW, Pulliam L, Robertson KR, Sacktor N, Valcour V, Wojna VE. Updated research nosology for HIV-associated neurocognitive disorders. Neurology. 2007 Oct 30;69(18):1789–1799. doi: 10.1212/01.WNL.0000287431.88658.8b. http://europepmc.org/abstract/MED/17914061. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Hinkin CH, Castellon SA, Durvasula RS, Hardy DJ, Lam MN, Mason KI, Thrasher D, Goetz MB, Stefaniak M. Medication adherence among HIV+ adults: effects of cognitive dysfunction and regimen complexity. Neurology. 2002 Dec 24;59(12):1944–1950. doi: 10.1212/01.wnl.0000038347.48137.67. http://europepmc.org/abstract/MED/12499488. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Hinkin CH, Hardy DJ, Mason KI, Castellon SA, Durvasula RS, Lam MN, Stefaniak M. Medication adherence in HIV-infected adults: effect of patient age, cognitive status, and substance abuse. AIDS. 2004 Jan 01;18 Suppl 1:S19–S25. doi: 10.1097/00002030-200418001-00004. http://europepmc.org/abstract/MED/15075494. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Vivithanaporn P, Heo G, Gamble J, Krentz HB, Hoke A, Gill MJ, Power C. Neurologic disease burden in treated HIV/AIDS predicts survival: a population-based study. Neurology. 2010 Sep 28;75(13):1150–1158. doi: 10.1212/WNL.0b013e3181f4d5bb. http://europepmc.org/abstract/MED/20739646. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Ettenhofer ML, Foley J, Castellon SA, Hinkin CH. Reciprocal prediction of medication adherence and neurocognition in HIV/AIDS. Neurology. 2010 Apr 13;74(15):1217–1222. doi: 10.1212/WNL.0b013e3181d8c1ca. http://europepmc.org/abstract/MED/20220123. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Ettenhofer ML, Hinkin CH, Castellon SA, Durvasula R, Ullman J, Lam M, Myers H, Wright MJ, Foley J. Aging, neurocognition, and medication adherence in HIV infection. Am J Geriatr Psychiatry. 2009 Apr;17(4):281–290. doi: 10.1097/JGP.0b013e31819431bd. http://europepmc.org/abstract/MED/19307857. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Anand P, Springer SA, Copenhaver MM, Altice FL. Neurocognitive impairment and HIV risk factors: a reciprocal relationship. AIDS Behav. 2010 Dec;14(6):1213–1226. doi: 10.1007/s10461-010-9684-1. http://europepmc.org/abstract/MED/20232242. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Iudicello JE, Woods SP, Cattie JE, Doyle K, Grant I, HIV Neurobehavioral Research Program Group Risky decision-making in HIV-associated neurocognitive disorders (HAND) Clin Neuropsychol. 2013;27(2):256–275. doi: 10.1080/13854046.2012.740077. http://europepmc.org/abstract/MED/23181946. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Thames AD, Streiff V, Patel SM, Panos SE, Castellon SA, Hinkin CH. The role of HIV infection, cognition, and depression in risky decision-making. J Neuropsychiatry Clin Neurosci. 2012;24(3):340–348. doi: 10.1176/appi.neuropsych.11110340. http://europepmc.org/abstract/MED/23037648. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Gorman AA, Foley JM, Ettenhofer ML, Hinkin CH, van GWG. Functional consequences of HIV-associated neuropsychological impairment. Neuropsychol Rev. 2009 Jun;19(2):186–203. doi: 10.1007/s11065-009-9095-0. http://europepmc.org/abstract/MED/19472057. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Kamminga J, Cysique LA, Lu G, Batchelor J, Brew BJ. Validity of cognitive screens for HIV-associated neurocognitive disorder: a systematic review and an informed screen selection guide. Curr HIV/AIDS Rep. 2013 Dec;10(4):342–355. doi: 10.1007/s11904-013-0176-6. http://europepmc.org/abstract/MED/24072534. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Mind Exchange Working Group Assessment, diagnosis, and treatment of HIV-associated neurocognitive disorder: a consensus report of the mind exchange program. Clin Infect Dis. 2013 Apr;56(7):1004–1017. doi: 10.1093/cid/cis975. http://europepmc.org/abstract/MED/23175555. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Cysique L, Bain M, Lane T, Brew B. Management issues in HIV-associated neurocognitive disorders. NBHIV. 2012 Jul;4:63–73. doi: 10.2147/NBHIV.S30466. [DOI] [Google Scholar]
  • 24.Joint United Nations Programme on HIV/AIDS (UNAIDS) Global Report: UNAIDS Report on the Global AIDS Epidemic 2013. Geneva, Switzerland: UNAIDS; 2013. [Google Scholar]
  • 25.World Health Statistics: Global Health Indicators. Geneva, Switzerland: World Health Organization; 2010. [Google Scholar]
  • 26.Heaton RK, Marcotte TD, Mindt MR, Sadek J, Moore DJ, Bentley H, McCutchan JA, Reicks C, Grant I, HNRC Group The impact of HIV-associated neuropsychological impairment on everyday functioning. J Int Neuropsychol Soc. 2004 May;10(3):317–331. doi: 10.1017/S1355617704102130. [DOI] [PubMed] [Google Scholar]
  • 27.Robbins RN, Remien RH, Mellins CA, Joska JA, Stein DJ. Screening for HIV-associated dementia in South Africa: potentials and pitfalls of task-shifting. AIDS Patient Care STDS. 2011 Oct;25(10):587–593. doi: 10.1089/apc.2011.0154. http://europepmc.org/abstract/MED/21919734. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Breuer E, Stoloff K, Myer L, Seedat S, Stein DJ, Joska J. Reliability of the lay adherence counsellor administered substance abuse and mental illness symptoms screener (SAMISS) and the International HIV Dementia Scale (IHDS) in a primary care HIV clinic in Cape Town, South Africa. AIDS Behav. 2012 Aug;16(6):1464–1471. doi: 10.1007/s10461-011-0067-z. [DOI] [PubMed] [Google Scholar]
  • 29.Zipursky AR, Gogolishvili D, Rueda S, Brunetta J, Carvalhal A, McCombe JA, Gill MJ, Rachlis A, Rosenes R, Arbess G, Marcotte T, Rourke SB. Evaluation of brief screening tools for neurocognitive impairment in HIV/AIDS: a systematic review of the literature. AIDS. 2013 Sep 24;27(15):2385–2401. doi: 10.1097/QAD.0b013e328363bf56. http://europepmc.org/abstract/MED/23751261. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Joska JA, Westgarth-Taylor J, Hoare J, Thomas KG, Paul R, Myer L, Stein DJ. Validity of the International HIV Dementia Scale in South Africa. AIDS Patient Care STDS. 2011 Feb;25(2):95–101. doi: 10.1089/apc.2010.0292. [DOI] [PubMed] [Google Scholar]
  • 31.Joska JA, Witten J, Thomas KG, Robertson C, Casson-Crook M, Roosa H, Creighton J, Lyons J, McArthur J, Sacktor NC. A comparison of five brief screening tools for HIV-associated neurocognitive disorders in the USA and South Africa. AIDS Behav. 2016 Aug;20(8):1621–1631. doi: 10.1007/s10461-016-1316-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Callaghan M, Ford N, Schneider H. A systematic review of task- shifting for HIV treatment and care in Africa. Hum Resour Health. 2010 Mar 31;8:8. doi: 10.1186/1478-4491-8-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Bekker L, Myer L, Orrell C, Lawn S, Wood R. Rapid scale-up of a community-based HIV treatment service: programme performance over 3 consecutive years in Guguletu, South Africa. S Afr Med J. 2006 Apr;96(4):315–320. http://blues.sabinet.co.za/WebZ/Authorize?sessionid=0:autho=pubmed:password=pubmed2004&/AdvancedQuery?&format=F&next=images/ejour/m_samj/m_samj_v96_n4_a19.pdf. [PubMed] [Google Scholar]
  • 34.Zachariah R, Ford N, Philips M, Lynch S, Massaquoi M, Janssens V, Harries AD. Task shifting in HIV/AIDS: opportunities, challenges and proposed actions for sub-Saharan Africa. Trans R Soc Trop Med Hyg. 2009 Jun;103(6):549–558. doi: 10.1016/j.trstmh.2008.09.019. [DOI] [PubMed] [Google Scholar]
  • 35.Robbins RN, Brown H, Ehlers A, Joska JA, Thomas KG, Burgess R, Byrd D, Morgello S. A smartphone app to screen for HIV-related neurocognitive impairment. J Mob Technol Med. 2014 Feb 01;3(1):23–26. doi: 10.7309/jmtm.3.1.5. http://europepmc.org/abstract/MED/24860624. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Robbins RN, Mellins CA, Leu C, Rowe J, Warne P, Abrams EJ, Witte S, Stein DJ, Remien RH. Enhancing lay counselor capacity to improve patient outcomes with multimedia technology. AIDS Behav. 2015 Jun;19 Suppl 2:163–176. doi: 10.1007/s10461-014-0988-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Remien RH, Mellins CA, Robbins RN, Kelsey R, Rowe J, Warne P, Chowdhury J, Lalkhen N, Hoppe L, Abrams EJ, El-Bassel N, Witte S, Stein DJ. Masivukeni: development of a multimedia based antiretroviral therapy adherence intervention for counselors and patients in South Africa. AIDS Behav. 2013 Jul;17(6):1979–1991. doi: 10.1007/s10461-013-0438-8. http://europepmc.org/abstract/MED/23468079. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Gouse H, Henry M, Robbins RN, Lopez-Rios J, Mellins CA, Remien RH, Joska JA. Psychosocial aspects of ART counseling: a comparison of HIV beliefs and knowledge in PMTCT and ART-naive women. J Assoc Nurses AIDS Care. 2017;28(4):504–517. doi: 10.1016/j.jana.2017.03.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Benedict RH, Schretlen D, Groninger L, Brandt J. Hopkins Verbal Learning Test-Revised: normative data and analysis of inter-form and test-retest reliability. Clinical Neuropsychol. 1998 Feb 1;12(1):43–55. doi: 10.1076/clin.12.1.43.1726. [DOI] [Google Scholar]
  • 40.Benedict RH. Brief Visuospatial Memory Test-Revised: Professional Manual. Odessa, FL: Psychological Assessment Resources, Inc; 1997. [Google Scholar]
  • 41.D'Elia L, Satz P, Uchiyama C, White T. Color Trails Test. Odessa, FL: Psychological Assessment Resources, Inc; 1996. [Google Scholar]
  • 42.Kongs S, Thompson L, Iverson G, Heaton R. Wisconsin Card Sorting Test-64 Card Version: Professional Manual. Odessa, FL: Psychological Assessment Resources; 2000. [Google Scholar]
  • 43.Wechsler D. WMS-III: Wechsler Memory Scale Administration and Scoring Manual. San Antonio, TX: Psychological Corporation; 1997. [Google Scholar]
  • 44.Wechsler D. WAIS-III: Wechsler Adult Intelligence Scale Administration and Scoring Manual. San Antonio, TX: Psychological Corporation; 1997. [Google Scholar]
  • 45.Reitan RM. Validity of the Trail Making Test as an indicator of organic brain damage. PMS. 1958 Dec;8(7):271. doi: 10.2466/PMS.8.7.271-276. [DOI] [Google Scholar]
  • 46.Tombaugh TN, Kozak J, Rees L. Normative data stratified by age and education for two measures of verbal fluency: FAS and animal naming. Arch Clin Neuropsychol. 1999 Feb;14(2):167–177. [PubMed] [Google Scholar]
  • 47.Strauss E, Sherman E, Spreen O. A compendium of neuropsychological tests: administration, norms, and commentary. Oxford: Oxford University Press; 2006. [Google Scholar]
  • 48.Heaton RK, Clifford DB, Franklin DR, Woods SP, Ake C, Vaida F, Ellis RJ, Letendre SL, Marcotte TD, Atkinson JH, Rivera-Mindt M, Vigil OR, Taylor MJ, Collier AC, Marra CM, Gelman BB, McArthur JC, Morgello S, Simpson DM, McCutchan JA, Abramson I, Gamst A, Fennema-Notestine C, Jernigan TL, Wong J, Grant I, CHARTER Group HIV-associated neurocognitive disorders persist in the era of potent antiretroviral therapy: CHARTER Study. Neurology. 2010 Dec 07;75(23):2087–2096. doi: 10.1212/WNL.0b013e318200d727. http://europepmc.org/abstract/MED/21135382. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Carey CL, Woods SP, Gonzalez R, Conover E, Marcotte TD, Grant I, Heaton RK, HNRC Group Predictive validity of global deficit scores in detecting neuropsychological impairment in HIV infection. J Clin Exp Neuropsychol. 2004 May;26(3):307–319. doi: 10.1080/13803390490510031. [DOI] [PubMed] [Google Scholar]
  • 50.Blackstone K, Moore DJ, Heaton RK, Franklin DR, Woods SP, Clifford DB, Collier AC, Marra CM, Gelman BB, McArthur JC, Morgello S, Simpson DM, Rivera-Mindt M, Deutsch R, Ellis RJ, Hampton AJ, Grant I, CNS HIV Antiretroviral Therapy Effects Research (CHARTER) Group Diagnosing symptomatic HIV-associated neurocognitive disorders: self-report versus performance-based assessment of everyday functioning. J Int Neuropsychol Soc. 2012 Jan;18(1):79–88. doi: 10.1017/S135561771100141X. http://europepmc.org/abstract/MED/22114912. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Carey C, Woods S, Rippeth J, Gonzalez R, Moore D, Marcotte T, Grant I, Heaton RK, HNRC Group Initial validation of a screening battery for the detection of HIV-associated cognitive impairment. Clin Neuropsychol. 2004 May;18(2):234–248. doi: 10.1080/13854040490501448. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Multimedia Appendix 1

NeuroScreen test descriptions.

mhealth_v6i1e5_app1.pdf (169.7KB, pdf)

Articles from JMIR mHealth and uHealth are provided here courtesy of JMIR Publications Inc.

RESOURCES