Improving detection of HIV-associated cognitive impairment: Comparison of the International HIV Dementia Scale and a Brief Screening Battery

Abstract

Objectives

The International HIV Dementia Scale (IHDS) was developed to screen for HIV-associated dementia (HAD), but it has been used more generally for HIV-associated neurocognitive disorder (HAND). This study sought to examine the accuracy of the IHDS in a cohort of Brazilian HIV-infected individuals and compare its performance to an alternative screening battery for detecting HAND.

Methods

108 participants (including 60 HIV-infected persons), completed the IHDS and a gold standard neuropsychological (NP) battery of 17 tests. As alternative screening method, all possible three-test combinations from the NP battery were examined and a superiority index (a marker of specificity and sensitivity) was calculated.

Results

Sensitivity and specificity to HAND using the standard IHDS cutpoint of 10 were 36% and 75% respectively. The best balance between sensitivity and specificity was accomplished with a modified cutpoint of 11.5, which yielded sensitivity of 72% and specificity of 58%. The top two most sensitive test combinations, compared to the gold standard NP battery, were Trail Making Test A, WAIS-III Digit Symbol (DS) and HVLT-R Total Recall (sensitivity 91%, specificity 96%), and DS, BVMT-R Total Recall and Grooved Pegboard Test-Dominant Hand (sensitivity 94%, specificity 91%).

Conclusions

Both test combinations can be administered in under 10 minutes and were more accurate than the IHDS in classifying HIV+ participants as NP impaired or unimpaired. These data suggest that demographically corrected T-scores from commonly used NP measures with modest time and material demands can improve identification of patients with HAND who may benefit from a more extensive NP examination.

HIV-associated neurocognitive disorder (HAND) continues to be a significant and debilitating complication of HIV infection that has profound implications for productivity^1,2, quality of life, and illness prognosis³. The introduction of combination antiretroviral therapy (cART) has brought dramatic improvements in HIV associated medical morbidity and mortality as well as HIV-associated dementia (HAD), although milder forms of HAND persist and affect up to half of all patients^4,5

While a comprehensive neuropsychological evaluation should be considered the gold-standard for diagnosing HAND, there is a need for brief cognitive screening measures that require minimal administration resources and can reliably identify those patients who need additional evaluation⁶. The International HIV Dementia Scale (IHDS) was designed for use in international, resource-limited settings, as an appropriate screening tool under different cultural, linguistic, and educational conditions and evaluates memory, motor speed, and psychomotor functioning⁷. It can be easily included within a medical visit and does not require administration by neuropsychologist or neurologist^7,8,9. The IHDS summary score (range = 0–12) is compared to a raw cut score of ≤ 10 to determine whether further evaluation for possible dementia is needed. Data from studies in India and Uganda support the use of the IHDS as a screening measure in resource-limited settings^7,10. Although the cutpoint of 10 was found to yield adequate sensitivity and specificity in these cohorts, it appears that this value may not be suitable across cultures. Joska et al. (2011) found in South Africa¹¹, evaluating HAND, that cutpoint yielded sensitivity and specificity of 45% and 79% respectively, while a cutpoint of 11 improved sensitivity somewhat to 53% with specificity 80%.

In a study conducted in Southeastern Brazil, using the cut-off of 10, the IHDS had 55% sensitivity and 80% specificity for detecting HAND⁹. Sensitivity was only 46% for detecting Asymptomatic Neurocognitive Impairment (ANI) and Mild Neurocognitive Disorder (MND). With a cut-off score of 11, overall sensitivity improved to 78% with 52% specificity, the sensitivity of detecting ANI/MND rose to 76%⁹. The IHDS was developed with the expressed goal of screening HAD⁷, but as HAD becomes rare and detecting milder forms of HAND is of greater need, it has been used to detect ANI/MND which are more prevalent in the current era of cART⁴. Nonetheless, the IHDS is largely used in clinical settings in Brazil, low income countries, and also in the U.S. to diagnose HAND, which is outside the scope of its intended use as a screening instrument. This highlights the need for further evidence regarding the utility of the IHDS and also warrants the examination of other approaches for screening HIV+ patients for HAND.

One approach is to develop a brief cognitive screening battery made up of commonly used tests whose combination yields acceptable classification accuracy rates¹². The psychometric properties of the battery will be enhanced by selecting reliable and validated NP measures with normative data available for the intended population¹³. The tests included in the brief battery should be selected using a theoretical framework that reflects the cognitive domains thought to be frequently affected by the disorder of interest. There is some evidence that this approach may be superior to using the IHDS. Carey and colleagues (2004)¹² set out to develop an empirically valid, psychometrically-sound, screening battery to detect HAND and found that the Hopkins Verbal Learning Test-Revised (HVLT-R) Total Recall score paired with the Grooved Pegboard Test nondominant hand or the WAIS-III Digit Symbol subtest, had 78% sensitivity and 75% specificity when compared to the “gold standard” larger NP battery. Furthermore, these two combinations were more accurate than the IHDS.

The objectives of the present study were to examine the accuracy of the IHDS for detecting HAND in southern Brazil and to compare the performance of the IHDS to that of an empirically derived brief screening battery compared to a comprehensive NP test battery as the gold standard.

METHOD

Participants

A group of 60 HIV+ participants were recruited from the Hospital de Clinicas UFPR (HC-UFPR), Curitiba, Southern Brazil. All HIV+ participants had serological confirmation of their HIV and hepatitis status according to guidelines published by the Brazilian Ministry of Health¹⁴. Control participants (n=48) were recruited from the HC-UFPR blood bank and tested serologically negative for HIV, hepatitis B (HBV), hepatitis C (HCV) and syphilis. HIV+ and HIV− groups were recruited to be comparable with respect to sex, age and years of education. Group characteristics are presented in Table 1. Exclusion criteria for both groups included loss of consciousness greater than 30 minutes, non-HIV-related neurologic injury or disorder (e.g., epilepsy, stroke, developmental delay), psychotic disorders, and significant levels of current substance use, defined as more than two alcoholic drinks per day over the past 30 days, or use of any illegal drugs in the past 30 days. CSF and plasma HIV RNA was quantified by HIV bDNA (Siemens). As described previously¹⁵, 26 subjects had HIV subtype B, 33 had HIV subtype non B (C-22, BF-7, BC-1, CF-1, F-2), in one subject the subtype was not identified.

Table 1.

Demographic and disease characteristics of the groups

Variable	HIV− (n=48)	HIV+ (n=60)	P value
Age1	42.9 (11.3)	42.5 (9.1)	0.82
Education1	9.1 (4.4)	9.1 (4.3)	0.996
Male n (%)	25 (52%)	30 (50%)	0.98
HCV+ n (%)3	0 (0%)	12 (20%)
Duration of HIV (months)	–	99 [29–140]	–
CD4 Nadir2	–	88 [28–268]	–
CD4 Current2	–	372 [193–547]
plasma HIV viral load2	–	1.7 [1.7–3.1]
Plasma HIV RNA ≤50 copies/ml n (%)	–	35 (58%)	–
CSF HIV viral load2	–	1.7 [1.7–2.8]
CSF HIV RNA ≤50 copies/ml n (%)	–	32 (53%)	–
on ARV n (%)	–	47 (78%)	–
AIDS diagnosis n (%)	–	48 (80%)	–

¹mean (SD);

²median [IQR];

³HIV− were selected to be HCV–

The local HC-UFPR and the national (CONEP) IRB approved this project. Written informed consent was obtained from study participants after the research procedure had been fully explained to them.

Neurobehavioral Assessments

The International HIV dementia scale⁷

IHDS instructions and stimuli were translated into Brazilian Portuguese and back-translated into English by bilingual Portuguese-English speakers, and reviewed by several Brazilian native Portuguese speakers to ensure cultural and linguistic appropriateness. For the memory section, we translated three of the original English words (dog, hat, bean) as cachorro, chapéu, feijão and replaced the fourth word, red, with the color yellow, amarelo, to avoid association with red bean that is a popular food in Brazil. The IHDS was administered to all participants by the same neurologist (SMA), according to instructions in the original publication⁷.

Neuropsychological (NP) test battery (Gold standard)

All participants received a NP evaluation assessing 7 ability domains comprising 17 NP tests widely used to study HIV infection in English- and non-English speaking countries^{16,17,18,19,20}. The domains assessed included: Executive Function (Category Test, Color Trails Test 2, Stroop incongruent condition; Wisconsin Card Sorting Test-64); Verbal Fluency (phonemic, semantic: animals and actions); Speed of Information Processing (WAIS-III Digit Symbol and Symbol Search, Trail Making Test A, Color Trails Test 1, Stroop Test): Attention/Working Memory (Paced Auditory Serial Addition Test-50 item, WMS-III Spatial Span); Learning and Memory (Brief Visuospatial Memory Test-Revised, Hopkins Verbal Learning Test-Revised); and Motor Performance (Grooved Pegboard Test). Instruments not already validated for use in Brazilian Portuguese were translated into Portuguese, back-translated into English, and reviewed by several Brazilian native Portuguese speakers to ensure cultural and linguistic appropriateness.

Demographic corrections for each NP test were developed using data from Brazilian HIV− controls. Norms were statistically derived by methods previously described¹. Briefly, raw scores for the individual tests were converted to normally distributed scaled scores, which have a mean of 10 and a standard deviation of 3. The scaled scores were then converted to T-scores (mean 50, SD 10), with regression-based adjustments for the effects of age, years of education, and sex. T-scores of 40 or above (i.e., no worse than −1 SD) were assigned a deficit score of 0 (normal). T-scores below 40 were converted to deficit scores as follows: 35–39 = 1(mild impairment); 30–34 = 2 (mild to moderate impairment); 25–29 = 3 (moderate impairment); 20–24 = 4 (moderate to severe impairment); T<20 = 5 (severe impairment). Deficit scores across the test battery were averaged to compute a Global Deficit Score (GDS). The GDS summarizes the number and severity of neurobehavioral deficits across the entire test battery. Based on previous literature, a GDS cut-off ≥0.50 was used to classify overall NP impairment^1,12,21.

HAND diagnoses were assigned according to the Frascati Criteria²² as described previously¹⁵, based on evidence of neurocognitive impairment and declines in everyday functioning. Activities of daily living (ADL) were determined with ADL form which was a modification of the Lawton and Brody ADL Questionnaire²⁶.

Examiner Training

To ensure standardization of NP test administration, examiners (neuropsychologists at UFPR) underwent training by the UCSD team, with at least one bilingual member to facilitate discussion. During the training session, each test or interview was demonstrated, and its purpose and administration nuances were discussed. Several rounds of “mock testing” were conducted. Certification sessions subsequently took place with staff or patient volunteers from the hospital as test subjects. All certifications were done in Portuguese. Quality assurance reviews were conducted on test forms for all participants.

Data Analysis

Group comparisons were made using t-test, chi-square, Fisher’s exact, and non-parametric Wilcoxon tests as appropriate. Correlation was calculated using Spearman coefficient.

To select the optimal three-test screening battery, we examined the sensitivity and specificity of all possible three-test combinations. Average deficit scores for all combinations were used to classify participants as impaired (deficit score ≥0.5). These classifications were compared to the “gold standard” (the full NP battery GDS, as described above) to calculate sensitivity and specificity values. A superiority index²³ was used to rank test combinations based on their accuracy performance in predicting HAND. The superiority index gives more weight to a combination that has high sensitivity and specificity, rather than a combination that has poor sensitivity and specificity or highly discrepant sensitivity and specificity (e.g., very high sensitivity but very low specificity). For the three measures used as screening batteries, see Supplemental Tables A1 and A2 for lookup tables and formulas to convert raw scores into demographically corrected T-scores.

Descriptive classification accuracy statistics (sensitivity, specificity, positive predictive power, negative predictive power, and overall predictive power) were generated for the IHDS and the test combinations relative to the full NP battery GDS. Receiver-operating characteristic (ROC) curves were generated for the IHDS and the most sensitive three-test combinations to determine whether the measures performed significantly better than chance at classifying HIV+ persons as NP impaired. Multiple cutpoints were examined for the IHDS.

RESULTS

The IHDS scores, GDS, and HAND classification in the groups of HIV+ and HIV− participants are shown in Table 2. Of the HIV+ individuals, 40% obtained perfect scores (i.e., 12 points) on the IHDS. On the gold standard NP battery, 40% of HIV+ participants were classified as NP normal. In the HIV+ sample 8.3% met criteria for HAD, 3.3% met criteria for MND, and 48.3% met criteria for ANI (see Table 2). In total, 60% (n=36) were NP impaired on the gold standard battery.

Table 2.

GDS, IHDS and HAND classification in the HIV+ and HIV− groups

Variable	HIV− (n=48)	HIV+ (n=60)	Cohen’s d2	P value
GDS1	0.23 (0.28)	0.87 (0.90)	0.96	<0.001
GDS ≥ 0.5	11 (23%)	36 (60%)		<0.001
HAND
normal		24 (40%)		–
ANI		29 (48.3%)		–
MND		2 (3.3%)		–
HAD		5 (8.3%)		–
IHDS Total1	11.5(0.8)	10.4 (2.4)	−0.62	0.001
IHDS Motor1	3.8 (0.4)	3.5 (0.8)	−0.53	0.006
IHDS Speed1	3.9 (0.3)	3.4 (1.1)	−0.67	<0.001
IHDS Recall1	3.8 (0.6)	3.6 (0.8)	−0.33	0.096

¹mean (SD);

²Groups differences presented as Cohen’s d effect sizes.

Note: ANI = Asymptomatic Neurocognitive Impairment; MND = Minor Neurocognitive Disorder; HAD = HIV-associated dementia.

IHDS classification accuracy

Based on the IHDS standard raw cutpoint of ≤10, 32% (n=19) of HIV+ participants were classified as impaired. A serial examination of cut-offs on the IHDS indicated that 11.5 was the cutpoint with the best balance between sensitivity and specificity, this means that the participant lost 0.5 point because needed clues to remember a word. Using the cutpoint 11.5, 60% (n=36) of HIV+ participants were classified as impaired. Figure 1 shows the ROC curve with the 10 and 11.5 cutpoints in the HIV+ sample.

Figure 1. Receiver operating characteristic (ROC) curve showing the 10 and 11.5 cutpoints in the HIV+ group.

In the HIV+ group (n=60), Sensitivity and specificity of the standard raw cutpoint (≤10) were 36% and 75%. Using the cutpoint of ≤11.5 improved the sensitivity to 72% but decreased the specificity to 58%. The positive predictive value (PPV) increased slightly from 68% (≤10) to 72% (≤11.5) and the negative predictive value (NPV) increased from 44% (≤10) to 58% (≤11.5). The overall accuracy for the standard cutpoint (≤10) was 52% while the 11.5 cutpoint provided an overall accuracy of 67%. Youden’s index (J) increased from 0.11 (≤10) to 0.30 (≤11.5); Youden’s index is the vertical distance between the ROC curve and the first bisector (or chance line).

Considering only the participants HIV+ with HAD (n=5) and using 11.5 cut-off, the sensitivity of IHDS was 40% (2 out of 5 got values below 11.5); while three (60%) got a perfect score of 12, but it is a very small number of subjects limiting the ability to test the accuracy.

IHDS associations with demographics and GDS

There was a negative linear correlation between the raw IHDS total score and GDS (ρ = −.43 p<.001) in the HIV− and HIV+ participants. In the HIV+ group, the raw IHDS total score was significantly associated with education (ρ=.47, p<.001) but not with age or sex (p’s>.3).

Three test screening battery

The 19 neuropsychological measures in the gold standard battery yielded 969 three-test combinations. Table 3 presents the top five combinations ranked on the basis of highest superiority index along with the PPV, NPV, and overall accuracy corresponding to each test combination. The combinations were reviewed to identify tests deemed feasible to administer at the bedside or during a clinician visit. Also shown is the proportion of HIV+ participants classified as impaired using each 3-test combination (based on an average deficit score ≥ .5). Figure 2 shows the ROC curves for these batteries in HIV+ participants.

Table 3.

Classification accuracy of the top five combinations based on highest superiority index.

Test combination	Superiority Index	Sensitivity	Specificity	PPV	NPV	Youden’s index (J)	Overall accuracy	Proportion classified as impaired
Trails A, WAIS-III DS, HVLT Total Recall	1885.0	0.91	0.96	0.97	0.88	0.87	0.93	0.54
WAIS-III DS, BVMT Total, Pegboard NDH	1639.0	0.94	0.91	0.94	0.91	0.85	0.93	0.61
Trails A, WAIS-III DS, Pegboard DH	842.0	0.85	1.0	1.0	0.83	0.85	0.91	0.49
Trails A, WAIS-III DS, Color Trails 2	842.0	0.85	1.0	1.0	0.83	0.85	0.91	0.49
Trails A, HVLT Total, Pegboard NDH	593.7	0.91	0.92	0.94	0.88	0.83	0.91	0.56

Tests: Brief Visuospatial Memory Test-Revised (BVMT) Total Recall (i.e., learning score), Color Trails Test 2, Hopkins Verbal Learning Test-Revised (HVLT) Total Recall (i.e., learning score), Grooved Pegboard Test – Dominant Hand (DH) and Nondominant Hand (NDH), Trail Making Test A, Wechsler Adult Intelligence Scale-III (WAIS-III) Digit Symbol (DS).

Figure 2.

Receiver operating characteristic (ROC) curves for the 5 best short test batteries in the HIV+ sample.

Discussion

There is a need for valid screening measures in clinical settings to facilitate identification of patients with cognitive impairment. The IHDS is a screening tool specifically designed to detect HIV-associated dementia. This measure is easy to use and has minimal time and material demands; however, there are concerns regarding changes in its psychometric properties across cultures¹¹ and low sensitivity in the context of milder forms of HAND, which are by far the more common manifestation in the era of cART. Consistent with the findings of Rodrigues et al (2013)⁹, who examined the performance of the IHDS in a cohort from a different region of Brazil, we found that this measure had marginal sensitivity that improved when a higher cutpoint was used. In our study, this higher cutpoint was 11.5 out of 12; i.e. anything less than a perfect score was indicative of impairment. Furthermore, we found that the total IHDS score was highly influenced by years of education. This highlights the need to factor years of education into the interpretation of IHDS scores, as not doing so will likely seriously impact the classification accuracy of this measure. Together, these data suggest that the IHDS has limited utility as a screening measure and is not appropriate as a stand-alone diagnostic tool for HAND.

As an alternative to the IHDS, we identified the top five most accurate three-test screening batteries, composed of commonly used and validated NP measures and using demographically corrected scores based on a Brazilian sample. These batteries were found to classify HIV+ participants as impaired or unimpaired with very good accuracy, based on the gold standard of a larger comprehensive neurocognitive battery. Specifically, the combination of Trails A, WAIS-III Digit Symbol, and HVLT-R total recall (i.e., learning score) demonstrated the highest diagnostic accuracy, followed by the WAIS-III Digit Symbol, BVMT-R total recall (learning score), and Pegboard dominant hand (Table 3). These two combinations also provided excellent sensitivity (91% and 94% respectively) and specificity (96% and 91% respectively). When the goal of an instrument is to identify as many potentially impaired individuals for follow up comprehensive testing, the negative predictive power is key. If a test has low negative predictive power, “unimpaired” test performances are more likely to be produced by truly impaired individuals (false negatives), who would consequently be overlooked for follow-up. Compared to the IHDS (cutpoint = 10; NPV = 44% or cutpoint 11.5; NPV = 58%), both three-test combinations also demonstrated superior negative predictive power (89% and 91% respectively). The top five test combinations have varying degrees of material demands but can all be administered within 10 minutes. As such, these five test combinations are likely suitable for rapid administration in the clinic. The clinician will need to balance the ease of administration (e.g., the combination of Trails A, WAIS-III Digit Symbol, Color Trails 2 has the least time and instrument demands but mostly tap psychomotor speed) with the desire to sample more ability domains (e.g., WAIS-III DS, BVMT-R Total, Pegboard NDH). The main disadvantage of this method is that results cannot be obtained immediately, unless a suitable computer program is available; raw scores have to be subjected to demographic adjustments (i.e., age, education and sex) prior to interpretation. However, taking into account demographic effects substantially improves the level of confidence that the identified impairment is due to the disorder and not confounding demographic characteristics¹³. The formulas for applying demographic corrections based on a group of healthy participants from southern Brazil are provided in the supplemental appendix (see supplemental digital content). Concerning the practicality, several of the 3-test alternatives require forms purchased from publishers; they also require translation, modification and validation to the specific population; for best results these tests require for testing environments free of distraction. The last three requirements are also necessary for the IHDS.

In diverse cohorts across the world, there is ample evidence supporting the presence of HIV-associated impairments in learning, motor ability, and processing speed^17,18,20,24 ; our data are consistent with these findings. It should be noted that 20% of our sample had HCV; the impairment in these HIV-infected individuals may be attributable in part to HCV²⁵. Overall, similar to the results of Carey and colleagues¹² and Moore et al.,²⁷ we observed that performance on measures of learning and psychomotor speed was particularly sensitive to HIV-associated NP impairment in this Brazilian cohort. Our procedure for extracting the most sensitive combination of batteries yielded tests those were very similar to the ones extracted from US studies, suggesting a robustness of this approach and its capacity to detect impairment.

The present cross-sectional study based on retrospective data is not without limitations. First, we had a small sample of HIV+ participants and few individuals had MND and HAD, thus limiting our ability to test the accuracy with which the IHDS and brief batteries identified those with more severe impactful impairments that are likely to be easier to detect; ANI is arguably more difficult to detect. The proportion of ANI in our sample was somewhat higher than would be expected based on U.S. cohorts such as the CHARTER¹⁸ (i.e., 48% vs 33%) This may be attributable to possible self-selection of higher functioning study participants, greater proportion of individuals with limited insight about their neurocognitive abilities, or greater proportion of individuals whose everyday functioning demands were not perceptively affected by their impairments. Again, given the sample size, these proportions of different HAND diagnoses would require replication. Additionally, normative corrections for the NP tests were developed on only 48 HIV− participants. While some basis for normative corrections is better than none, a larger representative sample of healthy participants would yield more robust estimates of demographic effects on NP performance in Brazil. Future studies with larger sample sizes should validate the brief screening batteries presented here. Finally, we utilized translated and adapted versions of cognitive tests, many of which are yet to be validated in Brazil. The extent to which this alters these findings ought to be addressed in future research.

To conclude, this study provides further evidence regarding the limited utility of the IHDS as a screening tool in HIV-infected Brazilian cohorts. As an alternative, we provide an initial validation of brief screening batteries made up of three tests each. The combinations presented have substantially superior classification accuracy compared to the IHDS. These findings may aid clinical and research assessments of cognitive functioning in HIV-infected Brazilians by allowing the provider/researcher to select a screening battery depending on the setting and testing resources available.