Limitations of the International HIV Dementia Scale in the Current Era

Abstract

Objective:

The International HIV Dementia Scale (IHDS) was developed as a tool to detect HIV-dementia in both industrialized and resource-limited settings. Studies employing the IHDS have produced mixed results, with recent data suggesting unusually high rates of dementia among Ugandans. This study aimed to define the performance characteristics of the IHDS in three African countries.

Design:

Cross-sectional study.

Methods:

We recruited 2208 HIV-infected and 429 HIV-uninfected individuals from East Africa (Kenya n=1384; Tanzania n=368; Uganda n=456) who underwent testing with the IHDS and a 30-minute neuropsychological testing battery. Cognitive impairment was defined as −1SD on two of 6 tests or −2SD on one test compared to demographically matched controls stratified by age and education. We examined predictive capacity of the IHDS to detect cognitive impairment using receiver operator characteristic (ROC) curve analysis.

Results:

The mean (SD) ages of the HIV-infected and HIV-uninfected groups were 39.7 (10.7) and 37.4 (10.4), respectively. Among HIV-infected individuals, 1508 (68%) were on combination antiretroviral therapy (cART), 1298 (61%) had plasma viral load <500 copies/ml and 884 (38%) met criteria for cognitive impairment. Using the customary IHDS cut-off of 10, 1136 (83%) of the HIV-infected participants met criteria for dementia resulting in 91% sensitivity but only 17% specificity. A modified cut-off score of 8 derived from the ROC resulted in low sensitivity (56%) and specificity (64%).

Conclusions:

The IHDS has poor performance characteristics for the identification of cognitive impairment in East Africa. Cultural-informed and sensitive screening tests are needed to detect mild cognitive dysfunctions in developing countries.

INTRODUCTION

Global statistics estimate 36.7 million people living with HIV, of which 19.1 million live in Eastern and Southern Africa [1]. Neurological complications affecting the central nervous system (CNS) are common in the context of HIV; however, the vast majority of studies on HIV-associated neurocognitive disorders (HAND) are performed in developed countries [2]. Published work demonstrates worse neuropsychological performance in HIV-infected individuals living in Sub-Saharan Africa compared to their HIV-uninfected counterparts [3–6]. In a study of HIV-infected Ugandans, approximately one third met criteria for HIV-associated dementia (HAD), the most severe form of HAND [7]. In the same study older age and lower CD4 cell count were associated with increased risk for HAD [7].

Neuropsychological evaluation is a critical component of HAND diagnosis, but it is time consuming and not readily available in resource-limited regions. Thus, HIV-related cognitive impairment often remains unrecognized given the lack of specialized personnel and diagnostic cognitive measures [8]. Brief and sensitive screening tools are needed to inform the cognitive status of HIV-infected patients living in resource-limited regions.

The HIV Dementia Scale (HDS) was initially developed as a brief screening tool to detect HAD. The screening measure includes four subtests: timed written alphabet to assess psychomotor speed, recall of four items for memory, cube copy time to evaluate construction, and anti-saccadic errors for executive functioning [9]. While the HDS was designed to detect the subcortical aspects of dementia typical of advanced HIV infection, more severe stages of cognitive impairment are less frequent in the era of combination antiretroviral therapy (cART), and milder stages of disease may require more sensitive measures. Many studies now document a higher frequency of global involvement beyond subcortical structures in HAND, thus accounting for a cognitive phenotype characterized by not only slowing psychomotor speed but also executive, planning, or multitasking dysfunction [10,11].

In 2005 a new cross-cultural screening test was developed, termed the International HIV Dementia Scale (IHDS), as a tool for both the industrialized and developing world [12]. This adaptation from the HDS eliminated items believed to be overly complicated for non-specialists to administer (i.e., antisaccadic error test) and items that were sensitive to cultural variance (alphabet writing and cube copy) [13,14]. The IHDS removed the anti-saccades test and replaced the alphabet writing and the constructional task with tests of motor and psychomotor speed, which can be properly executed across different cultures but also added emphasis on these aspects of HIV-related cognitive impairment. As initially described, a cut-point of 10 on this 12-point scale provided a sensitivity of 80% and specificity of 55% to detect cognitive impairment in 81 HIV-infected individuals in Uganda [12].

A more recent study that included 618 HIV-infected Ugandans, most of whom were on cART (75%), identified a high rate of “probable” HIV dementia (64%) using the IHDS, raising concerns for its utility in the current era [15]. By employing a comprehensive neuropsychological testing battery in Uganda and South Africa, other studies reported 31% and 25% prevalence of HAD, respectively [3,7]. Here, we investigated the psychometric characteristics of IHDS to detect neuropsychological abnormalities within the African Cohort Study (AFRICOS, n=2637). We hypothesized that the sensitivity and specificity of IHDS to detect cognitive dysfunction would be suboptimal and lead to an unacceptable high risk for false positives for utility as a screening tool.

METHODS

Study population

AFRICOS is a prospective cohort study that aims to enroll 3000 HIV-infected and 600 HIV-uninfected individuals at US Military HIV Research Program (MHRP) President’s Emergency Plan for AIDS Relief (PEPFAR)-associated clinical sites in Kenya, Tanzania, Uganda, and Nigeria. Its primary objective is to longitudinally assess the impact of clinical practices, biological factors and socio-behavioral issues on HIV infection and disease progression in an African context. This work has implications for HIV clinical care and treatment as well as exploration of the pathogenesis of HIV and its comorbidities in Africa. Inclusion criteria for HIV-infected adults included known HIV infection, age 18 years or older, signed informed consent, intent to be a long-term residency in the area, receipt of HIV care in a clinic and ability to provide contact information. Exclusion criteria included any significant condition that, in the opinion of the study investigators, would interfere with study conduct. The same inclusion and exclusion criteria were applied to the HIV-uninfected participants but they were required to consent to HIV testing and pre- and post-test counseling. Given that Nigerians (n=281) reported substantially more education compared to the other East African countries leading to challenges in creating comparative data among controls and since the IHDS was initially described in Uganda, we restricted our analyses to Uganda, Kenya and Tanzania.

Cognitive Assessment

All participants underwent cognitive screening with the IHDS followed by a 30-minute neuropsychological testing battery. The IHDS includes three sub-items: timed finger tapping, timed alternating hand sequence test and a 2-minute delayed recall of four words. Testers were trained on all cognitive tests, certified and re-certified every 6 months to assure consistent testing across all sites. Certifications were completed by trained investigators in person or by video. Specific instructions for the IHDS included the size (first finger and thumb wide open to fully closed) and speed of finger tapping as well as instructions to demonstrate exactly two full alternating hand sequences before timed testing. The cognitive battery consisted of World Health Organization (WHO) auditory verbal learning test (AVLT) trial 1 to assess attention, AVLT sum of trials 1–5 for learning efficiency and AVLT delayed recall for memory; Trails A to assess psychomotor speed; Grooved Pegboard to measure manual dexterity as well as psychomotor speed; and action fluency to assess verbal performance. Four study participants did not complete the AVLT test, 44 the Trails A, most due to illiteracy for numbers, 5 the Grooved Pegboard and 4 the action fluency task.

Comparative data were acquired from 429 co-enrolled HIV-uninfected individuals who also sought care at the same PEPFAR facilities and stratified to four levels of age (18–25; 26–34; 35–44; 45+) and two levels of education (primary or less; secondary or above). Since we did not find a significant effect of sex on neuropsychological performance, these comparative data tables were not stratified by sex. Cognitive impairment was defined as at least −1 SD below the mean on two measures or −2 SD below the age and education matched mean from the local comparative data on at least one test. Participants also completed the Center for Epidemiological Studies-Depression (CES-D) scale to examine mood [16]. Clinically significant depression was defined as a CES-D score ≥16 [17].

Statistical Analysis

We used one-way analysis of variance for continuous variables and chi-squared tests for categorical variables to compare demographic/clinical factors and neuropsychological testing performance between the HIV-infected and HIV-uninfected groups.

We calculated sensitivity, specificity and overall receiver-operator characteristic (ROC) area under the curve for the IHDS’s ability to predict cognitive impairment. Additionally, two sets of sensitivity analyses were completed. First, we repeated the statistical analyses restricting the sample to HIV-infected participants with performance −2 SD below the mean on two tests to examine the accuracy of the IHDS to detect more pronounced cognitive dysfunction since the IHDS was designed as a screening tool for HIV-related dementia [12]. Given that the IHDS was initially validated in Uganda, we limited the second sensitivity model to Ugandan participants, thus investigating whether cultural differences across African regions might lead to heterogeneous performance characteristics of the IHDS, influencing our analyses. All two-tailed p-values were considered significant at a threshold of p<0.05. Analyses were performed using SPSS 22.0 (SPSS Inc., Chicago, IL, USA) and STATA 15 (STATA: Release 15, College Station, TX: StataCorp LP).

RESULTS

Participants

We enrolled 2208 HIV-infected participants from Kenya (n=1384), Tanzania (n=368) and Uganda (n=456) and 429 HIV-uninfected individuals from Kenya (n=267), Tanzania (n=63) and Uganda (n=99; Table 1). Differences were noted between the infected and uninfected groups including the HIV-infected being slightly older (39.7 vs. 37.4 years; p<0.001) and less often literate (90% vs. 93%, p=0.033). The frequency of hepatitis C (HCV) antibody was lower in the HIV-infected group (2%, n=41 vs. 4%, n=15, p=0.035). Just over half of the participants (58%) were female.

Table 1.

Demographic and clinical characteristics of study participants

Characteristics	HIV+ n (%)	HIV− n (%)	Chi-square or T-test	P-value
N by country	N=2208	N=429	1.88	0.391
• Kenya	1384	267
• Tanzania	368	63
• Uganda	456	99
Age, mean (SD; range)	39.7 (10.7; 18–76)	37.4 (10.4; 18–69)		<0.001
Sex, female	1288 (58%)	246 (57%)	0.14	0.703
Education			2.33	0.127
• Primary or less	1418 (64%)	259 (60%)
• Secondary or above	789 (36%)	170 (40%)
Ability to read and write	1979 (90%)	399 (93%)	4.53	0.033
CES-D, mean (SD)	16.1 (6.4)	15.6 (7.1)	−1.46	0.145
Currently Employed	795 (36%)	161 (38%)	0.353	0.552
Recreational drugs	13 (3%)	64 (3%)	0.024	0.877
On cART	1508 (68%)	-	-
Estimated years on cART, mean (SD)	3.1 (2.8)	-	-
Hepatitis C Antibody (% positive)	40 (2%)	15 (4%)	4.46	0.035
Syphilis	117 (6%)	19 (5%)	0.62	0.432
Undetectable viral load (HIV RNA <500 copies/ml)	1298 (61%)	-	-
CD4 count, mean (SD)	437 (235)	-	-
Log10 plasma HIV RNA, mean (SD)	2.2 (2.1)	-	-
Estimated years of HIV infection, mean (SD)	2.9 (3.0)	-	-
WHO stage
• Stage 1	813 (38%)	-	-
• Stage 2	721 (34%)	-	-
• Stage 3	511 (24%)	-	-
• Stage 4	90 (4%)	-	-
Cognitive status
• Cognitive impairment*	844 (38%)	113 (26%)	21.73	<0.001
Neuropsychological performance
• Neuropsychological composite score, mean (SD)**	−0.20 (0.63)	0.01 (0.58)	6.28	<0.001
• Attention	−0.09 (0.99)	−0.00 (0.99)	1.56	0.118
• Learning	−0.23 (1.07)	−0.00 (0.99)	3.99	<0.001
• Memory	−0.18 (1.05)	−0.00 (0.99)	3.20	<0.01
• Psychomotor speed	−0.26 (1.08)	0.01 (0.97)	4.81	<0.001
• Manual Dexterity	−0.22 (1.06)	0.02 (0.93)	4.38	<0.001
• Language	−0.23 (0.93)	−0.00 (0.98)	4.72	<0.001

Abbreviation: cART: combination Antiretroviral Therapy; WHO: World Health Organization; CES-D=Center for Epidemiological Studies Depression Scale; IHDS: International Dementia Scale; Syphilis=Serum RPR or VDRL, n=209 missing values.

^*Defined as −1 SD or −2 SD below the mean on two or one test, respectively.

^**Calculated by averaging the z-scores on the individual tests.

Notes: P-values that reached statistical significance (p<0.05) are shown in bold

Among the HIV-infected participants, 1508 (68%) were on cART, 1298 (61%) had plasma viral load <500 copies/ml, and the mean (SD) CD4 t-lymphocyte count was 437 (235) cells/ml. In the subgroup of individuals on cART, 1237 (85%) had undetectable plasma HIV RNA defined as HIV RNA <500 copies/ml and had a mean (SD) CD4 cell count of 442 (229) copies/ml. The mean scores on the CES-D did not differ by group; however, the frequency of meeting the threshold for clinically significant depression was higher among HIV-infected individuals compared to the uninfected group (46% vs. 40%, p=0.022).

Accuracy of IHDS to predict poor neuropsychological performance

The frequency of meeting our definition of cognitive impairment was 38% (n=844) in the HIV-infected group and 26% (n=113) in the control group (Chi-squared=21.73; p<0.001). Among the HIV-infected group, the frequency of impairment was not statistically different between those on cART (38%, n=573) versus those not (39%, n=271; Chi-squared=0.12; p=0.728).

The standard IHDS cut-off of 10 led to 91% sensitivity and 17% specificity. By applying this cut-off in our cohort, 83% (n=1136) HIV-infected participants met IHDS criteria for HIV-related dementia. ROC curve analyses (AUC: 0.63; 95% CI: 0.61–0.66; p-value <0.0001) revealed no acceptable levels of both sensitivity and specificity of the IHDS to detect cognitive impairment (Figure 1; Table 2). Based on our analyses, we estimate that for every 1 case of impairment identified, we would concurrently have 1.18 false positive cases.

Figure 1. ROC curve for IHDS to detect neuropsychological abnormalities among 2208 HIV- infected African adults.

Area under the curve (AUC) was 0.63 (95% CI: 0.61–0.66; p-value <0.0001).

Table 2.

IHDS performance characteristics by cut-point

	≤10	≤9	≤8	≤7
Sensitivity	91%	73%	56%	36%
Specificity	17%	41%	64%	83%
PPV	40%	44%	49%	57%
NPV	75%	71%	70%	68%
Youden’s Index	0.08	0.14	0.20	0.19

Abbreviations: PPV: positive predictive value; NPV: negative predictive value.

In sensitivity analyses, we tested performance characteristic of a more severe definition of cognitive impairment. One hundred twenty eight HIV-individuals (6%) scored at least −2 SD below the HIV-uninfected group on 2 tests. In this sub-group of more impaired individuals, a cut-off of 10 on the IHDS resulted in a sensitivity of 95% and specificity of 14% in predicting cognitive impairment. An ROC analysis noted a suboptimal cut-point of 8 where 69% sensitivity and 58% specificity was achieved.

Restricting the analyses to the Ugandan cohort, the results were effectively unchanged. Specifically, the standard cut-off of 10 yielded a sensitivity of 95% and a specificity of 10%. Stated differently, 90% of individuals identified as impaired on the IHDS were not impaired on the larger cognitive battery (i.e., false positive errors).

DISCUSSION

The current study investigated the performance characteristics of the IHDS in predicting cognitive impairment in a large sample of HIV-infected Africans. Our findings revealed a high false positive rate when impairment was determined by the IHDS, suggesting that recent reports of high dementia frequencies defined by IHDS scores among HIV-infected individuals in Uganda are likely influenced by false positive errors.

We observed cognitive impairment in 38% of HIV-infected individuals independent of treatment status. The prevalence of HAND in resource-limited countries remains less well established [2]. Based on IHDS scores, previous studies estimated a substantially higher prevalence of cognitive impairment in HIV-infected patients living in Uganda [15,18]. Another study from South Africa observed a 42% rate of Mild Neurocognitive Disorder (MND) and 25% rate of HIV-associated Dementia (HAD) when applying a comprehensive neuropsychological testing battery [2,3]. Education and gender, predicted HAD, but current CD4 cell count was unrelated to the presence of HAD [3]. Similar results were reported in a separate cohort from Kenya, with MND reported in 47% of individuals and HAD in 20% [8]. Finally, application of a cut-off score of 9.5 or less on the IHDS, revealed HAD in 38% among HIV-infected study participants in Botswana, with increased age, lower education, and longer disease duration correlating to dementia status [4].

In a prospective study among cART-naïve HIV-infected Ugandans, 49% met criteria for HAND; however, the prevalence decreased to 15% after 2 years on cART [19]. Multiple factors such as mixed or low degrees of viral suppression, discordant classifications of cognitive impairment and different literacy levels might contribute to the noted discrepancy in HAND prevalence across studies. Furthermore, certain viral clades have been speculated to have more or less neuropathogenic qualities compared to others, though consistent clinical support for clade differences remains limited [20,21]. Higher rates of HAND in resource-limited settings might also be attributed to later access to cART compared to developed countries where guidelines recommend to begin cART as early as possible and regardless of CD4 cell count [3,6,11,22].

In our study, the IHDS demonstrated high sensitivity, but unacceptably low specificity for a screening tool, with 83% of the HIV-infected subjects in our study classified as cognitively impaired and 77% of the HIV-uninfected participants when applying the standard cut-off of 10. Ideally, an optimal cognitive screening test should have a high sensitivity and specificity, particularly in developing countries where the capacity to complete a more comprehensive neuropsychological assessment is limited. Therefore, a screening tool with a high sensitivity but weak specificity risks to overdiagnose cognitive impairment in clinical settings where cognitive status cannot be further evaluated. In our study, restricting the analyses to participants with more severe forms of neuropsychological abnormalities did not improve the IHDS accuracy, nor did limiting the analyses to Ugandan participants, where the tool was initially validated. The results of these sensitivity analyses suggest that the suboptimal psychometric properties of the IHDS observed in the current study are not likely due to obvious methodological factors.

Our results mirror those of a recent study conducted in a non-African setting that investigated the prevalence of cognitive deficits in a chronically well-treated HIV-infected cohort (the Dutch TREVI Cohort) with a large proportion of individuals having achieved viral suppression [23]. Investigators found that IHDS had suboptimal psychometric characteristics when applied alone or in combination with an adjunctive brief screening evaluation, leading the authors to recommend caution when evaluating patients with such measures [23]. More recently, a study found that the Cognitive Assessment Tool-rapid version (CAT-rapid) in association with the IHDS showed good performance for detecting HAD, probably due to the inclusion of measures of psychomotor speed, memory and executive functions [24]. In another study symptomatic forms of HAND were accurately predicted by a computerized assessment (CogState) that tapped sustained attention, processing speed, attention, working memory, verbal learning and memory [25,26]. While the gold standard for detecting HAND is a detailed neuropsychological testing battery, preferably tapping the cognitive areas known to be primarily affected in HIV, sensitive and cross-cultural cognitive screening tools are needed in resource-limited regions. Despite limitations of most of the screening tools designed to detect HAND, in clinical settings cognitive screening may help HIV physicians identify patients at higher risk of HAND and detect its early clinical manifestations [27,28].

Our study benefits from a large sample size and the co-enrollment of demographically similar seronegative individuals recruited at the same sites to inform normative comparisons. In the general population, 15% to 25% of individuals would be expected to perform poorly (−1 or −2 SD below the mean) on at least one measure in a six-test battery [29]. This is consistent with the 26% abnormality rate that we found among the uninfected participants, thus giving us confidence that these matched controls could serve as normative for the region and in the rate of cognitive impairment seen in the HIV-infected group (38%). Variability in neuropsychological performance is expected in the normal population, pointing to each individual’s relative cognitive strengths and weaknesses rather than the presence of acquired brain dysfunction [30,31]. Our study also highlights the importance of formulating normative data from groups of individuals that reflect the broad demographic and socioeconomic characteristics of the population under study. The inclusion of matched local controls is doubly beneficial since it might also lessen the impact of cultural and language challenges in neuropsychological testing. Further studies are needed to investigate the prevalence of cognitive impairment among HIV-infected individuals on cART with viral suppression in African settings.

This study is not without limitations. The HIV-uninfected group included individuals with clinically significant depressive symptomatology (n=168, 40%), major traumatic brain injury with loss of consciousness less than one hour (n=16) or greater than one hour (n=6; N=22, 5%) and current alcohol use (i.e. more than 3 drinks per day, n=36, 34%). Careful examination of these factors revealed minimal impact on cognitive measures; thus, these individuals were retained in our analyses. Moreover, except for mood status, none of these potentially confounding factors statistically differed between the two groups. Although we controlled for illiteracy and country of origin, it is possible that differences in methodological approach resulted in a lower rate of dementia in the present study. For example, psychometrists were instructed to complete the finger-tapping sub-test by having participants start with the thumb and the index finger in an “L” position, and to make large amplitude taps as fast as possible. This approach was not explicitly defined in the published instruction for administration of the IHDS, and it is possible that our approach restricted the window to detect impairment by reducing the number of total correct responses. However, lowering the cut-off to 8, which should mitigate any discrepancy, did not alter the core findings.

In conclusion, our findings provide evidence that the IHDS is not sufficient to use as a screening tool to identify cognitive impairment among HIV-infected individuals in East Africa in the cART era. The clinical interpretation of the cognitive performance on the IHDS should be cautious across all severity levels of HAND. Further efforts are needed to identify cultural-informed and sensitive screening tests to detect mild cognitive dysfunctions in resource-limited settings.