Predictive Validity of Demographically-Adjusted Normative Standards for the HIV Dementia Scale

Abstract

The aim of the current study was to develop and validate demographically-adjusted normative standards for the HIV Dementia Scale (HDS). Given the association between demographic variables and the HDS summary score, demographically-adjusted normative standards may enhance the classification accuracy of the HDS. Demographically-adjusted normative standards were derived from a sample of 182 seronegative healthy participants and were subsequently applied to a sample of 135 HIV-1 seropositive individuals with multidisciplinary case conference diagnoses of HIV-1-associated neurocognitive disorders (e.g., HIV-1-associated dementia and Minor-Cognitive/Motor Disorder) in proportions consistent with published epidemiologic reports. In the normative sample, age and education (and their interaction) emerged as the only demographic factors significantly associated with the HDS. In comparison to the traditional HDS cut score (raw score total ≤10), use of the demographically-adjusted normative standards significantly improved the sensitivity (from 17.2% to 70.7%, respectively) and overall classification accuracy (increasing the odds ratio from 3 to approximately 6) of the HDS for identifying participants with HIV-1-associated neurocognitive disorders. The application of demographically-adjusted normative standards on the HDS improves the clinical applicability and accuracy of this cognitive screening measure in the detection of HIV-1-associated neurocognitive disorders.

Introduction

The HIV Dementia Scale (HDS) was specifically developed as a brief screening instrument for the detection of dementia in HIV-1 seropositive individuals (Power, Selnes, Grim, & McArthur, 1995). Despite a decrease in the incidence of HIV-1-associated dementia (HAD) resulting from the introduction of highly active antiretroviral therapies (HAART), the prevalence of HAD and milder forms of HIV-1-associated neurocognitive impairment remains high (McArthur, 2004). Therefore, accurate assessment and detection of HIV-1-associated neurocognitive disorders continues to be an important undertaking. Although ideal, a comprehensive neuropsychological evaluation is not feasible in many clinical and research settings, resulting in a need for validating brief cognitive screening measures requiring minimal time and instrumentation that can reliably identify individuals in need of additional assessment (Woods & Grant, 2005).

Common standardized screening tests such as the Mini-Mental State Examimation (MMSE; Folstein, Folstein, & McHugh, 1975) are not particularly useful as screening tools for HIV-1-associated cognitive disorders. These tests were developed to target cognitive abilities that are commonly impaired in dementias with prominent posterior neocortical pathophysiology (e.g., naming), whereas HIV-related neurocognitive disorders more often involve cognitive dysfunction mediated by fronto-striatal circuits (e.g., processing speed). To address these limitations, the HDS was designed with four subtests targeting abilities that are often impaired in persons with HAD (i.e., attention, episodic memory, and psychomotor speed). The items that comprise the subtests include a measure of antisaccadic errors, recall of four items after a delay, and two timed measures: written alphabet and cube copy (Power et al., 1995). The HDS summary score is commonly compared to a raw cut score of ≤ 10 (HDS range = 0–16) to determine impairment. Using this convention, the HDS has demonstrated superior sensitivity to HAD when compared to the MMSE (Power et al., 1995). The HDS was also found to have comparable sensitivity to the Executive Interview (EXIT; Royall, Mahurin, & Gray, 1992) and the Grooved Pegboard test (GP; Klove, 1963), an established marker of HIV-1-associated neuropsychological impairment (Berguis, Uldall, & Lalonde, 1999; Power et al., 1995).

Despite findings in the literature supporting the use of the HDS in lieu of other cognitive screening measures, recent reports have suggested the HDS is not adequately sensitive to HIV-related cognitive impairment (Carey et al., 2004; Smith, van Gorp, Ryan, Ferrando, & Rabkin, 2003; Woods & Grant, 1995). More specifically, use of the current raw cut score of ≤ 10 has consistently yielded a high rate of false negatives (Berguis et al., 1999; Carey et al., 2004; Smith et al., 2003) calling into question the utility of the HDS in detecting milder forms of HIV-1-associated cognitive impairment. Importantly, research has shown an association between HDS performance and demographics, most notably age and education (Berguis et al., 1999, von Giesen, Haslinger, Rohe, Koller, & Arendt, 2005), yet demographic variables are not explicitly factored into the interpretation of HDS scores. Thus it is conceivable that psychometric consideration of demographic factors might enhance the classification accuracy of the HDS and increase sensitivity, thereby improving the ability of the HDS to detect varying degrees of neuropsychological impairment (Heaton, Miller, Taylor, & Grant, 2004), including subsyndromic deficits. For example, in the case of highly educated individuals experiencing mild cognitive decline, interpretation of individual scores relative to the performance of a demographically similar cohort could ameliorate ceiling effects that potentially result in misclassification of these individuals as neurocognitively normal when a raw cut score is used. In other words, an HDS raw score of 12 in a 35-year-old woman with 18 years of education likely reflects a decline from premorbid abilities, but would be inaccurately classified as normal under the current classification system (i.e., a false negative error). Although the detection of impaired individuals (i.e., sensitivity) is of primary importance for a screening measure, the application of demographic adjustments might also improve the specificity of the HDS, particularly for individuals at high risk for producing false positive test scores (e.g., older adults with limited education). The aims of the current project were therefore to develop and validate demographically-adjusted normative standards for the HDS. It was hypothesized that the predictive validity of the HDS using demographically-adjusted T-scores would be superior to the traditional ≤10 cut point in classifying HIV-1-associated neurocognitive disorders.

Method

Participants and Procedure

Two study samples were comprised of individuals enrolled in various longitudinal neuroAIDS studies at the HIV Neurobehavioral Research Center (HNRC) of the University of California at San Diego. All studies were approved by the human research protections program, and all participants provided written, informed consent. Potential study participants were excluded if they reported histories of non-HIV-related neurological (e.g., head injury with loss of consciousness greater than 30 minutes, seizure disorder, intracranial neoplasm, cerebrovascular disease) or psychiatric (e.g., psychosis, current substance dependence, or mental retardation) illness that might adversely affect cognitive functioning, or if a urine toxicology screen conducted on the day of testing was positive for illicit drugs. All participants were administered the HDS, which takes approximately 5–10 minutes to complete, by a clinical research nurse as part of a comprehensive evaluation of cognitive, neuromedical, and psychiatric status. The four subtests of the HDS include antisaccadic errors (attention; range 0–4), timed written alphabet (psychomotor speed; range 0–6), recall of four items (memory; range 0–4) and cube copy time (construction; range 0–2). The subtest scores are summed to create a summary score, with a range of 0 to 16.

The normative study sample was comprised of 182 seronegative healthy comparison (HC) participants. On average, participants in this sample were 36.6 years of age (SD = 12.0) with 13.8 years of education (SD = 2.5). The sample was 73% Caucasian and 68% male. In order to evaluate the predictive validity of the demographically-adjusted normative data derived from the HC sample, we also examined HDS performance in a group of 135 individuals with HIV-1 infection, as determined by enzyme linked immunosorbent assays and a Western Blot confirmatory test. Consensus diagnoses of HIV-1-associated neurocognitive disorders were assigned for each participant according to modified American Academy of Neurology (AIDS Task Force, 1991) and Grant and Atkinson (Grant & Atkinson, 1995) criteria, based upon comprehensive and standardized neurological, neuropsychological, and psychiatric examinations. Recent data support the interrater reliability (Woods et al., 2004) and construct validity (Moore et al., 2006) of this nosology. One hundred and thirty-five HIV-1-infected participants were randomly selected from the HNRC cohort in proportions that approximate published epidemiological reviews of HIV-1-associated neurocognitive disorders (McArthur, 2004; Woods & Grant, 2005); i.e., 77 (57%) neurocognitively normal individuals, 22 individuals (16%) with asymptomatic neurocognitive impairment (ANI), 22 individuals (16%) with Minor Cognitive-Motor Disorder (MCMD), and 15 individuals (10%) with HAD. Table 1 displays the demographic and HIV disease characteristics of the validation sample.

Table 1.

Demographic Characteristics of HIV Validation Sample

Variable	NP Normal (n = 77)	Asymptomatic NP Impairment (n = 22)	MCMD (n = 22)	HAD (n = 14)	Total Sample (n = 135)
Age (years)	39.0 (8.0)	38.5 (11.2)	44.3 (7.5)	40.2 (4.4)	39.9 (8.4)
Education (years)	12.8 (2.6)	12.2 (3.2)	14.3 (2.8)	12.5 (2.3)	12.9 (2.8)
Sex (% male)	76.6	86.3	95.5	92.9	83.0
Ethnicity (% Caucasian)	59.7	63.6	50.0	92.9	62.2
Proportion with AIDS	51.9	45.5	77.3	92.9	59.3
Proportion immunosuppressed	28.6	9.1	50.0	42.9	30.4
Proportion on HAART	58.9	77.3	59.1	71.4	63.4

Note. Data represent mean (SD) unless otherwise indicated. HAART = highly active antiretroviral therapies; NP = neuropsychological; MCMD = Minor-Cognitive/Motor Disorder; HAD = HIV-1-Associated Dementia

Results

Development of Normative Standards

Table 2 shows the distribution of the HDS summary score and each individual subtest in the HC sample. When entered into a linear multiple regression model, age and education (ps < .001) were significant independent predictors of HDS total score in the HC cohort, but sex and ethnicity were not (ps > .05). A follow-up multiple regression analysis was conducted to evaluate how well age, education, and their interaction were associated with HDS total score. A significant amount of variance in HDS total score was explained, F(3, 178) = 26.17, p < .001, R² = .31. The partial regression coefficient relating age to HDS total score was statistically significant, B = −.26, p < .001, CI = −.35 to −.17, as were the partial regression coefficients relating education (B = −.43, p = .001, CI = −.69 to −.18) and their interaction (B = .02, p < .001, CI = .01 to .02) to that HDS summary score.

Table 2.

Distribution of HDS Summary Score and Subtest Scores in HC Sample

HDS Score	Minimum	Maximum	Median	IQR	M	SD
Summary	7.5	16	16	15, 16	15.33	1.36
Antisaccadic errors	0	4	4	4, 4	3.84	0.60
Written alphabet	2	6	6	6, 6	5.85	0.59
Recall	4	1	4	4, 4	3.82	0.51
Cube Copy	0	2	2	2, 2	1.82	0.49

Accordingly, age and education and their interaction were entered into the normative model. The methodology for deriving demographically-adjusted normative standards provided by Heaton and colleagues (Heaton et al., 2004) was adapted for the current study. Raw HDS summary scores were converted to scaled scores (M = 10, SD = 3). Scaled score conversions are presented in Table 3. A linear multiple regression model was conducted with age, education, and their interaction as independent variables and HDS scaled scores as the dependent variable. Predicted scaled scores were generated using the residuals from this regression procedure and demographically-adjusted T-scores for the HDS summary score can be derived using the following formula:

• T-score = (((Observed Scaled Score – Predicted Scaled Scorea)/2.6) × 10) + 50

  ^aPredicted Scaled Score = 5.68 + ({years of education × 0.43} + {age × −0.04} + {[(age − 36.64) × (years of education − 13.81)] × 0.03}).

Table 3.

Normative conversion of raw HDS summary scores to scaled scores

Raw scores	Scaled scores
≤ 9.0	1
9.5 – 10.0	2
10.5 – 12.0	3
12.5	4
13.5	5
14.5	6
15.0	7
15.5	8
16.0	12

Table 4 provides a quick reference tool for approximating T-scores from scaled scores, which are grouped by broad age and education categories. These approximated T-scores were generated from the above regression formula by using the middle values from each age and education category (e.g., 10 years of education and 23 years of age were used to derive T-scores displayed in the first column). The shaded areas represent T-scores that fall in the impaired range (i.e., T < 40).

Table 4.

Demographically-Adjusted Normative Standards (T-scores, M = 50, SD = 10)

Education: 8–12 years
Age
SS	21–25	26–30	31–35	36–40	41–45	46–50	51–55	56–60	61–65
	N=17	n=8	n=6	n=8	n=8	n=6	n=2	n=5	n=3
1	13	16	19	22	25	28	31	34	37
2	17	20	23	26	29	32	35	38	41
3	21	24	27	30	33	36	38	41	44
4	25	28	30	33	36	39	42	45	48
5	28	31	34	37	40	43	46	49	52
6	32	35	38	41	44	47	50	53	56
7	36	39	42	45	48	51	54	57	60
8	40	43	46	49	52	55	58	61	64
12	55	58	61	64	67	70	73	76	79

Education: 13–15 years
Age
SS	21–25	26–30	31–35	36–40	41–45	46–50	51–55	56–60	61–65
	n=19	n=9	n=5	n=5	n=7	n=10	n=5	n=3	n=0
1	13	13	14	15	15	16	17	17	18
2	17	17	18	19	19	20	20	21	22
3	20	21	22	22	23	24	24	25	26
4	24	25	26	26	27	28	28	29	29
5	28	29	29	30	31	31	32	33	33
6	32	33	33	34	35	35	36	37	37
7	36	36	37	38	38	39	40	40	41
8	40	40	41	42	42	43	44	44	45
12	55	56	56	57	58	58	59	60	60

Education: 16–20 years
Age
SS	21–25	26–30	31–35	36–40	41–45	46–50	51–55	56–60	61–65
	n=11	n=9	n=5	n=9	n=6	n=5	n=10	n=3	n=0
1	12	11	9	7	6	4	2	1	0
2	16	15	13	11	10	8	6	5	3
3	20	18	17	15	13	12	10	9	7
4	24	22	21	19	17	16	14	12	11
5	28	26	24	23	21	20	18	16	15
6	32	30	28	27	25	23	22	20	18
7	35	34	32	30	29	27	26	24	22
8	39	38	36	34	33	31	29	28	26
12	55	53	51	50	48	46	45	43	41

Classification Accuracy: Raw Scores vs. Demographically-Adjusted T-Scores

The traditional cut score of ≤ 10 and the demographically-adjusted T-scores were applied to the representative validation sample of HIV-1 seropositive individuals in order to compare classification accuracy for discriminating impaired (n = 58) versus neurocognitively normal (n = 77) individuals. Consistent with published interpretive recommendations (Heaton et al., 2004; Woods et al., 2004), T-scores < 40 were considered to be in the impaired range, whereas T-scores ≥ 40 were classified as falling within normal limits. The predictive validity of demographically-adjusted T-scores was also evaluated for specifically discriminating neurocognitively normal individuals versus each diagnostic group (i.e., ANI, MCMD, HAD) separately. The use of demographically-adjusted T-scores significantly improved sensitivity for discriminating impaired versus neurocognitively normal individuals in comparison to the raw cut score (sensitivity was 17.2% with the raw cut score and 70.7% with demographically-adjusted T-scores). The associated decrease in specificity was also significant but relatively smaller (specificity was 93.5% with the raw cut score and 73.7% with demographically-adjusted T-scores). The odds ratio, representing the likelihood of accurate versus inaccurate classification of impairment (i.e., the probability that an individual with an HIV-1-associated neurocognitive disorder will score in the impaired range on the HDS relative to the probability than a neurocognitively normal individual will score within normal limits), was approximately doubled with the use of demographically-adjusted normative standards, increasing from 3.0 (CI = 1.0, 8.6) to 6.4 (CI = 3.0, 13.2). A similar pattern of results was revealed when discriminating each diagnostic group from neurocognitively normal individuals separately. Findings are displayed in Table 5.

Table 5.

Classification Accuracy Statistics

Diagnostic Groups and Scoring	Sensitivity	Specificity	Hit Rate	PPP	NPP	Odds Ratio
NP-normal versus Impaired
Raw cut score	17.2	93.5	60.7	66.7	60.0	3.0 (CI = 1.0, 8.6)
Demographically-adjusted T-score	70.7	73.7	71.9	66.1	76.1	6.4 (CI = 3.0, 13.2)
NP-normal versus ANI
Raw cut score	0.0	93.5	72.7	0.0	76.6	0.0 (CI = 0.2, 63.4)
Demographically-adjusted T-score	50.0	72.7	67.7	34.4	83.6	2.7 (CI = 1.0, 6.9)
NP-normal versus MCMD
Raw cut score	22.7	93.5	77.8	50.0	80.9	4.2 (CI = 1.1, 15.0)
Demographically-adjusted T-score	77.3	72.7	73.7	44.7	91.8	9.1 (CI = 2.8, 24.5)
NP-normal versus HAD
Raw cut score	35.7	93.5	84.6	50.0	88.9	8.0 (CI = 2.0, 29.7)
Demographically-adjusted T-score	92.9	72.7	75.8	38.2	98.2	34.7 (CI = 4.1, 137.0)

Note. PPP = Positive predictive power, NPP = Negative predictive power; NP = neuropsychological; ANI = Asymptomatic NP Impairment; MCMD = Minor-Cognitive/Motor Disorder; HAD = HIV-1-Associated Dementia

Discussion

The HDS is a screening instrument designed specifically for the detection of HIV-1-associated dementia. Given its brevity and ease of use, the HDS may be a helpful tool for researchers and practitioners alike, but has historically been limited by concerns regarding high false negative rates and possible demographic confounds. Considering that milder forms of HIV-1-related neurocognitive impairment are increasingly more common (McArthur, 2004), it is particularly important to address these limitations. As such, the purpose of this study was to develop and validate demographically-adjusted normative standards for the HDS and to examine their predictive validity in classifying individuals with HIV-1-associated neurocognitive disorders.

Consistent with prior reports of associations between demographic factors and the HDS (Berguis et al., 1999; von Giesen et al., 2005), the current study demonstrated a strong correspondence between the HDS summary score and age and education in an HIV seronegative cohort. Similar to other cognitive tests, interpretive adjustments based on age and education could enhance the clinical utility of the HDS by ensuring that test performance is more accurately interpreted through comparison of an individual’s test performance to that of a demographically-similar cohort. Accordingly, the ability of the HDS summary score to discriminate neurocognitively normal and impaired HIV-1-infected individuals could be improved.

Reported findings based on use of a raw cut score (≤ 10) have indicated that the HDS demonstrated excellent specificity, but generally poor sensitivity to HAD and milder forms of HIV-1-associated neuropsychological impairment (Berguis et al., 1999: Carey et al., 2004; Smith, et al., 2003; von Giesen et al., 2005). In fact, a recent study asserted that the HDS lacked sufficient predictive accuracy to be used as a screening tool for HIV-1-associated neurocognitive impairment beyond frank dementia (Smith et al., 2003). The use of demographically-adjusted T-scores in the current study revealed that sensitivity was significantly improved over the use of the raw cut score (i.e., an increase in sensitivity of 54%). In the present study, sensitivity refers to the proportion of neurocognitively impaired individuals (i.e., those with an ANI, MCMD, or HAD diagnosis) who scored in the impaired range on the HDS. An associated significant decrease in specificity (of only 20%) was also revealed, but specificity remained in the acceptable range. In addition, the mild increase in false positive errors was far outweighed by the considerable improvement in sensitivity. In terms of practical significance for a screening measure, false positive errors result in relatively few consequences as the individual’s neurocognitively normal status should become clear with the appropriate follow-up evaluations. Alternatively, the increase in sensitivity with the use of demographically-adjusted normative standards could address the high false negative rate reported in previous studies using the raw cut score. False negatives (i.e., individuals who are cognitively impaired were misclassified as normal) are particularly problematic because they indicate that some individuals who require additional assessment and/or treatment may not be identified, and the sharp increase in sensitivity observed with the use of the demographically-adjusted normative standards reduces this risk. As a result, the use of demographically-adjusted normative standards could potentially reduce the likelihood that HIV-1-associated neurocognitive disorders will go undetected. An evaluation of the predictive validity of the HDS with respect to specific diagnostic groups revealed a similar pattern of results for each subset, such that sensitivity was increased with an associated decrease in specificity.

The overall reduction in the likelihood of misclassification with the use of demographically-adjusted T-scores indicates that the HDS could indeed be a useful screening tool for HAD, as well as general HIV-1-associated neurocognitive impairment. Perhaps most impressively, the odds ratio (representing the likelihood that an individual with an HDS score in the impaired range has HIV-related neurocognitive impairment relative to an individual who performs within normal limits) was doubled with the use of the demographically-adjusted T-scores versus the raw cut score, increasing from approximately 3 to 6. That is, an individual who scored in the impaired range as defined by the demographically-adjusted T-scores was approximately 6 times more likely to be impaired than someone who performed within normal limits, as opposed to a likelihood of 3 using the raw cut score. Again, similar findings were yielded in the diagnostic subsets in which the odds ratios were considerably increased with the use of demographically-adjusted T-scores.

Even with demographic-adjustment, the HDS still may not be sensitive enough to detect milder forms of impairment. The improvement in sensitivity was most modest in the ANI subset, increasing only to 50.0%. Additionally, in the analyses using the complete HIV validation sample (i.e., all diagnoses combined), the majority of the false negative results were individuals with an NPI diagnosis. A post-hoc analysis revealed that there were no significant demographic or disease differences between misclassified individuals and those who were correctly identified (all ps > .10), indicating that ceiling effects may account for the relatively weak performance of the HDS in this diagnostic group. Armed with the knowledge that performance on the HDS could indicate a false negative result, in cases for which impairment is suspected the risk of misclassification can be reduced by supplementing the evaluation of such individuals with additional tests with low false negative rates (e.g., Hopkins Verbal Learning Test-Revised [Benedict, Schretlen, Groninger & Brandt, 1998; Carey et al., 2004]), or providing a referral for more comprehensive neuropsychological assessment despite a normal HDS score.

While some authors have argued that demographic corrections have had relatively little effect on the diagnostic accuracy of cognitive screening tests (Paolo, Tröster, Glatt, Hubble, & Koller, 1995), the demonstrated improvement in sensitivity and the odds ratio support the use of demographically-adjusted T-scores for the HDS. The fact that both sensitivity and the odds ratio were improved with the use of demographically-adjusted T-scores is particularly encouraging for the clinical utility of the HDS because these have been recognized as the most useful indices of classification accuracy for a screening measure (Elwood, 1993). Table 4 provides a user-friendly tool for referencing age and education corrected normative standards for the HDS summary score. Additionally, T-scores can be derived from raw scores with the use of the formula presented in the method section, which should be viewed as the optimal means of minimizing the impact of age and education on the HDS.

Demographic equivalence among the diagnostic groups comprising the HIV validation sample did not pose a limitation for the current study, as a post hoc analysis revealed that the diagnostic groups did not significantly differ with respect to any demographic variable. Nevertheless, a potential limitation of the current study is limited external validity resulting from the characteristics of the normative sample. Although there was no evidence of ethnicity or sex differences on the HDS in the current study, findings from these predominantly Caucasian and male volunteers may not adequately generalize to HIV-1 infected women and minorities, which is particularly relevant because ethnicity and sex effects are sometimes observed on neurocognitive tests of memory and information processing speed (Heaton et al., 2004). Another important limitation is the truncated range of age in the normative sample (maximum of 66 years of age). Due to the widespread use of HAART medications, people are living longer with HIV-1 infection, and therefore the development and validation of normative standards for the performance of older individuals on the HDS will be needed (McArthur, 2004). Given that the demographically-adjusted normative standards were generated using a regression-based approach, some of the education-grouped age cells have few or no participants, especially for the older age cells (see Table 4). In addition, the HIV validation sample was relatively free of commonly encountered comorbidities that may affect cognition (e.g., current substance-related disorders). Future studies are therefore needed to evaluate the ability of the HDS to discriminate between HIV-1-associated neurocognitive disorders and neuropsychological impairment due to non-HIV-related central nervous system pathology.

A brief and accessible cognitive screening measure for HIV-related cognitive impairment is a necessary tool for clinicians and researchers, especially in light of the potential increase in the prevalence of HIV-related neurocognitive disorders and the likelihood of presentation with subtle effects in the HAART era. The conclusion of the current study is that the clinical utility of the HDS is potentially enhanced by the consideration of demographic factors, with little to no reduction in the ease of use and accessibility of the measure. Refinement and validation of cognitive screening measures remains an important research goal and future research studies are warranted to continue in this endeavor. In particular, the International HIV Dementia Scale (IHDS; Sacktor et al., 2005) is an adapted form of the HDS, which was developed for use with non-Western populations by replacing the antisaccades, alphabet writing, and cube-copy subtests with more focused tests of motor and psychomotor speed. Development of demographically-adjusted normative standards for use of the IHDS in industrialized and developing countries could similarly improve the detection of impaired individuals and increase the clinical utility of the IHDS.