Repeatable Battery for Assessment of Neuropsychological Status in Early Parkinson’s Disease

Abstract

Adequate reliability and valid factor structure are prerequisites for appropriate use of a measure in a population. Although the Repeatable Battery for Assessment of Neuropsychological Status (RBANS) has been used to examine cognition in Parkinson’s disease (PD), its reliability and factor structure have not been examined in this population. This study examined the reliability and factor structure of the RBANS in participants with de novo PD recruited for two NIH Exploratory Trials in Parkinson’s Disease (NET-PD), using Cronbach’s α and factor analysis. Confirmatory factor analysis (CFA) was implemented on the factor structure proposed in the RBANS manual, and exploratory factor analysis (EFA) was conducted to identify a valid factor structure given the proposed one was not supported. The RBANS exhibited poor reliability in participants with NET-PD. Cronbach’s α ranged from 0.03 to 0.74 for the five domains and the full scale. CFA results indicated that the proposed factor structure in the RBANS manual was not supported in this sample. EFA identified a two-factor structure for six of the 12 RBANS items. Six items were eliminated due to low correlation with other items or severe ceiling effects. This new factor structure was validated by another CFA. The two domains have fair reliability. Cronbach’s α ranged from 0.65 to 0.74 for the two factors in the two datasets. These results suggest that the current RBANS domain and total scores may not provide valid measurement of the neuropsychological status for patients with early PD.

The Repeatable Battery for Assessment of Neuropsychological Status (RBANS) has been gaining wide acceptance as a tool for assessing cognitive functioning in clinical trials due to its short administration time, conormed domain scores, inclusion of a summary score, and availability of alternate forms.¹ Since its first description 10 years ago,² it has been used in at least 58 published clinical studies and in diverse clinical populations including stroke,^3,4 Alzheimer’s disease,² Parkinson’s disease,⁵ bipolar disorder,⁶ multiple sclerosis,^7,8 and schizophrenia.⁹ Studies have examined the validity and reliability of the RBANS^1,3,4 and the factor structure of the RBANS.^3,10–12 However, none of them supports the RBANS’ factor structure as proposed by its author.¹³

Besides motor symptoms, persons with Parkinson’s disease (PD) display cognitive changes that are less well understood and studied. The National Institute of Neurological Disorders and Stroke (NINDS) has put a priority on the development of improved methods for characterizing the nonmotor aspects of PD and for improving diagnostic assessment of cognitive impairments in patients with PD.¹⁴ Because the RBANS can be administered in a short period of time, appeared to tap into cognitive domains thought to be affected in PD, and has been used previously to assess cognitive impairment in patients with PD,⁵ it was chosen for use in the initial NIH-sponsored Neuroprotection Exploratory Trials in Parkinson’s Disease (NET-PD) futility trials,^15,16 as one of three tools to assess cognition. However, no study has reported on the factor structure validity of the RBANS in PD and particularly, in patients with early PD. This study was conducted to explore the psychometric characteristics (i.e., reliability and factor structure) of the RBANS, in a relatively homogeneous sample of participants with de novo PD recruited for two NET-PD trials to assess the suitability of the RBANS as a cognitive tool for use in early PD.

METHODS

Participants

Two PD clinical trials (NET-PD FS-1 and FS-TOO) recruited persons with early untreated PD from 42 sites in the United States and Canada. These trials were organized by the Clinical Trials Coordination Center at the University of Rochester, the Department of Biostatistics, Bioinformatics and Epidemiology at the Medical University of South Carolina, and NINDS. The NINDS sponsored the two trials. The NET-PD steering committee developed the protocols and consent forms with the participating sites and guided the implementation of the trials. The protocols and consent forms were approved by a NINDS–appointed oversight board, an independent data safety monitoring board, and the institutional review boards of each of the participating sites. The data safety monitoring board monitored the safety, data integrity, and progress of the trials.^15,16

The RBANS was administrated at the baseline and 12-month follow-up visits. The RBANS data collected in the two trials were pooled at baseline and 12-month follow-up, and the two pooled datasets had 413 and 339 participants. At baseline and 12-month follow-up, 11 and six patients had missing values on some RBANS items, and another 19 and 18 patients were detected as outliers from the distributions of the 12 RBANS items determined by Malhanobis distance.¹⁷ The final baseline and 12-month follow-up dataset had 383 and 315 participants.

Measurement Instrument

The RBANS is designed as a brief “neuropsychological screen”¹³ for neurocognitive status for adults aged 20 to 89 years. The author of RBANS suggested the use of this battery for screening for cognitive deficits in acute-care settings, tracking recovery during rehabilitation, tracking degenerative disease progression, and neuropsychological screening that could be performed by non-neuropsychologists. The 12 items on the RBANS assess five cognitive domains: immediate memory, visuospatial/constructional abilities, language, attention, and delayed memory. The first four domains are each measured by two items; the last domain is measured by four items. The five domain scores are combined as a total scale score implying a second-order factor structure. In this study, all items were administrated to the patients with PD by NET-PD investigators or staff trained in the use of this instrument and scored as defined in the RBANS manual. According to the RBANS manual, the possible values for the RBANS scores at the item, domain, and scale level are 0 to 89, 40 to 154, and 40 to 160.

Statistical Analysis

SAS(version 9.1) was used for data preparation and descriptive analysis, and Mplus (version 5) was used for factor analyses.

Normality and multicollinearity of raw item scores were checked. Basic descriptive statistics of the five domain scores and total scale score were calculated, along with the correlations among the 12-item scores.

Internal reliability of the five domain scores and total RBANS score was checked, using the Cronbach’s α. “Adequate” or “good” internal reliability is defined as a Cronbach’s α of >0.7 or 0.8.¹⁷

The RBANS manual proposed a second-order factor structure with the five domain scores (first-order factors) and total scale score (second-order factor). A confirmatory factor analysis (CFA) was conducted, to assess the proposed second-order factor structure in the current population with de novo PD. Validity of the factor structure was assessed using the following indices and cutoff values: comparative fit index (CFI) ≥ 0.95, Tucker-Lewis index (TLI) ≥ 0.95, standardized root mean square residual (SRMR) ≤ 0.08, and root mean square error of approximation (RMSEA) ≤ 0.08.¹⁸ When the proposed factor structure of the RBANS was not confirmed, an exploratory factor analysis (EFA) with varimax rotation was conducted on the baseline dataset. Any item score with a Pearson correlation coefficient < 0.40 with all other item scores^10,19 or a variance inflation factor > 10¹⁷ was eliminated from the EFA. We determined the number of factors to be retained in EFA by the following criteria^20,21: χ² goodness of fit test (P > 0.05), root mean square residual (RMR < 0.05), RMSEA (≤0.08), and the percentages of common variance accounted for by a single factor (≥5%). The validity of the new factor structure obtained from the EFA was assessed by a new CFA, separately using the baseline data and 12-month follow-up data.

RESULTS

Descriptive Analysis

Table 1 gives baseline demographic and disease characteristics of the 383 participants included and 30 excluded from the analyses. The 30 excluded participants tended to be less educated and more racially/ethnically diverse than those included. As in the early stage of PD, most of the participants had slight to mild symptoms, and half were tremor dominant.²²

TABLE 1.

Demographic and disease characteristics of participants at baseline

		Included (N = 383)		Excluded (N = 30)		P
Demographic characteristics
Age	Mean ± SD	61.6 ± 10.4		62.1 ± 10.8		0.81
	Range	32–87		39–81
Education	Mean ± SD	15.3 ± 3.1		13.2 ± 4.2		0.01
	Range	3–24		5–22
		N	%	N	%
Gender	Male	249	65.0	16	53.3	0.20
	Female	134	35.0	14	46.7
Race	White	362	94.5	22	73.3	<0.001
	Othersa	21	5.5	8	26.7
Ethnicity	Not Hispanic or Latino	374	97.6	24	80.0	<0.001
	Othersb	9	2.4	6	20.0
Disease characteristics
UPDRS total score (Parts I–III)	Mean ± SD	22.9 ± 9.3		23.6 ± 7.2		0.34
	Range	5–61		8–37
Schwab and England Activities   of Daily Living scale	Mean ± SD	93.2 ± 5.1		91.7 ± 7.0		0.41
	Range	70–100		80–100
		N	%	N	%
Tremor dominantc	Yes	198	51.7	12	40.0	0.22
	No	185	48.3	18	60.0
Hoehn and Yahr Staging	0	1	0.3	0	0	0.88
	1	198	51.7	15	50.0
	2	182	47.5	15	50.0
	3	2	0.5	0	0

^aOther races include India, Alaska Native, Asian, Black/African American, more than one race, and unknown or not reported.

^bOther ethnicities include Hispanic or Latino, and unknown or not reported.

^cA patient with PD was determined to be tremor dominant or not by using the method described by Jankovic et al.²⁰

Descriptive statistics for the 12-item raw scores, the five index (domain) scores, and the total scale score of the RBANS in the analyzed dataset (Table 2) indicate substantial ceiling effects in picture naming, list recognition, figure copy, and line orientation, where the percentage of participants who achieved the maximum scores are 91.1%, 51.2%, 35.5%, and 26.9%, for the test items named earlier.

TABLE 2.

Basic descriptive statistics of the item, domain, and total scores (baseline data, n = 383)

Domain	Item	Possible score rangea	Minimumb	Maximumb	Mean	SD
IM		40–152	40 (0.3%)	152 (0.3%)	98.10	16.37
	LL	0–40	9	40 (0.3%)	26.31	5.27
	SIM	0–24	6	24 (1.6%)	17.81	3.78
VC		50–136	58	136 (0.3%)	102.76	18.31
	FC	0–20	7	20 (35.5%)	17.36	3.11
	LO	0–20	5	20 (26.9%)	17.60	2.56
La		40–137	78	134	97.80	9.70
	PN	0–10	8	10 (91.1%)	9.91	0.31
	SF	0–40	8	40 (0.3%)	19.56	4.91
Att		40–154	56	150	102.96	16.25
	DS	0–16	3	16 (15.1%)	12.08	2.74
	Co	0–89	11	89 (0.8%)	42.37	11.39
DM		40–137	48	130	98.30	15.67
	LRL	0–10	0 (5.2%)	10 (4.4%)	5.48	2.53
	LRN	0–20	13	20 (51.2%)	19.08	1.27
	SDR	0–12	0 (0.3%)	12 (18.0%)	9.27	2.40
	FR	0–20	0 (0.3%)	20 (4.7%)	13.25	3.37
Total  scale		40–160	61	148	99.99	14.55

IM, immediate memory; VC, visuospatial/constructional; La, language; Att, attention; DM, delayed memory; LL, list learning; SIM, story immediate memory; FC, figure copy; LO, line orientation; PN, picture naming; SF, semantic fluency; DS, digit span; Co, coding; LRL, list recall; LRN, list recognition; SDR, story delayed recall; FR, figure recall.

^aThe item scores are raw scores recorded during the test of RBANS. The domain scores are converted from the item scores using conversion tables in the RBANS manual. The total score is converted from the sum of index scores using the appropriate table in the RBANS manual. For additional details, please refer to the RBANS manual (Randolph, 1998).

^bA percentage of subjects that have scores reached the limit of possible score range is included in parentheses, as an indicator of possible floor or ceiling effect.

The RBANS domain scores are scaled to have a mean of 100 and standard deviation of 15 for each age group.¹³ The distribution of the study data (Table 2) is similar to those reported in the RBANS manual.¹³

Six items in the domains of visuospatial/constructional, language, and attention have low correlation with other items (Pearson correlation coefficients varying from 0.03 to 0.41, Table 3). This suggests these six items will not load on any factor and may be measuring some other independent attributes of the patients with PD. Correlations between each pair of items within each of these three indices are very low (0.19, 0.11, and 0.15), suggesting that the two items within each pair may not be measuring a single latent trait (factor).

TABLE 3.

Correlation among the item scores and reliability of the domain and total scores

Domain	Item	Reliability coefficient (Cronbach’s α)	LL	SIM	FC	LO	PN	SF	DS	Co	LRL	LRN	SDR	FR
IM	LL	0.70	1.00
	SIM		0.56	1.00
VC	FC	0.32	0.18	0.24	1.00
	LO		0.20	0.28	0.19	1.00
La	PN	0.03	0.03	0.08	0.16	0.20	1.00
	SF		0.40	0.33	0.13	0.20	0.11	1.00
Att	DS	0.13	0.14	0.28	0.10	0.20	0.14	0.15	1.00
	Co		0.41	0.32	0.11	0.21	0.08	0.38	0.15	1.00
DM	LRL	0.66	0.74	0.49	0.11	0.16	−0.01	0.32	0.09	0.30	1.00
	LRN		0.49	0.33	0.07	0.12	−0.01	0.22	0.10	0.20	0.58	1.00
	SDR		0.50	0.74	0.22	0.25	0.06	0.29	0.26	0.29	0.50	0.37	1.00
	FR		0.40	0.46	0.52	0.36	0.18	0.28	0.14	0.25	0.37	0.28	0.43	1.00
Total scale		0.74	0.64	0.66	0.48	0.43	0.12	0.55	0.44	0.50	0.52	0.42	0.57	0.61

IM, immediate memory; VC, visuospatial/constructional; La, language; Att, attention; DM, delayed memory; LL, list learning; SIM, story immediate memory; FC, figure copy; LO, line orientation; PN, picture naming; SF, semantic fluency; DS, digit span; Co, coding; LRL, list recall; LRN, list recognition; SDR, story delayed recall; FR, figure recall.

Internal Reliability Analysis

The reliability coefficients of the five domain scores and of the total RBANS score are reported in Table 3. Although the Cronbach’s α for the total RBANS score, for the immediate memory domain score, and for the delayed memory domain are ^~0.70, they are extremely low for the three domain scores of visuospatial/constructional, language, and attention, with Cronbach’s α equals to 0.32, 0.03, and 0.13. This suggests that the three domains have very low internal reliability, and that the items from these domains may be measuring some other attributes individually and independently.

Factor Analysis

The initial CFA model fit indices for the second-order factor structure for the 12 items of the RBANS were poor for both of the baseline and follow-up data-sets, suggesting that the current factor structure of the RBANS is not supported in the current population studied. For the baseline dataset, CFI = 0.764, TLI = 0.682, SRMR = 0.075, RMSEA = 0.135; for the 12-month follow-up dataset, CFI = 0.699, TLI = 0.595, SRMR = 0.082, RMSEA = 0.158. In addition, none of the four factor structures previously published confirmed the originally proposed factor structure of the RBANS (Table 5), suggesting that the originally proposed factor structure of the RBANS¹³ is less than optimal.

TABLE 5.

Summary of studies on factor structure of the RBANS^a

Study	Duff et al.10	Wilde3	Garcia et al.11	Carlozzi et al.12	NET-PD15
Primary focus	Factor structure	Construct Validity	Component structure	Factor structure	Factor structure
Population profile	Community-dwelling  elders	Acute ischemic  stroke patients	Memory disorder patients	Veterans with  memory disorders	Early untreated  patients with PD
Sample size	824	210	351	175	383
Age	73.4 ± 5.8 (≥65)	61.91 ± 13.97	77.9 ± 7.5 (51–89)	74.1 ± 8.0 (44–87)	61.6 ± 10.4 (32–87)
Gender	Male: 43%	Male: 49.5%	Male: 41.3%	Male: 100%	Male: 65%
Race	Caucasian: 86%	Caucasian: 59.5%	Caucasian: 98.8%	Caucasian: 71.4%	Caucasian: 94.5%
		AA: 41.9%	Hispanic: 0.6%	AA: 28.6%	AA: 2.1%
		Hispanic: 7.6%	Asian: 0.6%		Asian: 2.3%
		Asian: 1.0%
Education	59% ≥ high school	12.27 ± 3.1	12.8 ± 2.8	11.3 ± 4.0	3–24 (15.3 ± 3.1)
					74% ≥ high school
Factor analytic techniques	CFA-EFA	Principal components  analysis	Principal components  analysis	CFA-EFA-Re-CFA	CFA-EFA-Re-CFA
CFA to original 5-factor  structure of RBANS	Failed	N/A	N/A	Failed	Failed
Items excluded from the new  factor structureb	PN SF DS	None	None	DS	FC LO PN SF DS Co
New factor structure identified	2-factor structure  on 9 items	2-factor structure  on 12 items	3-factor structure  on 12 items	2-factor structure  on 11 items	2-factor structure  on 6 items
Items with modified scoring  procedureb	FC FR	None	None	None	None

^aColumns 1, 2, and 4 are modified from Carlozzi et al.¹²

^bFC, figure copy; LO, line orientation; PN, picture naming; SF, semantic fluency; DS, digit span; Co, coding; FR, figure recall.

Next, EFA was used on the baseline dataset to determine whether a new factor structure of the RBANS in the current population with PD can be identified. EFA on all of the 12 RBANS items did not have a solution. Although the percentages of common variance accounted for criteria indicated a two factor structure for the 12 items, the P value for the χ² goodness of fit test is <0.001, the RMR is 0.06, and the RMSEA is 0.10. Based on correlation and the internal reliability analysis, six items from the visuospatial/constructional, language, and attention domain were eliminated from the EFA. The EFA on the six remaining RBANS item raw scores yielded a two-factor structure (Table 4). These two factors accounted for 100% of the initial estimate of common variance of the six items analyzed. The P value of the χ² test of model fit, the RMR and RMSEA for this two-factor structure was 0.117 (χ² = 7.39, df = 4), 0.016, and 0.047, suggesting the correlations among the six RBANS items analyzed are best explained by two factors. Although list learning had salient loading on both of the two factors, its loading on the first factor was >20% higher than that on the second factor (0.69 vs. 0.43) and was included in the first factor.

TABLE 4.

Factor loadings after varimax rotation

Item	Factor 1	Factor 2
List Learning (LL)	0.69	0.43
List Recall(LRL)	0.89	0.29
List Recognition(LRN)	0.59	0.20
Story Immediate Memory(SIM)	0.24	0.93
Story Delayed Recall(SDR)	0.33	0.71
Figure Recall(FR)	0.29	0.43

Bold values indicate the most salient factor loading for each of the six items.

The CFA on the new two-dimensional structure of the six-item raw scores for both the baseline RBANS data (n = 383) and the 12-month follow-up data (n = 315) rendered fair model-fit indexes. For the baseline data: CFI = 0.980, TLI = 0.962, SRMR = 0.029, RMSEA = 0.080; for the 12-month data: CFI = 0.952, TLI = 0.910, SRMR = 0.048, RMSEA = 0.122. These fit indices provide evidence for the adequacy of the newly derived factor structure in our analysis. These are superior to the indices for the factor structure from the RBANS manual suggesting that the new two-dimensional structure of the six remaining item raw scores is a valid measure of some neuropsychological attributes in the participants with PD currently studied and outperforms the previously proposed factor structure in the RBANS manual. Two factors each derived from three items may be useful to measure some aspect of neuropsychological status of patients with de novo PD (vide infra). However, a second-order factor structure based on these two factors derived from the six items analyzed could not be supported by either of the two datasets based on the model fit indices such as CFI, TLI, SRMR, and RMSEA (data not shown). Therefore, a total scale score based on the two factors derived from the remaining six items should not be derived for meaningful use in practice.

The reliability analyses for the new two-factor structure yielded fair results. For baseline data, the Cronbach’s α for the two factors are 0.68 and 0.74; for 12-month follow-up data, these values are 0.65 and 0.66. Compared with the Cronbach’s α for the original five factors (Table 3), these results indicate that on average, the two new factors have better internal consistency than the original five factors.

DISCUSSION

This study indicates that the RBANS seems to have a different factor structure in patients with de novo PD than the original theoretically derived structure as proposed in the RBANS manual. Six of the original 12 RBANS test items are excluded in the new structure because either they have low correlation with each other and with the remaining six items, the three domains assessed by them have low internal consistency, or they have severe ceiling effects.²³ Therefore, the standard RBANS index scores and total score as defined in the RBANS manual do not seem to be appropriate for use in patients with de novo PD.

Valid internal structure (acceptable reliability and factor structure) is one of the most important pieces of evidence for the justification of the intended use and interpretation of scores from a measure.^19,24 Although the RBANS is gaining popularity as a brief instrument for measurement of cognitive impairment in a variety of clinical populations, only four of the 58 published studies using this measure examined its factor structure^3,10–12 before making inferences based on the scores obtained from it. Consistent with our current findings, none of these four factor structures previously published confirmed the originally proposed factor structure of the RBANS (Table 5) suggesting that the originally proposed factor structure of the RBANS¹³ is less than optimal.

Our study examined the second-order factor structure of the RBANS as initially proposed by its author.¹³ This second-order factor structure could not be confirmed in either the current baseline dataset or in the 12-month follow-up dataset. In the new two-factor structure identified by our analysis, six of the 12 RBANS items were excluded. Each of these six excluded items had low correlation with each other and with the remaining six items (Table 3). However, these six items may be measuring some unique attributes of the population with de novo PD currently studied, and may have potential use in future studies of this population if the items can be shown to have clinical relevance. The extremely low reliabilities of the three domain scores in our participants with de novo PD (i.e., visuospatial/constructional, language, and attention) suggest the need for additional research on the utility of the RBANS and the specific tests assessing these cognitive domains in patients with de novo PD.

The two factors that were identified in the present analysis are related to verbal learning and rote verbal memory (i.e., list learning, list recall, and list recognition) and to memory for verbal information with semantic and syntactic content (story memory). One might assume that different test items (e.g., list learning and story memory) assessed in similar time frames (e.g., immediate or delayed) are measuring the same cognitive domain (e.g., immediate verbal memory or delayed verbal memory), just as the items in the two memory factors (immediate memory and delayed memory) are suggested to do in the current RBANS manual. However, list learning and memory and story memory, although both verbal tasks, are tapping into different cognitive processes that use different neural circuits. Similarly, immediate and delayed recall and recognition tasks tap into different process and provide different types of information about ability to store versus retrieve information from memory. In this study, items related to the same domain (e.g., verbal memory) are joined together into one factor. This imposed a challenge for confirming the factor structure of a measure that involves both the immediate memory domain and the delayed memory domain. One possible way around this problem would be to do a CFA within each of the two memory domains separately. However, given the paucity of items within each of the two memory domains in the RBANS, the failure of confirming a single factor structure within each of these two domains is not surprising (data not shown).

In the RBANS, there are two domains related to memory (i.e., immediate memory and delayed memory), and test items related to these domains are administered at the beginning (immediate memory) and end (delayed memory) of the test session. Factor analysis is based on the correlation among the items, so it is not surprising that items from different domains in the RBANS but representing the same item tested either at different time points or in different forms loaded saliently on the same factor. These correlations may have led to a domain structure obtained from factor analysis that differs from the original structure proposed by the author of the RBANS. This scenario was also observed in other four studies.^3,10–12

Although the number is small, the high proportion of Latinos in the study that did not complete the RBANS suggests the possibility that development and use of a Spanish version of the RBANS for this minority population might be warranted. In general, the high proportion of racially/ethnically diverse participants who did not complete the RBANS also suggests a study of the profile of the RBANS in racially/ethnically diverse populations is warranted.

The current analysis of the RBANS in patients with de novo PD suggests that the RBANS has poor reliability in patients with de novo PD, the factor structure described in the RBANS manual is not supported in the current sample, and that the RBANS domain and total scores as originally designed may not provide valid measurement of the neuropsychological status of early patients with PD. Thus, the RBANS may not be an appropriate cognitive tool for assessing cognition in patients with early, de novo PD.