Reliability of Three Benton Judgment of Line Orientation Short Forms in Idiopathic Parkinson’s Disease

Abstract

Individuals with Parkinson’s disease (PD) often exhibit deficits in visuospatial functioning throughout the course of their disease. These deficits should be carefully assessed as they may have implications for patient safety and disease severity. One of the most commonly administered tests of visuospatial ability, the Benton Judgment of Line Orientation (JLO), consists of 30 pairs of lines requiring the patient to match the orientation of two lines to an array of 11 lines on a separate page. Reliable short forms have been constructed out of the full JLO form, but the reliability of these forms in PD has yet to be examined. Recent functional MRI studies examining the JLO demonstrate right parietal and occipital activation, as well as bilateral frontal activation and PD is known to adversely affect these pathways. We compared the reliability of the original full form to three unique short forms in a sample of 141 non-demented, idiopathic PD patients and 56 age and education matched controls. Results indicated that a two-thirds length short form can be used with high reliability and classification accuracy in patients with idiopathic PD. The other short forms performed in a similar, though slightly less reliable manner.

INTRODUCTION

The Benton Judgment of Line Orientation (JLO; Benton, Varney, & Hamsher, 1978) is a widely-used neuropsychological tool for assessing visuospatial deficits. Clinicians often employ the JLO test with Parkinson’s disease patients to determine their level of visuospatial functioning (Benton, Varney, & Hamsher, 1978; Bondi, Kaszniak, Bayles, & Vance, 1993; Cronin-Golomb & Braun, 1997; Montse, Pere, Carme, Francesc, & Eduardo, 2001), and Parkinson’s disease patients have classically demonstrated impairment on this test when compared to age-and-education-matched controls (Boller, Passafiume, Keefe, Rogers, Morrow, et al., 1984; Levin, Llabre, Reisman, Weiner, Sanchez-Ramos, et al., 1991). The JLO has many advantages for use in PD patients, including minimal motor involvement, high construct validity, and high test-retest reliability in both normal controls and PD (Benton, Hamsher, Varney, & Spreen, 1983; Montse, et al., 2001). A major weakness of the full JLO is its relatively lengthy administration time, estimated in a survey of neuropsychologists to be about 21 minutes (Lundin & DeFilippis, 1999). Clinicians may wish to reliably shorten the length of the JLO test for use in impaired patient populations that undergo multiple neuropsychological evaluations.

Currently there are three short form tests available for use, but none have been examined in PD (Mount, Hogg, & Johnstone, 2002; Qualls, Bliwise, & Stringer, 2000; Vanderploeg, LaLone, Greblo, Schinka, 1997; Winegarden, Yates, Moses, Benton, & Faustman, 1998; Woodard, Benedict, Roberts, Goldstein, Kinner, & Capruso, 1996). These three short forms have been shown to be reliable, internally consistent, and have relatively high clinical diagnostic accuracy for use in mixed patient populations (Winegarden, et al., 1998; Woodard, et al., 1996), as well as in patients with focal brain lesions (Calamia, Markon, Denburg, & Tranel, 2011).

The two most common short form tests are comprised of the odd and even numbered items (heretofore named OF and EF, respectively) of the JLO. Benton originally designed the JLO in such a way that item difficulty increases with each successive item. Consequently, short forms consisting of either odd or even-numbered items preserve the grading-by-difficulty of the full form. Winegarden and colleagues proposed a two-thirds length short form comprised of the latter 20 items of the 30 item test (heretofore referred to as TF) which is cited as more reliable and internally consistent with the original full form than the OF or EF in a sample of mixed neurologically disordered patients (Winegarden, 1998).

Recent functional MRI studies examining the JLO demonstrate right parietal and occipital activation, as well as bilateral frontal activation among individuals performing this task (Kesler, Haberect, Menon, Warsofsky, Dyer-Friedman, et al., 2004; Lee, Liu, Hung, Pu, Ng, et al., 2005). PD is known to adversely affect dorsal frontostriatal pathways in the brain (Zgaljardic, Borod, Foldi, & Mattis, 2003) and cause visuospatial impairment that may appear early in the disease (Boller, Passafiume, Keefe, Rogers, Morrow, et al., 1984; Hovestadt, De Jong, & Meerwaldt, 1987) and may worsen with increasing disease duration (Levin, Llabre, Reisman, Weiner, Sanchez-Ramos, et al., 1991; Lee, Harris & Calvert, 1998). Thus, there is good rationale for including JLO in a PD test battery, though full test administration time may be additionally problematic in this population due to attendant cognitive and motor slowing.

The present study examined the reliability, sensitivity and specificity, and classification accuracy of three proposed short form tests of the Benton JLO for use in idiopathic PD. Based on the previous short form studies, we hypothesized that each of the three short forms (OF = Odd Form, EF = Even Form, TF = Two-thirds Form) will be sufficiently reliable with the full form in a cohort of non-demented, idiopathic PD patients, as well as in a group of age-matched controls. We further examined the relationships between JLO test performance and disease severity, fatigue, and time since diagnosis. The clinical classification accuracy for identifying impairment was compared between the three short forms and the full form. We also wanted to provide clinically useful PD normative data for all four forms.

METHOD

Participants

This study was approved by the University of Florida Institutional Review Board and followed the Principles of the Declaration of Helsinki. For this retrospective investigation, we used data from individuals with PD (n=141) and healthy age-matched volunteers without PD (n=56) who had signed consent forms allowing their data to be used for research purposes. The cohort of idiopathic PD outpatients and research participants were drawn from an outpatient, university-affiliated movement disorders clinic (MDC) and two research investigations. A movement disorder neurology specialist within the Center for Movement Disorders and Neurorestoration (CMDNR) at the University of Florida completed all diagnostics for idiopathic PD, with PD diagnosis guided by the United Kingdom PD Society Brain Research Criteria (Gibb & Lees, 1988). From the UF CMDNR research database and affiliated NINDS funded investigations, a total of 526 patients in the MDC database were reviewed to obtain the 141 patients who met inclusion criteria. Study inclusion criteria for PD required Hoehn and Yahr scale range of 1–3, being “on” medication at time of testing, and no dementia (Dementia Rating Scale, Second Edition (DRS-2)score ≥ 130 raw, no history of deep brain stimulation (DBS), and no other neurological disorder history. Non-PD “healthy” peers were recruited from newspaper advertisements and community memory screenings, and were involved in separate federally funded research investigations. The final two participant groups were similar in age (p > 0.05; Cohen’s d = −0.26) and years of education (p > 0.05; Cohen’s d = −0.25; Table 1.)

Table 1.

Participant Demographics

	Idiopathic PD Patients a	Healthy Controls b
	Mean (SD)	Mean (SD)	p-value	Cohen’s d
Age	67.93 (6.28)	69.54 (5.64)	.096	−0.26
Education	15.10 (3.17)	15.89 (3.02)	.111	−0.25
Gender	76% male	68% male	-	-
Disease Years a	8.25 (4.91)	-	-	-
Fatigue Severity Score b	38.48 (13.33)	-	-	-
UPDRS e Score c	29.63 (16.45)	-	-	-
Hoehn and Yahr d	1.55 (.689)	-	-	-

^aParkinson’s disease, N = 141;

^bN = 56 ;

^cN = 38;

^dN = 33;

^eUnified Parkinson’s Disease Rating Scale

Note. The term Disease Years refers to the length of time in years at evaluation that each participant has had a formal diagnosis of Parkinson’s Disease (PD) made by a medical doctor. P-values are based upon independent samples t-test.

Materials

All participants completed a set of neuropsychological measures as part of a clinical movement disorder protocol or a research investigation. A neuropsychologist or a trained psychometrician completed test administration according to standardization procedures. Primary tests of interest for the current study included:

The Benton Judgment of Line Orientation (Benton et. al, 1978; form H) requires examinees to match two angled line portions on the top page to an array of eleven target lines. The total number of correct matches is recorded, and both lines need to be correctly matched to be counted as correct. For the OF and EF, correct responses were summed and multiplied by two. For the TF, the correct responses from item numbers H11–30 are summed, followed by the application of Lindquist’s equipercentile method (Lindquist, 1951). The TF to full form conversion table is shown in Table 2. For analyses of diagnostic accuracy, an age-and-gender-corrected score of 20 and below was classified as “impaired”, while 21 and above was “unimpaired”, as suggested in the JLO normative tables provided by Benton, Sivan, Hamsher, Varney, & Spreen (1994).

Table 2.

JLO Two-Thirds Form (TF; H11-H30) to Full-Form Score Conversions

JLO TF (H11-H30)	JLO Full (H11-H30)
0	3
1	5
2	6
3	7
4	9
5	10
6	11
7	13
8	15
9	17
10	18
11	19
12	20
13	22
14	23
15	24
16	25
17	26
18	28
19	29
20	30

Note. Based on the Lindquist Equipercentile method (1951)

The Fatigue Severity Scale (FSS; Krupp et al., 1989) is a self-report questionnaire used to assess factors related to tiredness in patients with movement disorders. The dependent variable was the summed total of items on a 1–7 Likert scale that were endorsed by the patient as resulting in fatigue during their daily activities.

Unified Parkinson’s Disease Rating Scale (UPDRS; Gibb & Lees, 1988) is a neurological assessment of functioning in patients with Parkinson’s disease. The dependent variable is the UPDRS total score “on” medication at the time point closest to the neuropsychological evaluation.

Statistical Approach

IBM SPSS Statistics v 20.0 was used for descriptive statistics, Pearson correlation coefficients, alpha reliability, Area Under the ROC Curve (AUC), and linear regression analyses. Using the values obtained from SPSS, corrected correlations (Levy, 1967) were performed by hand so that error terms from the non-independent test administrations are kept independent while retaining the true-score overlap. This method of correlation correction preserves the relationship between the short and full form as the degree to which the short form and full form are measures of a single construct (Kaufman, 1977). JLO raw scores were slightly negatively skewed but met normality requirements for kurtosis (skewness range = −.655 to −.454, Std. Error = .186; kurtosis range = −.422 to .075, Std. Error = .369). We calculated Cronbach’s alpha to assess short and full form JLO reliability. For both PD patients and controls, Fisher r-to-Z transformation compared PD and control groups for differences in short and full form reliability coefficients. Equivalent full form score calculations varied by short form test. A McNemar change test was employed to determine if there was a significant change in impairment classification for each short form relative to the full form score. As an additional measure of classification accuracy the Brier score, or mean squared error (MSE), was calculated for each form to demonstrate it’s ability to provide an accurate measurement of group membership based on each forms’ demographic corrected score total. In other words, this statistic provides a probability for how well the particular test form is able to discriminate between impaired patients, unimpaired patients, impaired controls, and unimpaired controls. It is specifically defined as the mean of (observed – expected)² and is equal to zero for a perfect prediction and equals 1 for a prediction that is always incorrect.

RESULTS

Group Differences in JLO Performance

The PD group performed lower than the control group on the full JLO, as well as on all short forms (Table 3). Effect sizes (d, Cohen, 1988) for group differences ranged from −0.48 to −0.54, which correspond to “medium” effects based on Cohen’s (1988) rules of thumb.

Table 3.

Participant Performances by Form

	Idiopathic PD Patients a	Healthy Controls b
	Mean (SD) c	Mean (SD) c	p-value	Cohen’s d
JLO d (full score)	22.78 (4.79)	25.27 (4.07)	<.01	−0.54
Odd Form (OF)	23.02 (5.12)	25.39 (4.38)	<.05	−0.48
Even Form (EF)	22.54 (5.46)	25.14 (4.19)	<.01	−0.51
Two-thirds Form (TF)	22.87 (4.87)	25.32 (4.29)	<.01	−0.52

^aParkinson’s Disease, N = 141;

^bN = 56;

^cMean raw score without demographic correction;

^dJudgment of Line Orientation Test

Note. P-values based on independent samples t-test

Reliability of Short Forms to Full JLO

Internal consistency reliability data (Cronbach’s alpha) for the full form JLO and all three short forms are presented in Table 4. Cronbach’s alpha was .81 for both PD and controls in this sample, demonstrating sufficient reliability for further analysis using moderately stringent criteria (Nunnally, 1978). Alpha values for each of the short forms were slightly below the moderately stringent cutoff of .80 for short form reliability for both groups. Short and full forms were highly correlated for the PD (OF: r=.90; EF: r=.91; TF: r=.97) and the control group (OF: r=.95; EF: r=.95, TF: r=.96). However, these correlations are spuriously high as a result of the short forms being imbedded within the full form. Therefore, these correlations have been corrected to account for shared error variance between the short and long forms (Levy, 1967). These corrected correlations are provided in Table 4 along with the uncorrected correlation coefficients. Lastly, we note that the PD group had a significantly lower correlation relative to the control group for the OF but not for the EF and TF (OF: p<.05; EF, p=.06; TF, p=.36; Table 4).

Table 4.

Pearson Correlations, Fisher’s r to z P-value, and Cronbach’s Alpha for Short Forms and Full Form Demographic Corrected Scores of Judgment of Line Orientation Test

	Odd Form	Even Form	Two-Thirds Form	Full Form
PD a Correlation with Full JLO b	.90 d	.91 d	.97 d	-
PD a Corrected Correlation with Full JLO b	.55	.58	.75 d	-
HC c Correlation with Full JLO b	.95 d	.95 d	.96 d	-
HC c Corrected Correlation with Full JLO b	.60	.55	.75 d	-
Fisher’s r to z p-value of PD a vs. HC c	.02	.06	.36	-
PD a Cronbach’s α	.68	.71	.77	.81 e
HC c Cronbach’s α	.67	.61	.79	.81 e

^aParkinson’s disease;

^bJudgment of Line Orientation Test;

^cHealthy Control;

^dCorrelation ≥ .75;

^eCoefficient alpha exceeds the moderately stringent value of .80 (Nunnally, 1978).

Participant Performances by Form

JLO Short Form Scores and Covariate Considerations

Neither age at testing, fatigue (FSS), nor disease severity (UPDRS score) had a significant effect on performance on any of the JLO forms. However, time since PD diagnosis was negatively correlated with performance on all short forms of the JLO as well as the full form (R² =−.417, p<.01 ; Table 5).

Table 5.

Pearson Product Moment Correlations between Disease Measures and Judgment of Line Orientation scores

	Odd Form	Even Form	Two-Thirds Form	Full Form
Disease Years	p<.01 r = −.417	p<.01 r = −.348	p<.01 r = −.405	p <.01 r = −.416
UPDRS a Score	p=.061 r = −.231	p=.372 r = −.149	p=.210 r = −.208	p=.115 r = −.260
Fatigue Severity	p=.844 r = .027	p=.485 r = .095	p=.478 r = .097	p=.625 r = .067

^aUnified Parkinson’s Disease Rating Scale

Note. The term Disease Years refers to the length of time in years at evaluation that each participant has had a formal diagnosis of Parkinson’s disease made by a medical doctor.

Sensitivity, Specificity, and Clinical Classification Accuracy

Impaired and unimpaired short and full form scores were obtained for each PD patient using the JLO normative tables provided by Benton, Sivan, Hamsher, Varney, & Spreen (1994). The short form classification (impaired or unimpaired) was compared to the full form classification for each PD patient. The OF correctly classified 94.2% of patients (114/121) as unimpaired and specified 85.0% of patients (17/20) as impaired when compared to the full form JLO score . The EF correctly classified 86.7% of patients (105/121) as unimpaired and specified 65.0% of patients (13/20) as impaired when compared to the full form JLO score. The TF classified 97.5% of patients (117/120) as unimpaired and specified 76.2% of patients (16/21) as impaired when compared to the full form JLO score. Results of a McNemar change test indicate that none of the short forms produce a significant change in diagnostic accuracy when compared to the full JLO score. The OF produced a total of 16% of patients who were misclassified (p=.51), the EF produced a total of 20% misclassification (p=.09), and the TF produced a total of 13% misclassification (p=.73). All Area Under the Curve (AUC) analyses exceeded acceptable minimum values and further demonstrated that when using the impairment cutoff score of 20 provided by Benton, the TF, OF, and EF were able to correctly classify 98%, 97%, and 90% of patients in reference to their full-form score, respectively. These sensitivity, specificity, and clinical classification data are provided below (Table 6). Additionally, the brier score for the full form, OF, EF, and TF were 0.09, 0.11, 0.14, and 0.10, respectively.

Table 6.

Impairment Classification Accuracy of Short Forms Relative to Full Form Score for Parkinson’s Disease Patients

	Odd Form	Even Form	Two-Thirds Form
Impaired to Unimpaired (False Negative)	3 (2%)	7 (5%)	5 (4%)
Impaired to Impaired (True Positive)	17 (12%)	13 (9%)	16 (11%)
Unimpaired to Impaired (False Positive)	6 (4%)	16 (11%)	3 (2%)
Unimpaired to Unimpaired (True Negative)	115 (82%)	105 (75%)	117 (83%)
Classification Change Significance a	p = 0.508	p = 0.093	p = 0.727
Sensitivity	.85	.65	.76
Specificity	.95	.88	.98
Positive Predictive Value	.74	.45	.84
Negative Predictive Value	.98	.94	.96
Area Under the ROC Curve	.98 b	.90 b	.97 b
Brier Score c	.11	.14	.10

^aBased on the McNemar change test;

^bp < .01;

^cFull-form = .09

Note. A cut-off score of 20 and lower after the application of age and gender correction was used to classify patients as impaired (Benton et al., 1994). These calculations consider “impaired full Judgment of Line Orientation score” as the condition predicted by each Short Form Judgment of Line Orientation score.

PD Normative Data by Group and Age

Age-and-education-stratified normative data for the three short forms and the full form of the JLO in this sample of 141 idiopathic Parkinson’s disease patients are provided in the supplemental materials for this article. These normative data can be useful for clinicians who wish to make a comparison of their PD patient’s JLO score to that of a normative sample of 141 non-demented, idiopathic PD patients (See Supplemental Materials).

DISCUSSION

The three proposed short forms exhibited differences in test characteristics that require consideration before clinical application. Next to the full form, only the two-thirds form (TF) exhibited minimally acceptable internal reliability. The TF also had the highest correlation (corrected and uncorrected) with the full form and produced the least amount of diagnostic misclassification compared to the other proposed short forms. Additionally, the TF was able to correctly classify 97% of patients as impaired or unimpaired when utilizing standard cutoff criteria provided by Benton, et al. Based on the Fisher r to z transformation, there were no significant differences between short and full form correlations for PD and HC groups, including when using the TF. Therefore, the high level of agreement with the full form, clinically adequate reliability, relatively low diagnostic misclassification, and the strong level of agreement with the control group make the TF the most reliable, and diagnostically accurate short form for use in patients with Parkinson’s disease.

The even more abbreviated odd and even forms (OF and EF) had overall lower reliability than the TF but their total scores were well-correlated with those from the full form for PD patients. These two shorter forms were also similar in their classification of PD patients as impaired versus unimpaired, with the OF being slightly more reliable in terms of diagnostic classification when compared to the full form. Area under the curve (AUC) analyses determined that the OF exceeded the TF for correctly classifying impairment relative to that of the full form classification. Although statistical corrections for shared error-variance corrections reduced the reliability correlations between short and long forms, the AUC analyses continue to suggest good diagnostic classification with the OF and TF.

For all forms, longer PD duration was associated with fewer correct responses. These findings validate those of previous publications on visuospatial function in PD (Boller, Passafiume, Keefe, Rogers, Morrow, et al., 1984; Levin, Llabre, Reisman, Weiner, Sanchez-Ramos, et al., 1991). There were no associations for any test form, however, with increased age, fatigue, or UPDRS score. The consistency of these findings across all three measures suggest that, despite questionable internal consistency for the OF and EF forms due to their reduced number of items, these two shorter forms are still useful for assessing line orientation in non-demented individuals with idiopathic Parkinson’s disease.

Consistent with previous literature on visuospatial functioning in PD, individuals with PD in this sample performed significantly lower than did the controls on the full JLO. This was also true for each of the three short forms. The part-whole correlation between the OF and full form was significantly lower for PD patients compared to controls, but this group difference in part-whole correlations was not observed for the EF or TF. This difference likely results from increased variability in JLO scores in the PD group, though it is likely trivial from a clinical perspective since all uncorrected part-whole correlations were >.90 for both groups. A potentially important finding is that OF and EF do not predict impairment in full form JLO score with the same degree of accuracy. In particular, the EF demonstrates a 7% and 8% reduction in the AUC when compared to the TF and OF, respectively. Thus, these analyses demonstrate that the use of the EF may result in a higher percentage of predicted impaired scores when the full score is unimpaired than do the other two forms.

LIMITATIONS AND FUTURE DIRECTIONS

There are study limitations. Many of the findings in this study may be sample-dependent. As determined by the Benton standard cut-score for impairment, our sample had few visuospatially impaired participants. . Caution is therefore necessary when applying our data to patient samples. It is possible that this small number of impaired patients relative to unimpaired patients may have an affect on classification accuracy statistics. Additionally, we did not examine the JLO short forms for their ability to predict impairment severity. We encourage future research in this area. Until then, we encourage clinicians to convert the short form score to a full form score, to use the demographic correction provided by Benton (Benton, 1994). Additionally, the current study focused only on, form H of the Benton JLO. The alternate JLO form (V) uses the same items in a different order. Follow-up studies examining Form V short form reliability are needed. Readers should also be aware that the TF, although shown to be the more reliable short form, is limited by its use of 20 of the 30 available items. This form may have limited application during non-overlapping serial administrations.. Despite these limitations, however, we consider the study data useful.

With the growing cost of neuropsychological evaluations, as well as cuts to healthcare budgets, test administration length is a concern. Patients with PD frequently undergo extensive neuropsychological examinations as part of routine cognitive check-ups, or for pre-surgical evaluations. For this reason, it has been proposed that by splitting the test into two equal and reliable short forms (odd and even), a clinician could use them as alternate forms for serial assessment (Woodard, 1996). While a comprehensive battery is important, reliably shortening tests ensures that the patient is able to give optimal effort throughout the entire battery. In a disease population that experiences symptoms of mental fatigue (Friedman, 1993; Lou, Kearns, Oken, Sexton, & Nutt, 2001; Chaudhuri & Behan, 2000), visuospatial function impairment both with executive functioning impairment (Bondi, 1993), and without executive functioning impairment (Cronin-Golomb, 1997; Waterfall, 1995); it is important to ensure that cognitive evaluations remain brief. With certain cautions discussed above, the three short forms discussed in this paper appear to be reliable and effective ways of reducing the time required to administer this test to Parkinson’s disease patients, and may also reduce the ill effects of a lengthy test administration. These short forms also give clinicians the ability to administer the odd and even forms as reliable alternate forms for use in serial testing of PD patients. The JLO data provided in the appendices of this paper may be useful as a normative group of non-demented, idiopathic, pre-DBS, PD patients. Future research should be conducted on patients with more advanced disease (i.e. H&Y >3) to evaluate the effects that higher levels of fatigue and disease severity on JLO performance for short form variations. Other possible future studies include the primary application of these short forms in clinical research to determine their performance in reflecting effects of medical or surgical treatments for PD and their effects on motor and non-motor disability in this population.