An empirically based proposal to identify a short battery to detect neuropsychological impairment in a general adult practice

Abstract

Objectives:

To demonstrate that 1) models based on small numbers of tests can be statistically developed to identify neuropsychological impairment in a general adult neuropsychology clinic and 2) those models show strong predictive validity on replication in a slightly different sample.

Method:

Latent Class Analyses (LCA) were used to determine neuropsychological classification in 231patients referred to general adult neuropsychology services. A clinical rating scale was also used to approximate clinical decision-making. Regression models were constructed in a training sample (n=127) drawn from an adult neuropsychology clinic using test scores from seven different a priori test battery combinations to predict group membership or clinical rating. The utility of the seven models was assessed in a testing sample (n=104) from another independent adult neuropsychology clinic.

Results:

The LCA yielded a two class solution characterized by impaired versus non-impaired performance on neuropsychological tests. A seven test battery provided the best balance of accuracy and length in predicting LCA group with a sensitivity of 84.4% and a specificity of 90%. Sensitivity and specificity were slightly attenuated using the clinical rating scale as the criterion, but the seven test battery still provided good accuracy (AUC=.906).

Conclusions:

Test protocols based on only five to eight test scores can accurately identify most patients with clinical impairment in a diverse adult neuropsychology clinic. Development of short protocols with adequate sensitivity and specificity will become increasingly important to address long waiting lists in light of the COVID pandemic against the general backdrop of increasing demand for neuropsychological services.

INTRODUCTION

The only thing that is constant is change. Heraclitus

Nowhere is Heraclitus’ observation more apparent than in healthcare, including the field of neuropsychology. Over recent years there has been growing recognition of the neurocognitive and neurobehavioral consequences of many disorders which has increased the demand for neuropsychological services. This is evident in the increasing numbers of neuropsychologists who are members of diverse integrated care teams and the related pressure to see more patients quickly (Festa, 2018; Kubu et al, 2016). Some clinical teams have addressed this need by integrating brief cognitive screening assessments in primary care settings (see Lanca, 2018; Morley et al., 2015) whereas others have developed computer based cognitive screening assessments (Busch et al, 2017; Busch et al, 2019; Gorodeski et al, 2019; Macaron et al., 2020) to be used in large medical populations. It is likely that increased use of cognitive screening exams will trigger even more referrals for standard, detailed neuropsychological assessment, particularly given the base rate of cognitive dysfunction in the elderly population (Plassman et al., 2008). Most recently, the COVID pandemic resulted in closing many neuropsychological clinics; thereby expanding the waiting list for neuropsychological services resulting in access challenges. As neuropsychological clinics start to resume onsite testing, there will be increased pressure to see even more patients to address access challenges. In addition, there will be pressure to move patients quickly through neuropsychological assessments to mitigate potential disease transmission by minimizing exposure in close testing settings. The confluence of all of these factors, in the context of relatively limited resources to address the demand for neuropsychological services, highlights the need for neuropsychologists to use short neuropsychological batteries based on standard tests to address referral questions related to diagnosis and treatment recommendations. Best professional and ethical practices dictate that those models are based on empirical data (cf., Meehl, 1954; Dawes, 2005).

The goals of neuropsychological assessment vary depending on the clinical service and patient needs (e.g., assessment of cognitive risk associated with epilepsy surgery, diagnosis of dementia, determination regarding presence or absence of memory or attention impairment, etc.). However, core to all neuropsychological assessments is a determination regarding the normality or abnormality of an individual’s cognitive and neurobehavioral function. Clearly, abnormalities in cognitive function can vary with respect to cognitive domain(s) involved and severity. Ideally, neuropsychological assessment includes reliable and well-validated measures in which there are sufficient convergent data to support a clinical diagnosis since the potential risks associated with a misdiagnosis can be high (see Roebuck-Spencer et al., 2017). There is an inherent tension between identifying a sufficient number of tests to accomplish the assessment goal(s) with adequate breadth to assess important domains versus extending the neuropsychological battery to the point that it is no longer cost efficient or patient friendly. Development of short test batteries with good accuracy based on independent, stand-alone neuropsychological measures will help neuropsychologists address the increased demand for neuropsychological services in the context of insufficient resources and other challenges specific to the COVID pandemic.

Others (Benedict et al, 2002; Funabiki et al, 2002; Welsh et al, 1994) have identified short neuropsychological batteries for use in specific populations with demonstrated clinical utility. Lanca (2018) described a triage neuropsychology service in a primary care setting in which a cognitive screening assessment may trigger referral for a very brief neuropsychological assessment that then might trigger more in-depth neuropsychological evaluation; to our knowledge there are no data regarding the sensitivity and specificity of the brief neuropsychological assessment employed in the primary care setting. We are also unaware of any similar studies which have examined the predictive accuracy of a brief neuropsychological assessment protocol in the context of a diverse adult neuropsychology clinic. The goal of this retrospective study was to identify whether briefer neuropsychological batteries comprised of existing standard tests with established reliability and validity could be identified that show good predictive accuracy in detecting abnormal neuropsychological function in a general adult neuropsychology clinic. The current study utilized regression analyses to develop regression equations for seven proposed short neuropsychological batteries in a large training sample drawn from a general adult neuropsychology clinic. The value of those models in predicting abnormal neuropsychological function was evaluated in an independent testing sample using two separate yet complementary criteria: statistically derived latent class group membership and clinical judgment.

Latent class analysis (LCA; Thomas, Lanyon, & Millsap, 2009; Goetchebeur et al, 2000) was used to identify group membership in a large sample of general adult neuropsychology patients based on a set of neuropsychological test scores. LCA is useful in identifying the underlying relationships between observable (i.e., neuropsychological test scores) and categorical latent variables (e.g., normal brain function). In the context of our study, LCA is useful in the absence of an independent gold-standard criterion for generating one or more latent classes that represent individuals with abnormal neuropsychological function (Goetghebeur et al., 2000). Prior literature suggests that in the absence of a definitive or gold-standard criterion, LCA can be used to develop an empirical criterion. LCA models generate a probability score for every individual that indicates the likelihood they are a member of an underlying category. In a two-class model with strong indicators, probabilities are non-normally distributed – tending toward 0 (e.g., normal) and 1 (e.g., abnormal) and, therefore, can be dichotomized based on a .50 probability cut score to empirically generate a gold-standard.

Although test scores are critically important to neuropsychological clinical decision-making, they do not capture all the data that neuropsychologists use to make clinical judgments and diagnoses (cf., Bauer et al, 2012; Matarazzo, 1990). Consequently, a clinical rating scale was developed and implemented to more closely approximate actual clinical decision-making. The clinical rating scale included consideration of the patient’s history, interview findings, behavioral observations, quantitative and qualitative test data; consequently, it incorporates much more information than the individual test scores which formed the basis of the latent class analyses.

METHODS

Participants:

All patient data were retrieved from a large IRB approved data registry. Patients were referred to a general adult neuropsychology service from different referral sources. A total sample of 231 patients were included and represented recent general adult referrals to two board certified neuropsychologists who utilize a similar battery but have slightly different referral bases within the neuropsychology service. The sample was divided based on the staffing neuropsychologist’s clinic which resulted in a training sample of 127 and a testing sample of 104. The decision to group the training and testing samples by the staffing neuropsychologist provides a more conservative test of the predictive validity of statistical models and the potential utility of brief batteries in a general adult neuropsychology service since different providers’ clinics may differ with respect to many factors that could influence model generalizability, including different interviewing procedures, different administration orders, and distinct patient referral patterns and rates of neuropsychological abnormalities. Thus, using training and testing samples with different characteristics is a more conservative test of validity shrinkage in future applications, particularly those with slight differences in sample or clinic characteristics from those used to develop and test short battery models in this study.

Latent Class Membership:

LCA was used to empirically identify class membership in the full sample (N=231). To generate LCA models and probabilities, a set of 10 neuropsychological variables were submitted as indicators. The decision to include the specific variables was based on the pattern of inter-correlations and clinical rationale (i.e., tests that assessed different domains or core features within those domains). Specifically, for tests with several highly inter-correlated measures, typically only one measure with the most useful clinical interpretation was chosen. Similarly, if measures across tests showed high correlations indicating potential multi-collinearity, one measure from the most useful test (in this case this was typically the test that provided the most clinical information per authors’ consensus) was chosen. Successive LCA models with 1–3 classes were estimated and the optimal number of classes was determined by iteratively comparing the model with k and k – 1 classes using multiple fit indices: the Akaike and sample-size adjusted Bayesian information criterion indices, entropy, and the bootstrap likelihood ratio test (Lo, Mendell, & Rubin, 2001; McLachlan & Peel, 2000; Nylund, Asparouhov, Muthen, 2007). Class size was also considered in interpretation since very small classes are unlikely to be useful in comparing optimal test batteries. LCA analysis was conducted in MPlus version 8.0 (Muthen & Muthen, 2017). The empirically derived LCA classifications were the primary outcome variable.

Clinical Rating Scale:

All patient files were assigned a clinical rating score based on the following criteria: 0 = Generally Normal, No impairments or an isolated borderline score/qualitative finding; 1 = Borderline Impaired, A pattern of subtle inefficiencies (e.g., more than one borderline score [i.e., 6–9^th percentile], a pattern of qualitative findings consistent with executive cognitive inefficiencies such as an erratic learning curve or interference effects on memory testing, losses of cognitive set or perseveration); 2 = Focal Impairment, A pattern of focal, impairment (i.e., </= 5^th percentile) on more than one test in any cognitive domain); 3 = Multifocal Impairment, A pattern of impairment per above involving more than one domain but not global impairment; 4 = Global Impairment, A pattern of impairment involving most, if not all, domains. The clinical rating scale score was determined by systematic review of all of the available test data from the standard evaluation, qualitative findings (e.g., an erratic learning curve, false positive errors), documented behavioral observations and interview notes. The clinical rating scale was designated the secondary outcome variable since it is not strictly independent of other clinical data and potential rater bias. However, it should be noted that clinical evaluations occurred at a different time from clinical rating scale completion and did not overly emphasize one particular measure or score.

Neuropsychological Data:

Core sets of tests were identified in both samples and the corresponding test scores were converted to z scores for most measures based on appropriate normative data including adjustments for demographic variables (e.g., age, education, gender) when available. Exceptions included tests that are not normally distributed (e.g., Boston Naming Test, Judgment of Line Orientation Test) or in which z scores are not helpful in interpretation (e.g., Beck Anxiety Inventory, Beck Depression Inventory). Not surprisingly since the data were drawn from a clinical sample, there were some missing data. The missing data rate was 4.0% out of all potential test scores for all patients. A series of correlations were run to determine if any demographic or test data were significantly related to missing data. Data were significantly more likely to be missing in patients who were older for the following tests: Wisconsin Card Sort Test (WCST, n=17), Trail Making Test Part B (Trails B, n=22), and Matrix Reasoning (MR, n=19). This finding is not surprising since it is likely that if a patient was unable to complete Trails B successfully, the WCST would not be administered. Similarly, if a patient was struggling or slow to test, it is likely that MR would have been omitted due to time constraints. All other tests had six or fewer missing scores. All variables were included in the multiple imputation process to satisfy the missing at random assumption, to reduce bias, and to increase the precision of imputed scores (Little & Rubin, 2002; Schafer & Graham, 2002; Baskaran & Smeeth, 2014). Five imputed datasets were computed using the SPSS missing values analysis option with a fully conditional specification using Markov Chain Monte Carlo estimation. One of the imputations did not converge, thus that imputation was excluded. Results were highly consistent between the original and imputed datasets. Unless otherwise stated, all statistics reflect those from the pooled results.

Model Estimation and Testing:

All statistics other than the LCA were run using SPSSv25. A set of seven test combinations were identified based on correlational analyses (removing highly inter-correlated measures) and clinical rationale (e.g., sampling of different domains). As a first step to test combination model building, non-parametric Spearman bivariate correlations were computed between the ten neuropsychological variables and LCA Class membership in the training sample to identify tests that were most highly correlated with LCA Class (Supplemental Table 1). Six different models using scores from five to eight tests were identified based on the pattern of correlations and clinical rationale. In addition, another model (Model 6B) that did not require manipulation of test materials, such as blocks or a computer keyboard, was identified to minimize potential exposure in light of the COVID-19 pandemic. Table 1 provides a list of the models.

Table 1.

Proposed Models

Model	Tests
5A	MR + Rey Delay + SIM + LM-I + BD
5B	MR + Rey Delay + SIM + LM-I + BNT
6	MR + Rey Delay + SIM + BNT + LM-I + BD
6B	MR + Rey Delay + SIM + BNT + LM-I + Trails B
7A	MR + Rey Delay + SIM + BNT + LM-I + BD + Trails B
7B	MR + Rey Delay + SIM + BNT + LM-I + BD + WCST Pers Errors
8	MR + Rey Delay + SIM + BNT + LM-I + BD + Trails B + WCST Pers Errors

Legend: MR = Matrix Reasoning; Rey Delay = Rey Auditory Verbal Learning Test, Delayed Recall; SIM = Similarities; LM – I = Logical Memory Immediate Recall; BNT = Boston Naming Test; BD = Block Design; Trails B = Trail Making Test, Part B; WCST Pers Err = Wisconsin Card Sorting Test, Perseverative Errors

The predictive validity of test combinations was examining by running a series of logistic regressions using the LCA class membership as the dependent variable in the training sample. For each test combination, the model estimates the relative accuracy of that combination in identifying abnormal neuropsychological function or impairment – hereafter test combination models. The resulting estimates from these analyses were used to derive predicted logit scores which were then applied to the testing sample [e.g., Predicted Logit Model X = Constant + (test score1*B test score1) + (test score2*B test score 2) + (test score3*B test score 3) + ……]. The testing sample predicted logit scores were transformed to predicted probability scores using Euler’s number [i.e., Predicted Probability Model X = 2.718281828**predicted logit Model X/ (1+2.718281828**predicted logit Model X)]. Bivariate correlations were compared between models using the test for significance of the difference in dependent correlations. Receiver operating characteristic (ROC) curve area under the curve (AUC) analyses were used to estimate diagnostic accuracy for impairment classifications across the seven models.

A similar set of supplemental analyses was done using the Clinical Rating Scale as the dependent variable. A series of regressions were computed in the training sample. The resulting estimates from those analyses were used to predict clinical rating scores for each test combination model in the testing sample. To estimate the AUC for the Clinical Impairment Scale, scores were dichotomized using ≥2 corresponding to an impaired score and 0–1 corresponding to an unimpaired score.

Power analysis:

The study’s primary goal was to identify the predictive value of the various test models in predicting LCA class membership. The testing sample (n=104) had excellent power (.95) to detect a significant AUC of ≥.69 (.05, one-tailed). Thus, the testing sample is well powered, and likely over-powered to detect clinically useful accuracy levels anticipated in this study. Specifically, we anticipated that all test combination models would likely exceed an AUC of .70 and, thus, the focus was on identifying the models with highest accuracy relative to the number of tests required to predict impairment.

RESULTS:

Patient Characteristics:

Demographic and descriptive clinical data regarding the training and testing samples are presented in Table 2. There were some differences between training and testing samples on specific neuropsychological variables as well as neurological diagnoses (Table 3) although there were no significant differences in referral sources for the two samples (Table 4). These differences further reinforce the conservative nature of adopting a model of independent training and testing samples in this study. If model accuracy remains high in the test sample, it is likely that these findings will generalize across different clinical settings, including settings with slight differences in procedures, patients, and disparate impairment base rates.

Table 2.

Training and Testing Sample Demographic and Neuropsychological Data

Variable	Training Sample (n=127)	Testing Sample (n=104)	Statistic	p value
Gender %M	40.94%	45.19%	X2(1,231)=421	.516
Age (years)	55.89	54.06	t(229) = −.89	.374
Education (years)	14.39	14.18	t(229) = −.58	.561
Probability LCA Class Impaired	.35	.62	t(229) = 4.56	<.0001
Clinical Impairment Score	1.72	2.14	t(229) = 2.49	.013
Beck Depression Inventory (raw)	16.54	16.58	t(223) = .03	.979
Beck Anxiety Inventory (raw)	12.79	13.13	t(226) = .22	.826
WRAT: Reading (z score)	.02	−.09	t(218) = −.85	.397
Trail Making Test, Part A (z score)	−.38	−.50	t(217) = −.79	.427
Trail Making Test, Part B (z score)	−.50	−.64	t(217) = −.83	.430
WCST Pers Errors (z score)	−.59	−1.09	t(229) = −3.40	.001
Digit Span (z score)	−.24	−.66	t(201) = −3.29	.001
Arithmetic (z score)	−.20	−.55	t(193) = −2.61	.010
Symbol Search (z score)	−.31	−.36	t(186) = −.33	.741
Coding (z score)	−.39	−.47	t(188) = −.52	.605
Vocabulary (z score)	.28	−.09	t(196)= −3.01	.003
Similarities (z score)	−.12	−.33	t(229) =−1.83	.068
BNT (raw score)	51.52	50.13	t(229) = −.1.17	.242
Phonemic Word Fluency (z score)	−.44	−.86	t(229) = −3.01	.003
Block Design (z score)	−.26	−.51	t(229) = −1.92	.056
Matrix Reasoning (z score)	.11	−.26	t(229) = −2.68	.008
JOLO (raw score)	24.29	23.36	t(209) = −1.49	.136
Logical Memory - I (z score)	−.46	−.39	t(229) = −.32	.752
Logical Memory – II (z score)	−.51	−.46	t(224) = .39	.697
Rey AVLT Total (raw score)	42.03	38.06	t(228) = −2.58	.011
Rey AVLT, List B (z score)	−.19	−.76	t(228) = −4.17	<.0001
Rey AVLT, Immediate (z score)	−.43	−.78	t(228) = −1.95	.052
Rey AVLT, Delay (z score)	−.48	−.99	t(228) = −3.03	.003

Data presented were drawn from imputation 1 which was representative of all of the data

Legend: WCST Pers Errors = Wisconsin Card Sort Test, Perseverative Errors; BNT = Boston Naming Test; JOLO=Judgment of Line Orientation Test; Rey AVLT=Rey Auditory Verbal Learning Test

Table 3.

Frequency Counts of Neurological Diagnoses^* by Training and Testing Sample

Diagnosis	Training Sample	Testing Sample
Stroke, TIA	13	6
Encephalopathy	1	4
Brain Tumor, MRI abnormality	4	5
Multiple Sclerosis	4	7
Epilepsy/Seizure	1	20
Movement Disorder	18	2
Headache, Pain	5	4
TBI/CHI/PCS	9	16

Legend: TIA=Transient Ischemic Attack, TBI=Traumatic Brain Injury, CHI = Closed Head Injury, PCS = Post-concussion Syndrome

^*Not all patients had a clear neurological diagnosis noted in the chart prior to referral for neuropsychological assessment. In addition, cognitive diagnoses (e.g., MCI, dementia) were not included in this table since this was often the referral question associated with the request for neuropsychological evaluation.

Table 4.

Training and Testing Sample Referral Sources

Referral Source	Training Sample (n=127)	Testing Sample (n=104)
Neurology	80 (63.0%)	68 (65.4%)
Psychiatry	15 (11.8%)	9 (8.7%)
Primary Care, Geriatrics, Internal Medicine	21 (16.5%)	16 (15.4%)
Other Specialty	8 (6.3%)	9 (8.7%)
Self, Not known	3 (2.4%)	2 (1.9%)

Chi²(4, 231) = 1.129, p=890

Latent Class Analyses Results:

A two group LCA solution fit the data best (Table 5). Descriptive statistics for the neuropsychological variables, associated effect sizes representing the differences between neuropsychological abnormal (i.e., impaired) and normal (i.e., nonimpaired) classes, and confidence intervals are provided in Table 6.

Table 5.

Goodness-of-Fit Criteria for solutions specifying 1–3 classes, n=231

Classes	1	2	3
AIC	9397	9043	8824
BIC	9465	9150	8968
Entropy	-	0.809	0.883
BLRT	-	p<0.001	p<0.001
% per class	100	54/46	51/49/<1

Goodness-of-fit criteria from latent profile analysis/mixture model

AIC = Akaike information criteria; BIC = sample-size adjusted Bayesian information criteria; BLRT = bootstrap likelihood ratio test p-value for k vs k-1 classes

Table 6.

Means and Effect Size by Class Membership

	Class 1 (n=107)		Class 2 (n=124)		Effect Size (d) (95% CI)
Variable*	Mean	SD	Mean	SD
WCST Trials to First Category, raw	56.06	48.99	21.69	20.99	0.95 (0.66, 1.23)
WCST Perseverative Errors, z score	−1.39	1.01	−0.29	0.99	1.10 (0.81. 1.39)
Trails B, z score	−1.18	1.12	0.07	0.99	1.19 (0.90, 1.49)
Boston Naming Test, raw score	46.70	10.31	54.54	3.86	1.03 (0.76, 1.31)
Similarities, z score	−0.76	0.75	0.28	0.79	1.35 (1.06, 1.64)
Phonemic Word Fluency, z score	−1.06	1.02	−0.25	1.00	0.81 (0.54, 1.08)
Block Design, z score	−0.91	0.66	0.11	0.89	1.29 (1.00, 1.57)
Matrix Reasoning z score	−0.80	0.68	0.65	0.79	1.96 (1.63, 2.29)
Logical Memory- Immediate, z score	−1.04	1.00	0.16	2.00	0.74 (0.47, 1.01)
Rey AVLT Delay z score	−1.45	1.11	−0.07	1.08	1.26 (0.98, 1.55)

^*All means significantly different between classes at p<0.001 based on two-sided t-test.

Legend: WCST = Wisconsin Card Sort Test; Trails B = Trail Making Test, Part B; Rey AVLT = Rey Auditory Verbal Learning Test

Reliability of the Clinical Rating Scale:

Forty-six of the patient files were coded using the Clinical Rating Scale by two independent raters to assess inter-rater reliability. Spearman correlational analyses indicated that the ratings were significantly correlated (r=0.82, p<0.001). Kappa statistics were also computed and the linear weighted Kappa was 0.52 (p<0.001) indicative of moderate agreement. A quadratic Kappa was higher (Kappa = 0.70, p<0.001) indicating good inter-rater reliability and highlighting that most of the misses between the two raters were by a single category. Across all cases, the Clinical Rating Scale had a positive correlation with the LCA Probability Score [r(229)=0.755, p<.001]. This correlation suggests that clinical ratings are convergent but not redundant with LCA classes. The testing sample tended to be more impaired as a group on specific measures than the training sample on both the LCA criterion and the Clinical Rating Scale (Table 7).

Table 7.

Clinical Impairment Rating Scale Scores in Training and Testing Samples

Clinical Impairment Score	Training Sample n=127	Testing Sample n=104
0, Unimpaired	32 (25.2%)	12 (11.5%)
1, Borderline Impaired	30 (23.6%)	18 (17.3%)
2, Focal Impairment	26 (20.5%)	27 (26%)
3, Multifocal Impairment	20 (15.7%)	37 (35.6%)
4, Global Impairment	19 (15%)	10 (9.6%)

X²(4, 231)=17.86, p=0.001, Imputation 1

Predicting LCA Impairment Class Membership:

All models significantly predicted class membership in both the training and testing samples. As would be expected, the relationships were slightly attenuated in the testing sample, but still highly significant and the associated effect sizes were very large (all r>.87, R²≥.76). Figure 1 illustrates the relationships between each test combination model and LCA classes. Figure 2 presents AUC values across all models. Together, correlation and AUC values suggest that the optimal models based on accuracy relative to number of tests were models 6B and 7A. Specifically, model 7A had greater validity (significantly higher correlation) than all models with five or six tests, was also superior to model 7B, and had comparable validity to model 8. Using a probability cut score of .5, Model 7A demonstrated a sensitivity of 84.4% and a specificity of 90% in the test sample. Model 6B had slightly lower accuracy than model 7A but the difference was very modest and implementation of Model 7A required only one more test to be administered. It is also worth noting that, while models significantly differed in validity, all models demonstrated very good overall accuracy for impairment detection (i.e., all AUCs ≥ .923).

Figure 1. Non-parametric Spearman correlations between each test combination model predicted probability and LCA impairment classes.

* Combination Model 7A had a significantly higher correlation with LCA class than Combination Models 5a – 6b

Figure 2.

ROC with test combination models predicting LCA impairment classes

Clinical Rating Scale Analyses:

All models significantly predicted clinical rating in both the training and testing samples with very large effect sizes (all r≥.76 R²≥.58). Once again, the relationships were modestly attenuated in the testing sample per expectations. Figures 3 and 4 illustrate the corresponding bivariate relationships and AUC values between each test combination model and the clinical rating. There is not as clear a distinction between the different models using the clinical rating scale criterion but, once again, Model 7A appears to be among the top models. This is supported by the ROC analyses as well with all AUCs > .884. Model 7A had a sensitivity of 77% and a specificity of 93% for detecting impairment using the clinical rating scale.

Figure 3.

Non-parametric Spearman correlations between each test combination model predicted probability and Clinical Impairment Rating

Figure 4.

ROC comparing Model Probability Scores and Clinical Impairment Rating Criterion

DISCUSSION

Neuropsychologists will need to develop alternate service delivery models to address the anticipated growing need for our services. Inevitably, this will entail development of short test batteries based on existing well-established neuropsychological measures that are useful for diagnostic purposes and treatment recommendations. The most professional and ethical practices dictate that those decisions are based on empirical data. Our data demonstrate that test protocols using as few as five to eight test scores can identify patients who have some degree of neuropsychological abnormality based on either an empirically-derived LCA criterion or clinical judgment. The optimal model based on the LCA-derived criterion included the following tests: Trail Making Test, Block Design, Matrix Reasoning, Similarities, Boston Naming Test, Rey Auditory Verbal Learning Test, and Logical Memory. A sample of experienced psychometrists (range 2 – 25 years’ experience) estimated that these tests could be completed in 60–75 minutes. The current study provides the initial step in empirically determining the predictive validity of short protocols. The findings suggest that a core protocol consisting of these measures supplemented by other important measures that do not require substantial additional evaluation time and/or could be completed by patients prior to the visit or online (including those that did not discriminate between the groups such as the Beck Depression Inventory, Beck Anxiety Inventory, tests of pre-morbid function or symptom validity measures) may be helpful in guiding test choice decision-making when time is limited in a general adult neuropsychology clinic. Our data also indicate that batteries comprised of as few as five tests also provide good accuracy when clinicians are especially pressed for time. Clearly, if time is not an issue and/or there is clinical need, additional measures can and should be administered. Further, performance on these brief protocols may help with triaging neuropsychological services and determining who needs access to more services. All of these models will require additional validation and should not be used in clinical practice until further validation is complete. Specifically, additional work is necessary to ascertain if the short batteries are sufficient to address other important clinical questions beyond abnormal or normal brain function. Full-sample regression models based on 5–8 tests are available from the authors for future research and validation.

The present study also demonstrates that, in the absence of an independent gold-standard criterion (as is the case with neuropsychological evaluations), LCA methods can provide a helpful approach for empirically estimating abnormal neuropsychological function. The models and clinical rating scale were intentionally designed to accommodate evaluation of neuropsychological abnormality or impairment across different cognitive domains. The results of the LCA confirmed a statistically derived clear separation in the data of two distinct groups – one with abnormal findings and one with normal findings. The data did not indicate a pattern of domain specific classes. Such patterns may be apparent if similar analyses are applied to more homogenous clinical populations (e.g., patients with epilepsy, patients with Parkinson’s disease). However, our focus was to identify a short neuropsychology battery applicable in a general adult neuropsychology clinic.

We intentionally decided to focus on examining the predictive accuracy of a short set of standard, reliable and validated stand-alone measures rather than commonly used “brief” measures such as the Repeatable Battery for the Assessment of Neuropsychological Status (Randolph, 2012) for many of the same reasons articulated by Benedict and colleagues (2002). Namely, the individual measures can be interpreted in isolation based on normative data, do not suffer from a limited range, and have well established reliability and validity. Further, these tests are well known such that the transition to using a short battery consisting of these measures would be relatively easy for many practicing clinical neuropsychologists.

This study has a number of limitations. First, there is an inherent circularity with using neuropsychological test scores to statistically define class membership and then applying a subset of those scores in a regression model to examine clinical utility of those measures in identifying class membership. This is a challenge that exists in any similar study in which there is lack of an independent gold-standard criterion for determining group membership. We argue that this is particularly problematic in neuropsychology since neuropsychological evaluation (which includes consideration of test scores) is the gold standard criterion. One could argue that we could have adopted a separate set of neuropsychological test scores to predict group membership and then tested the utility of another set of tests against that criterion, but it is well known that many neuropsychological measures are inter-correlated which does not completely solve the circularity challenge (i.e., memory test scores are related to memory test scores). Consequently, we chose to adopt others’ (Thomas, Lanyon, & Millsap, 2009; Goetchebeur et al) example and used LCA to help address this conundrum. LCA provides the most objective, empirical method to determine class membership. Further, in order to mitigate the circularity associated with having neuropsychological test scores predict neuropsychological test performance, we included a clinical rating scale to more closely mirror clinician judgment and approximate the true gold standard. The clinical rating scale incorporates consideration of the test scores used in the various models but also includes consideration of many more test variables, qualitative findings, interview and behavioral observations in an attempt to more closely mirror the inherently difficult to quantify clinical judgment entailed in neuropsychological interpretation. Despite the relatively crude nature of our clinical rating scale, we demonstrated moderate to good inter-rater agreement using Kappa statistics suggesting that the scale has reasonable reliability. Future work should extend on this preliminary work to develop a quantitative clinical rating scale that incorporates all clinically important aspects of the neuropsychological assessment in a more refined and rigorous manner, including the level of impairment (i.e., mild, moderate, severe) which is a limitation of our scale.

As we noted in the introduction, neuropsychological evaluations are requested to address specific questions for specific patient populations which often includes detailed consideration of cognitive domains. We readily admit that there are limitations to the criterion variables we used (i.e., LCA class determinations, clinical rating scale) in that neither takes into account degree of impairment or domain specific patterns of strengths and weaknesses that are routinely incorporated into neuropsychological case conceptualization. Although our criterion variables are limited in this manner, we contend that a common, core component of all neuropsychological evaluations is a determination regarding abnormal cognitive and/or neurobehavioral function. This assumption was also supported by the LCA which yielded a two group solution and not a multi-group solution indicative of focal and distinct patterns of cognitive strengths and weaknesses. Consequently, our criterion variables have some clinical and empirical utility and are relevant in a diverse, general adult neuropsychological clinic in which the referral questions and underlying neuropsychological patterns may vary tremendously.

We acknowledge that other criteria have been used historically to examine the predictive value of neuropsychological assessment including determinations regarding the laterality of epileptic focus or site of lesion. However, these external criteria are limited in a similar way to the criterion variables we used in that they do not adequately reflect the full value of neuropsychological assessment such as characterizing cognitive strengths and weaknesses (as well as being less sensitive and specific than neuroimaging, see Ruff, 2003). Nonetheless, it may be that different criteria based on objective data may be more applicable to other patient populations and/or specific referral questions than the criteria we adopted. This fact does not diminish the value of applying empirical methods, such as those shared here, to help address clinically relevant, critical decisions regarding test selection in the context of limited resources. In fact, our study highlights the need to develop an empirical and clinically relevant gold standard criterion for neuropsychological evaluation.

We could have adopted the approach of conducting our analyses using a highly homogenous, well-defined population but in doing so, we limit the generalizability of our findings and the potential clinical utility of these results in helping neuropsychologists who may be struggling now to accommodate high patient referral demands in general adult neuropsychology clinics. We fully support other investigators who wish to prospectively test the sensitivity and specificity of our proposed protocols in different patient populations and will supply the relevant equations to those who request them. In addition, we provide supplemental analyses (Supplemental Tables 2–3) that illustrate the sensitivity and specificity of Model 7A in predicting LCA Class in two diagnostic subgroups of patients (i.e., seizure disorder, head injury) in the testing sample.

In a related vein, it is possible, if not likely, that a different set of neuropsychological tests used as input to LCA would generate a different class structure. Future work is needed to either iteratively examine different LCA class structures using different test combinations to evaluate the stability and resistance of this criterion to the specific measures used as input or to identify a criterion that is completely independent per above. It is important to note that we partially mitigated the impact of over-fitting to the impairment criteria by using independent training and testing samples derived from separate neuropsychological clinical services. This approach provides a more rigorous test of the generalizability of our results to different clinics which may have different procedures and base rates of clinical impairment. It will be imperative that these findings are replicated prospectively in other settings and samples including patients with diverse demographics (which is a limitation of our study) in future studies.

Finally, inclusion of secondary analyses using a criterion based on clinical judgment helps address the concern that the LCA criterion (as a sole measure of abnormal function) may not be realistic. We also argue that inclusion of the clinical rating scale is more reflective of real-world clinical decision making. Clinical neuropsychological decision making is not solely based on test scores; clinical neuropsychological practice requires interpretation and integration of multiple pieces of data. A short protocol consisting of the seven tests included in Model 7A contains a wealth of clinically relevant data not accessible using only test scores. For example, data from the Boston Naming Test include not only the total score but evidence of the number correct with stimulus and/or phonemic cues as well as paraphasic errors, conduit d’approche, etc. Standard administration of Trails B requires administration of Trail Making Test, Part A and both provide an indication of errors which are useful qualitative data that are not reflected in the scores. Similarly, administration of Logical Memory I should also include administration of delayed recall given the low cost associated with adding Logical Memory-II. Finally, the Rey AVLT delay score is predicated on completing all five learning trials, List B, and immediate recall with all of the accompanying qualitative data such as evidence of an erratic learning curve, intrusion errors, interference effects and so on. Thus, administration of these seven tests provides a wealth of clinically important data beyond the test scores used in the model.

The goal of our study was to show that models based on small numbers of tests can be statistically developed that show strong predictive validity on replication in a slightly different testing sample. These data illustrate how the combination of clinical judgment, psychometrics, and statistics can identify a brief neuropsychology protocol based on independent stand-alone measures that can assist with clinical decision making and help neuropsychologists provide services to a greater number of patients. Relying on empirical methods to assist with clinical decision making is not new (c.f., Meehl, 1954) and is an ethical duty (Dawes, 2005). We hope that our research will lead to similar kinds of empirical studies so that the field can address our ethical and fiduciary responsibilities to our patients, the profession, and society as demand for our services increases. It is best ethical practice that decisions regarding which patient gets which kind of an evaluation are based on sound data.