Evaluation of Screening Tests for Cognitive Impairment in Patients Receiving Maintenance Hemodialysis

Significance Statement

Cognitive impairment is common among individuals receiving maintenance hemodialysis, but few data exist regarding how well screening tests for cognitive function perform in this population. The authors assessed the ability of the Mini Mental State Examination, the Modified Mini Mental State Examination, the Montreal Cognitive Assessment, the Trail Making Test Part B, the Mini-Cog test, and the Digit Symbol Substitution Test to predict severe cognitive impairment in a cohort of 150 patients on dialysis whose cognitive status had been first defined with a battery of neurocognitive tests. The Montreal Cognitive Assessment was the best-performing overall screening test, and the authors recommend it as the preferred test to screen for severe cognitive impairment in patients receiving maintenance hemodialysis. Identification of such impairment may then facilitate optimal medical management and discussion of relevant issues with patients and family members.

Visual Abstract

Abstract

Background

Neurocognitive testing shows that cognitive impairment is common among patients receiving maintenance hemodialysis. Identification of a well performing screening test for cognitive impairment might allow for broader assessment in dialysis facilities and thus optimal delivery of education and medical management.

Methods

From 2015 to 2018, in a cohort of 150 patients on hemodialysis, we performed a set of comprehensive neurocognitive tests that included the cognitive domains of memory, attention, and executive function to classify whether participants had normal cognitive function versus mild, moderate, or severe cognitive impairment. Using area-under-the-curve (AUC) analysis, we then examined the predictive ability of the Mini Mental State Examination, the Modified Mini Mental State Examination, the Montreal Cognitive Assessment, the Trail Making Test Part B, the Mini-Cog test, and the Digit Symbol Substitution Test, determining each test’s performance for identifying severe cognitive impairment.

Results

Mean age was 64 years; 61% were men, 39% were black, and 94% had at least a high-school education. Of the 150 participants, 21% had normal cognitive function, 17% had mild cognitive impairment, 33% had moderate impairment, and 29% had severe impairment. The Montreal Cognitive Assessment had the highest overall predictive ability for severe cognitive impairment (AUC, 0.81); a score of ≤21 had a sensitivity of 86% and specificity of 55% for severe impairment, with a negative predictive value of 91%. The Trails B and Digit Symbol tests also performed reasonably well (AUCs, 0.73 and 0.78, respectively). The other tests had lower predictive performances.

Conclusions

The Montreal Cognitive Assessment, a widely available and brief cognitive screening tool, showed high sensitivity and moderate specificity in detecting severe cognitive impairment in patients on maintenance hemodialysis.

Multiple studies show that cognitive impairment is very common among patients receiving maintenance hemodialysis.^1,2 Within this population, cognitive impairment is associated with a greater incidence of depression,³ higher mortality,⁴ and likely contributes to poor quality of life.⁵ Reflective of a high overall burden of vascular disease in patients with kidney disease, cerebrovascular disease is thought to be the primary major contributor to the excess risk of cognitive impairment.⁶ Consistent with this hypothesis, we previously demonstrated that patients on hemodialysis perform significantly worse compared with population norms on cognitive tests assessing executive function, and that vascular disease and its risk factors are associated with lower executive function scores.^2,7 These findings were supported by a recent longitudinal study which showed that dialysis initiation was associated with loss of executive function but not with change in other cognitive domains.⁸

There is currently no recommended screening test for cognitive impairment in patients on hemodialysis, despite recognition by the ESKD Functional Status Technical Expert Panel in 2014 that “cognition is an important aspect of functional status and determinant of the patient’s ability to manage his/her disease, live independently, and participate in care decisions.”⁹ Comprehensive neurocognitive testing to diagnose cognitive impairment requires the use of multiple cognitive tests and a significant time commitment. A screening test that is readily available and easy to administer would allow for widespread screening for cognitive impairment in dialysis units.

However, there is reason to believe that not all cognitive tests used in the general population for neurocognitive screening perform well in patients with kidney disease. Some screening tests, such as the Mini Mental State Examination (MMSE), focus primarily on memory and do not include an executive-function component. In the hemodialysis population, where executive function is the most frequent manifestation of cognitive impairment, this could lead to underestimation of the degree of cognitive impairment.² Conversely, a test which solely assesses executive function may not capture impairment in memory, which becomes more prevalent in older individuals.¹⁰

We therefore performed detailed neurocognitive testing in 150 patients receiving maintenance hemodialysis to assess whether they had cognitive impairment using criteria that assesses both the breadth and depth of cognitive impairment across multiple cognitive domains, including memory, attention, and executive function. We also administered the MMSE, the Modified Mini Mental State Examination (3MS), the Montreal Cognitive Assessment (MoCA), Trail Making Test Part B (Trails B), the Mini-Cog test, and the Digit Symbol–Coding Test to identify a potential screening test. Using area-under-the-curve (AUC) analysis, we determined which screening test was most predictive of severe cognitive impairment using the detailed battery.

Methods

Study Population

Outpatients who were ≥18 years old and receiving maintenance in-center hemodialysis at four Dialysis Clinic Inc. units and one hospital-based outpatient unit (St. Elizabeth’s Medical Center) in the greater Boston area were evaluated for entry into the screening portion of the Cognition and Dialysis Study, with study enrollment occurring from February 2015 to September 2018. Eligibility criteria included English fluency as well as adequate visual and hearing acuity to complete full neurocognitive testing and allow for comparison to normative data. To minimize floor effects and to maximize the likelihood of study completion, individuals with MMSE scores ≤10 (considered severe dementia in the general population) and/or prior diagnosis of dementia based on medical record review were excluded. To minimize the possibility of uremia or acute medical illness influencing test scores, temporary exclusion criteria included nonaccess-related hospitalization within 1 month of screening, receipt of hemodialysis for <1 month, and single-pool Kt/V <1.0. The Tufts Medical Center/Tufts University Institutional Review Board approved this study, and all participants who agreed to undergo detailed cognitive testing signed informed consent. The clinical and research activities reported are consistent with the Declaration of Helsinki.

Baseline Demographics and Clinical Characteristics

Demographic, clinical, and laboratory factors were ascertained at the time of cognitive testing. Education (<12th grade, high-school graduate to <2 years of college, and ≥2 years of college) was obtained via self-report questionnaire. History of cardiovascular disease, defined as a composite of either coronary artery disease and/or peripheral vascular disease, was determined by patient history and/or documentation in an individual’s electronic or paper chart. Patients were specifically asked about personal history of myocardial infarction and coronary revascularization, which were used to define coronary disease, and about intermittent claudication and peripheral vascular disease, which were used to define peripheral vascular disease. Additional medical history including primary cause of ESKD, hemodialysis vascular access type, dialysis vintage (time since hemodialysis initiation), diabetes, heart failure, and history of stroke were obtained from the Dialysis Clinic Inc. or St. Elizabeth’s electronic record. Serum albumin level and single-pool Kt/V most proximate to the time of cognitive testing were obtained from participant medical records. To assess the presence of depression, each participant was administered the Center for Epidemiologic Studies Depression Scale. Use of either the dominant versus nondominant hand for completion of cognitive testing was recorded at the time of testing.

Neurocognitive Assessment

Participants were administered a battery of neurocognitive tests by research coordinators after a period of training by the study neuropsychologist (T.S.). To limit subject fatigue, all testing was completed during the first hour of hemodialysis. Using the same battery of tests, we have previously demonstrated similar performance regardless of whether testing was performed during the first hour of dialysis or before the start of dialysis.¹¹ Furthermore, this time point, as opposed to a nondialysis day, was chosen because the vast majority of education and instruction occurs while an individual is receiving a hemodialysis treatment. Neurocognitive testing was performed in a private room or in as quiet an environment as possible. To fit testing within the first hour of dialysis, neurocognitive testing was conducted over two dialysis sessions (detailed neurocognitive assessment at the first visit, which included the Trails B and Digit Symbol tests; the remaining screening tests were conducted at the second visit), with individual sessions no more than 1 week apart. No individual tests were repeated on the subsequent visit and no feedback was given to participants at either session on any test.

Definition of Cognitive Impairment

The neurocognitive battery used to define cognitive impairment included cognitive tests that are commonly used in the general population (Table 1) and are reported to possess high inter- and intrarater reliability. The neurocognitive battery consisted of the Wechsler Memory Scale–III Word List Learning subtest,¹² the Wechsler Adult Intelligence Scale–III Block Design¹² and Digit Symbol–Coding subtests,¹² the Digit Span test (forward and backward),¹² and the Trail Making Tests Parts A and B (Trails A and B).¹³ For Trails B, as is typically done, a 300-second time limit was imposed, with those unable to complete the test during this time period considered “noncompleters.” Trails B was evaluated both as a continuous score (noncompleters treated as a score of 301) and as a binary variable (completer versus noncompleter). Cognitive impairment was defined based on the criteria developed by Murray et al.¹ who administered a similar but not identical neurocognitive battery in a prevalent cohort of patients on hemodialysis. Published normative data—which are standardized by age, sex, and education—were used to classify cognitive test results. We classified each test into one of three cognitive domains—memory, attention/psychomotor speed, and executive function—based on the primary function that each test is designed to assess.^14–16 Normal cognitive function was defined as performance of no less than 1.5-SD below the mean in comparison to normative data on all tests in all domains of cognitive function. Mildly impaired cognitive function included performance of 1.5–1.99 SD on one or more tests in a single domain. Moderately impaired cognitive function included performance of 1.5–1.99 SD on one or more tests in two or more domains or ≥2 SD on one or more tests in one domain. Severely impaired cognitive function included ≥2 SD on one or more tests in two or more domains.

Table 1.

Cognitive tests used in the detailed neurocognitive battery (gold standard), categorized by the primary cognitive domain evaluated

Domain Assessed	Cognitive Test	Scoring	Test Details
Memory	Immediate recalla	Total initially correct	Assessment of memory in which a list of 12 words is presented during four trials, and retention of these words is tested after a delay of 25–35 min
	Delayed recalla	Total number recalled after delay
	Delayed recognitiona	Number of correctly identified words
Attention and psychomotor speed	Digits Span	Total correct number	Participants are asked to repeat lists of numbers read out loud, and to repeat different lists in reverse order
	Digit Symbol–Codingb	Number of copied symbols in 2 min	Symbols are decoded by matching a given symbol to a digit provided in an answer key
	Trails A	Time to completion	“Connect the dots” for a consecutive number sequence from 1 to 25
Executive function	Block designb	Number completed, weighted for time	Participants are required to reproduce depicted patterns using a set of colored blocks
	Trails B	Time to completion	Connect the dots, alternating between numbers (1–13) and letters (A to L); limit 300 s

^aDerived from the Word List Learning subtest of the Wechsler Memory Scale–III.

^bFrom the Weschler Adult Intelligence Scale.

Screening Tests

Six individual tests were chosen as possible screening tests (Table 2). These tests were chosen because they have been widely used in other populations, are easily administered, have cut points established for diagnosis of dementia or mild cognitive impairment, and some incorporate measures of executive function.^13,17,18 Importantly, they are all readily available to clinicians and they have been considered as potential screening tools for cognitive impairment in patients on dialysis.¹⁹

Table 2.

Potential screening tests for assessment of cognitive function

Tests	Description	Time (min)
MMSE20	Measures global cognitive function	5
3MS21	Extended version of the MMSE	15
MoCA17,18	Measures multiple cognitive domains resulting in a composite score	10
Mini-Cog22	Combination of three-word recall and clock drawing	6
Digit Symbol12	Decoding symbols matching it to a digit provided in an answer key	4
Trails B13	Connect the dots, alternating between numbers and letters	5

Statistical Analyses

Characteristics of the study population were described with frequency counts and proportions for categoric and binary variables, means with SDs for continuous normally distributed variables, and medians with interquartile ranges for skewed variables. For the two screening tests (Trails B and Digit Symbol) that were contained within the detailed neurocognitive battery, our primary analyses removed each respective cognitive test from the overall battery because inclusion of these tests in the “gold standard” would bias toward these items being more accurate screening tests. Our primary analysis was the ability of each screening test to predict severe cognitive impairment (compared with no, mild, or moderate cognitive impairment). Specifically, we used receiver-operating-characteristic (ROC) analysis to explore the utility of each of the screening tests as a diagnostic test for identifying neurocognitive deficits. For the screening test with the highest overall AUC, we then calculated sensitivity, specificity, and negative and positive predictive value by varying the test scores used as cut points. For the screening test with the highest overall AUC, we then calculated sensitivity, specificity, negative and positive predictive values, classification error rate, and accuracy by varying the test scores used as cut points. Classification error and accuracy rates were calculated as the proportion of incorrectly and correctly classified patients, respectively, for each given cut point of the test score.

To better understand the predictive ability of each screening test, we also examined the performance of each screening test within the three distinct cognitive domains. For this analysis, severe cognitive impairment within each of the domains was defined as a score of <2 SD below the norm on at least one test. Similar ROC analysis to the above was used to calculate an AUC. We also examined the ability of each screening test to predict a combined outcome of both moderate and severe cognitive impairment in secondary analyses. Finally, in a sensitivity analysis, we added the Trails B and Digit Symbol back to the gold-standard neurocognitive battery and re-examined the predictive ability of each of these tests. Significance was assessed using a two-sided α of 0.05 and results were presented with 95% confidence intervals. All analyses were performed using SAS Enterprise Guide version 7.12 (SAS, Cary, NC) and R language version 3.3.1 (R Foundation for Statistical Computing, Vienna, Austria).

Results

Baseline Characteristics

A total of 290 individuals receiving maintenance hemodialysis were screened for participation in the study (Figure 1). A total of 64 individuals did not meet inclusion criteria for the study, with the most common reasons being non-English speaking (n=52), severe dementia (n=6), and recent illness (n=6). A total of 74 individuals met entry criteria but were not interested in participating in the study. A total of 152 participants agreed to initial study consent. Two participants withdrew consent before cognitive testing, leaving 150 participants who underwent full cognitive testing.

Figure 1.

Flow diagram of participant screening. Two hundred ninety individuals were screened for study entry, with 74 individuals not interested in participating, 64 individuals not meeting entry criteria, and two individuals withdrawing consent, leaving 150 participants who underwent cognitive testing.

The mean (SD) age of study participants at enrollment was 64 (14) years; 61% were men, 39% were black, and 94% had at least a high-school education (Table 3). Of the included participants, 96% (146 of 150) were able to complete all cognitive tests, including all screening tests. Based on our prespecified criteria, 32 (21%) participants had normal cognitive function, 25 (17%) had mild cognitive impairment, 49 (33%) had moderate impairment, and 44 (29%) had severe impairment. Those participants with more severe cognitive impairment were less likely to have higher educational attainment and had lower serum albumin concentrations, although these trends were not statistically significant across the range of cognitive impairment categories. When examining which cognitive domains contributed to the diagnosis of severe impairment, attention and executive function were severely impaired in 43% of participants, attention and memory in 18% participants, memory and executive function in 12%, and all three domains in 27% of participants.

Table 3.

Demographics and clinical characteristics by degree of cognitive impairment

Characteristics	Total Cohort (n=150)	Normal (n=32; 21%)	Mild (n=25; 17%)	Moderate (n=49; 33%)	Severe (n=44; 29%)	Trend P Value
Age, yr	64±14	64±16	65±15	63±14	64±13	0.7
Female	39%	50%	32%	43%	32%	0.2
Black	39%	34%	40%	37%	44%	0.5
Education
<12th Grade	6%	0%	8%	2%	14%	0.09
High-school graduate	42%	44%	32%	49%	39%
>2 Yr college	52%	56%	60%	49%)	48%
Pre-SBP monthly average, mm Hg	138±19	135±20	136±18	141±21	139±18	0.4
Pre-DBP monthly average, mm Hg	73±11	72±12	73±11	74±11	74±10	0.7
Stroke	17%	16%	12%	20%	16%	0.8
Peripheral vascular disease	7%	13%	12%	0%	9%	0.3
Diabetes	51%	41%	56%	51%	55%	0.3
Heart failure	25%	25%	24%	33%	18%	0.7
Coronary artery disease	18%	19%	8%	29%	11%	0.9
Primary cause of ESKD
Diabetes	37%	21%	40%	43%	41%
Hypertension	26%	28%	28%	22%	27%	0.3
GN or other	37%	52%	32%	35%	32%
Dialysis vintage, yr	2.7 (1.3, 5.6)	3.6 (1.6, 7.2)	2.1 (1.2, 5.0)	2.5 (0.9, 4.9)	2.3 (1.5, 5.6)	0.6
Vascular access
Fistula	83%	84%	88%	84%	77%
Graft	9%	6%	8%	8%	11%	0.6
Catheter	9%	9%	4%	8%	11%
Body mass index, kg/m2	28±7	28±7	29±10	28±5	27±6	0.7
Single-pool Kt/V	1.51±0.25	1.63±0.34	1.45±0.16	1.49±0.24	1.50±0.22	0.2
Albumin, g/dl	4.0±0.4	4.0±0.3	4.1±0.2	4.0±0.5	3.9±0.4	0.1
CES-D	14.0±9.3	16.0±10.3	10.8±7.4	14.6±8.8	13.4±9.9	0.5
Use of nondominant hand for tests	6%	4%	6%	5%	5%	0.5

Data are presented either as n (%), mean±SD, or median (25th, 75th percentiles). SBP, systolic BP; DBP, diastolic BP; CES-D, Center for Epidemiologic Studies Depression Scale.

Screening Test Scores and Performance

Scores for screening tests are shown in Table 4. Each test showed a significant trend toward worse screening test scores as the severity of cognitive impairment increased. MoCA had the highest predictive ability for severe cognitive impairment (AUC, 0.81; 95% CI, 0.73 to 0.89), followed closely by the Digit Symbol test (AUC, 0.78; 95% CI, 0.68 to 0.88; Table 5). Similarly, Figure 2 shows graphically via the ROC curve that the MoCA, compared with the Digit Symbol and Trails B, has the highest AUC for predicting severe cognitive impairment across the range of scores. The 3MS, Mini-Cog, and Trails B tests all had similar predictive ability for severe impairment, whereas the MMSE had the lowest predictive ability. When test score cut points were examined for the MoCA, a score of ≤21 was found to maximize the combination of sensitivity and specificity for severe impairment (sensitivity, 86%; specificity, 55%; Table 6). Using the same cut point of 21, the MoCA displayed a positive predictive value of 44% and a negative predictive value of 91% (Table 6).

Table 4.

Distribution of screening test scores by degree of cognitive impairment

Screening Tests	Total (n=150)	Normal (n=32; 21%)	Mild (n=25; 17%)	Moderate (n=49; 33%)	Severe (n=44; 29%)	Trend P Value
MoCA	20.4±4.1 (9, 29)	22.6±3.0 (17, 29)	22.4±3.1 (16, 29)	21.1±2.6 (16, 26)	16.9±4.6 (9, 29)	<0.001
Digit symbol substitution	35.7±15.8 (0, 80)	43.5±12.3 (20, 71)	41.0±14.3 (24, 74)	37.3±14.4 (13, 80)	24.7±15.2 (0, 57)	<0.001
Trails B completeda
No	31 (21%)	2 (6%)	0 (0%)	2 (4%)	27 (61%)	<0.001
Yes	119 (79%)	30 (94%)	25 (100%)	47 (96%)	17 (39%)
Trails B score among completers, s	141.3±64.1 (22, 285)	104.1±49.2 (22, 219)	138.0±63.7 (41, 259)	152.3±59.9 (68, 285)	181.4±69.4 (62, 269)	<0.001
3MS	84.9±11.0 (50, 100)	89.3±7.6 (68, 99)	89.9±5.9 (75, 98)	86.5±7.0 (70, 100)	77.3±14.4 (50, 99)	<0.001
MMSE	25.6±3.3 (14, 30)	26.9±1.8 (22, 29)	27.0±2.2 (22, 30)	25.9±2.2 (21, 30)	23.5±4.4 (14, 30)	<0.001
Mini-Cog	2.2±1.6 (0, 5)	2.9±1.4 (0, 5)	2.6±1.2 (0, 5)	2.3±1.7 (0, 5)	1.3±1.3 (0, 5)	<0.001

Cognitive impairment determined by comprehensive neurocognitive battery. Presented as mean±SD and (min, max) scores.

^aTime limit of 300 s.

Table 5.

Performance of screening tests in predicting severe cognitive impairment

Screening Test	Cognitive Battery (Gold Standard)	Na	nb	AUC (95% CI)
MoCA	Complete battery	148	43	0.81 (0.73 to 0.89)
Digit symbol substitution	Battery minus Digit Symbol	144	36	0.78 (0.68 to 0.88)
Trail making part B (continuous)	Battery minus Trails B	141	19	0.73 (0.59 to 0.87)
Trails making part B noncompleters	Battery minus Trails B	141	19	0.76 (0.66 to 0.86)
Mini-Cog	Complete battery	148	43	0.73 (0.65 to 0.81)
3MS	Complete battery	148	44	0.72 (0.61 to 0.82)
MMSE	Complete battery	148	44	0.69 (0.58 to 0.80)

Complete battery: Wechsler Memory Scale–III Word List Learning subtest,¹² the Wechsler Adult Intelligence Scale–III Block Design¹² and Digit Symbol–Coding subtests,¹² Digit Span (forward and backward) test, and Trails A and B.¹³

^aN=total number of participants with sufficient data.

^bn=number of participants with severe impairment.

Figure 2.

ROC curve for prediction of severe cognitive impairment. The MoCA had the highest overall area-under-the-curve (AUC), followed by the Digit Symbol, and then Trails B.

Table 6.

Test characteristics by varying MoCA score to predict severe cognitive impairment

Score Equal To or Less Thana	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Classification Error (%)	Accuracy (%)
15	40	100	100	80	18	82
16	44	96	83	81	19	81
17	49	94	78	82	19	81
18	58	85	61	83	23	77
19	65	79	56	85	25	75
20	74	66	47	86	32	68
21	86	55	44	91	36	64
22	93	40	39	93	45	55
23	95	29	35	94	52	48
24	98	19	33	95	58	42
25	98	11	31	92	64	36
26	98	5	30	83	68	32
27	98	2	29	67	70	30
29a	100	0	29	0	71	29

Maximum MoCA score is 30. PPV, positive predictive value; NPV, negative predictive value.

^aNo individual had a score of 28.

Individual Cognitive Domains

Examining the performance of each of the screening tests within the three cognitive domains demonstrated that the MoCA had the highest and most consistent performance across each of the domains (Table 7). The Digit Symbol and Trails B tests performed well within the executive and attention domains, but not within the memory domain. The remaining three tests had consistently worse performance across the three domains.

Table 7.

Performance of screening tests in predicting severe cognitive impairment within a single domain

Screening Test	Cognitive Domain	Na	nb	AUC(95% CI)
MoCA	Executive	145	45	0.74 (0.65 to 0.83)
	Memory	148	38	0.80 (0.72 to 0.87)
	Attention	148	53	0.72 (0.63 to 0.81)
Digit Symbol substitutionc	Executive	143	43	0.70 (0.60 to 0.81)
	Memory	144	37	0.59 (0.48 to 0.70)
	Attention	144	37	0.80 (0.70 to 0.89)
Trails Bd	Executive	139	8	0.70 (0.46 to 0.95)
	Memory	141	33	0.63 (0.52 to 0.74)
	Attention	141	48	0.76 (0.67 to 0.84)
Mini-Cog	Executive	145	45	0.71 (0.62 to 0.80)
	Memory	148	39	0.71 (0.61 to 0.80)
	Attention	148	52	0.63 (0.54 to 0.72)
3MS	Executive	146	46	0.66 (0.55 to 0.77)
	Memory	148	38	0.71 (0.61 to 0.81)
	Attention	148	53	0.71 (0.62 to 0.80)
MMSE	Executive	146	46	0.70 (0.60 to 0.80)
	Memory	148	38	0.69 (0.59 to 0.80)
	Attention	148	53	0.64 (0.54 to 0.74)

^aN=total number of participants with sufficient data.

^bn=number of participants with severe impairment.

^cDigit Symbol removed from attention domain, AUC=0.84 (95% CI, 0.77 to 0.91) when included.

^dTrails B removed from executive domain, AUC=0.89 (95% CI, 0.83 to 0.96) when included.

When each screening test was used to predict moderate or severe impairment, the Trails B (AUC, 0.74; 95% CI, 0.66 to 0.82) and MoCA (AUC, 0.73; 95% CI, 0.65 to 0.81) had similar performance (Supplemental Table 1). This can also be seen graphically in Supplemental Figure 1, which shows the predictive ability of the MoCA, Trails B, and Digit Symbol tests for moderate or severe impairment across a range of test scores. The remaining screening tests did not perform as well and overall showed fairly similar predictive ability for moderate or severe cognitive impairment. When using a cut-point score of ≤21, MoCA had a sensitivity of 70% and specificity of 64% for either moderate or severe impairment.

Sensitivity Analyses

When the full neurocognitive battery was used (adding back Trails B or Digit Symbol, respectively) each screening test performed better in predicting severe cognitive impairment than when it had been removed, as expected. Trails B displayed an AUC of 0.87 (95% CI, 0.79 to 0.95), whereas the Digit Symbol test displayed an AUC of 0.78 (95% CI, 0.69 to 0.87; Supplemental Table 2).

Discussion

In patients receiving maintenance hemodialysis, the MoCA was best able to predict those individuals with severe cognitive impairment as compared with several other common cognitive tests. The Trails B and Digit Symbol tests had similar but slightly worse performance compared with the MoCA, whereas the MMSE, 3MS, and Mini-Cog did not perform as well. When a combination of moderate and severe cognitive impairment was examined, MoCA and Trails B had similar overall performance.

The MoCA, Digit Symbol, and Trails B tests all share similar test characteristics that may explain their superior performance. Each of these cognitive tests in part assesses executive function, a cognitive domain involved in planning and carrying out tasks. Because patients with kidney disease are more likely to display deficits in executive function,² these tests may better capture cognitive deficits when compared with tests that do not include a significant executive-function component (such as the MMSE and 3MS). This observation points to the underlying pathophysiology that may explain the high burden of cognitive impairment in patients with kidney disease. That is, cerebrovascular disease, including stoke and subclinical vascular disease, is believed to be a primary reason for the development of impaired executive function, both in the general population^23,24 and in those with kidney disease.² It is well known that patients with CKD and especially those with kidney failure are at high risk for both cardiovascular²⁵ and cerebrovascular disease,²⁶ and that the presence of each of these conditions is associated with cognitive impairment.^7,26 Although executive dysfunction is prevalent among patients receiving hemodialysis, our definition of severe cognitive impairment required significant impairment on at least two cognitive domains, indicating that both memory and attention are also impaired. When examining performance within distinct domains, the MoCA was most consistent in predicting global cognitive impairment across all three domains rather than simply detecting impaired executive function.

There are only a few prior publications that have examined screening tests for cognitive impairment in patients receiving maintenance hemodialysis. Kurella et al.²⁷ assessed the Kidney Disease Quality of Life Cognitive Function (KDQOL-CF) scale as a screening test to identify cognitive impairment in 79 participants with kidney failure receiving in-center hemodialysis and 78 participants with stage 3–5 CKD. The 3MS was used as the gold-standard measure of global cognitive function. KDQOL-CF scores were correlated with 3MS scores, with 52% sensitivity and 81% specificity. There are several important limitations of this study. First, the KDQOL-CF is a self-reported measure that poorly predicts overall cognitive function.⁵ An additional limitation of this study is the use of 3MS scores as the gold standard, because it is likely that the 3MS underestimates the true prevalence of cognitive impairment in patients with kidney disease due to its heavy reliance on components which assess memory. Tiffin-Richards et al.²⁸ evaluated the MoCA and MMSE as potential screening tests in 43 patients on hemodialysis compared with 42 healthy controls. Similar to our study, the gold standard for determining cognitive impairment was a battery of multiple cognitive tests including the California Verbal Learning Test, the Medical College of Georgia Complex Figures tests, Digit Forward test, Trails A and B, and the Boston Naming test, among others. Severe cognitive impairment was defined as >2-SD below standardized scores of the healthy controls in more than two cognitive tests. The MoCA was found to be superior to the MMSE. When using a cut point of <24, the MoCA had an AUC of 0.75, with a sensitivity of 76.7% and specificity of 78.6%. In comparison, the MMSE had an AUC of 0.70, with a sensitivity of 55.2% and a specificity of 75.0%. The major limitations of this study were the small sample size, which resulted in only 19 patients being classified as having severe impairment, and use of a healthy control group as the reference rather than population-based norms.

Determining the appropriate cutoff for the MoCA is an important consideration. A lower cutoff score will increase the specificity for detecting severe impairment, but this will occur at the expense of decreased sensitivity. For example, if a cutoff of ≤15 was chosen for MoCA, then only those with severe impairment would be detected (100% specificity), but this would occur at the expense of missing the diagnosis of severe impairment nearly two thirds of the time (40% sensitivity). In comparison, a cutoff score of <21 produces a sensitivity of 86% and specificity of 55%, indicating that few participants with severe impairment are missed, but nearly half of the participants may not have severe impairment. Given that the prevalence of severe cognitive impairment is high (nearly 30% of this cohort), we would favor a cutoff of ≤21 because it both identifies the majority of patients with severe impairment and it rules out the majority without severe impairment (negative predictive value, 91%). Translating these numbers over to the studied cohort, using a MoCA score cutoff of ≤21 resulted in six false negatives out of the 150 total participants.

The strengths of our study include a larger sample size than the existing literature, use of a comprehensive battery to diagnose cognitive impairment, and a cohort that is fairly similar to the overall United States hemodialysis population.²⁹ The major difference in this cohort compared with the United States hemodialysis population is a higher-than-average education level, which may affect the prevalence of cognitive impairment. However, we have no a priori reason to believe that this should affect the predictive ability of an individual cognitive test. There are several additional limitations to our study. We note that nearly 50% of screened individuals either refused to participate or were excluded (mostly due to being non-English speaking). We cannot rule out the possibility that test performance of the screening tests may be affected by the enrolled participants, although we have no reason to suspect this bias. However, it is important to note that the goal of our study was not to evaluate the prevalence of cognitive impairment but rather to evaluate the characteristics of a screening test; therefore, internal validity is as important, and arguably more important, than generalizability. We also acknowledge the possibility of practice effects from administration of multiple tests with similar questions, although individuals did not receive feedback on their performance and thus would be unaware of either correct or incorrect responses. We did not perform a formal clinical evaluation for dementia. We did, however, exclude participants with very low MMSE scores or a prior diagnosis of dementia; these exclusion criteria were considered necessary due to the inability of participants with very severe deficits to complete the full cognitive battery (although even with this exclusion, we observed that 30% of participants had severe impairment). This necessary requirement may in part limit generalizability to the hemodialysis population as a whole. In addition, we administered cognitive tests during the first hour of hemodialysis, which could hypothetically affect cognitive performance. Although there are some studies showing poorer cognitive performance during hemodialysis,³⁰ a prior trial we conducted with similar cognitive tests showed there was no difference in performance during the first hour of hemodialysis versus before the start of hemodialysis.¹¹ Finally, two of our screening tests, the Trails B and Digit Symbol, were also included in the full cognitive battery. We have attempted to address this by repeating analyses with and without these tests in the full battery. On the one hand, removing the tests limits our ability to compare predictive ability across screening tests. In particular, performance on Trails B contributes to nearly half of the severe outcomes; removing this test results in a decrease in the number of severe outcomes, and may bias against the performance of this particular screening test. On the other hand, including the Trails B and Digit Symbol scores as both a predictor and a component of the outcome results in a falsely elevated predictive performance. Due to this bias, we have elected to remove each individual test from the full cognitive battery for our primary results.

Based on these results, we suggest that the MoCA is our preferred screening test for cognitive impairment in a hemodialysis population. In addition to its performance in this study, the MoCA has been widely used in the general population as well as in select populations with other chronic diseases,³¹ it is free to use (but does require a fee for administrator-training certification), and has been translated into >50 languages.¹⁸ The Trails B and Digit Symbol tests had similar but slightly worse performance compared with the MoCA and could be alternative choices; however, there is less literature regarding their use as screening tests in the general population. The MMSE, 3MS, and Mini-Cog tests displayed worse performance and our results do not support their use as screening tests in patients on dialysis.

We believe these results can be of substantial use to providers who care for individuals with kidney failure requiring hemodialysis because it is unrealistic to administer a prolonged battery of cognitive tests to every patient receiving hemodialysis. A suitable screening test that can be administered in 5–10 minutes and that can reliably identify individuals with cognitive impairment could allow for the following: First, the very high prevalence of cognitive impairment observed when administering an extended neurocognitive battery indicates that it may be reasonable to administer a screening test such as the MoCA to all individuals receiving maintenance hemodialysis. Second, individuals found to have severe or even moderate cognitive impairment by the screening test could be referred for more detailed neurocognitive testing and for additional medical workup for reversible causes of cognitive impairment. Finally, and likely most importantly, the diagnosis could be disseminated to all members of the dialysis care team (such as the nephrologist, nurses, social worker, dietician, and technicians), allowing for closer follow-up care, evaluation of measures to improve compliance, and discussion with the patient and family members about goals of care and strategies to improve application of complex medication regimens, such as fluid restrictions and diabetes management.

In conclusion, we found that the MoCA showed the best performance in predicting severe cognitive impairment among patients receiving maintenance hemodialysis, with good sensitivity and reasonable specificity at a cutoff score of 21. Based on the MoCA’s wide availability, ease of use, and low cost, we recommend use of this cognitive test to screen for cognitive impairment in persons requiring maintenance hemodialysis. Future research should focus on the feasibility and manner in which cognitive testing could be routinely performed within an outpatient dialysis unit.

Disclosures

Dr. Weiner reports salary support from Dialysis Clinic, Inc., during the conduct of the study. All remaining authors have nothing to disclose.

Funding

The study was funded by the Paul Teschan Research Fund of Dialysis Clinic, Inc. (to Dr. Sarnak). Support for work on this study was also funded by the National Institute of Diabetes and Digestive and Kidney Diseases via K23DK105327 (to Dr. Drew). The funders of this study had no role in the study design, collection, analysis and interpretation, writing, or decision to submit the manuscript for publication.

Supplemental Material

This article contains the following supplemental material online at http://jasn.asnjournals.org/lookup/suppl/doi:10.1681/ASN.2019100988/-/DCSupplemental.

Supplemental Table 1. Performance of screening tests predicting moderate or severe cognitive impairment.

Supplemental Table 2. Performance of trails B and digit symbol predicting severe cognitive impairment using the complete cognitive battery.

Supplemental Figure 1. Receive operator curve for prediction of moderate or severe cognitive impairment.