Cognitive Testing in Patients with CKD: The Problem of Missing Cases

Abstract

Background and objectives

Cognitive testing is only valid in individuals with sufficient visual and motor skills and motivation to participate. Patients on dialysis usually suffer from limitations, such as impaired vision, motor difficulties, and depression. Hence, it is doubtful that the true value of cognitive functioning can be measured without bias. Consequently, many patients are excluded from cognitive testing. We focused on reasons for exclusion and analyzed characteristics of nontestable patients.

Design, setting, participants & measurements

Within the Choice of Renal Replacement Therapy Project (baseline survey: May 2014 to May 2015), n=767 patients on peritoneal dialysis (n=240) or hemodialysis (n=527) were tested with the Trail Making Test-B and the German d2-Revision Test and completed the Kidney Disease Quality of Life Short Form cognition subscale. We divided the sample into patients with missing cognitive testing data and patients with full cognitive testing data, analyzed reasons for nonfeasibility, and compared subsamples with regard to psychosocial and physical metrics. The exclusion categories were linked to patient characteristics potentially associated with missing data (age, comorbidity, depression, and education level) by calculation of λ-coefficient.

Results

The subsamples consisted of n=366 (48%) patients with missing data (peritoneal dialysis =62, hemodialysis =304) and n=401 patients with full cognitive testing data (peritoneal dialysis =178, hemodialysis =223). Patients were excluded due to visual impairment (49%), lack of motivation (31%), and motor impairment (13%). The remaining 8% did not follow instructions, suffered from medical incidents, or had language difficulties. Compared with patients with full cognitive testing data, they were more likely to have depression; be treated with hemodialysis; be older, nonworking, or more comorbid; and experience poorer shared decision making. Reasons for exclusion were not related to levels of age, comorbidity score, depression score, or education level.

Conclusions

We excluded almost one half of eligible patients from cognitive testing due to visual, motivational, or motor difficulties. Our findings are consistent with exclusion categories reported from the literature. We should be aware that, because of disease-related limitations, conclusions about cognitive functioning in the CKD population may be biased. In the future, nonvisual and nonverbal cognitive testing can be a valuable resource.

Introduction

Neurocognition and CKD

Cognitive testing (CT) in CKD has received considerable research attention in recent years. For example, Koushik et al. (1) present a detailed review of cognitive functioning (CF) in CKD, outlining neurocognitive functioning in CKD at all stages. The authors report that patients on hemodialysis (HD) and patients on peritoneal dialysis (PD) show poorer CF than matched healthy controls (2,3) and that patients on HD show performance fluctuations in CT that depend on the time point of testing (4,5). Poorer CF was observed shortly before a dialysis session; best CF occurred shortly after, which is supported by recent work (6,7). However, in patients on PD, no such performance fluctuation was observed (4,5). Compared with patients on PD, patients on HD perform worse regarding attention, concentration, and visual processing speed (8,9). Memory capacity seems to decrease with longer duration of HD (>2 years) (10). CF increases regarding processing speed, symbol processing, cognitive switching, attention, and working memory when shifting to nocturnal HD (11), whereas in a more recent trial, executive functioning did not improve with more frequent HD (12). Additionally, current studies confirm better CF for PD compared with HD (13) and that patients on dialysis presumably are more impaired in their executive functioning than memory (14). In sum, these findings suggest a well researched map of CF in patients on dialysis. However, how reliable is the literature on standard CT in the population of patients with CKD?

The Issue of Missing Patients

Assessing CF in patients on dialysis is of high clinical relevance, because CF is evidently associated with positive patient-centered outcomes, such as self-care capacity and shared decision making (15,16). CF has a key role in favorable medical outcomes and the adherence to medication prescriptions and dietary and fluid advice (17). However, standard testing, like the Mini-Mental State Examination (MMSE) or other paper and pencil tests, is only valid in patients who are visually, physically, and motivationally able to participate. Patients on dialysis usually suffer from various limitations. Hence, it is doubtful whether the true value of their CF can be measured without bias. They are likely to have impaired vision due to their higher average age and diabetic complications, motor difficulties due to dialysis shunt or paresis, and depression or other reasons for lack of motivation. This may lead to the exclusion of eligible and cognitively high-performing individuals with visual or motor impairments. These missing patients should be considered in addition to the patients who were excluded a priori due to study site selection and preclusion criteria on the patient level. Typically, the application of a priori criteria varies depending on the research question, with patient age or duration of dialysis treatment as common criteria. However, there also exists extensive exclusion on the basis of various medical conditions without clear indication of the underlying objective to do so. Neither the resulting bias due to a priori exclusion criteria nor the resulting bias due to online exclusion during the CT procedure received much attention in the literature so far, although it may be a major problem for the generalizability of the aforementioned findings. In practice, the patients’ cognitive abilities may be over- or underestimated, and patients may, therefore, receive suboptimal counseling or decision making. Hence, some of them can experience a less individualized treatment and even an unfavorable disease course. Some authors already indicate that the exclusion of patients on the basis of extensive criteria and the varying application of exclusion thresholds can lead to a misestimation of CF in the CKD population (6,14,18). In this article, we will address this aspect in more detail, focusing on factors potentially leading to exclusion during CT in an empirical study and analyzing characteristics of these excluded patients and the potential differences compared with their included counterparts.

Materials and Methods

Study Design and Sampling

The analysis was carried out within the Choice of Renal Replacement Therapy (CORETH) Project (funded by the German Federal Ministry of Education and Research), a multicenter observational survey registered in the German Clinical Trials Register (DRKS00006350) (16). The project addresses the decision making for either HD or PD focusing on the perspectives of recently initiated patients on dialysis. Detailed information about the study design has been already published (16,19,20). Patients were recruited from May of 2014 to May of 2015 from 55 dialysis units all over Germany. Local nephrologists screened patients on dialysis, and two trained study nurses obtained written informed consent. They surveyed the cohort using a 25-page standardized psychosocial questionnaire. Moreover, two standardized cognitive tests were conducted. The temporal sequence of the questionnaire versus the cognitive tests was randomized for each patient.

The time of patients’ study entry was set a priori at 6–24 months after initiation of dialysis to ensure the absence of acute complications or adaptation problems during the very early phase of treatment. Moreover, inclusion criteria (absence of acute psychiatric symptoms, ability to read and understand the questionnaire, ability to provide written consent, and age ≥18 years old) ensured that patients were able to perform the questionnaire. The inclusion criteria for general study participation were on the basis of preceding nephrologists’ screenings for eligibility, and the investigators made the final judgement for CT participation during the assessment. Applying these criteria, 6421 of 7312 screened patients on dialysis had to be excluded, because the vast majority was not in line with the time criterion (Figure 1); n=111 patients were unwilling to provide written consent. Hence, 780 potential patients remained for CT. By means of this preclusion, 11% of the nationwide screened patients were eligible for the CORETH data collection. From this cohort, n=13 patients had to be excluded, because they completed the survey at home without a study nurse (CT not valid). Finally, the analysis refers to a total of 767 patients.

Figure 1.

The Choice of Renal Replacement Therapy (CORETH) dialysis units in Germany and flow diagram of sampling. CT, cognitive testing. Reprinted from ref. 42, with permission.

Instruments and Outcome Measures

Within the CORETH approach, CF was operationalized through three indicators: patient-reported CF, cognitive switching, and selective attention. The Kidney Disease Quality of Life Short Form (KDQoL-SF [21]) is a self-report measure for patients with kidney disease. The three-item subscale “cognitive functioning” that proved good test quality was used in this study. The Trail Making Test-B (TMT-B [22]), a well established neuropsychologic paper and pencil test, was used for the assessment of cognitive switching. The test d2-Revision (d2-R [23]) was applied for assessment of selective attention and is an economic, well validated paper and pencil test in Germany. The d2-R consists of 14 lines with a total of 658 items. Each line is made up of the letters d (targets) and p (distractors), with one to four small dashes arranged either individually or in pairs above or below. Measuring selective attention, participants have to identify and cross out any target letter d with two dashes surrounding it within 20 seconds per line. The test manual (23) certifies high reliability (Cronbach’s α: r=0.96; retest reliability after 10 days: r=0.94) and widely evidenced validity. Norm data exist for a German sample (N=4.024) with an age range from 9 to 60 years old. The d2-R has been successfully applied in different settings, such as driver fitness assessment (24), attention testing in children (25), older adults from 60 to 82 years old (26), and the oldest old from 70 to 103 years old (27). It has also been mentioned as an assessment option within the CKD context (17).

Results of CT are, however, not the focus of this article, but rather, the focus is the reason why CT was not feasible in a certain proportion of enrolled patients. During data collection, the two trained study nurses documented reasons for nonfeasibility of CT in the form of written statements (for example, “patient not motivated to perform CT after instruction” or “CT procedure impossible because patient failed to hold pen due to shunt”; translated from original statements). The study nurses were instructed to document only one main reason for CT failure per patient.

Statistical Analyses

The sample was divided into patients with missing CT data and patients with full CT data. We analyzed the reasons for nonfeasibility of CT and the characteristics of patients with missing CT data. Therefore, relevant CORETH metrics were used (the study protocol [19] has detailed description and referencing). Mean values (Ms) and SDs were calculated for the subsamples regarding continuous variables. The percentage distribution was calculated for categorical variables. ANOVA was conducted for continuous variables, and chi-squared testing was used for categorical variables. Error probability was set to α=0.05. All analyses were carried out with SPSS 22.0 and R 2.15.0 for Windows.

By means of a previous literature screening (Supplemental Material), we reviewed typical reasons for patient exclusion in existing studies on CF in patients with CKD. According to the literature, the documented statements of the study nurses were categorized by content, and their frequency (percentage) was calculated in relation to all missing patients with statements.

Additionally, the reasons for exclusion were linked to the patient characteristics age, comorbidity, depression, and education level. For stratification, we formed three groups classifying them as follows: lower level, characteristic ≤(M−1⋅SD); medium level, (M−1⋅SD)< characteristic <(M+1⋅SD); and high level, characteristic ≥(M+1⋅SD). The education level correlated significantly positive with all cognitive metrics (KDQoL-SF cognition subscale: r=0.11; TMT-B time inverted: r=0.17; d2-R: r=0.34; P<0.001). In other words, individuals with higher education levels were more likely to perform better in CT than patients with medium or lower education levels. However, if highly educated patients without cognitive impairment have to be excluded from CT just as their cognitively impaired counterparts, this might further bias the true value of CF in the CKD cohort. A similar bias should be true for other factors (e.g., exclusion due to comorbidity). Moreover, we investigated the association between patient characteristics and the reasons for missing data by means of the λ-coefficient, a measure of associations between variables determined by the proportional reduction of error ranging from 0.00 to 1.00. A λ>0 reflects that the knowledge of the characteristics level in one patient would allow for conclusions about the reason for his or her exclusion.

Ethical Considerations

The study was carried out in accordance with the Code of Ethics of the Declaration of Helsinki and approved by the leading Ethics Committee of the University of Halle-Wittenberg. The ethics committees at every study site also approved the study protocol. Data safety in accordance with good clinical practice regulations has been guaranteed by the Coordination Centre for Clinical Studies Halle.

Results

There were n=366 missing patients with a documented reason for nonfeasibility in one of the cognitive tests. The sample with missing data consisted of patients with a missing value in the TMT-B or the d2-R (Table 1). There were no missing data for the patient-administered KDQoL-SF cognition subscale. A total of n=192 patients did not complete any test.

Table 1.

Sample of patients with missing data (n=366) derived from the total sample (n=767)

Missing Data in Cognitive Indicators	n
Patient-reported cognitive functioning score only (KDQoL-SF cognition)	0
Cognitive switching speed only (TMT-B)	2
Selective attention score only (d2-R)	172
Cognitive switching speed and selective attention score	192
Cognitive switching speed or selective attention score and documented reason	366

One of two patients who only performed the d2-R test and not the TMT-B was analphabetic and hence, unable to connect the TMT-B letters, and the other one misunderstood the instructions of the TMT-B. KDQoL-SF, Kidney Disease Quality of Life Short Form; TMT-B, Trail Making Test-B; d2-R, d2-Revision.

Table 2 illustrates the characteristics of patients with missing CT data and compares them with the characteristics of patients with full CT data (n=401). The patients with missing CT data, on average, were 11 years older, had a different distribution of their education levels, were nonworking, and were treated with HD. There were no significant differences with respect to the distribution of sex or duration of therapy.

Table 2.

Sample characteristics of patients with missing cognitive testing data and patients with full cognitive testing data

Characteristic	Sample with Missing CT Data, n=366	Sample with Full CT Data, n=401	P Value
Mean age (SD)	69.1 (12.0)	57.9 (15.4)	<0.001
Sex, %
Women	34.4	31.2	0.19
Men	65.6	68.8
Education, %
Lower	20.2	29.5	<0.001
Medium	69.9	49.0
High	9.8	21.5
Working status, %
Working	7.1	26.5	<0.001
Nonworking/retired	92.9	73.5
Dialysis modality, %
Hemodialysis	83.1	55.6	<0.001
Peritoneal dialysis	16.9	44.4
Mean months of treatment (SD)	15.1 (5.9)	14.6 (6.3)	0.30

CT, cognitive testing.

Furthermore, patients with missing CT data and patients with full CT data were compared regarding physical and psychosocial metrics from the CORETH Project (19). The comparison showed that patients with missing CT data were more comorbid; indicated a lower physical quality of life; lived more closely to the dialysis unit; had a smaller living space, a smaller social network, and fewer persons living in the same household; were less participation oriented; experienced poorer shared decision making; and were more likely to have depression (Table 3).

Table 3.

The Choice of Renal Replacement Therapy metrics expressed as median (25th; 75th percentiles) for patients with missing cognitive testing data and patients with full cognitive testing data

CORETH Metrics (19)a	Sample with Missing CT Data, n=366	Sample with Full CT Data, n=401	P Value
CCIb	7.0 (5.0; 8.0)	5.1 (3.0; 6.5)	<0.001
Self-assessed comorbidityc	9.0 (6.0; 13.0)	8.0 (5.0; 12.0)	0.01
Physical QoLd	34.1 (26.9; 42.7)	39.5 (32.4; 48.0)	<0.001
Distance to dialysis unite	10.0 (4.1; 16.8)	14.0 (7.0; 23.0)	<0.001
Living spacef	79.0 (60.0; 110.0)	86.0 (61.2; 122.3)	0.003
Size of social networkg	2.0 (2.0; 4.0)	3.0 (2.0; 6.0)	<0.001
Persons living in same householdh	1.0 (0.0; 1.0)	1.0 (1.0; 2.0)	<0.001
Depressioni	4.0 (2.0; 8.0)	4.0 (2.0; 6.0)	0.01
Participation preferencej	27.8 (16.7; 44.4)	38.9 (22.2; 55.6)	<0.001
Shared decision makingk	66.7 (11.1; 93.3)	88.9 (60.0; 100.0)	<0.001

The nonparametric Mann–Whitney U test was used to compare differences between groups. CORETH, Choice of Renal Replacement Therapy; CT, cognitive testing; CCI, Charlson Comorbidity Index; QoL, quality of life.

^aHigher values indicate better outcomes. For the other CORETH metrics, no differences appeared (psychologic QoL, anxiety, social functioning and support, and treatment satisfaction [19]).

^bCCI (minimum =0, maximum =37).

^cSelf-Administered Comorbidity Questionnaire German Version, Total Score (minimum =0, maximum =39).

^d12-Item Short Form Health Survey (SF-12) physical sum scale (minimum =0, maximum =100).

^eDistance in kilometers.

^fLiving space in square meters.

^gNumber of persons in social network.

^hNumber of persons living in the same household.

ⁱHospital Anxiety and Depression Scale German Version, subscale depression (minimum =0, maximum =21).

^jAutonomy Preference Index, subscale participation preference (minimum =0, maximum =100).

^kShared Decision–Making Questionnaire German Version (minimum =0, maximum =100).

The content-driven categorization of reasons for nonfeasibility in the sample of patients with missing CT data (Table 4) is presented in Figure 2. The empirically derived reasons were similar to those reported in the literature (Supplemental Figures 1, 2, Supplemental Table 1).

Table 4.

Empirical reason categories

Reason for Missing Data	Explanation/Example
Visual impairment	Patients could not read the test despite visual aids or were blind
Refused consent/lack of motivation	Patients were unwilling to participate in CT despite motivation by the study nurse and successful participation in the CORETH survey
Motor condition	Patients were unable to hold the pencil due to the dialysis shunt or paresis
Did not follow instruction	Patients wrote noninstruction conform symbols or did not start responding despite consenting to CT
Technical or medical incident	CT had to be interrupted due to alarms of dialysis machine or sudden drop of BP
Language difficulties	Patients were analphabetic or spoke different languages

CT, cognitive testing; CORETH, Choice of Renal Replacement Therapy.

Figure 2.

Frequency distribution of reasons for missing cognitive testing data (n=366).

The calculation of λ for the link between reasons for missing data and patient characteristics yielded that none of the characteristics corresponded statistically significantly with the exclusion categories. Speaking in terms of content, for example, highly educated, older, or comorbid individuals with less depression were also excluded due to different reasons (Table 5). However, regarding the percentage with the largest deviation from the reference distribution of the total sample of patients with missing CT data (n=366), some relative trends emerged: younger patients were less frequently visually impaired but more frequently had language difficulties or a technical incident during CT, fewer comorbid patients were less likely to refuse consent, and patients with a lower education level tended to misunderstand the instruction a bit more.

Table 5.

Distribution of reason categories among stratified patient characteristics in the sample with missing cognitive testing data (n=366)

Characteristic	Level (Range)												Reference
	Age, yr			Comorbidity (CCI)			Depression (HADS)			Educationa
	Low (≤57)	Medium (57<yr<81)	High (≥81)	Low (≤4.3)	Medium (4.3<CCI<8.7)	High (≥8.7)	Low (≤1.4)	Medium (1.4<HADS<9.0)	High (≥9.0)	Low	Medium	High
N	57	270	39	69	229	68	61	236	69	74	256	36	366
Reason for missing data, %
Visual impairment	35	51	54	36	51	53	51	50	44	51	48	44	49
Refused consent/lack of motivation	39	30	26	41	30	22	38	28	32	23	33	28	31
Motor condition	14	13	10	12	12	15	7	14	15	12	11	22	13
Did not follow instruction	2	5	10	3	4	10	2	5	9	11	4	3	5
Language difficulties	7	1	0	6	1	0	2	2	1	1	2	0	2
Technical or medical incident	4	1	0	3	1	0	2	1	0	1	1	3	1
λ (P value)	0.01 (0.76)			0.02 (0.68)			0.0			0.0

CCI, Charlson Comorbidity Index; HADS, Hospital Anxiety and Depression Scale German Version, subscale depression.

^aEducation level range: low, without graduation; medium, middle school; high, high school diploma.

Discussion

We had to exclude 48% of patients from CT due to visual, motivational, or motor difficulties. The reasons for missing data did not differ according to levels of age, comorbidity score, depression score, or education. Our findings support those from previous studies (Supplemental Material): compared with their included counterparts, patients excluded from CT show significantly worse psychosocial and physical conditions. However, both included and excluded patients self-indicated a good cognitive status, with an M of 87 of 100 points on the KDQoL-SF cognition subscale. These results strengthen earlier statements that frequently applied exclusion criteria at CT of patients with CKD can lead to bias in the estimation of their CF (6,14,18). The findings even suggest that existing studies about CF in CKD are of limited generalizability to the CKD population. However, we can only speculate about the extent of that bias. Excluding patients with cognitive impairment most likely leads to an overestimation of CF, whereas the effect of the remaining selection mechanisms (visual, motivation, motor, and language) on the measurement of CF is not clear and may result in both over- and underestimation. Other than the necessity of precluding patients from standard CT (i.e., due to proven dementia), additional reasons can occur during the CT procedure that lead to extensive patient exclusion. Some of these reasons have been described in the literature, but so far, research is missing systematic classification and factors associated with missing data in CT. A conceivable classification may be factors resulting in missing CT data that are not unique to patients with CKD but possibly more prevalent and factors that may be unique to those with CKD. Factors not unique to patients with CKD, such as refused consent and lack of motivation, likely cause bias in all research fields. Depression is also more prevalent in CKD but will affect enrolment in all CF studies. Patients with CKD will have a high prevalence of comorbid conditions, including decreased vision and hearing. The language problem can occur during CT and depends on the goal of the study. Some tests have been developed in only one particular language, or the objective is to compare the cohort results with norms. In light of socioeconomic status as a risk factor, CKD may be associated with illiteracy to some extent, which can result in misunderstanding the instructions. Contrarily, factors that may be unique to those with CKD are testing during dialysis (i.e., exclusion due to alarms of the dialysis machine) as well as issues resulting from dialysis access in the dominant arm and not being able to complete CT if writing is required. Moreover, the inability to complete CT because of proven dementia should be considered as a separate issue and result in precluding patients. In the future, these patients should just be included in the numerator of those with cognitive impairment.

To draw a more precise and representative picture of the CF in typical patients on dialysis, whose visual, language, or motor abilities are rather poor, these individuals should be tested with adequate instruments in future studies. Sophisticated nonvisual and nonverbal CT may be a valuable resource here. A body of literature refers to these forms of CT (28–35). Another approach is the application of auditory screening instruments, such as the Telephone Interview for Cognitive Status (TICS [36]), which can be administered face to face or by telephone (37). The TICS is especially known as a reliable alternative screening instrument for dementia (38,39) and also, validly applicable for poststroke patients (40). Compared with the MMSE, the TICS shows fewer ceiling effects (37) and more predictive value for memory capacity (41). In sum, operational suggestions are to consider alternative CT that all patients with CKD can perform, consider the inclusion of tests that are less dependent on vision or writing, perform CT on nondialysis days with the understanding that one may lose generalizability for those who enroll, and translate validated tests into other languages.

Limitations

Several limitations of our approach have to be discussed. Within the CORETH Study, we precluded a large amount of screened patients, but we cannot provide the exact portion with proven dementia. The expense of documentation would have been too extensive regarding the exclusion of >6000 ineligible patients within 1 year of data collection. We estimate an amount of >95% preclusion due to the time criterion of 6–24 months with dialysis treatment. This may be somewhat difficult for an article focusing on a missing analysis on the first view. However, it should be kept in mind that we derived CT data from the CORETH Project, which is, in turn, the only up to date multicenter study on psychosocial and neurocognitive factors in the early phase of dialysis throughout Germany with a comparably large patient-centered data pool (16,19,20).

Almost all missing data were in the d2-R score, whereas no data were missing in the KDQoL-SF cognition subscale. Here, we can only speculate about the reasons. It is very likely that the TMT-B is easier to understand and that its symbols and instructions are more clear to the participants than those of the d2-R. Unfortunately, we cannot invoke norm data of the d2-R scores for patients with CKD. However, it has already been successfully applied in older adults (>60 years old [26]), although the authors do not indicate a clear statement about patients exclusion. Because the TMT-B was performed before the d2-R in our CT procedure, we cannot rule out sequence effects. Performing the TMT-B may have demoralized some patients, even if it is the easier of two. Because the patient-administered KDQoL-SF cognition subscale contains only three handy items, it could have been the easiest for all patients. The application of only two cognitive tests instead of a comprehensive neurocognitive test battery is another limitation of the study. In CKD, this is particularly relevant because of patient exclusion. On the one hand, by including many tests, one can deal with some missing data on an individual test. On the other hand, our literature screening (Supplemental Material) yielded that, empirically, inconsistent and varying utilizations of terms, like inclusion, exclusion, eligible, etc., across studies tend to hamper the transparency of missing analyses and the actual amount of missing data.

Because lumping together heterogeneous aspects is always in some way subjective, our derived exclusion categories are only one possible solution. It is likely that there exist intercorrelations between them (e.g., motivational aspects versus depression), leading to common intersections. Hence, all that we can provide are reason clusters and their frequency regarding the question of why preselected patients are excluded online from a CT procedure.

We conclude that there are unavoidable but also, avoidable exclusions for CT in the CKD population. Although the correct classification and documentation may be challenging, the reasons for unavoidable exclusion potentially related to cognitive impairment should be specified and listed numerically to allow the reader to draw conclusions to its subgroups and their magnitude. However, patient exclusion due to avoidable reasons, such as language difficulties or visual/hearing or motor impairment, should be addressed by fair CT in the future. Our frequency analysis is just the initial trial to sharpen the focus on the extent of bias in research on cognitive abilities in patients with CKD. In practice, non–CT-eligible patients with CKD may have a higher need for support and differentiated counseling. Consulting staffs should be aware of the problem of missing data in CT in the CKD population, because patients with misestimated CF may receive suboptimal shared decision making. They may even be subject to suboptimal choice of dialysis modality (i.e., HD versus PD) due to an under- or overestimation of their cognitive abilities.

Disclosures

None.