Telephone-Based Cognitive-Behavioral Screening for Frontotemporal Changes in Patients with Amyotrophic Lateral Sclerosis (ALS)

Abstract

Objective

To establish a valid and reliable battery of measures to evaluate frontotemporal dementia (FTD) in patients with ALS over the phone.

Methods

Thirty-one subjects were administered either in-person or telephone-based screening followed by the opposite mode of testing two weeks later, using a modified version of the UCSF Cognitive Screening Battery.

Results

Equivalence testing was performed for in-person and telephone-based tests. The standard ALS Cognitive Behavioral Screen (ALS-CBS) showed statistical equivalence at the 5% significance level when compared to a revised phone-version of the ALS-CBS. In addition, the Controlled Oral Word Association Test (COWAT) and Center for Neurologic Study-Lability Scale (CNS-LS) were also found to be equivalent at the 5% and 10% significance level respectively. Similarly, the Mini-Mental State Examination (MMSE) and the well-established Telephone Interview for Cognitive Status (TICS) were also statistically equivalent. Equivalence could not be claimed for the ALS-Frontal Behavioral Inventory (ALS-FBI) caregiver interview and the Written Verbal Fluency Index (WVFI).

Conclusions

Our study suggests that telephone-based versions of the ALS-CBS, COWAT, and CNS-LS may offer clinicians valid tools to detect frontotemporal changes in the ALS population. Development of telephone-based cognitive testing for ALS could become an integral resource for population-based research in the future.

Introduction

Cognitive impairment and behavioral changes in patients with Amyotrophic Lateral Sclerosis (ALS) have been well-recognized symptoms of the disease for the past few decades (1). Frontotemporal dementia (FTD), or frontotemporal lobar degeneration (FTLD; referring to the neuropathology of FTD), is often considered a comorbid disorder of ALS (2–5) with approximately 15% of ALS cases possessing both diagnoses (4). Related cognitive and behavioral disturbances in ALS have a prevalence rate of up to 50% and mirror clinical symptoms found within the subtypes of FTLD (behavioral variant FTD, progressive non-fluent aphasia, and semantic dementia) (6, 7). Recent literature, suggesting that ALS and FTLD share pathological, clinical, and genetic features, corroborates not only their comorbidity but the likely existence of a spectrum or continuum of impairment between the disorders (1, 2, 8–13).

Neuropsychological testing for the ALS population thus must be sensitive enough to detect cognitive and behavioral changes within this continuum, which may present throughout the disease course. Due to the physical impairments of ALS patients, typical neuropsychological screening tests are inappropriate to use (1, 14). In the past decade, a number of ALS-specific screening tests have been developed to measure the heterogeneous nature of frontotemporal changes in ALS by eliminating physical tasks that are difficult for patients and assessing the relevant domains of cognitive and behavioral impairment. These validated screening tests include the ALS-Cognitive Behavioral Screen (ALS-CBS) (15), the Edinburgh Cognitive Behavioural ALS Screen (ECAS) (16), the Penn State cognitive screen (17), the Cambridge Behavioral Inventory (18), the ALS Frontotemporal Dementia Questionnaire (ALS-FTD-Q) (19), and the MiND-B (20).

Despite this recent growth in ALS-specific screening instruments, the utility of these tests remains limited to in-person evaluations. Because of the severe physical disabilities characteristic to the disease, testing can become costly, time-consuming, and fatiguing for patients who travel to their appointments with a plethora of issues and limited clinic time (21). Telephone-based, cognitive tests for neurodegenerative diseases have shown to successfully relinquish some of these associated burdens and costs and may increase generalizability of results by increasing sample sizes and reducing selectivity (22). Unfortunately, most telephone-based screening tools assess age-related, Alzheimer’s-type memory impairment and are not entirely applicable to the screening of FTD for ALS patients.

The development of a telephone-based screening battery to detect frontotemporal changes in ALS is necessary and will help relinquish the burdens associated with in-person testing. Additionally, a telephone screening battery for FTD specifically for ALS patients has never been undertaken in previous studies and can be an invaluable tool for epidemiological research. The University of California San Francisco (UCSF) Screening Battery has demonstrated effectiveness in screening for cognitive and behavioral impairment in patients with ALS (23). Because telephone-based frontotemporal screening has never been developed, we modified the UCSF Screening Battery to be used over the phone and tested its reliability.

Materials

UCSF Screening Battery

Assessment of Cognitive Functioning

ALS-CBS

The ALS Cognitive Behavioral Screen (ALS-CBS) was developed by Woolley et al. (2010) and identities patients with cognitive and behavioral changes with 71% specificity and 85% sensitivity (21). The ALS-CBS contains both cognitive and behavioral sections to assess executive functioning. Patients for this particular measure are not required to write down words. The cognitive section is completed by the patient while the behavioral section (discussed below) is completed and self-administered by the patient’s caregiver. This measure serves as a valid alternative to general comprehensive neuropsychological test batteries that are considered too time-consuming and fatiguing for patients with ALS (21). This test can be administered in 5 minutes and yields a total cognitive score ranging from 0–20, generated from four subtests: initiation and retrieval, concentration, attention, and tracking-monitoring.

For the purpose of this study, the cognitive section of the ALS-CBS was modified to administer over the phone. We eliminated sections which rely on visual assessment, such as direct commands and removal of eye movements. The saccades and anti-saccades tasks were replaced with tapping commands to assess cognitive inhibition. All other components of the cognitive section remained the same.

WVFI and COWAT

The Written Phonemic Fluency Test (WVFI) has been used to assess executive functioning and intrinsic response generation in ALS patients (24). Patients are instructed to write down as many 4-letter, “C” words they can think of in four minutes. After the initial four minutes is up, the patients must then re-write all the words to determine their “copy time” with the interviewer. Their verbal fluency index score is then calculated by subtracting their “copy time” from 4 minutes and dividing this total by the correct number of words. Thus, “thinking time” is determined by adjusting the score with the patient’s writing speed, controlling for both dysarthria and hand weakness (24). When administered over the phone, patients must relay to the interviewer when they have finished copying. Additionally, for this study, patients were asked to mail back their written responses in a pre-stamped envelope provided by the test interviewer. Since some patients are unable to write due to their physical limitations, the Controlled Oral Word Association Test (COWAT) was also administered (25). The in-person and telephone versions of the WVFI and COWAT did not differ with regard to instruction or content. All participants completed both the WVFI and COWAT for comparison purposes.

Assessment of Behavioral Functioning

ALS-CBS-CG

The behavioral section of the ALS Cognitive Behavioral Screen (ALS-CBS) is completed by the patient’s caregiver, which we label as ALS-CBS-CG. The caregiver rates the level of witnessed behavioral change based on established norms. For the telephone version of this section, an additional question was added for the assessment of language difficulties, inquiring whether the patient has been saying the wrong words, making up new words, and has demonstrated frequent spelling errors. This addition and update to the ALS-CBS-CG was found to be more valid among ALS patients, who demonstrate language issues throughout their disease course (26). Because the newer version of the ALS-CBS-CG totals to 15 items compared to the original 14 items, different analyses were administered to account for total differences (45 vs. 42 total points).

ALS-FBI

The Frontal Behavioral Inventory (FBI) is administered to a caregiver by trained interviewers and assesses behavioral and personality changes in patients (27). A modified version of the FBI specifically for patients with ALS was utilized (23) (23). This version contains clarifying questions that assist caregivers in assessing symptoms and behaviors unrelated to physical impairments caused by the disease itself. Behavior change was rated from 0–3 on 24 items, grouped into negative behavior and disinhibition subscales. The in-person and telephone FBI-ALS did not differ with regard to instruction or content.

Supplemental Measures

CNS-LS

The Center for Neurological Study-Lability Scale (CNS-LS) was developed by Moore et al. (1997) and serves as a well-validated, self-report assessment of pseudobulbar affect, a neurological condition that afflicts around 50% of ALS patients presenting bulbar symptoms (28). The in-person and telephone CNS-LS did not differ with regard to instruction and content.

MMSE and TICS

The Mini-Mental State Examination (MMSE) is the most widely-used, in-person instrument to assess global cognitive impairment (29). The Telephone Interview for Cognitive Status (TICS) developed by Brandt et al. 1988 is a telephone-based instrument that was originally developed to evaluate cognitive functioning for Alzheimer’s disease (30). The TICS is now administered internationally and has become the most widely used cognitive screening test for epidemiological surveys (30, 31). For this study, the version TICS-40 was used, containing 40 items and including the addition of a delayed word recall task. These tests were not modified for this study and were used for comparative purposes to the UCSF Screening Battery.

Methods

The Institutional Review Board (IRB) of Columbia University approved the study protocol and instruments. Clinic charts were pre-screened, and eligible patients and caregivers were approached during routine clinic visits. Eligibility criteria included patients 1) diagnosed with sporadic ALS (definite, probable and possible ALS based on the El Escorial Criteria), progressive muscular atrophy, suspected primary lateral sclerosis, bulbar palsy, or predominantly upper motor neuron disease; 2) diagnosed by the attending neurologist; 3) with a disease duration of ≤ 18 months after symptom onset; 4) aged 20 years or older; 5) reliable family caregiver who could give independent informed consent for providing information and assisting with testing; 6) fluent in English; 7) capacity to consent -- can understand and sign the Informed Consent approved by the IRB and the form for HIPAA regulations. Researchers reviewed the consenting process with the patients and their caregivers, discussed the study protocol, and answered any questions that arose. Patients and caregivers both gave their written consent.

Patients were administered both in-person and telephone-based testing and assigned to reverse-order groups, first receiving in-person or telephone evaluations followed by the opposite mode of testing two weeks later (tests included as supplementary files). For telephone screening, patients were prompted to practice tapping the phone to ensure they would be able to complete the tasks required for some of the measures prior to actual testing. Voice volume and quality were also checked. Assignment to an order of screening was initially randomized; however, this had to be changed due to limited time availability of participating patients. Two interviewers, trained by Dr. Jennifer Murphy, PhD to administer the screening battery, performed the testing with one interviewer carrying out both modes of testing for each patient. Visit type (telephone or in-person) and sequence of visits were recorded.

Statistical Analyses

The mean age for this sample of patients was 62, 65% of the sample was male, and most participants were college educated, as demonstrated in Table 1. Models for analyses were adjusted for sex, age, and education (dichotomized to college education vs. others); none were found to be significant.

Table 1.

Sample Characteristics

Variable	Patients
Demographics	n=31
Age
Mean (SD)	62 (8)
N (%) male	20 (65%)
Education	n=31
College degree	21 (68%)
Some college but no degree	2 (6%)
Associate’s degree	1 (3%)
High school degree or GED	5 (16%)
Some high school	1 (3%)
Grade school	1 (3%)
ALSFRS Score	n=31
Mean (SD)	37 (9)
FVC %	n=31
Mean (SD)	85 (17)

n = total number of patients

SD = standard deviation

Equivalence testing methods were used to compare the battery of instruments across two testing modes. These methods are rigorous alpha-level analyses used by the FDA to compare generic drugs to standard drugs (32). Equivalence testing requires the analyst to determine a priori what is considered equivalent; e.g., if the ratio of group means is between equivalence bounds of 0.8 and 1.25 (=1/0.8). Analyses are based on the judgment that the confidence interval that is narrow enough to be within the pre-determined equivalence bounds demonstrates equivalence. Therefore, when the sample size is too small and/or the confidence intervals are too wide, equivalence cannot be claimed.

Equivalence bounds were defined a priori as lower equivalence bound, EL=0.8, and upper equivalence bound, EU=1.25. Generally, m0 was defined as the mean of an instrument with visit type 0 (telephone testing) and m1 was defined as the mean of the instrument with visit type 1 (in-person testing) with a ratio of m0/m1. Equivalence between the two tests is claimed at significance level α if and only if the 100 (1–2α)% confidence interval for the mean ratio is contained completely within (EL,EU). All data were log transformed and for each instrument, we used a mixed model to estimate the differences between the in-person and telephone versions of each test, controlling for visit type. Mixed models allow for correlated within-subject observations, while assuming independence across subjects. Confidence intervals were constructed for differences in the means (log-transformed values) and asymmetric intervals were estimated as [exp(L), exp(U)].

The ALS-CBS was selected as the primary instrument a priori; analyses of other instruments were secondary tests and thus adjustment for multiple testing was not necessary. Due to the different scales for the MMSE and TICS and the ALS-CBS-CG, we calculated the percentages of the totals (30 points for the MMSE and 41 points for TICS) for each subject. Similar calculations were done for the ALS-CBS-CG, which contains 42 total points for the in-person score and 45 total points for the telephone score.

For tests not found to be equivalent, a post-hoc sample size study was performed to determine the number of subjects necessary to make the confidence interval within the required range. The given mean and standard deviation of the differences in the log transformed scales were assumed and the minimum sample size was such that the $\mathit{mean}-t_{0.95,N-1}\sqrt{\frac{\mathit{STD}}{n-1}}>log(0.80)=-0.223$ and $\mathit{mean}+t_{0.95,N-1}\sqrt{\frac{\mathit{STD}}{n-1}}<log(1.25)=0.223$.

In secondary analyses, intra-class correlation coefficients were estimated for each instrument – using the percent values for instruments on different scales. Finally, we tested whether the sequence of testing made a difference for the primary instrument and those tests that failed to show equivalency. A mixed-effects model was parameterized to include sequence (e.g., telephone then in-person testing or in-person then telephone testing) and visit type (telephone or in-person).

Results

Thirty-one patients and their caregivers completed both in-person and telephone testing. The mean age (±SD) of the patients was 62 (± 8.7), 35% of the patients were women, the mean ALSFRS-R score was 36.7 (± 5.5), and the mean %FVC was 84.7 (±16.6). Following equivalence testing methods, the ALS-CBS demonstrated equivalency at the 5% significance level. In further analyses, there was no significant sequence effect in the evaluation of ALS-CBS (p=0.56).

The COWAT and CNS-LS were found to be statistically equivalent across the two testing modes at the 5% level. Similarly, the MMSE and TICS were statistically equivalent when the scales were on the percent of total values; and the ALS-CBS caregiver tests on the transformed percent scale were also equivalent across testing procedures. The variability in the ALS-FBI and WVFI was such that equivalence between the two testing methods could not be claimed. The sequence of testing (i.e., telephone then in-person or in-person then telephone) was not significant for these two tests (p=0.143 and p=0.559, respectively).

For future planning, the sample size required for the confidence intervals to be narrow enough to claim equivalence were calculated, assuming the observed mean and variance of the log ratio of the means. For the ALS-FBI, a sample size of 61 subjects with both in-person and telephone-based testing is associated with a narrow enough confidence interval to be claimed equivalent. For the WVFI, a sample size of 78 subjects with both in-person and telephone-based testing is associated with a narrow enough confidence interval to be claimed equivalent. Intra-class correlation coefficients were estimated for the six instruments on the same scale between in-person and telephone testing (Table 2).

Table 2.

Results from equivalence testing across visit types. Equivalence is claimed with 5% significance (*) when the 90% confidence interval on the mean ratio is completely contained within the equivalence bounds of [0.80, 1.25]. For comparison, 80% confidence intervals are displayed for a test of equivalence with 10% significance (#) for those not significant at the 5% level.

Instrument	ICC	Asymmetric 90% Conf Int on ratio	Asymmetric 80% Conf Int on ratio
ALS-CBS	0.50	[1.00, 1.11] *
COWAT	0.34	[0.90, 0.99]*
CNS-LS	0.79	[0.94, 1.09] *
FBI-ALS	0.54	[0.72, 1.14]	[0.76, 1.08]
WVFI	0.76	[0.95, 1.32]	[0.99, 1.27]
MMSE_TICS (%)	b	[0.84, 0.93] *
ALS-CBS-CG (%)a	0.79	[0.95, 1.00] *

^aA constant value of 1 was added to accommodate the 0s in log transformed values.

^bThe ICC could not be estimated due to ill-conditioning of the covariance matrix of the random effects.

Discussion

The results indicate that some instruments within this battery are equivalent to in-person testing and effectively assess cognitive functioning over the phone. Both the cognitive and behavioral subscales of the ALS-CBS showed equivalence to in-person testing, as did the COWAT and the CNS-LS. For our comparative tests, the MMSE and TICS also demonstrated equivalency, as shown in past investigations (30, 33, 34). Parallel to the Alzheimer’s screening literature, where the TICS can dependably replace the in-person MMSE, we have demonstrated that instruments within the telephone-based battery to assess frontotemporal changes in the ALS population are equivalent to in-person testing.

On the other hand, the ALS-FBI and WVFI did not pass equivalence testing. Even when using less stringent analyses, namely intraclass correlation coefficients (ICCs) across all six measures, the ALS-FBI and WVFI still failed to show significant levels of agreement. It is important to note that not all ICCs corresponded with equivalence testing. This is primarily due to the fact that ICCs do not account for variability in the measurements, while equivalence testing accounts for both mean estimates and variability. Although the ALS-FBI and WVFI did not differ in content between the two testing modes, there are limitations and differences associated with the administration of these instruments.

The ALS-FBI assesses the caregiver’s subjective observations concerning the patient’s behavior and personality. Although high inter-rater reliability has been associated with this test, test-retest reliability has not been established (23). Caregivers may indicate different responses at different times depending on examples they relay to the interviewer. Their general mood and stress levels the day of testing could have also played a factor in their responses. Additionally, their perception and ability to discern symptoms of the disease from behavioral changes may have altered from the first round of testing. Caregivers, although interviewed separately from the patient in both conditions, may offer more honest or alternatively more guarded information when being interviewed over the phone in their home, as opposed to being in an office setting. Administration of the ALS-FBI is an in-depth, semi-structured interview between the interviewer and caretaker. Alternatively, the caregiver portion of the ALS-CBS-CG, which assesses many of the same behavior and personality changes as the ALS-FBI and was deemed equivalent, is self-administered by the caregiver and worded in a simple, straightforward manner. Thus, the inherent subjectivity and possible lack of test-retest reliability of the ALS-FBI may have accounted for the observed differences between testing.

The WVFI also failed to show equivalency. Test-retest reliability has not been established for this instrument. Since only two weeks separated the different testing modes, participants may have remembered their previous words and had more time to think and generate new words during the second round. In addition, the evaluator can visibly see when the patient has finished the task in-person and can write down an accurate copy time, while the patient has to relay completion of the task to the evaluator over the phone, creating a slight, possible delay. The evaluator can control for any distractions that may occur when attempting to complete the task in-person. These reasons may have accounted for the observed differences in our sample between in-person and telephone administration.

There were a few limitations to this study. A major limitation included the small number of study participants. With a sample size of 31, the ALS-FBI and WVFI did not show equivalence. For these specific tests, a larger sample size could narrow the confidence intervals enough for these measures to be possibly deemed equivalent. Additionally, all patients were recruited from one ALS center and are not representative of the entire patient population. Assignment of whether the patient had in-person or telephone-based testing first was not randomized and depended on the availability of the patient; however, analyses showed no order effects for the ALS-CBS, ALS-FBI and WVFI. Further analyses across all measures that include establishing test-retest reliability should be undertaken, particularly for the ALS-FBI.

It is important to note that we did not test this sample of patients for FTD or cognitive impairment prior to testing, since we only wished to compare scores between modes of administration. Additionally, 95% of the scores among the measures were below the maximum. For future use of this battery, however, clinical heterogeneity of the population must be taken into account when analyzing the scores from this measure. The sample of patients who participated in this study did not show high levels of respiratory impairment or bulbar symptoms, as indicated by the mean %FVC and scores on the ALSFRS-R. Patients with low ALS-FRS scores or those with a low %FVC may have a difficult time performing some of the tasks involved. Test administrators can experience difficulty understanding patients who have bulbar symptoms; a problem that can become exacerbated over the phone. Simple clarifications can be made in these instances to ensure the patient is understood. The addition of a tapping task in the ALS-CBS was predicted to be difficult for some patients who showed motor deficiencies. Test administrators were asked to practice this task with the patient before testing to determine if the task was feasible and audible over the phone. Instructions were included in the revised telephone version to accommodate these changes. Precautions concerning all of these issues must be taken into account before administering these instruments.

Despite the limitations of this study, development of telephone-based cognitive testing in the ALS population has never been undertaken and could become an integral resource, especially to large, population-based, research studies. This type of testing could be particularly useful for patients with severe and progressive physical disabilities and high caregiver demands such as ALS. Additional validation and reliability studies to support these findings are crucial and will greatly benefit the field.