A validation study for remote testing of cognitive function in multiple sclerosis

Abstract

Objectives

Determine the validity and reliability of a remote, technician-guided cognitive assessment for MS, incorporating the Symbol Digit Modalities Test (SDMT) and the California Verbal Learning Test, Second Edition (CVLT-II).

Methods

In 100 patients, we compared conventional in-person testing to remote, webassisted assessments, and in 36 patients, we assessed test-retest reliability using two equivalent, alternative forms.

Results

In-person and remote administered SDMT (r=0.85) and CVLT-II (r=0.71) results were very similar. Reliability was adequate and alternative forms of SDMT (r=0.92) and CVLT-II (r=0.81) produced similar results.

Conclusions

Findings indicate remote assessment can provide valid, reliable measures of cognitive function in MS.

INTRODUCTION

Full characterization of cognitive ability in MS requires psychometric assessment using tests that require a human interface that is a neuropsychologist or similarly trained professional. This is not practical for large clinical or epidemiologic studies, which need data collected on several hundreds or thousands of participants. Thus, an easily administered, inexpensive and sensitive assessment tool that could be utilized remotely would facilitate large studies of risk factors for cognitive health in MS. Such a tool could also facilitate clinical management by early identification of cognitive deficits.

Several computerized assessments for cognitive function were validated in MS samples although extent of examiner oversight is debated.¹ Further,computer-assistedmethods do not incorporate voice recognition methodology, and as such, memory tests are restricted to recognition, not recall memory. The California Verbal Learning Test, Second Edition (CVLT-II)² and the Symbol Digit Modalities Test (SDMT)³ are validated performance tests of episodic memory and processing speed that comprise part of the recommended Brief International Cognitive Assessment for MS (BICAMS).⁴ Alternative forms of the SDMT show excellent reliability and equivalence.⁵

We developed a technician guided, web-based tool to measure cognition using the CVLT-II and the SDMT. Previously, we determined the feasibility of a remote assessment protocol⁶ by comparing telemedicine CVLT-II data from MS patients to data acquired in person from an independent sample of cases and controls. CVLT-II assessments by conventional in-person and remote assessment yielded indistinguishable results. Herein, we aimed to replicate these results and extend them to the SDMT.

METHODS

MS cases were adult members of the Kaiser Permanente Medical Care Plan in Northern California Region (KPNC), an integrated health services delivery system. A total of 100 MS cases were studied for cognitive assessment validation. An independent sample of 36 cases were studied for cognitive assessment reliability. Study protocols were approved by the Institutional Review Boards at UC Berkeley and Kaiser Permanente Division of Research. All study participants provided informed consent. The staff interviewer was formally trained in administering the SDMT, CVLT-II, and multiple equivalent forms of SDMT and CVLT-II, and conducted all remote assessments using the same procedure. For further details, see Supplementary Materials.

To validate remotely administered SDMT and CVLT-II, we assigned 50 participants to receive the in-person assessments followed by remote assessment, and 50 participants to receive the reverse order. Similarly, to test the reliability of two equivalent, alternative forms of remote-administered SDMT and CVLT-II, we assigned 18 participants to receive Version 1 followed by Version 2 (SDMT) and standard followed by alternative forms (CVLT-II), and 18 participants to receive the reverse order, respectively. All in-person and remote assessments were scheduled 1–2 weeks apart.

RESULTS

Subject characteristics are shown in Table 1. Remote and in-person assessments of SDMT were strongly and significantly correlated (r=0.85, p<1.0×10⁻²⁸) (Figure 1A). Similar results were observed for CVLT-II testing (r=0.71, p<1.0×10⁻¹⁵) (Figure 1B). Correlations did not significantly change depending on order of assessment: in-person (SDMT p<1.0×10⁻¹³, CVLT-II p<1.0×10⁻⁸) or remote (SDMT p<1.0×10⁻¹², CVLT-II p<1.0×10⁻⁸) administered first (data not shown). Parallel versions of SDMT (r=0.92, p<1.0×10⁻¹⁴) and CVLT-II (r=0.81, p<1.0×10⁻¹⁰) were highly correlated (Figures 1C and 1D).

Table 1.

Characteristics of Kaiser Permanente Northern California Region (KPNC) MS casesin validation and reliability study samples

	Mean (SD) or Frequency (%)
Characteristic	Validation Sample	Reliability Sample
Number of individuals	100	36
Age at assessment (years)	58.2 (9.4)	56.8 (9.2)
Sex (Female)	81 (81%)	25 (69.4)
Disease subtype (RRMS and SPMS)	86 (86%)	28 (77.8)
Education (any college)	93 (93%)	31 (86.1)
Race and ethnicity (White, non-Hispanic/Black/Other)	86 (86%)/13 (13%)/1 (1%)	36 (100%)/0/0/0
Disease duration (years)	25.5 (10.0)	22.9 (9.4)
PHQ-9 score	16.8 (5.6)	16.7 (4.8)
MSNQ score	18.9 (11.7)	19.0 (12.0)

The remote assessment included the Patient Health Questionnaire (PHQ-9)⁹ and the MS Neuropsychological Screening Questionnaire (MSNQ).¹⁰ Abbreviations: KPNC, Kaiser Permanente Northern California Region; RRMS, relapsing remitting MS; SD, standard deviation; SPMS, secondary progressive MS

Figure 1.

Comparison of remote and in-person A) SDMT assessments and B) CVLT-II assessments, and comparison of remote administered equivalent, alternative forms of C) SDMT and D) CVLT-II. Pearson correlations were used to assess relationships between remote and in-person administrations of the SDMT and CVLT-II (validation study) and alternative forms of the remote-administered SDMT and CVLT-II (reliability study). Statistical analyses were conducted using R.

Linear regression results showed that males and individuals with no college education had worse CVLT-II scores (ß=−7.36, p=0.003; ß=-7.92, p=0.03, respectively) compared to female and college educated individuals (Supplementary Table 1). A self-reported history of recent relapse was associated with worse CVLT-II (ß=−6.32, p=0.02) and SDMT (ß=−5.36, p=0.06) scores. Longer disease duration was associated with worse SDMT scores (ß=−0.29, p=0.02).

DISCUSSION

Results indicate SDMT and CVLT-II data collected through remote, web-based, technician-guided administration yield similar results to standard, validated in-person measures of the same assessments. Further, two alternative, parallel forms of remotely assessed SDMT and CVLT-II show high reliability, indicating either form can be used to obtain equivalent SDMT or CVLT-II results. These web-based assessments show promise for appraisal of cognition when circumstances prevent patient attendance in the neurology clinic.

Our work represents an initial effort to address the challenges of cognitive testing in MS by measuring episodic memory and processing speed remotely. While the MS data indicate the remote and conventional data are roughly equivalent, it remains to be seen whether new normative data are needed for clinical utilization. Notably, the BICAMS recommended visual memory test, the Brief Visuospatial Memory Test Revised, was not included in this study because a manual response (drawing figures from memory) is required. While one can envisage how current videoconference technology might overcome this limitation, we have not pursued such a course, to date. We acknowledge that the BVMTR is recognized as a sensitive test in MS. Furthermore, this research was supported by national funding. It remains to be seen if the work will translate to an insurance-based, routine clinical practice, that can retain adherence to these protocols to maintain reliability.

Experiencing a self-reported recent relapse was associated with worse CVLT-II and SDMT scores. Notably, ~18% of MS cases reported having a relapse within 4 weeks prior to cognitive testing; ~75% of these individuals were relapsing at time of study (overall ~13% of all participants). Cognitive decline occurs in relapses and can be missed by routine neurological care. In a recent study, SDMT scores significantly declined from baseline when MS patients experienced a relapse and recovered to slightly below baseline levels three months later.⁷ CVLT-II scores had similar but trending results. Our results strongly agree with these findings; however, more work is need. Assessing patients in a guided, remote manner ensures reliable assessment of cognition on an as-needed basis and could detect cognitive relapses.

By assessing memory in patients prior to a clinic visit using a web-based approach, we may be able to meet the goal of routine neurocognitive monitoring of MS patients, as recently suggested.⁸ A positive test could identify individuals with early cognitive impairment who are at increased risk of job loss, acute disease activity and rapid disease progression. MS clinic staff could be alerted prior to patient visits. Patients who are at greatest risk for cognitive impairment or who present with cognitive impairment could be targeted for intervention strategies to improve cognitive function.

RHBB received honoraria or speaking fees from Biogen, Celgene, EMD Serono and Novartis, provided consultancy for Biogen, Celgene, Medday, Novartis, Roche, Sanofi, and Verasci, and has received research support from Biogen, Genentech, Genzyme, Mallinckrodt, and Novartis.