Telemedicine and the Mini-Mental State Examination: Assessment from a Distance

The Mini-Mental State Examination (MMSE), a cognitive function assessment tool, was administered via telehealth with the assistance of a face-to-face collaborator. The study included 73 patients with type 2 diabetes. MMSE scores were approximately the same for both remote and in-person assessment, indicating a high correlation. Some words such as “quarter” were misunderstood by the telemedicine patient. The results indicate that telehealth for cognitive assessment by MMSE is a useful tool.

Abstract

The objective of this study was to determine the reliability of the Mini-Mental State Examination (MMSE) administration via telehealth with a focus on the auditory and visual test components. Reliability was assessed through use of an in-person collaborator and by assessment of faxed test copies. The MMSE was administered via telehealth with the assistance of a face-to-face collaborator. Patient responses were recorded by both the remote and in-person nurse and compared item by item; total scores for each subject were also compared. Visual items were assessed through a blinded separate scoring of a faxed copy. Percent agreement per item and total score were calculated and correlations between scores were determined by Pearson correlation coefficients. Mean score differences and associated 95% confidence intervals were calculated. Eighty percent of individual items demonstrated remote to in-person agreement of >95% and all items were >85.5% in agreement. Pearson correlation coefficients demonstrated high correlations (>0.86) between 80% of the items examined. Mean differences in scored test items were not significantly different from zero. This study demonstrates the utility of using telehealth for cognitive assessment by MMSE. It supports the use of telehealth to improve healthcare access among patients for whom distance, cost, and mobility are potential barriers to attending face-to-face clinical visits. Continued validation and reliability testing is warranted to ensure that all healthcare provided via telehealth maintains an equal quality level to that of in-person care.

Introduction

In the face of an aging population and rising healthcare costs, telehealth technology continues to grow rapidly throughout the United States. For patients living in rural or frontier areas, telehealth can translate to increased access to vital healthcare services, e.g., clinical care, consultation, counseling, and education. As telehealth expands, so does the need to validate and reliability test patient care delivered in this manner, to ensure that equal levels of quality healthcare can be provided using telehealth as through face-to-face encounters.

Cognitive testing through telehealth is one way to improve rural healthcare using this technology. A rapidly aging population translates to increasing numbers of individuals experiencing memory problems. There is, therefore, a serious need to develop convenient, yet valid and reliable, methods of cognitive assessment in rural areas, including screening questionnaires, e.g., the Standardized Mini-Mental State Examination (MMSE).

Several studies have demonstrated good correlation between results of various cognitive tests using telehealth technology as compared to face-to-face interview.^1–9 However, many are limited by small sample size or fluctuating clinical states of subjects between tests. This study improves upon past studies by using a larger sample size and focusing on the auditory and visual components of the technology in a single testing session.

The objective of this study was to determine the reliability of MMSE administration via telehealth, focusing on the auditory and visual test components. Reliability was assessed through use of an in-person collaborator and by assessment of faxed test copies. A single MMSE was conducted on each subject.

Materials and Methods

Eastern Montana Telemedicine Network

The Eastern Montana Telemedicine Network (EMTN) was initiated in 1993 in eastern Montana to improve access to medical and behavioral specialty services through the use of interactive videoconferencing. Today EMTN has 33 partners in 20 rural and frontier communities throughout eastern/central Montana and northern Wyoming. Since its inception, EMTN has conducted 17,837 clinical encounters averaging over 1,200 encounters annually.

Standardized Mini-Mental State Examination

The MMSE is a widely used tool to assess cognitive function during clinical visits.¹⁰ The 12-item scale tests five areas of cognitive function: orientation, registration, attention and calculation, recall, and language. The maximum score is 30; a value of <23 is a positive screen for cognitive impairment. The test takes 5–10 minutes to administer and has been validated and extensively used in clinical practice and research.¹⁰

Study Design

Study participants were part of a larger diabetes self-management study, in which rural patients agreed to have regular clinical visits with members of a diabetes management team using EMTN telehealth technology. Baseline and annual assessments included the administration of the MMSE to identify potential cognitive impairment that might preclude continued study participation. Four participating rural sites agreed to participate. MMSE reliability testing occurred on all new rural patients from February 2006 to May 2007.

Instead of the standard two-rater reliability test, differences in recorded patient responses were identified by adding a face-to-face collaborator to the remote assessor. Patient responses, as recorded by both the remote and in-person nurse, were compared item-by-item; total scores for each subject were compared. The test items, as read by the remote nurse, were recorded by the in-person nurse, for clarity and understanding. Visual items, i.e., drawing and writing, were scored twice, once at time of assessment via telemedicine, and later using a faxed copy. Both were scored by the remote assessor who was blinded during the second scoring. Item-by-item comparisons were made to identify specific MMSE items that might be more vulnerable to error under the remote assessment condition. Percent agreement per item and total score were calculated and correlations between scores were determined by Pearson correlation coefficients. Mean score differences and associated 95% confidence intervals were calculated. Stratification by site was not possible due to sample size.

This study was approved by the Billings Institutional Review Board.

Results

Seventy-three eligible patients were included in the study. Fifty-six percent of subjects were female; average age was 61 years (range 36–90 years). All patients had type 2 diabetes; 17% were previously diagnosed with depression.

Mean and standard deviation MMSE score by remote and in-person assessment was 28.7 ± 1.61 (range 24–30) and 28.7 ± 1.51 (range 24–30), respectively. No patients received a score indicating cognitive impairment, and scores did not differ significantly from study subjects with pre-existing depression (28.4 ± 2.07, p = 0.7), nor from the larger study population assessed in-person (28.5 ± 1.67, p = 0.9). Eighty percent of individual items demonstrated remote to in-person agreement of ≥95% and all items were ≥85.5% in agreement. Pearson correlation coefficients demonstrated high correlations (≥0.86) between 80% of the items examined. No items read by the remote nurse were recorded as unclear by the in-person nurse. Among patients, the word “quarter” in the three-word memory item was sometimes not understood in the telemedicine condition. Figure 1 demonstrates the mean score difference on each of the 12 test items, and the associated 95% confidence intervals. None of score differences was significantly different from zero.

Fig. 1.

Standardized Mini-Mental State Examination score mean differences* (remote versus in-person) and 95% confidence intervals (n = 73). UCL, upper confidence limit; LCL, lower confidence limit.

The sample size of 73 patients would yield 80% power to detect a small-to-medium effect size¹¹ using a two-tailed paired t test with type I error rate = 0.05.

Discussion

Few studies have demonstrated the reliability or validity of cognitive testing via telehealth. Loh and colleagues⁶ measured the inter-rater reliability between face-to-face and telehealth administration of the MMSE and the Geriatric Depression Scale and found a 90% correlation between direct and remote MMSE scores. However, in 40% of cases, the difference between telehealth and direct assessments was two points or more. Another recent study by Loh et al.⁴ demonstrated similar results with good agreement (κ = 0.8, p < 0.0001) for diagnosing Alzheimer's disease between telemedicine and direct assessment. Both studies were limited by small sample sizes.

Utilizing slightly different methods, this study clearly demonstrated a high correlation between in-person and remote recording of individual MMSE items. Mean differences in scores were well below those demonstrated in past studies,⁶ and were not significantly different from zero. While it did not affect the results, the word “quarter” was not always understood by the telemedicine patient. The authors suggest substituting a different word, as permitted by the instrument creators.

This study demonstrates the utility of using telehealth for cognitive assessment by MMSE, albeit across a narrow range of functioning. However, if telehealth error were to occur, it would likely produce false positives, which would initiate further testing. This supports the use of this technology for lower functioning patients because cognitive impairment is unlikely to be missed. This study supports the use of telehealth to improve healthcare access among patients for whom distance, cost, and mobility are potential barriers to attending face-to-face clinical visits. Continued validation and reliability testing is warranted to test more variable patient populations and to ensure that all healthcare provided via telehealth maintains an equal quality level to that of in-person care.

Acknowledgments

The authors would like to acknowledge the nurses and clinic staff at the four participating rural clinics in this study as well as the entire Center for Clinical Translational Research staff. We also acknowledge the EMTN, without which this study would not have been possible. Funding for this study was provided by the National Institute of Diabetes and Digestive and Kidney Diseases.

Disclosure Statement

No competing financial interests exist.