Patient-Powered Reporting of Modified Rankin Scale Outcomes Via the Internet

Abstract

Background and Purpose:

The modified Rankin Scale (mRS) is a common and resource-intensive measure of functional outcome in stroke-related conditions. In this observational prospective cohort feasibility study, mRS scores are generated using a patient-powered online survey and compared to scores obtained by structured telephonic interview.

Materials and Methods:

Fifty-one patients with subarachnoid hemorrhage (SAH) or their surrogates responded to an online survey following discharge from the hospital. These responses were used to generate an mRS score and then compared to blinded telephonic assessments by trained personnel. A weighted kappa (K_w) with confidence intervals (CIs) was calculated.

Results:

The K_w between the patient/surrogate and the trained personnel scores was 0.85 (95% CI, 0.74-0.95, P < .001).

Conclusion:

This study provides first evidence that patient/surrogate survey responses may be an efficient and reliable alternative to generate mRS scores compared to trained personnel after SAH.

Introduction

The modified Rankin Scale (mRS) is the most common measure of functional outcome in stroke-related conditions.^1,2 Long-term functional outcomes are increasingly recognized as essential end points to study the effectiveness of interventions for acute neurological injury. However, there are limited means for systematically obtaining long-term functional outcomes in patients with brain injury. Existing methods such as face-to-face office visits, telephone calls, or regular mail surveys are resource intensive or less feasible due to technological changes. Currently available online tools do not feature a hierarchical design and yield error messages listing potential conflicting responses.³ In this study, we developed an inexpensive patient-powered online survey to generate long-term mRS scores. We hypothesized that a simple, rapid, online survey completed by a patient or a surrogate could generate an accurate mRS score when compared to trained personnel conducting telephonic structured interviews.

Materials and Methods

The “Survey Monkey” Web site service (www.surveymonkey.com) was used to develop a survey featuring an adaptive hierarchical design that minimized the number of responses required to generate an mRS score. A sample of the survey is available at: https://www.surveymonkey.com/s/VNPQRBT and may be accessed using subject ID number “0000.” In the initial phases of the survey design, direct feedback was solicited from a focus group of patients and surrogates to ensure that the questions in the survey were easily comprehended. Fifty-one patients enrolled in the Columbia University Subarachnoid Hemorrhage Outcomes Project provided informed consent and were included in this study. Requests to complete the survey were sent from July to October 2013 via e-mail at 14, 90, or 365 days following discharge from the hospital. Telephonic interviews by 3 trained personnel who were blinded to the mRS survey score were also conducted during this time period. Each patient was identified using a unique, anonymous identification number. A Cohen quadratic weighted kappa (K_w) was calculated to compare the interrater reliability between the trained personnel and the patient/surrogate groups. The study was approved by the institutional review board.

Results

Characteristics of the 51 patients with subarachnoid hemorrhage (SAH) are shown in Table 1. The patients’ ages ranged from 19 to 83. The K_w between the patient/surrogate and the trained personnel scores was 0.85 (95% confidence interval, 0.74-0.95, P < .001). The number of questions on the Internet survey required to generate an mRS score ranged between 2 and 14. The survey responder was the patient in 53% and a surrogate in 47% of cases. Two hundred and four e-mail requests were sent to obtain the 51 total responses. A wide range of survey mRS scores were represented (Figures 1 and 2). Exact agreement occurred between the patient/surrogate and trained personnel groups in 71% of cases. In all, 53% (10 of 19) of disagreements occurred with surrogate entries and 47% (9 of 19) occurred with patient entries. Seventy-nine percent (15 of 19) of disagreements deviated by a mRS scale of 1. Eighty percent of survey entries (41 of 51) were completed in less than 5 minutes.

Table 1.

Patient Characteristics and Demographics.

Age, mean	53.9 years
Female gender	62.7%
Hunt and Hess scale
1-2	59%
3	20%
4-5	21%

Figure 1.

Modified Rankin scores obtained by online survey.

Figure 2.

Modified Rankin scores obtained by telephonic interview.

Discussion

Simple, efficient, and cost-effective methods for measuring long-term functional outcome after stroke and other serious illnesses can vastly improve the value of hospital-based registries and observational cohort studies. Existing methodologies for obtaining basic measures of functional outcome and recovery, such as the mRS, require hired and trained staff to conduct telephonic or in-person interviews. Given the ubiquity of Internet access in the modern era and the straightforward criteria that differentiate various levels of the mRS, we hypothesized that self- or surrogate-reported mRS scores obtained via a brief Internet survey would have similar reliability to that of 2 different observers conducting an in-person interview.

In this sample of patients with SAH, there was very good interrater reliability (K_w = 0.85, 71% exact agreement) between the patient/surrogate and trained personnel groups. By comparison, prior studies on interrater reliability using structured mRS assessments have reported interrater variability (nonweighted Kappa) ranging between 0.50 and 0.74 with exact agreements in the range of 63% to 81%.^4-7 In one study from 1991, a nonstructured interview technique yielded a K_w = 0.90 with 80% exact agreement.⁸ In our sample, disagreements between the 2 groups were relatively evenly split between the patient responders (9 of 19; 47%) and the surrogate responders (10 of 19; 53%), suggesting that neither of the 2 groups were more likely to report an erroneous assessment.

Strengths of this study include the validation of a novel and relatively automated process to obtain long-term functional outcomes. In addition, online, patient-powered tools such as the one used in this study are consistent with a recent emphasis on electronic communications to engage patients in their own care.⁹ Limitations of this study include the small sample size, and the fact that only patients or surrogates literate in English were included. A direct translation to a different language may not yield identical results if, for example, the colloquialism is not adequately captured in the translation. It remains unclear how often patients and families will respond to Internet-based outcome surveys, compared to telephone, postal, or face-to-face interviews. These results need to be extended to patient populations other than SAH. Finally, the majority of patients in this study had low mRS scores, which may have introduced bias toward higher concordance.

In summary, this study provides first evidence that patient/surrogate survey responses may be an efficient and reliable alternative to generate mRS scores compared to trained personnel after SAH. Future studies are needed to replicate and test the validity of this type of survey in other languages, larger sample size, and different patient populations.