Compliance with Electronic Patient Reported Outcome Measure System Data Collection Is 51% Two-years After Shoulder Arthroscopy

Abstract

Objective

To determine patient compliance in completing electronic patient-reported outcome measures (PROMs) following arthroscopic shoulder surgery and identify risk factors for noncompliance.

Methods

A retrospective review of compliance data was performed for patients who underwent arthroscopic shoulder surgery by a single surgeon in a private practice setting from June 2017 to June 2019. All patients were enrolled in Surgical Outcomes System (Arthrex) as a part of routine clinical care, and outcome reporting was integrated into our practice electronic medical record. Patient compliance with PROMs was calculated at preoperative, three-month, 6-month, 1-year, and 2-year follow-up time points. Compliance was defined as a complete patient response to each assigned outcome module in the database over time. Logistic regression for compliance at the one-year timepoint was performed to assess for factors associated with survey compliance.

Results

Compliance with PROMs was highest preoperatively (91.1%) and decreased at each subsequent time point. The largest decrease in compliance with PROMs occurred between the preoperative and 3-month follow-up time points. Compliance was 58% at 1 year and 51% at 2 years after surgery. Overall, 36% of patients were compliant at all individual time points. There were no significant predictors of compliance with regard to age, sex, race, ethnicity, or procedure.

Conclusions

Patient compliance with PROMs decreased over time with the lowest percentage of patients completing electronic surveys at the traditional 2-year follow-up for shoulder arthroscopy. In this study, basic demographic factors were not predictive of patient compliance with PROMs.

Clinical Relevance

PROMs are commonly collected after arthroscopic shoulder surgery; however, low patient compliance may affect their utility in research and clinical practice.

Introduction

Patient reported outcome measures (PROMs) are an important component in orthopaedic registries. The use of PROMs in orthopaedics offers a way to better measure surgical outcomes by providing an objective assessment of the patients’ subjective experience, including measurements of pain, function, range of motion, and quality of life.¹ This patient-centered approach to post-operative care is not only beneficial to the patient and the care team, but it has the potential to contribute to value-based care models in the future.² Although there is increasing utilization of PROMs in the orthopaedic shoulder literature, there are limitations to the widespread use of PROMs in orthopaedic practice, including cost, lack of standardized outcome sets, and missing data.³

While significant resources are required to implement and maintain an outcomes based registry, recent literature suggests that compliance with PROMs is low in arthroscopy registries.⁴ Previously reported barriers to completion of PROMs in orthopaedics include age, race, and type of surgery.⁵ The PROM Working Group of the International Society of Arthroplasty Registries (ISAR) recommends data collection immediately before and 1 year after surgery, a threshold of sixty percent for acceptable frequency of response, documentation of non-responders, and documentation of incomplete or missing data.⁶ Arthroscopy Journal prefers eighty percent follow-up at two years. However, two-year follow-up rates for total hip and total knee arthroplasty in a private practice setting have been reported at less than sixty percent with decreasing rates at longer intervals.⁷ This study will offer community-based surgeons a standard for comparison of patient compliance with PROMs and identify opportunities to improve compliance in groups identified as at-risk for noncompliance.

The purpose of the current study is to determine patient compliance in completing electronic patient reported outcome measures (PROMs) following arthroscopic shoulder surgery and identify risk factors for noncompliance. Our hypothesis is that PROM completion will decrease over time. We additionally hypothesize that age less than 65 and male sex will be significant risk factors for noncompliance with electronic PROMs, as they have previously been associated with noncompliance after other orthopaedic procedures.

Methods

A retrospective review of outcome reporting data was conducted for elective arthroscopic shoulder surgery by a single surgeon (R.U.H.) in a private practice setting from June 2017 through June 2019. The study period was from the integration of PROMs into the practice electronic medical record until 2 years prior to study initiation. Inclusion criteria for the study included all patients undergoing arthroscopic shoulder surgery, including rotator cuff repair, subacromial decompression, biceps tenodesis, and/or distal clavicle excision. No patients were excluded from the study if they met the inclusion criteria during the study period. Retrospective review of the electronic medical record was completed to obtain basic demographic and procedural information. This study was exempt from institutional review board approval.

The primary outcome measure of the study was the rate of compliance with PROMs at standard follow-up intervals for shoulder arthroscopy. Once patients were scheduled for surgery, they were automatically enrolled into the electronic outcome reporting system (Surgical Outcomes System [SOS], Arthrex) and assigned to validated outcome modules based upon their indication for surgery. The integration of SOS into the practice electronic medical record allowed for ease of enrollment and monitoring. All patients completed modules assigned by the operating surgeon (Table 1). Patients with instability were also assigned the Western Ontario Shoulder Instability Index (WOSI). Patients received an automated, electronic notification by email and text with a link to complete the assigned order set(s) at preoperative, 3-month, 6-month, 1-year, and 2-year follow-up time points. Patients had the opportunity to complete the assigned order set within a set timeframe determined by SOS, as outlined in Table 1. Patients received reminder notifications until the requested forms were completed or the eligibility period ended. Compliance was defined as a complete patient response to the assigned outcome logged in SOS database at each individual time point. Patients that did not open, did not complete, or completed the module outside of the defined time window were categorized as noncompliant. Patients were encouraged to complete the outcome measures at follow-up appointments with a fellowship-trained shoulder surgeon (R.U.H.), who also had one medical assistant assigned to contact patients flagged by SOS for noncompliance.

Table 1.

Compliance Windows for Completion of Patient-Reported Outcome Measures in the SOS System

Follow-Up Appointment	Time to Complete Module(s)	Module(s) Assigned
Preoperative	Through the day of treatment	Standard preoperative form, VAS, ASES, SANE, VR-12
3 months	±2 weeks	VAS, ASES, SANE
6 months	±1 month	VAS, ASES, SANE, VR-12
1 year	±2 months	Standard postoperative form, VAS, ASES, SANE, VR-12
2 years	±2 months	Standard postoperative form, VAS, ASES, SANE, VR-12

ASES, American Shoulder and Elbow Surgeons Shoulder Score; SANE, Single Alpha-Numeric Evaluation; VAS, visual analog scale pain; VR-12, Veterans RAND 12 Item Health Survey.

Individual patient compliance with PROMs was calculated by the number of complete responses divided by the number of assigned forms. Patient response rates were also aggregated at each individual time point and evaluated based on basic demographic and procedural data.

Statistical Analysis

Statistical analyses were performed in the R statistical programming language (version 3.6.2)⁸ using the RStudio integrated development environment.⁹ The tidyverse¹⁰ and janitor¹¹ packages were used to transform data prior to analysis. Continuous variables are reported as median and interquartile range, and categorical and ordinal variables are reported as proportions of the total cohort. Logistic regression for survey compliance at the one-year timepoint was conducted using the mgcv¹² package, and McFadden’s pseudo-R squared for the model was calculated using the pscl¹³ package.

Results

Two-hundred twenty-five patients underwent elective shoulder arthroscopy during the 2-year study period. Demographic data are reported in Table 2. The most common procedure was an arthroscopic rotator cuff repair with biceps tenodesis (40%). The median age at time of surgery was 57 years [IQR 48 to 66], and the majority were male (64%). Most patients in the study population were White (84%), and a notable percentage of White patients self-identified as Hispanic or Latino (34%). Compliance with PROMs was highest preoperatively (91.1%) and decreased at subsequent time points (Fig 1). The largest decrease in compliance with PROMs occurred between the preoperative and 3-month follow-up time points. Overall, 36% of patients were compliant at all individual time points. There were no significant predictors of follow-up with regard to age, sex, race, ethnicity, or procedure.

Table 2.

Shoulder arthroscopy patient demographics, reported as number (%) or Median (Interquartile Range)

Characteristic	n = 225
Sex
Female	81 (36%)
Male	144 (64%)
Age at Surgery	57 (48, 66)
Race, Ethnicity
White, not Hispanic or Latino	110 (49%)
White, Hispanic or Latino	68 (30%)
White, patient declines to specify	1 (0.4%)
Black or African-American	5 (2.2%)
Other race, not Hispanic or Latino	4 (1.8%)
Other race, Hispanic or Latino	4 (1.8%)
Patient declines to specify	32 (14%)
Patient declines to specify, not Hispanic of Latino	1 (0.4%)

Fig 1.

Compliance rates with patient-reported outcome measures after shoulder arthroscopy from preoperative to 2 years postoperative.

Discussion

The main finding of this study is that compliance decreases over time and is roughly 50% at 2 years. While compliance decreased over time, this study did not support our hypothesis that younger age and male sex would be associated with noncompliance. The results of this study confirm that patient compliance with PROMs following shoulder arthroscopy remains a limiting factor in the practicality of their implementation and usage in the private practice orthopaedic setting.

A major concern regarding the collection of PROMs is loss of follow-up and resulting bias in registry data due to certain populations being excluded from outcome reporting studies. The current study was unable to replicate previous data that associated basic demographic or procedural data with patients at risk for noncompliance with PROMs. The most complete PROM studies come from large arthroplasty registries that often include thousands of patients. Compliance with PROMs has not been as well studied in arthroscopy, and sample size may be a barrier to identifying subtle differences. One study of anterior cruciate ligament reconstruction included 313 patients, while another in shoulder arthroscopy included 143 patients.^14,15 The current study included 225 patients, and we must consider the possibility of type II error due to the study being underpowered.

Multiple studies have identified different risk factors for PROM completion depending on the procedure, practice setting, and patient-specific factors. The most cited demographics for PROM noncompletion following elective orthopaedic procedures are male sex, younger age, and non-White race.^5,14,16, 17, 18, 19 Although these are commonly cited, a recent study analyzing 9 factors concluded no single patient characteristic can accurately predict loss to follow-up.²⁰ This study failed to replicate the results of previous orthopaedic studies; however, one publication specific to shoulder arthroscopy did not find differences in PROM compliance, according to patient age or gender.¹⁵ Certain patient-specific factors, such as socioeconomic status, access to technology, and native language, affect minority populations at disproportionate rates and can contribute to health disparities in orthopaedic literature.^21,22 Of note, greater than one-third of the patients included the current study identified as Hispanic or Latino and less than 20% of patients were non-White. One study of an electronic shoulder arthroplasty registry even described subpopulations that were more or less likely to complete PROMs at different follow-up intervals.¹⁹ Only one-third of our cohort completed assigned PROMs at all follow-up time points, and we did not identify any trends in patient noncompliance over time.

In addition to patient barriers, there are inherent characteristics of PROM surveys that influence completion rates and data quality, such as survey length. The outcome modules utilized in this study were assigned by the surgeon and standardized by procedure. All patients completed the Veterans RAND 12 Item Health Survey (VR-12), in addition to shoulder-specific outcome modules at each time point. Despite the notion of survey fatigue, a meta-analysis of different PROM tools utilized in shoulder surgery suggests that multiple PROM scales are necessary for patients undergoing rotator cuff repair and instability procedures because the efficacy of any given score is dependent on the condition being treated.^23,24 In short, there is no single PROM tool that encompasses the complexity of any given pathology. The Patient-Reported Outcomes Measurement Information System (PROMIS) from the National Institutes of Health attempts to address this issue and standardize outcome reporting, but it requires clinicians to adopt a common outcome reporting system.²⁵ Our current practice utilizes core measures within Surgical Outcomes System and has not transitioned to PROMIS scores.

The remaining logistical concerns required to implement and maintain a registry are of concern to smaller practices that may not have the funding or staff support to obtain adequate PROM data. Many studies have evaluated the difference in PROM response rate with a variety of collection methods to maximize data collection and efficiency. Electronic administration of PROMs has been adopted due to decreased patient and provider burden, as well as immediate feedback and ease of charting outcomes over time.²⁶ However, the use of combined manual and electronic collection of PROMs has been shown to significantly improve response rate.²⁷ In addition to electronic communication, there are multiple third-party platforms that can be used to assist with data collection and registry management. Despite these technological advances, there is no substitute for patient engagement.²⁸ One study in shoulder arthroscopy documented an increase of 20% in compliance rates with the effort of a dedicated research assistant.¹⁵ Our practice benefitted from the support of a team member dedicated to contact SOS nonresponders, which likely contributed to the PROM completion rates reported in this study.

Several remaining questions in orthopaedic outcomes research include the following: 1) how can PROMs be used to change the standard of orthopaedic follow-up care? and 2) what is an acceptable completion rate for PROMs in standard orthopaedic practice? The use of PROMs may allow surgeons to decrease the number of follow up appointments by providing additional data on patients after their last in-person appointment. For example, the majority of patients achieve a minimal clinically important difference at 6 months after rotator cuff repair, as well as substantial clinical benefit and Patient Acceptable Symptomatic State less than 1 year following surgery.²⁹ These findings suggest that 1- and 2-year follow-up appointments may be unnecessary, especially if PROM scores are maintained or continue to improve during 2-year follow-up.

At the standard 80% follow-up, Zelle et al. described biased results in over one-fourth of randomized samples in a trauma database simulation.³⁰ On the basis of our data, the official recommendation by ISAR of 60% PROM completion at 1-year follow-up seems reasonable for research purposes. There is still a need for improvement in the measurement of surgical outcomes, and a suggestion to improve PROM compliance is to simplify outcome measures to include only clinically relevant questions for common orthopaedic procedures.^31,32 However, the documentation of PROM nonresponders gives the surgeon some insight into patient outcomes. In two arthroplasty studies where nonresponders were contacted, patients who initially did not respond reported poorer outcomes.^17,33 This finding has recently been challenged by a single study that did not identify biased outcome reporting when analyzing a cohort of patients with only 50% follow-up using automated methods.³⁴ While not generalizable to all settings, it is possible that outcomes-based research could be performed with a goal of 50% loss to follow-up.³⁵ Our study confirms that 50% follow-up is attainable in clinical practice; however, noncompliance may be associated with patient-specific characteristics that were not identified in this study.

Limitations

The primarily limitations of the study are selection bias and nonresponse since the outcomes represent the results of a single surgeon in a private practice setting and require patient survey completion. These findings may not be applicable to academic-based practices or different patient populations. Roughly 50% of patients did not complete PROMs at 2-year follow up, and these patients were not contacted to gain insight into the reasons for noncompliance. Other limitations include the retrospective nature of the study and the lack of ability to quantify the influence of the research assistant. The current study used multiple PROMs assigned by the surgeon depending on the procedure and did not evaluate the effect of survey length on patient compliance. Another limiting factor is the sample size of the study. This study may have been underpowered to detect any significant differences, which would lead to a type II (B) error.

Conclusions

Patient compliance with PROMs decreases over time with the lowest percentage of patients completing electronic surveys at the traditional 2-year follow-up for shoulder arthroscopy. In this study, basic demographic factors were not predictive of patient compliance with PROMs.