Capturing patient-reported outcomes: paper versus electronic survey administration

Abstract

Objective: To compare the capture rates and costs of paper patient-reported outcomes (pPRO) administered in-clinic and electronic PROs (ePRO) collected through emails and texts.

Design: Retrospective review.

Setting: Level 1 trauma center.

Patients/Participants: The pPRO program enrolled 2164 patients for postsurgical follow-up in 4 fracture types: ankle, distal radius, proximal humerus, and implant removal from 2012 to 2017. The ePRO program enrolled 3096 patients in 13 fracture types from 2018 to 2020. Among the patients enrolled in the ePRO program, 1296 patients were matched to the 4 original fracture types and time points.

Main Outcome Measures: PRO capture rates in 4 fracture types by matched time point and estimated cost of each program per enrolled patient.

Results: At first follow-up, pPRO provided a higher capture rate than ePRO for 3 of 4 fracture types except for implant removal (P < 0.05). However, at 6-month and 1-year follow-ups, ePRO demonstrated statistically significant higher capture rates when compared with pPRO for all applicable modules (P < 0.05). The average cost for the pPRO program was $171 per patient versus $56 per patient in the ePRO program. Patients were 1.19 times more likely to complete ePRO compared with pPRO (P = 0.007) after controlling for age, sex, fracture type, and time point.

Conclusion: The electronic PRO service has improved long-term capture rates compared with paper PROs, while minimizing cost. A combined program that includes both in-clinic and out of clinic effort may be the ideal model for collection of PROs.

Level of Evidence: Level 3.

1. Introduction

Patient-reported outcomes (PROs) are validated questionnaires completed by patients that measure health status, functions, and quality of life from the patient's perspective. PROs can be disease or injury-specific or can more globally assess domains of mental, physical, and emotional health.^[1] In recent decades, PROs have increasingly been collected for research and clinical practice.^[2–8] They are now required for Part II board certification by the American Board of Orthopaedic Surgeons, encouraged by the Centers for Medicare & Medicaid Services for joint replacement, serve as efficacy end points in clinical trials and support labeling claims in medical product development.^[1,9–13] PROs can also be used to benchmark patient progress, informing clinical and shared decision-making in real time.^[14–17]

Given the broad application and utility of PROs, optimizing data capture is imperative. Previous studies have highlighted limitations of the traditional paper-based patient-reported outcomes (pPRO),^[18,19] including administrative burden, secondary data entry errors, challenges translating responses into easily-accessible information, and incomplete data sets.^[20–24] In the orthopaedic trauma patient population, the PRO capture rate is commonly low because of the limited number of clinic follow-ups when the surveys are administered, and many patients are not able to return to all postoperative visits.^[25,26]

Because email and smart devices have become ubiquitous, electronic data collection for PROs becomes a possibility allowing PRO delivery even without clinic visits. Multiple platforms have been developed to facilitate electronic PRO (ePRO) data collection^[27–29]; however, limited information on them is published in the orthopedic trauma literature.^[30] The purpose of this study was to compare the longitudinal capture rate of ePROs with traditional pPROs within our orthopaedic trauma population at a single level 1 trauma center. We examined factors associated with survey completion and adjusted the capture rate based on patient characteristics to reflect a more generalizable capture rate. The secondary objective was to look at the cost associated with implementing and maintaining both programs.

2. Methods

The approval for the study was obtained from Institutional Review Boards. This is a retrospective chart review of our trauma registry from 2012 to 2020. From 2012 to 2017, pPROs were administered in-clinic at an academic level 1 trauma center to patients with trauma in 4 types of surgery: operatively treated ankle fractures, distal radius fractures, and proximal humerus fractures and elective implant removal for any fractures. Each fracture type had 3 designated follow-up time points: Ankle follow-up was collected at 6 weeks, 6 months, and 1 year; distal radius at 6 weeks, 3 months, and 1 year; proximal humerus at 3 months, 6 months, and 1 year; and implant removal was collected at 6 weeks, 3 months, and 6 months. There was no mail or phone follow-up for patients who did not complete their ePROs during clinic or did not attend their in-clinic appointment.

The ePRO program was implemented in 2018 with PRO surveys administered through emails and short message service (sms) text messages (see Appendix, Supplemental Digital Content 1, http://links.lww.com/OTAI/A58). No surveys were conducted in clinic. The program was contracted through a HIPAA-compliant third-party vendor, CODE Technology (Minneapolis, MN). The company manages the PRO process including survey implementation, customization of a platform for data collection, and dashboards for reporting results. Patients were scheduled to receive 3 emails (on the date a survey is due, day 3, and day 6) and a reminder call.

Patients without an email were called to obtain an email address. The communication activities were scheduled based on the date of the procedure and were not linked to the appointments. PRO data are stored separately from the electronic medical records with our PRO vendor. The dashboard provides individual patient PRO scores graphed against the average at each time point for patients with the same fracture type within the department. Capture rate and PRO reports are sent as feedback to providers on a quarterly basis. The ePRO registry has 13 fracture types including the 4 implemented in the pPRO registry with additional 9 types (humerus, chest wall, foot, hip, knee, tibia, pelvis, operative scapula, and nonoperative scapula). Each fracture type module collected both an anatomy-specific PRO and a global health PRO.

Demographic variables were extracted from the electronic health record. Raw capture rates for pPRO and ePRO were calculated by the number of responses divided by all patients who were eligible to complete their PRO at each time point whether they were in clinic or not. Comparisons between ePRO and pPRO capture for the 4 matched fracture types and time points were performed with t tests and χ² as appropriate. Statistical significance was set at an alpha term of 0.05. All statistical analysis was conducted in SAS 9.4 (SAS Institute, Cary, NC).

Capture data were then modeled using a generalized linear mixed model, PROC GLIMMIX, to estimate a marginal adjusted mean capture rate and odds ratios by program and time point. A random effect for subject nested within fracture types was used to control for correlation within subject across multiple time points. All odds ratios and estimates were adjusted with the Tukey method for multiple testing. Adjusted estimates by time point and program allow us to compare survey capture in a theoretically balanced sample, where age, sex, and fracture types are distributed evenly across both programs. Adjusted odds ratios were then also calculated by time point between ePRO and pPRO.

System cost for traditional pPROs was calculated by summing the time required by staff to distribute and collect the paper survey in addition to 1 full-time employee dedicated to screening charts for fracture diagnoses and manually entering outcome data. For the ePRO, costs included our institution's contract with Code Technology and the time required by an employee to screen for fracture diagnoses and transfer contact information to the vendor each week. There was a one-time start-up cost associated with the first year of the ePRO service, in addition to an ongoing per-surgeon monthly subscription fee. All personnel costs were estimated using the time-driven activity-based costing method with a practical capacity adjustment of 85% to the total theoretical capacity, using fully loaded labor rates to estimate total theoretical capacity cost.^[31]

3. Results

From 2012 to 2017, the pPRO program enrolled 2164 patients in 4 fracture types while the ePRO program enrolled 3096 patients in 13 fracture types from 2018 to 2020. Among the patients enrolled in the ePRO program, 1296 patients enrolled in the 4 original fracture types and time points. Comparisons of the 4 matched fracture types between the pPRO and ePRO programs demonstrated that patients in the 2 programs were not statistically different in mean age but have different age group distributions (Table 1). There was a statistically significant higher percentage of male patients in the ePRO program (43.8% ePRO vs. 36.6% pPRO, P < 0.001). The distribution of fracture types was different between the 2 programs (P < 0.001) with distal radius being the most common fracture in the pPRO program (40.9%) while ankle was the most common fracture in the ePRO programs (48.6%).

TABLE 1.

Demographic Information of Patients in the pPRO Program and Patients in the 4 Matched Fracture Categories in the ePRO Program.

Variable	pPRO (N = 2164)	ePRO (N = 1296)	P
Age at procedure
Mean (SD)	53.66 (17.65)	52.50 (17.54)	0.09
Age by decade			0.004
18-24	135 (6.2%)	79 (6.1%)
25-34	261 (12.1%)	166 (12.8%)
35-44	257 (11.9%)	195 (15.0%)
45-54	364 (16.8%)	222 (17.1%)
55-64	556 (25.7%)	272 (21.0%)
65-74	351 (16.2%)	234 (18.1%)
75-84	142 (6.6%)	92 (7.1%)
85-94	94 (4.3%)	34 (2.6%)
95-104	4 (0.2%)	2 (0.2%)
Sex			<0.001
Female	1372 (63.4%)	728 (56.2%)
Male	792 (36.6%)	568 (43.8%)
Fracture category			<0.001
Ankle	771 (35.6%)	630 (48.6%)
Distal radius	885 (40.9%)	122 (9.4%)
Hardware removal	269 (12.4%)	146 (11.3%)
Proximal humerus	239 (11.0%)	398 (30.7%)

At the first follow-up point, pPRO provided better capture rates than ePRO for ankle fractures (76.7% vs. 46.6% P < 0.001), distal radius (70.3% vs. 42.0% P < 0.001) at 6 weeks, and proximal humerus (56.5% vs. 45.6% P = 0.008) at 3 months (Table 2). Electronic PRO provided a better capture rate than pPRO for only 1 fracture type (implant removal) at first follow-up at 6 weeks (48.6% vs. 37.9%, P = 0.037). However, at all 6-month and 1-year follow-ups, ePRO demonstrated statistically significant higher capture rates when compared with pPRO for all fracture types: ankle fractures (42.0 vs. 35.3%, P = 0.015 at 6 months; 38.2% vs. 15.3%, P < 0.001 at 1 year), distal radius (32.1% vs. 10.4% P < 0.001 at 1 year), proximal humerus (43.5% vs. 29.3%, P = 0.001 at 6 months; 43.6% vs. 17.9%, P < 0.001 at 1 year), and implant removal (37.3% vs. 22.8% P = 0.003 at 1 year). The highest capture rates were pPRO ankle fractures at 6 weeks (76.7%) and pPRO distal radius also at 6 weeks (70.3%) while the lowest capture rates were pPRO at 1 year for distal radius (10.4%) and ankle fractures (15.3%). All mean capture rates for ePRO range between 32.1% and 48.6%.

TABLE 2.

Response Rate Comparisons Between Patients in the pPRO Program and Patients in the 4 Matched Fracture Categories in the ePRO Program.

Fracture module	pPRO	ePRO	P
Ankle
6 week	591/771 (76.7%)	285/612 (46.6%)	<0.001
6 month	245/695 (35.3%)	233/555 (42.0%)	0.015
1 year	90/590 (15.3%)	149/390 (38.2%)	<0.001
Distal radius
6 week	622/885 (70.3%)	50/119 (42.0%)	<0.001
3 month	446/865 (51.6%)	54/118 (45.8%)	0.237
1 year	71/684 (10.4%)	27/84 (32.1%)	<0.001
Proximal humerus
3 month	135/239 (56.5%)	181/397 (45.6%)	0.008
6 month	63/215 (29.3%)	158/363 (43.5%)	0.001
1 year	33/184 (17.9%)	109/250 (43.6%)	<0.001
Hardware removal
6 week	102/269 (37.9%)	69/142 (48.6%)	0.037
3 month	60/260 (23.1%)	63/143 (44.1%)	<0.001
6 month	53/232 (22.8%)	50/134 (37.3%)	0.003

Factors associated with the survey capture rate were age category (P < 0.001), follow-up time point (P < 0.001), program (ePRO or pPRO P = 0.007), fracture type (P < 0.001), and an interaction between time and program (P < 0.001). After adjusting for age, sex, and fracture type, pPRO capture rates remain highest at 6 weeks at 63.2% (95% CI: 58.1%–68.0%) and was significantly different from each follow-up time point, settling at 8.3% (95% CI: 6.6%–10.4%) at 1 year, while ePRO shows a slight decline over time, it is not significantly different from 6 weeks until 1 year with mean capture rates decreasing from 37.7% (95% CI: 32.2%–44.5%) at 6 weeks to 27.8% (95% CI: 23.0%–33.2%) at 1 year (Fig. 1). The mean adjusted probability of overall survey capture for pPRO was 30.0% (95% CI: 26.0%–34.3%) while ePRO was significantly higher at 33.8% (95% CI: 29.4%–38.9%, P = 0.007). Odds ratios showed patients were 1.19 (95% CI: 1.05–1.35) times more likely to complete an ePRO versus a pPRO overall.

Figure 1.

Adjusted capture rates of pPRO and ePRO programs.

The annual cost for the pPRO program was $61,643 from 2012 to 2017. The ePRO program cost was $65,812 in 2018 and $54,312 in 2019 and 2020, averaging $58,145 annually. All pPRO program costs (100%) were from personnel, while personnel accounted for 9.6% of ePRO costs in 2018 ($6312/$65,812) and 11.6% of total costs in 2019 and 2020 ($6312/$54,312). The one-time start-up cost for the installation of the ePRO trauma package was $11,500 (17.5% of costs in 2018). The ongoing service contract costs were $48,000 from 2018 through 2020. The ePRO program incurred a lower annual average, per patient and per fracture type costs compared with the pPRO program (Table 3).

TABLE 3.

Cost Comparisons Between pPRO and ePRO.

Program element	pPRO	ePRO
Years implemented	6	3
Average annual cost	$61,643	$58,145
Fracture categories	4	13
Cost per fracture category	$15,410	$4473
Patients enrolled	2164	3096
Cost per patient	$171	$56

Estimates were rounded to the nearest dollar.

4. Discussion

We found that the pPRO program yielded a better capture rate in clinic for the first follow-up in 3 of the 4 fracture types compared with the ePRO program while ePRO outperformed at the longer follow-up points. Higher capture rates from ePROs have been reported in internal medicine and arthroplasty surgery when compared with traditional pPRO methods.^[18,23,32] Similar to our institution, other health systems and hospitals have made the transition of PRO collection to electronic platforms such as patient smart device or websites.^{[20–22,24,27,32]} Several studies have reported advantages of ePRO over traditional pPRO including real-time data availability, decreased response burden, increased satisfaction, and fewer missing data.^{[28,29,33–35]}

Our study is the first to compare performance and cost of ePROs against traditional methods in an orthopaedic trauma population. It is worth noting, however, that the difference in capture rates was significant between the 2 programs at the 6-week follow-up. Among the 4 fracture types, the highest capture rate of in-clinic pPROs was 76.7% while the highest capture rate of ePRO was 48.6%. None of the ePRO collections exceeded 50% suggesting that ePRO has a threshold capture rate for the initial start-up years. A low capture rate is among the challenges for PRO implementation in orthopaedic surgery, and results with <50% capture rate may not be reflective of the overall patient population.

As health care shifts toward a value-based system and PRO collection becomes mandated, there remain concerns about the costs of PRO collection.^[1,9,10] Our study found that even in the short term and accounting for significant start-up costs, costs associated with the ePRO program were comparable with those of the pPRO program. The annual cost for the ePRO program decreases over time because the start-up cost is incurred only once. In addition, the marginal cost of additional patients in the ePRO program is negligible because service fees account for nearly 90% of ePRO costs and are unchanged regardless of the number of patients.

Our study has limitations. First, our experience in a single institution study may not be reproducible in other settings. Although our model demonstrated multiple factors associated with survey completions including age, sex, and fracture types, the implication of these findings is not clear. We only performed comparisons in the 4 common fracture types and time points between the 2 programs while our ePRO has 9 more fracture types and additional time points. We were only able to measure the first 2 years of ePRO data, compared with 6 years of historical data on in-clinic pPRO questionnaires. Because pPRO was administered in clinic and not by mail, we acknowledge that pPRO capture rates seem to be increased by in-person collection but limited, at later time points, by clinic follow-up rates; however, ePRO could be completed by patients regardless of their follow-up. In addition, because pPRO collection was tied with clinic visits, the higher capture rates of ePRO at later follow-ups biased against patients who were doing well and were discharged from the clinic. This process can be seen as an advantage of the ePRO system in that it captures more accurately how patients are doing instead of selecting patients who still require late follow-ups. Because ePRO is collected through email and texts, its collection is limited in patients who do not have smart devices and emails or prefer not to engage with these technologies. Our cost analysis was limited by estimation, although the model used for calculation was validated. Although the cost for fracture type seemed to be arbitrary, this calculation reflects how our program was billed based on the number of fracture types and not based on capture rates or number of patients. Finally, our study only focused on the capture rates and cost but did not investigate the application of PRO results during clinical practice.

Our model demonstrated a higher adjusted completion rate at initial postoperative visits for pPRO and later follow-ups for ePRO suggesting that a blended model capturing patients both in and outside of clinic may best serve our orthopedic trauma population. In-clinic PRO collection may yield better capture rates at early follow-ups while ePRO collections through emails and texts at later time points allow patients to complete PRO without returning to clinic. This advantage is especially applicable to the trauma patient population who are known to not keep their regular follow-up appointments.^[25,26] Patients can complete ePRO at their convenience instead of a specific time in clinic for the pPRO. Given rapidly evolving COVID protocols and virtual remote clinic visits, ePRO also serves as an additional measure of functional outcomes for patients who are not able to return for an in-person clinic visit.

In conclusion, addition of the ePRO programs to an orthopaedic trauma PRO program offers many advantages including higher long-term capture rates and lower cost. The collection of PROs in patients with trauma continues to be a challenge with limited capture rates in both ePRO and pPRO models. Our data suggest a combined program that includes both in-clinic and out of clinic effort may be the ideal model for collection of PROs.