Patient-reported outcome measures in total joint arthroplasty: defining the optimal collection window

Abstract

Background

The purpose of this study was to determine the optimal window for collection of patient-reported outcome measures (PROMs) after total joint arthroplasty (TJA).

Methods

Our prospectively collected institutional joint registry was queried for patients who underwent primary, unilateral TJAs. The primary outcomes were the net changes in WOMAC, SF-12 MCS, SF-12 PCS, OHS, KSCRS, and UCLA activity rating system at 6, 12, and 24 months postoperatively. Secondary outcomes were data acquisition costs and follow-up attrition rates.

Results

Eight hundred sixty-six procedures (450 total hip arthroplasties, 416 TKAs) were analyzed. A consistent plateau in all PROMs was noted by 6 months postoperatively–except for SF-12 MCS which showed no significant changes at any time interval. For TKA, the percentage of overall improvement achieved by 6 months was 88.7%, 84.5%, 100%, and 90.5% for the WOMAC, SF-12 PCS, UCLA, and KSCRS, respectively. For total hip arthroplasty, these values were 92.7%, 83.5%, 88.0%, and 89.8% for WOMAC, SF-12 PCS, UCLA, and OHS, respectively. There were marginal improvements from 6 to 12 months and no improvement from 12 to 24 months. Follow-up rates at 6, 12, and 24 months were 85%, 69%, and 40%, respectively. Our institutional costs for collecting a complete data set per patient were $128, $158, and $272 for 6, 12, and 24 months, respectively.

Conclusions

Most of the improvement in PROMs after primary TJA occurs within the first 6 months. In addition, limiting PROMs collection to 6 months appears to be cost-efficient owing to increased attrition rates beyond this time interval.

Introduction

Total joint arthroplasty (TJA) is one of the most commonly performed orthopedic procedures and the utilization is projected to increase 4-fold by 2030 [1]. Along with this growing demand comes higher scrutiny regarding the value of care delivered [2]. The US healthcare system is in a midst of change from a volume-based, fee-for-service model to a patient-centered, value-based model [3,4]. The success of TJA has traditionally been measured from the surgeon’s perspective based on outcomes such as implant survival and complications [5,6]. With the shift to value-based care, the success of TJA is now increasingly linked to patient-reported outcome measures (PROMs) [7]. This has significant administrative, research, clinical, and financial implications, which makes understanding PROMs behavior in the postoperative period essential [[7], [8], [9]].

Value-based payment arrangements that reward quality and efficiency of care are beneficial to both surgeons and patients. However, it is important that as physicians we lead the charge in defining precedents for proper use and implementation of PROMs, else this responsibility will fall to alternative stakeholders [10,11]. With the increase in PROMs utilization, it is prudent to develop a detailed understanding of their responsiveness to surgery such that implementation can be executed in a responsible, cost-effective, and mutually beneficial manner for all stakeholders. Although many validated PROMs exist, there remains no gold standard measure, alluding to the complexity of the task at hand and need for increased research and understanding [2,3]. In addition, it remains unclear how long PROMs need to be collected after surgery. Specifically, at what time point following TJA do PROMs begin to reflect an accurate estimate of “success”? The answer to this question has several implications. First, it allows us to select an optimal time point in the postoperative period at which PROMs reasonably reflect the value added to patients. Second, this information has implications on research design and publications in journals that may require longer PROMs follow-up intervals, which may prove to be unnecessary [7]. Third, if performance and reimbursement are to be determined based on PROMs, it is crucial that the timing of PROMs collection is such that the results provided represent the most accurate estimate of value added [9].

The primary objective of this study is to present our 5-year joint registry experience collecting a panel of commonly used PROMs at a single tertiary care center in the United States. Focus was placed on when various PROMs peaked during the postoperative period and the associated registry maintenance costs, with the intention of better defining the optimal timing for PROMs collection following TJA. We hypothesized that significant improvements in PROMs occur early in the postoperative period, which may afford significant cost-saving opportunities.

Material and methods

Institutional Review Board approval was obtained. Our prospectively collected institutional joint registry was queried for patients who underwent primary hip and knee arthroplasty with a minimum of follow-up of 6 months. Patients undergoing nonelective, bilateral, unicompartmental, revision, or tumor-related procedures were excluded. In total, 866 procedures met these criteria and were included in the study: 450 total hip arthroplasties (THAs) and 416 total knee arthroplasties (TKAs). Procedures were performed between February 2012 and August 2017. Patient inclusion in the joint registry required consent and willingness to participate in regular surveys. All procedures were performed by 5 fellowship-trained arthroplasty surgeons. Patient characteristics collected in the study were age, sex, body mass index, American Society of Anesthesiologist physical classification system score, educational level, marital status, ethnicity, smoking status, and alcohol use. Patients had baseline Short Form-12 (SF-12) mental and physical component scores (MCS and PCS), Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), and University of California Los Angeles (UCLA) activity scores. Patients undergoing THA also had baseline Oxford Hip Score (OHS) and those undergoing TKA had baseline Knee Society Clinical Rating System (KSCRS) scores. All PROMs were then routinely collected at up to 6 months postoperatively and again at 12 and 24 months.

The WOMAC is a 24-item questionnaire used to assess the severity of osteoarthritis of the hip and knee [12]. The SF-12, an abbreviated version of the original Short Form 36, is a measure of general health that can be used to generate physical and mental composite scores [13]. The UCLA activity rating is a single question measure of a patient’s overall activity level. The patient selects 1 of 10 distinct activity levels, 1 being the least and 10 the most active [14,15]. The KSCRS is a measure specific to the knee that is divided into a Knee Score and a Function Score [16]. It is unique in that the Knee Score contains an objective physician-derived component, while the Function Score is derived purely from the patient. For the purpose of simplification, the combined score was reported in this study. The OHS is a 12-item questionnaire specific to assessing function and pain for patients undergoing THA [17,18]. Minimal clinically important difference (MCID) in TKA has been previously reported to be 15 points for WOMAC [19], 4.3 points for SF-12 PCS [20], 3.1 points for SF-12 MCS [21], and 11.4 points for KSCRS [22]. The MCID for THA was previously reported to be 7.6 points for WOMAC [23], 4.6 points for SF-12 PCS, 6.0 points for SF-12 MCS [24], and 5 points for OHS [25]. Although no MCID has been defined for UCLA activity rating, it can be assumed that an increase of 1 level constitutes a meaningful clinical improvement given the structure of the questionnaire.

At our institution, data collection, entry, and management are performed by a dedicated research assistant. The research assistant personally administers the questionnaires to patients via pen and paper method during joint class before surgery and again when patients return to their postoperative visits. Patients who do not show up to their appointments receive reminder phone calls. Due to the number and length of questionnaires, these are not collected over phone. The registry data are housed and maintained by a data management vendor (Ortech, London, Canada). Costs collected in this study included the salary for the research assistant paid at 0.75 full-time equivalent (FTE) as well as licensing and data maintenance fees. The 0.75 FTE was determined based on projected time needed to collect 4 datasets per patient (baseline and again at 6, 12, and 24 months postoperatively) for an estimated 300 patients per year after the commencement year. For situations where the research assistant was not available, 2 support research personnel were available for coverage. To encourage patient participation in the registry, the purpose/benefit of collecting outcomes data along with time commitment for questionnaire completion was discussed with patients by their surgeons. Patients were encouraged to sign up at the time of their joint class. Enrollment was limited to patients who did not require interpretation services.

Data were summarized with mean and standard deviation for continuous variables and frequency and proportion for categorical variables. Changes in PROMs over time were analyzed with mixed effects linear regression. The alpha level for all analyses was set at 0.05.

Results

In total, 866 patients undergoing TJA (450 THAs and 416 TKAs) were included in the analysis. Table 1 summarizes the baseline demographic characteristics for the study group.

Table 1.

Baseline demographic characteristics of the study group.

Age (y)	60.2 (±12.2)
ASA	2.3 (±0.5)
BMI	31.5 (±5.5)
Sex
Female	443 (51%)
Male	423 (49%)
Procedure
Total hip	450 (52%)
Total knee	416 (48%)
Education
Secondary	406 (49%)
Higher	422 (51%)
Marital status
Married or living with significant other	378 (45%)
Not living with significant other	461 (55%)
Ethnicity
White	657 (76%)
Black/African American	126 (15%)
Hispanic/Latino	53 (6%)
Other	30 (3%)
Smoking
Yes	47 (12%)
No	357 (88%)
Alcohol
None/occasional	716 (86%)
Daily	117 (14%)
Back pain
No	373 (43%)
Yes	493 (57%)

ASA, American Society of Anesthesiologists; BMI, body mass index.

Values are given as mean and standard deviation or frequency and proportion.

Changes in PROMs over time for TKA

The WOMAC improved by an average of 36.3 points at 6 months (P < .001). A clinically insignificant improvement of 4.1 points (P = .002) was observed between 6 and 12 months and no changes thereafter (P = .736). Similarly, the SF-12 PCS experienced its greatest increase, 14.3 points, by 6 months (P < .001). No significant changes were observed at 12 or 24 months (P = .058 and .237, respectively). The SF-12 MCS did not demonstrate improvement at any time interval (P = .912, .435, and .747 at 6, 12, and 24 months, respectively). The UCLA activity score increased by 1 point (P < .001) at 6 months and no changes were found beyond that point (P = .567 and .991 at 12 and 24 months, respectively). Finally, the KSCRS experienced the greatest improvement (62.2 points) by 6 months (P < .001). A clinically insignificant change of 9.3 points (P = .001) was observed between 6 and 12 months with no changes beyond 12 months (P = .501). The percentage of overall improvement achieved by 6 months was 88.7%, 84.5%, 100%, and 90.5% for the WOMAC, SF-12 PCS, UCLA, and KSCRS, respectively. Table 2 details the PROM scores over time for TKA. A subanalysis for implants used revealed minimal differences that were neither statistically nor clinically significant.

Table 2.

Patient-reported outcome scores over time for total knee arthroplasty.

Outcome measure/time interval	Mean	95% Confidence interval	P value
WOMAC
Baseline	55.6	53.7-57.6
6 mo	19.4	17.3-21.5	<.001
12 mo	15.3	13.0-17.7	.002
24 mo	14.7	11.4-18.1	.736
SF-12 PCS
Baseline	28.2	27.1-29.3
6 mo	42.5	41.4-43.7	<.001
12 mo	43.9	42.6-45.2	.058
24 mo	45.1	43.2-46.9	.237
SF-12 MCS
Baseline	56.4	55.3-57.4
6 mo	56.3	55.2-57.4	.912
12 mo	55.8	54.5-57.1	.435
24 mo	55.5	53.8-57.2	.747
UCLA
Baseline	4.3	4.1-4.5
6 mo	5.3	5.1-5.5	<.001
12 mo	5.3	5.1-5.5	.567
24 mo	5.3	5.0-5.6	.991
KSCRS
Baseline	87.4	82.7-92.0
6 mo	149.6	145.1-154.1	<.001
12 mo	158.9	153.8-164.1	.001
24 mo	156.1	148.5-163.8	.501

P values represent the statistical comparison of the indicated time interval compared to the previous time interval. Lower WOMAC scores indicate better health.

Changes in PROMs over time for THA

The WOMAC improved by an average of 46.4 points at 6 months (P < .001). A clinically insignificant improvement of 3.2 points (P = .017) was observed between 6 and 12 months and no changes thereafter (P = .788). Similarly, the SF-12 PCS experienced its greatest increase, 17.1 points, by 6 months (P < .001). A clinically insignificant increase of 2.1 points (P = .011) was observed between 6 and 12 months. The SF-12 MCS experienced significant improvement by 6 months (P < .001) although this change was not clinically significant. No improvements in SF-12 MCS were observed at 12 or 24 months (P = .578 and .122, respectively). The UCLA activity score experienced an average increase of 1.7 points (P < .001) at 6 months and no further changes at 12 or 24 months (P = .376 and .373, respectively). The OHS improved by an average of 22.7 points (P < .001) at 6 months. Improvements between 6 and 12 months were below not clinically significant (1.9 points, P = .001) and no changes were observed at 24 months (P = .419). The percentage of overall improvement achieved by 6 months was 92.7%, 83.5%, 88.0%, and 89.8% for WOMAC, SF-12 PCS, UCLA, and OHS, respectively. Table 3 summarizes the PROM scores over time for THA. Figure 1 details the changes in PROMs over time for TKA and THA. A subanalysis for surgical approach or implants used revealed minimal differences that were neither statistically nor clinically significant.

Table 3.

Patient-reported outcome scores over time for total hip arthroplasty.

Outcome measure/time interval	Mean	95% Confidence interval	P value
WOMAC
Baseline	62.9	61.0-64.8
6 mo	16.5	14.4-18.6	<.001
12 mo	13.3	11.0-15.7	.017
24 mo	12.8	9.5-16.2	.788
SF-12 PCS
Baseline	26.4	25.4-27.3
6 mo	43.5	42.4-44.6	<.001
12 mo	45.4	44.1-46.6	.011
24 mo	46.9	45.1-48.6	.130
SF-12 MCS
Baseline	52.7	51.6-53.9
6 mo	55.9	54.7-57.2	<.001
12 mo	55.5	54.1-56.9	.578
24 mo	53.8	51.9-55.8	.122
UCLA
Baseline	3.7	3.5-3.8
6 mo	5.4	5.2-5.6	<.001
12 mo	5.5	5.3-5.7	.376
24 mo	5.7	5.4-6.0	.373
OHS
Baseline	43.5	42.7-44.4
6 mo	20.8	19.9-21.7	<.001
12 mo	18.9	17.8-19.9	.001
24 mo	18.2	16.7-19.7	.419

P values represent the statistical comparison of the indicated time interval compared to the previous time interval. Lower WOMAC and OHS scores indicate better health.

Figure 1.

Changes in patient-reported outcomes plotted over time demonstrate a consistent plateau effect at 6 mo postoperatively. (a) Western Ontario and McMaster Universities Osteoarthritis Index; (b) Short Form-12 physical component summary; (c) Short Form-12 mental component summary; (d) University of California Los Angeles activity level rating; (e) Oxford hip score; (f) Knee Society clinical rating system. Lower WOMAC and OHS indicate better health.

Changes in MCIDs over time

Within the study period, there was a clear pattern that the MCIDs for all assessed PROMs mostly occurred within the first 6 months after surgery. Table 4, Table 5 describe the percentage of patients meeting the MCIDs over time for THA and TKA respectively.

Table 4.

Percentage of patients meeting the minimum clinically significant differences over time for total hip arthroplasty.

Outcome measure/time interval	6 mo	12 mo	24 mo
WOMAC	94%	95%	95%
SF-12 PCS	82%	85%	90%
SF-12 MCS	30%	33%	33%
UCLA	75%	75%	76%
OHS	95%	96%	97%

Table 5.

Percentage of patients meeting the minimum clinically significant differences over time for total knee arthroplasty.

Outcome measure/time interval	6 mo	12 mo	24 mo
WOMAC	85%	87%	87%
SF-12 PCS	83%	84%	84%
SF-12 MCS	31%	33%	35%
UCLA	48%	51%	55%
KSCRS	100%	100%	100%

Follow-up rates and costs associated with PROMs collection

Our annual institutional costs for collecting PROMs data were $78,500. These included the salary for a dedicated research assistant paid at 0.75 FTE ($75,000) and licensing/data maintenance fees ($3500). Based on a target enrollment of 300 patients per year with 4 datasets/patients (1200 datasets per year), the allocated cost per dataset was $65.42. There was significant loss to follow-up at each successive time interval postoperatively. Follow-up rates were 85%, 69%, and 40% at 6, 12, and 24 months, respectively. Fifteen percent of patients opted out of the registry within the first 6 months after surgery. Loss to follow-up beyond 6 months was primarily due to patients not returning to their follow-up visits despite receiving reminder phone calls. As a result, the actual cost per dataset increased to $128, $158, and $272 at 6, 12, and 24 months, respectively. Table 6 summarizes our annual return on investment (allocated costs divided by number of completed datasets) for collecting PROMs, which decreased significantly over time.

Table 6.

Annual return on investment for collecting outcomes data in primary total joint arthroplasty.

Total allocated cost per yeara	$78,500
Projected patient enrollment per year	300
Actual patient enrollment per yearb	180

	Projected datasets	Actual datasetsc	Allocated cost per dataset	Actual cost per datasetd	Annual investment loss
Baseline	300	180	$65.42	$109	−$7850
6 mo	300	153	$65.42	$128	−$9616.74
12 mo	300	124	$65.42	$158	−$11,513.92
24 mo	300	72	$65.42	$272	−$14,915.76
Total	1200	529	$65.42	$148	−$43,896.82

^aThe total allocated costs reflect the salary of a dedicated research assistant paid at 0.75 full-time equivalent ($75,000) and data maintenance/licensing fees ($3500).

^bPatient participation in the database is voluntary and requires willingness to participate in surveys during follow-up visits. Patients can opt out at any time if they wish.

^cActual number of datasets is based on actual follow-up rates of 100%, 85%, 69%, and 40% at baseline, 6 mo, 12 mo, and 24 mo, respectively.

^dCost per dataset is derived by dividing the total costs for 3 time periods (0-6 mo, 0-12 mo, or 0-24 mo) by the number of complete datasets generated during each period.

Discussion

This study provided evidence that the vast majority of clinically relevant improvement in a wide range of PROMs following primary TJA occurred within the first 6 months postoperatively. With the exception of the SF-12 MCS, which underwent minimal change from baseline throughout the 2-year follow-up period, all other PROMs experienced significant improvements by 6 months after surgery with only marginal changes thereafter. In addition, there was significant increase in registry costs combined with high rate of follow-up loss beyond 6 months. These findings suggest that limiting PROMs collection to 6 months postoperatively appears to be a cost-efficient and clinically relevant mark to assess the quality of care delivered.

Our findings are consistent with previous reports. Ramkumar et al. [2] conducted a systematic review and meta-analysis of 15 reports on PROMs after TJA. The group found that there were no statistically significant changes in Harris Hip Score, WOMAC, Knee Society Score, and SF scores (SF-12, SF-36, and SF-6D) from 12 to 24 months. In our study, we found that the improvements in outcomes began to plateau even sooner, as early as 6 months. Kagan et al. [26] prospectively reviewed 91 patients undergoing primary TKA using Patient-Reported Outcomes Measurement Information System physical function and pain interference tests. The authors found that 89% of improvement in physical function and 90% of pain interference occurred by 6 months after surgery. Data from international studies also support this trend. Van der Wees et al. [27] collected Harris Hip Score, OHS, WOMAC, and Knee Society Score Function Score following TJA at 3, 6, and 12 months postoperatively. The group showed a plateau in improvement for all four PROMs around 6 months postoperatively. In another study, Dailiana et al. [28] reported on WOMAC scores following TJA at 6 weeks as well as 3, 6, and 12 months postoperatively. The authors likewise showed a plateau effect around 6 months after surgery.

The data presented in this study by no means rules out continued clinical improvement beyond 6 months postoperatively. However, it provides evidence that a patient’s response to TJA at 6 months may be a good indicator of value of the surgical intervention as the vast majority of improvement in PROMs was reached by this time point. The possibility of making accurate estimates of overall outcomes at an earlier time point has multiple implications. The collection of PROMs and maintenance of a total joint registry is no trivial matter and poses many obstacles both financially and logistically as has been described by others [6,8,29,30]. Minimizing the number of data collection points, as well as collecting data at an earlier time point should help make maintaining a registry a more feasible and affordable feat, hopefully incentivizing more centers to collect such data. Measuring outcomes at 6 months rather than 12 or 24 months also provides a distinct advantage of increased response rates as our study showed a significant attrition beyond 6 months. It should be noted that our findings pertain to short-term outcome collection. This does not preclude the importance of obtaining mid- and long-term data, as this may help provide an additional metric for clinical performance and implant surveillance as some of the major modes of failure typically occur beyond the short-term window [31,32].

A central challenge identified by this study is how to improve survey completion rates. The first important area for improvement is minimizing the burden placed on both the patient and the institution to complete the surveys. Fifteen percent of our patients opted out of the registry due to the time-consuming nature of these surveys. Limiting the data collection to 1 general health questionnaire and 1 disease-specific questionnaire may help decrease survey fatigue as well as administrative burden for data entry and maintenance. A second area for improvement is administrating the surveys in a digital platform that allows patients to complete them at their convenience. Our pen and paper method proved to be not cost-effective at later time points as it required both the patient and the research assistant to be present to ensure the questionnaires are fully answered. However, as our study and previous reports indicate [[26], [27], [28]], evidence to date show that limiting data collection to 6 months is both clinically relevant and cost-efficient.

This study has some important limitations. The results reflect the experience of a single public academic institution which treats a patient population with increased disease burden. Its conclusions are dependent on collection of several questionnaires using the pen and paper method. As discussed, this approach resulted in a significant attrition rate at 12 and 24 months. The implementation of alternative methods such as simplified questionnaires and electronic administration may improve response rates and has the potential to alter the trends seen in this study. This could also have effects on registry maintenance costs and ultimately return on investment.

Conclusions

In conclusion, this study adds support to previous reports that most of the improvement in PROMs after primary TJA occurs within the first 6 months. In addition, limiting PROMs collection to 6 months appears to be cost-efficient due to increased attrition rates beyond this time interval. Although this conclusion is limited to the data collection method we used, it sheds light on an important and elusive element in PROMs collection—defining the optimal collection window! Future research on PROMs should bridge this gap in our knowledge given its important clinical, economic, and health policy implications in a value-based healthcare system.