Computerized Adaptive Testing for Patient Reported Outcomes in Ankle Fracture Surgery

Abstract

Background:

Advantages of using computerized adaptive testing (CAT) include decreased survey-burden, diminished floor and ceiling effect, and improved ability to detect the minimal clinical significant difference (MCID) among patients. The goal of this study was to compare the legacy patient-reported outcome measures (PROMs) to the Patient-Reported Outcomes Measurement Information System (PROMIS) scores in terms of ability to detect clinically significant changes in patients who have undergone surgery for ankle fractures.

Methods:

Patients who underwent osteosynthesis for an unstable ankle fracture between 2013–2016 and completed legacy outcome scores (Foot and Ankle Outcome Score [FAOS], Olerud and Molander Ankle Score [OMAS], and Weber Score) along with the PROMIS Physical Function (PF) and PROMIS Lower Extremity (LE) CATs postoperatively were included. Correlation between the scores at 3-month, 6-month, and 1-year intervals, as well as floor and ceiling effects, in addition to MCIDs were calculated for each instrument. A total of 132 patients were included in the study.

Results:

There was no observed floor or ceiling effect in either the PROMIS PF or the PROMIS LE scores. Clinically significant changes in the PROMIS LE score were detected in patients between 6-month and 12-month postoperative visits (P = .0006), whereas the reported OMAS score and Weber scores did not identify a clinically significant difference between patients at their 6-month and 12-month visit.

Conclusion:

The results of this study indicate that the PROMIS LE was superior for evaluating patients following ankle fracture surgery in terms of lower floor and ceiling effects and greater ability to distinguish clinically significant changes in patients between time points following surgery.

Level of Evidence:

Level III, comparative study.

Patient-reported outcome measures (PROMs) are increasingly used in both the research and the clinical setting to determine efficacy of treatment from the patient perspective. There are numerous PROMs currently used in foot and ankle surgery, many of which require patients to complete long, burdensome surveys.²² Furthermore, several studies have reported issues with floor and ceiling effects in several PROMs that are commonly used in foot and ankle surgery, including the Foot and Ankle Outcome Scores (FAOS) and the Olerud and Molander Ankle Score (OMAS).^16,31 Floor effect occurs when a large proportion, over 15%, of a study population achieves the lowest possible score, while ceiling effect occurs when over 15% of the study population achieves the highest possible score.

The use of computerized adaptive testing (CAT) for PROMs is rapidly developing among all medical specialties, including orthopedic surgery.^1,6,11,15 The advantages of using CAT include decreased survey-burden, diminished floor and ceiling effect, and improved ability to detect the minimal clinical significant difference (MCID) among patients.^8,10,13,14

The Patient-Reported Outcomes Measurement Information System (PROMIS) was developed using item response theory (IRT) to make a unidimensional test as specific as possible for the issue being tested and minimize the influence from other health domains.² When administered electronically as a CAT, the PROMIS is offered as a dynamic test, which selects questions based on a patient’s answer to preceding questions. This allows the PROMIS to customize each individual’s test, limiting the number of questions administered to each patient, and theoretically providing a more precise representation of a patient’s state.

PROMIS CATs have compared favorably against traditional PROMs used in orthopedics.^7,8,20 Papuga et al²⁸ recently found that PROMIS Physical Function (PF) detects changes in patients following ACL reconstruction that are not detected by the International Knee Documentation Committee (IKDC) measure. The PROMIS PF CAT was studied in patients undergoing meniscal surgery, and it correlated strongly with currently used PROMs while demonstrating no floor or ceiling effect.⁷ The PROMIS Lower Extremity (LE) CAT was derived from the PROMIS PF question bank and is isolated to items pertaining to lower extremity function, but has not been commonly used in assessing orthopedic patients to date.¹¹

The goal of this study was to determine if the legacy PROMs used in foot and ankle surgery would correlate with the PROMIS CAT scores in patients who had undergone ankle fracture surgery. We hypothesized that the PROMIS scores would demonstrate less floor and ceiling effect compared to traditional instruments in this patient population. Furthermore, we hypothesized that the PROMIS LE CAT would demonstrate enhanced sensitivity compared to currently used PROMs to detect improvements individuals between follow-up visits after ankle fracture surgery.

Methods

Following Institutional Review Board review, all patients who underwent osteosynthesis for unstable ankle fractures between 2013–2016 were prospectively enrolled in this study. Patients followed a standard postoperative protocol involving a 2-week period of immobilization followed by early range of motion exercises. After a total of 6 weeks of non-weightbearing, patients then began weightbearing and strengthening.

Patients first completed the legacy foot and ankle outcome scores that were administered at our institution, including OMAS, Weber Score, and FAOS. Patients also electronically completed the PROMIS PF v1.2 and PROMIS LE CATs. The PROMIS had specific tests for physical health, mental health and social health. Each PROMIS test was designed such that a 50 is the mean with standard deviation set to 10 to facilitate interpretation. The PROMIS PF CAT question bank consisted of 124 questions in total. The PROMIS LE CAT consisted of a bank of 79 items specifically addressing lower extremity function that were extracted from the larger PROMIS PF bank.^11,12 The average number of items required before score derivation in the PROMIS LE is 9.¹² The PROMIS was administered using an iPad tablet (Apple, Cupertino, CA). Follow-up visits occurred at 3-month, 6-month, and 12-month intervals after ankle fracture surgery.

Between 2013 and 2016 there were 156 patients prospectively enrolled in the database and confirmed as having ankle fracture surgery operated on by a single surgeon. Thirty-four patients did not complete PROs. A senior orthopedic resident confirmed that the injury was a rotational ankle fracture treated operatively and classified the fracture pattern according to the Lauge-Hansen classification system.¹⁸ This was verified with review of the operative record and the diagnosis given by the treating surgeon (DGL). 132 patients with operatively treated ankle fractures completed at least 1 PRO at either the 3-month, 6-month, or 12-month visit and were included in the study. 19 patients completed PROs at all 3 time points, 51 patients completed PROs at 2 time points, and 62 patients completed PROs at only 1 time point.

In all, 98 patients had supination-external rotation (SER) fractures (12 of which were SER3 and 86 were SER4 patterns), 23 patients had pronation-external rotation (PER) fractures (1 PER3 and 22PER4), 8 patients had supination adduction fractures, 2 patients had isolated medial malleolar fractures, and 1 patient had a pronation abduction fracture. The average age was 46.7 (17.9) years, and there were 79 (59.8%) females and 53 (40.2%) males.

Statistical Methods

Descriptive statistics were calculated for all demographic variables. Clinically significant outcome improvement was determined using the minimal clinically important difference (MCID), which was calculated using a distribution-based method based on study population parameters.²² Previous studies demonstrate that the MCID can be reliably predicted by using half the standard deviation for the outcome score for a given instrument in a patient cohort. The MCID was calculated using the distribution of scores at the 12-month time point.³ As all of the outcome scores measured used a 0–100 scale, the floor and ceiling for each score was defined as either 0 or 100 score. A significant floor or ceiling effect was one in which at least 15% of the participants scored either the highest or lowest score for any given outcome measure.

The correlation between the various outcome scores was calculated using the Spearman correlation coefficient. Correlation coefficients less than or equal to .3 were considered weak, between .31 and .39 moderate-weak, between .4 and .6 moderate, between.61 and .69 high moderate, and over .7 high.³⁰ Differences in patient-reported outcomes between time-points were analyzed using the Wilcoxon signed-rank test. The Wilcoxon signed-rank test uses pairwise comparisons that specifically analyzes individual patient changes between time points, in this case between both the 3-month to 6-month and the 6-month to 12-month time points, rather than comparisons between the whole group means at each time point. All statistical tests were performed using Stata 14.0 (StataCorp, College Station, TX).

Results

Figures 1 and 2 demonstrate the progression of the cohort’s average PROs at the 3-month, 6-month, and 12-month follow-up points as measured by PROMIS PF, PROMIS LE, Weber, and OMAS.

Figure 1.

Average PROMIS PF CAT and PROMIS LE CAT scores at each follow-up time point. Note: the Wilcoxon signed-rank test used pairwise comparisons from 6-month to 12-month time points, not comparisons between the means.

Figure 2.

Average Weber and OMAS scores at each follow-up time point. Note: the Wilcoxon signed-rank test used pairwise comparisons from 6-month to 12-month time points, not comparisons between the means.

The correlation between the PROMIS CATs (both PF and LE CAT) and the OMAS was high (P = .72, P = .73, respectively) (Table 1). Similarly, the correlation between the PROMIS PF and the PROMIS LE was high (P = .76). The correlation between the PROMIS CATs and the FAOS was moderate to moderate-high (P = .46-.69), and the correlation between PROMIS CATs and Weber was moderate (P = .63, P = .60) (Table 2).

Table 1.

MCID and Repeated Measure Comparisons at 3- to 6-Month and 6- to 12-Month Intervals.

Score	MCID	Difference between 3–6 months	P value (3–6 months)	Difference between 6–12 months	P value (6–12 months)
PROMIS PF	4.3	5.4	<.0001	1.8	.37
PROMIS LE	3.7	4.1	.0001	3.9	.006
FAOS ADL	8.0	6.2	.0001	5.7	.0006
FAOS Sports	10.7	18.9	<.0001	12.6	.004
FAOS QOL	12.3	11.8	<.0001	7.4	.0491
FAOS Pain	9.2	8.5	<.0001	5.9	.002
FAOS Symptoms	9.3	7.1	.0009	4.7	.04
Weber	9.4	12.4	<.0001	4.8	.07
OMAS	11.5	17.6	<.0001	3.8	.09

Abbreviations: ADL, Activities of Daily Living; FAOS, Foot and Ankle Outcome Score; LE, Lower Extremity; MCID, minimal clinically important difference; OMAS, Olerud and Molander Ankle Score; PROMIS, Patient-Reported Outcomes Measurement Information System; PF, Physical Function; QOL, Quality of Life.

Table 2.

Correlation between PROMIS and Olerud and Molander, FAOS, Weber Ankle Scores.

Measures	Correlation (rho)	P value	R2	N
PROMIS PF-PROMIS LE	.76	<.001	.61	220
PROMIS PF-OMAS	.72	<.001	.48	140
PROMIS PF-Weber	.63	<.001	.36	156
PROMIS PF-FAOS Symptoms	.46	<.001	.22	220
PROMIS PF-FAOS ADL	.63	<.001	.35	194
PROMIS PF-FAOS Quality of Life	.61	<.001	.35	216
PROMIS PF-FAOS Pain	.56	<.001	.31	217
PROMIS PF-FAOS Sports	.62	<.001	.37	167
PROMIS LE-OMAS	.73	<.001	.50	140
PROMIS LE-Weber	.60	<.001	.33	156
PROMIS LE-FAOS Symptoms	.50	<.001	.23	220
PROMIS LE-FAOS ADL	.69	<.001	.38	194
PROMIS LE-FAOS Quality of Life	.63	<.001	.38	216
PROMIS LE-FAOS Pain	.61	<.001	.35	217
PROMIS LE-FAOS Sports	.65	<.001	.39	167

Abbreviations: ADL, Activities of Daily Living; FAOS, Foot and Ankle Outcome Score; LE, Lower Extremity; OMAS, Olerud and Molander Ankle Score; PROMIS, Patient-Reported Outcomes Measurement Information System; PF, Physical Function.

Smaller changes in outcome score were required to achieve MCID on the PROMIS LE CAT compared to legacy PROMs (Table 1). Calculated 1-year follow-up MCID values on the PROs were 4.3, 3.7, 9.4, and 11.5 on the PROMIS PF, PROMIS LE, Weber and OMAS, respectively. Between the 3-month and 6-month follow-up, all instruments detected a statistically significant improvement in score, but the FAOS Activities of Daily Living (ADL), FAOS Quality of Life (QOL), FAOS Pain, and FAOS Symptoms did not detect clinically significant improvements (Table 1). Clinically significant changes in the PROMIS LE were detected in patients both between 3-month and 6-month follow-up and between the 6-month and 12-month follow-up (P = .0001, P = .0006) (Table 1, Figure 1), whereas neither the Weber nor the OMAS detected a clinically significant difference among individual patients’ outcomes between the 6-month and the 12-month follow-up (Table 1, Figure 2). The only domain of the FAOS that detected a clinically significant difference between 6 to 12 months was the sports domain (P = .0006, Table 1).

There was no observed floor or ceiling effect in either the PROMIS PF or the PROMIS LE scores (Table 3). While none of the PROMs had significant floor or ceiling effect (more than 15%), there was mild (less than 15%) ceiling effect and/or floor effect with all domains of the FAOS score (Table 3), and there was mild (less than 15%) floor effect with both the Weber and the OMAS.

Table 3.

Floor and Ceiling Effects for the Various PROMs.

Score	Floor (%)	Ceiling (%)
PROMIS PF	0	0
PROMIS LE	0	0
FAOS ADL	0	8.5
FAOS Sports	3.2	5.1
FAOS QOL	3.0	1.9
FAOS Pain	0	8.5
FAOS Symptoms	0	1.1
Weber	0	2.0
OMAS	0.5	2.3

Abbreviations: ADL, Activities of Daily Living; FAOS, Foot and Ankle Outcome Score; LE, Lower Extremity; OMAS, Olerud and Molander Ankle Score; PROMs, patient-reported outcome measures; PROMIS, Patient-Reported Outcomes Measurement Information System; PF, Physical Function; QOL, Quality of Life.

Discussion

The PROMIS CATs have performed favorably with regards to reliability, accuracy and decreased patient burden compared to legacy PROMs.^{1,8,15,19,27,28} One major drawback to commonly used PROMs is their inability to detect small but clinically meaningful differences in outcomes. Health care providers and payors have shifted emphasis onto PROMs to determine efficacy of treatments, and part of this shift may involve proving efficacy in these terms to justify treatment strategies. It is imperative that the instruments used to measure PROs are sufficiently sensitive, responsive to changes, and reliable. The goal of this study was to determine if PROMIS LE is able to detect clinically significant improvements in patients having undergone ankle fracture surgery that legacy scores are unable to detect.

In this population of patients recovering from surgery for ankle fractures, the only complete outcome score that was able to detect a clinically significant improvement in patients between 6 and 12 months was the PROMIS LE. Among the legacy PROMs, only 1 domain of the FAOS score, the sports component, was able to detect an MCID between 6 and 12 months. The PROMIS CATs, including the PROMIS LE and the PROMIS PF, had no ceiling or floor effect, while the legacy outcome scores demonstrated varying degrees of floor and ceiling effects. We can conclude from this finding that the PROMIS LE is more sensitive to detecting changes in a patient’s health status related to lower extremity function compared to PROMIS PF, FAOS, Weber, and OMAS.

To date, relatively few studies have used the PROMIS CATs to evaluate patients with lower extremity pathology. Hung et al¹³ tested the psychometric properties of the PROMIS LE CAT and concluded that it outperformed legacy hip PROs in terms of unidimensionality, lack of sex bias, and floor or ceiling effects. Ho et al⁸ used the PROMIS PF and PROMIS Depression CATs to predict which patients who were undergoing elective foot and ankle procedures would have clinically significant improvements from surgery. They found that the PROMIS PF improved on average 8 points from before to after surgery, which was clinically meaningful according to their distribution-derived MCID. However, patients with depression scores below 41.5 had 90% probability of failing to achieve MCID.⁸ Dean et al⁴ used the PROMIS PF to identify independent predictors of poor outcome following operative ankle fractures. Fuchs et al⁵ used PROMIS PF in ankle fracture patients to detect if arthroscopy at the time of operative fixation resulted in improved outcome scores and found no difference. Hancock et al⁷ compared the PROMIS PF CAT to legacy knee scores, including the Knee Injury and Osteoarthritis Outcome Score (KOOS) and the Marx Knee Activity Rating Scale, and found that the PROMIS PF CAT has moderate correlation with both scores, but PROMIS PF CAT had no floor or ceiling effect. Most recently, Koltsov et al validated PROMIS Pain Interference (PI), mobility and PF CATs in 232 patients who underwent elective foot and ankle surgery. The authors found that the PROMIS CATs correlated strongly with FAOS and were responsive to changes between preoperative and postoperative status in patients undergoing surgery for hallux valgus, ankle arthritis, flatfoot, hallux rigidis, osteo-chondral defects of the talus or ankle instability.

Aside from further validating PROMIS LE CAT in ankle fracture patients, the other major finding of this study was that patients continue to experience clinically significant improvements up to 12 months following surgery. This is consistent with previous studies, and is useful data to help guide patient expectations. Nilsson et al²¹ and Hong et al⁹ both reported persistent residual functional limitations in ankle fracture patients up to 12 months following injury. In a randomized multicenter trial comparing operative to nonoperative treatment of unstable lateral ankle fractures, Sanders et al²⁹ used the OMAS as well as the Short Form-36 (SF-36) as the main outcome measures, and documented continued improvement in both groups up to 24 months following injury. Interestingly, Sanders et al failed to detect a significant difference between the 2 treatment groups, operative versus nonoperative, at any time point, but as the authors discussed, they could not be certain that the outcome scores captured the subtle difference in function between the groups. Additional studies are warranted to compare these treatments now that more sensitive PROMs are available, such as the PROMIS LE CAT.

There is an increasing emphasis on the role of patient-reported outcome for measuring clinical success.^24–26 Health policy experts have advocated for tying reimbursement and health care rankings specifically to improvements in patient-reported outcome.^17,23 Our findings add to the growing body of literature demonstrating that legacy PROMs in foot and ankle surgery may have limited sensitivity for detecting clinically significant outcome improvement. In this study, the MCID was calculated using distribution-based methods as opposed to an anchor based approach. Approaches that focus on an individual’s perception of improvement can also be used, and could be the subject of future research in this area.³² Based on the findings of this study we propose that value based health care initiatives tied to clinically significant outcome improvement should incorporate PROMIS measures given their demonstrated lack of ceiling effects and improved responsiveness.

This study has several limitations. As this was a population of trauma patients, many patients did not follow-up at each scheduled visit, which limited the number of patients available with PROMs at multiple time-points. Patients who underwent surgery and never returned for any follow-up appointments could also not be captured. However, despite the limited patients with which pairwise comparisons could be assessed, there still were clinically and statistically significant improvements as measured by the PROMIS LE. Second, given the structure of our clinic, we could not assess time to completion for each of the individual PROs, nor could we guarantee the order with which patients completed the PROs. This could have introduced bias in terms of survey fatigue. However, the role of fatigue effect for PROMs is not well characterized and fatigue is likely mitigated in our setting where a research assistant helps patients fill out PROMs. Finally we only offered the PROs in English, and we did not record how many patients were unable to complete the outcome measures secondary to a language barrier.

Conclusion

In this cohort of patients undergoing ankle fracture surgery, there was a clinically significant improvement in PROs between the 6 and 12-month follow-up timepoints with the PROMIS LE that could not be detected by the Weber, Olerud and Molander, or PROMIS PF scores. These findings add to the growing literature demonstrating the advantages of PROMIS CATs in terms of responsiveness and convenience. Orthopedic surgeons should consider utilizing the PROMIS CATs when assessing postoperative outcomes in patients.