Comparison of Generic, Musculoskeletal-Specific, and Foot and Ankle–Specific Outcome Measures Over Time in Tibial Plafond Fractures

Abstract

Background:

This study performed a psychometric analysis assessing and comparing the responsiveness of the relevant components of a generic (Short Form–36 [SF36]), a musculoskeletal-specific (Short Musculoskeletal Functional Assessment [SMFA]), and a foot and ankle–specific (Foot and Ankle Outcome Score [FAOS]) outcome score when evaluating surgically treated tibial plafond fractures over time.

Methods:

Fifty-one patients were followed for 12 months after their tibial plafond fracture. Responsiveness, or the ability to detect clinical change in a disease, was evaluated through the standardized response mean (SRM), the proportion meeting a minimal clinically important difference (MCID), and floor and ceiling effects.

Results:

The SRM of the SF36–Physical Component Summary (PCS) was significantly greater than the SMFA–dysfunction index (DI) (P < .01) and FAOS–Activities of Daily Living (ADL) (P = .01) between baseline and 6 months, whereas the SRMs of only SF36-PCS and FAOS-ADL differed (P = .01) between 6 and 12 months. The proportion of patients achieving an MCID for SF36-PCS was higher than FAOS-ADL (P = .03) between baseline and 6 months and higher than SMFA-DI (P = .04) between 6 and 12 months. The FAOS-ADL showed substantial ceiling effects at baseline (88.2%) but much less at 6 months (5.9%) and 12 months (9.8%). Smaller ceiling effects were observed for the SMFA-DI (11.8%) at baseline, whereas none were observed for the SF36-PCS.

Conclusions:

This study found that the SF36-PCS had greater responsiveness in assessing tibial plafond fractures compared to the SMFA-DI and FAOS-ADL, particularly in the first 6 months after surgery. In addition, limitations were revealed in the SMFA-DI and FAOS-ADL. This study illustrates the necessary diligence required for selection of outcome measures, as musculoskeletal and anatomy specific scores are not necessarily superior.

Level of Evidence:

Level II, prospective cohort study.

Introduction

Fractures of the tibial plafond, or pilon fractures, are relatively rare, comprising less than 1% of all lower extremity injuries. ^4,22 They result from high-energy axial loads, such as motor vehicle accidents or fall from height, and are often associated with other skeletal and nonskeletal injuries seen in polytrauma. There is a significant effect on patient function, with 43% reporting the inability to return to prior work even 2 years following the injury. ²⁵ Ankle stiffness, swelling, and pain are all documented complications following pilon injuries. Although there is a general consensus that operative fixation is the preferred method of treatment for these injuries, the type and timing of fixation is still being investigated as an active area of research.

The standard in measuring results of treatment in orthopedic trauma, and across all of medicine, has become the use of validated functional outcome scores (FOS). ²⁵ The Short-Form 36 (SF-36) is a validated generic functional outcome score used to assess a variety of medical and surgical diseases. ^23,34,35 The Physical Component Score (PCS) has been used extensively in previous orthopedic literature to evaluate musculoskeletal injuries and their recovery. ⁶ While generic outcome measures allow comparison across disease states, new outcome measures are often tailored to a specific disease state, theoretically improving the responsiveness, or the ability to detect small but clinically significant changes in a specific disease. ³³ In orthopedic surgery, numerous anatomic- and injury-specific outcome measures have been developed, albeit with varying degrees of scientific rigor. The Short Musculoskeletal Function Assessment (SMFA) is a musculoskeletal outcome scoring system that has been tested for validity and reliability ^26,32 and is endorsed by the American Academy of Orthopaedic Surgeons. ¹ The Foot and Ankle Outcome Score (FAOS), which is adapted from the Knee Injury and Osteoarthritis Score (KOOS), has been shown as a valid ankle-specific outcome score. ^11,28,29

The generic and disease-specific outcome scores are proven valid and reliable methods for evaluating musculoskeletal injury. In contrast, there is a paucity of literature assessing the responsiveness of outcome scores. ⁹ This is likely due to the fact that studies assessing responsiveness require data on multiple outcome score assessments at multiple time points. Using multiple scores significantly increases the burden on the patient and clinician. We hypothesized that anatomy specific FOS would be more responsive in assessing orthopedic injuries than generic FOS. This study aimed to compare the responsiveness of the generic SF-36, musculoskeletal-specific SMFA, and the foot and ankle–specific FAOS scores in the setting of tibial plafond fractures.

Methods

A prospective study was conducted at a single level 1 trauma center between 2008 and 2015 on all patients who received operative fixation for a tibial plafond fracture. All patients older than age 18 years who sustained an isolated tibial plafond fracture and underwent open reduction and internal fixation as definitive treatment were approached for enrollment in this study. This study was approved by the Institutional Ethics Committee at our institution. Informed consent was obtained from all patients prior to enrollment. Exclusion criteria included any subjects with concomitant injuries in the ipsilateral limb, subjects who could not speak English, and subjects with follow-up of less than a year.

Three FOSs were used in this study. The SF36 is a generic health measure that evaluates 8 domains ^23,34,35 : physical functioning, role limitations due to physical health, bodily pain, general health perceptions, vitality, social functioning, role limitations due to emotional problems, and general mental health. From these domains, raw scores are transformed into a scaled score from 0 to 100, where in each case a lower score equates to more disability. These scaled scores are standardized to z scores (based on a US population), linearly transformed to a mean of 50 and an SD of 10, and then combined using specified factor score coefficients into two summary measures, the Physical Component Summary (PCS) and the Mental Component Summary. ³⁵

The SMFA has been used in numerous studies in the orthopedic literature and has been shown to be a valid and reliable method of evaluating a variety of musculoskeletal disorders, and is one of the most frequently utilized outcome measures in the foot and ankle literature. ^6,17,25 Adapted from the Musculoskeletal Functional Assessment Instrument, it is composed of 46 questions that can be summarized into 4 categories (daily activities category, emotional status category, arm and hand function category, mobility category) or in one of 2 index scores (dysfunction index [DI] and bother index). ^10,20 The raw scores are converted to a norm-based score from 0 to 100 for each category or index score based from AAOS mean scores and standard deviations. In the normal population, the mean SMFA score is 50, where a higher score equates to more disability.

The FAOS, an adaptation of the KOOS, is a 42-item questionnaire evaluating for 5 dimensions: Pain, Other Symptoms, Activities of Daily Living (ADL), Sport and Recreational Activities, and Foot and Ankle Related Quality of Life. Each dimension is represented by a normalized score from 0 to 100, where a lower score equates to more disability. ^28,29

All 3 surveys were provided in paper form to the patient during her or his initial hospital admission and during their 6- and 12-month postoperative clinic visits. Patients were asked to score their preinjury function score (baseline) within 2 weeks of the injury. At 6 months and 1 year, the patient repeated each questionnaire for their current function. A research coordinator was readily available to assist the patient in the event of difficulties understanding or completing the questions for any reason. To reduce respondent burden and survey fatigue, ²⁷ only questions used in the scoring of the selected component or dimension of the FOS were supplied to the patient. This study used the PCS of the SF36, the DI of the SMFA, and the ADL portion of the FAOS as these are the measures of physical recovery afforded by each instrument and are commonly used domains reflective of surgical decision making and management. These components of the outcome scores are commonly reported in the orthopedic trauma literature. Conceptually, these components are most similar as they primarily measure physical function and capabilities, making them ideal for comparison. All patients who completed questionnaires at 3 time points were included in the responsiveness analysis. Pearson correlation coefficients (r) were calculated between all pairs of scores, with r >0.6 defined as strong correlation and r <0.3 defined as weak correlation.

There were 89 patients recruited for this study. Fifty-one (57%) of the patients had complete data for the SF-36, SMFA, and FAOS at baseline, 6 months, and 12 months (Table 1). In the patient group with complete data, 35 (69%) of the patients were male, the mean age was 40.1 years, and the mean Injury Severity Score (ISS) was 8.8. There was no statistically significant difference in mean age, sex, or mean ISS between patients with complete versus incomplete data. However, a higher proportion of patients with incomplete data had an ISS greater than 9 (P = .049) and greater than 18 (P = .038) (Table 1). To ensure that all the results are based on the same cohort of patients, these are limited to the 51 patients with complete data.

Table 1.

Demographic Data of Study Patients Comparing Groups Based on Completeness of Data.

Characteristics	Patients With Complete Data (n = 51 [57.3%])	Patients With Incomplete Data (n = 38 [42.7%])	P Value
Sex, n (%)
Male	35 (68.6)	27 (71.1)	>.99a
Female	16 (31.4)	11 (28.9)	>.99a
Age
Mean (SD)	40.1 (14.1)	42.8 (14.1)	.87b
Median (range)	34.0 (19-70)	42.5 (21-74)
Injury Severity Score
Mean (SD)	8.8 (2.6)	9.8 (5.6)	.26b
Median (range)	9.0 (4-18)	9.0 (4-29)
ISS >9, n (%)	3 (5.9)	8 (21.1)	.049a
ISS >18, n (%)	0 (0.0)	4 (10.5)	.038a

Abbreviation: ISS, Injury Severity Score.

^a Fisher exact test.

^b Student t test.

Responsiveness was analyzed through the standardized response mean (SRM), the minimal clinically important difference (MCID), and the floor and ceiling effects. The SRM is the mean score improvement divided by the SD of score improvement. ¹⁸ An SRM was calculated between baseline and 6 months, as well as between 6 months and 1 year, for all scores. The SRMs for different scores over the same time period were compared using the paired t-test on the patient-specific standardized score improvements. The MCID is the smallest change in score that reflects a clinically significant difference and is specific to the disease. To our knowledge, there are no established data on the MCID in the above outcome measures in a population with tibial plafond injuries. In these scenarios, an accepted method for estimating MCID is to use one-half the SD of the patient scores when they are maximally affected by the disease. ²⁴ Hence, for this study, the MCID was calculated based on data at the 6-month time point. The proportions of patients meeting MCID on different scores over the same time period were compared using the McNemar test. To describe ceiling and floor effects for the measures, the proportions of patients achieving the maximum and minimum level of functioning detectable by each outcome score are reported at each time point.

All statistical analyses were carried out using the R statistical computing environment (R Foundation for Statistical Computing, Vienna, Austria), with P values <.05 considered statistically significant.

Results

The distribution of scores for the SF36-PCS, SMFA-DI, and FAOS-ADL at each time point is displayed in Figure 1. The scores were quite strongly correlated at both follow-up time points with all Pearson correlations (r) >0.62 in magnitude, whereas the correlations were weaker at baseline, particularly between the SMFA-DI and FAOS-ADL (r = 0.24) (Table 2). This low correlation is due, in part, to the large proportion of ceiling values on FAOS-ADL at baseline (see below).

Figure 1.

Distribution of scores at each time point.

Table 2.

Pearson Correlations Between the SF36-PCS, SMFA-DI, and FAOS-ADL for Patients With Complete Data.

Measures Compared	Time Point
	Baseline	6 Months	12 Months
SF36-PCS and SMFA-DI	–0.37	–0.77	–0.82
SF36-PCS and FAOS-ADL	0.59	0.62	0.79
SMFA-DI and FAOS-ADL	–0.24	–0.83	–0.83

Abbreviations: FAOS-ADL, Foot and Ankle Outcome Score–Activities of Daily Living; SF36-PCS, Short Form–36 Physical Component Summary; SMFA-DI, Short Musculoskeletal Functional Assessment–dysfunction index.

The standardized response mean (SRM) was calculated between baseline and 6 months as well as between 6 and 12 months (Figure 2). The SRM of the SF36-PCS was significantly greater in magnitude than the SMFA-DI and FAOS-ADL between baseline and 6 months (P < .01 and P = .01, respectively). Between 6 and 12 months, the SRM of SF36-PCS was significantly greater in magnitude than the FAOS-ADL (P = .01), but it was not significantly different from the SRM of SMFA-DI (P = .16). There were no differences in the SRM between SMFA-PCS and FAOS-ADL in either time period.

Figure 2.

Comparison of the magnitude ¹ of the standardized response mean for SF36-PCS, SMFA-DI, and FAOS-ADL. FAOS-ADL, Foot and Ankle Outcome Score–Activities of Daily Living; SF36-PCS, Short Form–36 Physical Component Summary; SMFA-DI, Short Musculoskeletal Functional Assessment–dysfunction index.

1. The mean improvements are all negative from baseline to 6 months and positive from 6 months to 12 months.

The minimal clinically important difference (MCID) was calculated for each score between time points, using one-half of an SD of the outcome scores at 6 months, when the patient was most affected from the disease (Figure 3). For the SF36-PCS, 84.3% of patients reached MCID between baseline and 6 months, which was significantly greater than the proportion reaching MCID for FAOS-ADL (64.7%, P = .03). Between 6 and 12 months, 58.8% of the patients had recovery greater than MCID for the SF36-PCS compared to 37.3% for the SMFA-DI (P = .04). There was no difference in the proportion of patients reaching MCID between the SMFA-DI and FAOS-ADL at either time point.

Figure 3.

Percentage of patients achieved minimal clinically important difference (MCID) between timepoints for SF36-PCS, SMFA-DI, and FAOS-ADL. FAOS-ADL, Foot and Ankle Outcome Score–Activities of Daily Living; SF36-PCS, Short Form–36 Physical Component Summary; SMFA-DI, Short Musculoskeletal Functional Assessment–dysfunction index.

With regard to the floor effects, no patient achieved the lowest level of functioning that could be assessed by any of the outcome scores at any time point. With regard to ceiling effects (Table 3), no patient achieved the highest level of functioning that could be assessed by the SF36-PCS at any time point. On the other hand, at baseline, 11.8% of the patients achieved the highest level of functioning measurable by the SMFA-DI at baseline, with considerably smaller effects at 6 months (2.0%) and 12 months (9.8%). It should be noted that a ceiling effect assesses distribution at the most positive possible health state, which corresponds to the lowest possible score on the SMFA-DI. The FAOS-ADL had a large ceiling effect at baseline (88.2%) but much less at 6 months (5.9%) and 12 months (9.8%).

Table 3.

Ceiling Effects at All Time Points: Number (%) of Patients at the Highest Possible Level of Functioning.

Outcome Measures	Time Point
	Baseline	6 mo	12 mo
SF36-PCS	0 (0.0)	0 (0.0)	0 (0.0)
SMFA-DIa	6 (11.5)	1 (1.96)	2 (3.92)
FAOS-ADL	45 (88.2)	3 (5.9)	5 (9.8)

Abbreviations: FAOS-ADL, Foot and Ankle Outcome Score–Activities of Daily Living; SF36-PCS, Short Form–36 Physical Component Summary; SMFA-DI, Short Musculoskeletal Functional Assessment–dysfunction index.

^a For SMFA-DI, the highest possible level of functioning corresponds to the lowest possible score.

Discussion

The use of both generic and disease-specific questionnaires to assess functional outcome has been adopted in the orthopedic literature. Although generic outcome scores can be used to compare function across disease states, it is thought that orthopedics-specific scores would be more responsive to small clinical changes specific to musculoskeletal injury. ¹² All 3 scores were well correlated when assessing injury at 6 and 12 months postoperatively. The poor correlation between the scores at baseline is related to the ceiling effects of the SMFA-DI and FAOS-ADL, which implies that these scores cannot distinguish between patients at higher levels of function. These ceiling effects also translated to a lesser responsiveness when assessing changes from baseline function for SMFA-DI and FAOS-ADL, as reflected by the SRM and MCID. As a result, the SF36-PCS, a generic outcome score, was more responsive than the musculoskeletal-specific SMFA-DI and foot and ankle–specific FAOS-ADL, especially between baseline and 6-month assessments. Over both time periods, the SF36-PCS had the largest magnitude with regard to SRM, significantly greater than the SMFA-DI and FAOS-ADL between baseline and 6 months, and significantly greater than the FAOS-ADL between 6 and 12 months. Again, between baseline and 6 months, the SF36-PCS had the largest proportion of patients reaching MCID. Although both the SMFA-DI and FAOS-ADL showed substantial ceiling effects at baseline, the SF36-PCS showed no floor or ceiling effects. It should be noted that the FAOS-ADL had a ceiling effect of more than 80% at baseline, showing it is a poor outcome score when assessing healthy or minimally symptomatic patients. This is not surprising as the FAOS is adapted from the Knee Injury and Osteoarthritis Outcome score, which aims to distinguish between patients with varying severity of a chronic degenerative condition, rather than between healthy and diseased patients. This is reiterated at the 12-month follow-up, where a ceiling effect of 9.8% is seen again as patients return to a higher level of functioning.

Our results are consistent with previous psychometric analysis in the orthopedic literature regarding outcomes following trauma involving the tibia. For both tibial plateau and tibial shaft fractures, no advantages were observed using a disease-specific score over the generic SF36 score. ^6,8 The same limitations for the SMFA seen in this study were also noted in a prior study for tibial shaft fractures with regard to the ceiling effect seen for preinjury scores. ⁶ There have been no studies to our knowledge assessing the responsiveness of the FAOS in trauma involving the tibia.

However, there have also been previous studies that suggest the SF36 has inferior responsiveness than the SMFA. This is likely due to the methodology used when comparing outcome scores. In the nontrauma population, the SMFA was found to have larger effect sizes than the SF36 in patients with primary osteoarthritis undergoing a total knee replacement. ¹⁶ This study assessed the SF36 by each domain, whereas our study focused on the more relevant summary PCS only. Also, Kirschner et al ¹⁶ assessed a population with a chronic degenerative disease. There is no “preinjury” or healthy baseline outcome score, which is where the ceiling effect was seen for the SMFA-DI in our trauma population. It has also been observed that the SF36 has limited questions assessing for upper extremity function, which may explain why Martin et al found the SMFA to be more responsive in their study that included 121 injuries to the upper limb. ²¹

As patients are frequently the primary stakeholders of research studies in the surgical field, the use of patient-reported outcomes have been increasing, with radiographic and clinical examination outcomes being used as secondary outcomes if at all. FOS should be selected based on their validity, reliability, and responsiveness specifically with regard to the pathology or disease of interest. Unfortunately, there is a wide breadth of FOS being used in foot and ankle research, with varying degrees of psychometric analysis. ³⁰ Since 2012, the most commonly used FOS is the American Orthopaedic Foot & Ankle Society Score (AOFAS), ³⁰ which has a surprising lack of literature demonstrating its validity, ³¹ reliability, ¹³ or responsiveness. ⁷

In an attempt to standardize outcome measurement, the Patient-Reported Outcomes Measurement Instrumentation System (PROMIS) was developed. This tool uses computer adaptive technology to tailor the surveys to the patients to decrease the administration time, limiting the burden on patients while maximizing utility. ³ Psychometric analysis is beginning to be performed on this novel FOS in foot and ankle patients ^2,14 and continued assessment of its validity, reliability, and responsiveness across foot and ankle pathologies should continue to be studied.

Our study did have a few limitations. Forty-three percent of patients enrolled in the study did not have complete data including 1-year follow-up and were not included in the data analysis. No differences were found between sex, age, and average ISS between groups. In addition, this level of loss to follow-up is common in trauma studies as trauma patients are often younger, mobile, have high rates of substance abuse, and are less reliable with regard to follow-up compliance. This study only included patients with pilon fractures that were fixed with open reduction and internal fixation, as well as those with no other concomitant injury in the ipsilateral limb. The tibial plafond fracture is uncommon to begin with, and with our inclusion criteria, the sample size was limited. In addition, given the specific patient population, the generalizability to other musculoskeletal diseases, particularly non-trauma, is questionable. The majority of the benefit seen in the SF36-PCS responsiveness compared to the SMFA-DI and FAOS-ADL were seen between a baseline, or healthy preinjury state, to a 6-month postinjury level of function. This sudden and drastic change in function is usually only seen in a trauma setting, in contrast to chronic degenerative diseases. Lastly, the preinjury baseline outcome scores were obtained postoperatively prior to discharge from the hospital. This could result in recall bias. However, a study evaluating patients’ ability to recall their preoperative level of function in elective knee arthroscopy found patients to be quite accurate when using the SF36. ⁵ This was also shown to be true for recall of preoperative outcome scores following total hip arthroplasty. ¹⁹

In conclusion, for tibial plafond fractures, the disease- and anatomy-specific outcome scores, SMFA-DI and FAOS-ADL, were less responsive than the generic SF36-PCS. This is consistent with other psychometric analysis of outcome scores following traumatic fractures of the tibia. With numerous novel outcome scores, such as PROMIS, being developed and utilizing new techniques including computerized adaptive techniques, ¹⁵ this study illustrates that it is still necessary to ensure proper psychometric analysis has been performed on the outcome measure with relation to the disease of interest, as not all musculoskeletal- and anatomy-specific scores are necessarily superior to generic scores.