Patient-Reported Outcomes Measurement Information System physical function correlates with Toronto Extremity Salvage Score in an orthopaedic oncology population

Abstract

Background

The National Institute of Health's Patient-Reported Outcomes Measurement Information System (PROMIS) uses computerised-adaptive testing to reduce survey burden and improve sensitivity. PROMIS is being used across medical and surgical disciplines but has not been studied in orthopaedic oncology.

Questions/purposes

The aim of the study was to compare PROMIS measures with upper extremity (UE) and lower extremity (LE) Toronto Extremity Salvage Score (TESS) by assessing the following: (1) responder burden, (2) correlation between scores and (3) floor/ceiling effects.

Patients and methods

This cross-sectional trial analysed all 97 adult patients treated surgically for a bone or soft tissue tumour at a tertiary institution between November 2015 and March 2016. TESS (UE or LE) and PROMIS (Physical Function, Pain Interference and Depression) surveys were administered preoperatively. Pearson correlations between each PROMIS domain and TESS were calculated, as were floor/ceiling effects of each outcome measure.

Results

(1) Completion of three PROMIS questionnaires required a mean total of 16.8 (+/− 5.8 standard deviation) questions, compared with 31 and 32 questions for the LE and UE TESS questionnaires, respectively. (2) The PROMIS Physical Function scores demonstrated a strong positive correlation with the LE TESS (r = 0.84; 95% confidence interval [CI], 0.72–0.91; p < 0.001) and moderate positive correlation with the UE TESS (r = 0.64; 95% CI, 0.34–0.83; p = 0.055). The PROMIS Depression scores demonstrated a weak negative correlation with both the LE TESS (r = −0.38; 95% CI, –0.61 to −0.10; p = 0.010) and with UE TESS (r = −0.38; 95% CI, –0.67 to −0.01; p = 0.055). The PROMIS Pain Interference scores demonstrated a strong negative correlation with the LE TESS (r = −0.71; 95% CI, –0.83 to −0.52; p < 0.001) and a moderate negative correlation with the UE TESS (r = −0.62; 95% CI, –0.81 to −0.30; p = 0.001). (3) The UE TESS had a range of scores from 16 to 100 with a 27% ceiling effect and no floor effect, and the LE TESS had a range from 10 to 98 with no floor or ceiling effect. There was no floor or ceiling effect for any PROMIS measures.

Conclusions

In an orthopaedic oncology population, the PROMIS Physical Function and Pain Interference scores correlate with the TESS and have the benefit of reduced survey burden and ceiling effect. The PROMIS Depression scores may provide additional information regarding patient outcomes not captured by the TESS.

Level of Evidence

Level III.

The translational potential of this article

Patient reported outcome measures asses patients' symptoms, function and health-related quality of life and are designed to capture more clinical information than can be gathered by objective medial testing alone. As reimbursements and the understanding of patient outcomes are becoming tied to performance on PROMIS measures, it is an important step to establish how PROMIS measures correlate and compare to traditional legacy measures.

Introduction

Historically, the speciality of musculoskeletal oncology has relied on physician-dependent measurements regarding patient function to compare outcomes of treatment. In the 1980s, Enneking [14] described a clinically based system to evaluate postoperative function of sarcoma patients and to compare outcomes across different treatment centres. Known as the Musculoskeletal Tumor Society (MSTS) rating scale, it is completed by the clinician who rates the patient on seven parameters: pain, range of motion, strength, joint stability, deformity, emotional acceptances of the surgical procedure and general functional ability. It has subsequently been revised but is still completed by the physician. Thus, the MSTS does not evaluate the patient's perceived health or functional capacity. Recently, the value of measuring a patient's perception of his or her function in response to treatment has been widely recognized. In the last 2 decades, there has been a transition from clinician-reported outcome measures towards patient-reported outcome (PRO) surveys as the former tends to underestimate symptoms and physical function [6], [4], [28].

The Toronto Extremity Salvage Score (TESS) is a physical disability measure developed specifically for patients undergoing surgery for extremity tumours and has been shown to have superior measurement properties compared with other commonly used scales [12], [10]. It was developed in an effort to improve on the MSTS scoring system and address patient perspectives; thus, the item content was based on patient suggestions and feedback. It consists of 31 questions that gauge the patient's physical function by querying the difficulty of performing activities of daily living and recreation. The TESS questionnaire was developed by the members of a multidisciplinary sarcoma treatment team including surgeons, physical and occupational therapists and nurses of the University of Toronto. This group worked together to identify specific types of functional difficulty experienced by sarcoma patients within the domains of daily living, work or school, leisure, mobility and sexual activity. A preliminary questionnaire was then mailed to the sarcoma patients, who rated the level of difficulty experienced with each particular activity, as well as the importance of the activity. Difficulty and importance frequencies were calculated, and items rated “total unimportant” or “not applicable” by 30% of the respondents were eliminated. Validation testing also assessed internal consistency and test–retest reliability of the TESS [12]. The TESS questionnaire is administered on paper and scored using a Microsoft Excel algorithm.

Although the TESS is currently the most commonly reported patient-reported measure in musculoskeletal oncology, several systems are currently used, which increases the challenge of comparing and combining data [11], [10], [27]. In addition, musculoskeletal oncology presents unique challenges for developing a patient-reported survey, as the heterogeneity of the population is quite broad across age, diagnosis, extent of limb involvement, type of resection and reconstruction or prosthetic options. A standard patient-reported survey would need to be exhaustive to capture the diverse spectrum of outcomes in musculoskeletal oncology, leading to long completion times, survey fatigue and relatively high floor and ceiling effects. The Patient-Reported Outcomes Measurement Information System (PROMIS), a computerised-adaptive testing (CAT) measure recently developed by the National Institutes of Health, represents a potentially universal measurement tool with the advantage of reduced burden on patients [32]. Computer adaptive tests use item response theory, a technique that selects each survey question based on the respondent's previous answer. It eliminates the need to respond to overlapping and repetitive items yet still yields precise measurements regarding patient function [3]. CAT is completed when the consistency of the answers is sufficient for scoring. PROMIS CAT has been studied in several orthopaedic populations [20], specifically including hand and shoulder [1], [2], [8], [13], [24], [25], [32], foot and ankle [18], sports [16], [28], adult reconstruction [20], spine [30] and trauma [19], [18], [21], [33], [29], [1], [2], [8], [13], [16], [20], [25], [31], as well as various oncologic subspecialities [15], [34], [5], [9]. There is currently a widespread initiative for adoption and inclusion of PROMIS domains in all Federal Drug Administration–funded and industry-funded clinical oncology outcome research. In addition, the Federal Drug Administration is working to approve the PROMIS Physical Function score for use as a clinical outcome assessment in adults with advanced solid tumours. If successful, this will lead to a standard PRO measure of physical function for use in future industry-sponsored clinical trials, intended for registration of new oncology drugs.

Despite the growing importance of PROMIS in oncology and orthopaedics, its role in orthopaedic oncology has not yet been established. The purpose of this study was to compare the performance of PROMIS relative to the upper extremity (UE) and lower extremity (LE) TESS by quantifying responder burden, determining the correlation between scores and assessing floor and ceiling effects for each outcome measure.

Methods

Approval was obtained from the institutional review board at Washington University in Saint Louis School of Medicine, and informed consent was obtained from all patients. This cross-sectional study included all new patients who underwent surgical treatment for primary and metastatic bone or soft tissue tumour and who were consecutively evaluated in the outpatient setting by the orthopaedic oncology service at a single university-based tertiary care institution between November 2015 and March 2016. The patients whose preoperative evaluation took place only in the emergency department or inpatient setting were excluded. Pregnant women and non-English speaking individuals were excluded.

Preoperative TESS and PROMIS (Depression, Pain Interference and Physical Function) scores were self-administered to all patients within 60 days of surgery. The TESS survey was completed on paper and manually entered into a secure database, whereas the PROMIS survey was administered using a tablet computer (mini iPad; Apple, Palo Alto, CA), collected over a secure wireless network, immediately scored and deposited into the electronic medical record. The TESS questionnaire was administered on paper and scored using a Microsoft Excel algorithm. The PROMIS measures are available in a computerized adaptive testing (CAT) format and are normalized to a standard population distribution with a mean of 50 and a standard deviation of 10 which simplifies comparisons between the treatment groups. PROMIS assessments are scored so that a higher score represents more of each item being measured. For example, a higher Depression score represents more depressive symptoms, whereas a higher Physical Function score represents better function. Of the 97 patients (70 LE and 27 UE patients), two LE patients were excluded for an incomplete TESS survey.

Our sample size calculation indicated that 47 patients would provide 80% power to detect a minimum of a moderate correlation (r = 0.40) with an alpha of 0.05 [17]. Correlation between each PROMIS measure and the LE and UE TESS was assessed using the Pearson coefficient using the statistical program R (R Foundation for Statistical Computing, Vienna, Austria), and the floor and ceiling effect of each PROMIS measure was calculated and compared with that of the LE and UE TESS. Correlation coefficients were calculated with 95% confidence intervals (CIs) and interpreted according to Evans [24] (0.00–0.19, very weak; 0.20–0.39, weak; 0.40–0.59, moderate; 0.60–0.79, strong; and 0.80–1.00, very strong).

Results

Our population of 97 patients had a mean age of 53 years and a mean tumour size of 6.7 cm. There were more soft tissue tumours than bone tumours, more LE tumours than UE tumours, and more malignant tumours than benign tumours (Table 1). Completion of PROMIS questionnaires required a mean of 4.4 questions (+/− 1.3 standard deviation [SD]) for Physical Function, 6.8 (+/−3.5 SD) for Pain Interference and 5.6 (+/−3.0 SD) for Depression. The combined PROMIS measures, including all three domains, required a mean of 16.8 (+/− 5.8 SD) questions to complete, compared with 31 and 32 questions for the LE and UE TESS questionnaires, respectively (Table 2).

Table 1.

Demographics and tumour characteristics in the patient population.

Patient/tumour characteristics	Mean (range)
Patient age (years)	53 (12–91)
Tumour size (cm)	6.7 (0.5–23)
Number (%)
Upper extremity tumours	27 (28%)
Lower extremity tumours	70 (72%)
Soft tissue tumours	55 (57%)
Bone tumours	42 (43%)
Benign tumours	37 (38%)
Malignant tumours	60 (62%)

Table 2.

Number of questions required for evaluation using PROMIS and TESS surveys.

Number of questions to complete the survey	Mean	SD +/−	Range
Physical function CAT	4.4	1.3	4–12
Pain interference CAT	6.8	3.5	4–12
Depression CAT	5.6	3.0	4–12
Total PROMIS (3 domains)	16.8	5.8	12–36
TESS LE	31	n/a	n/a
TESS UE	32	n/a	n/a

CAT, computerised-adaptive testing; LE, lower extremity; PROMIS, Patient-Reported Outcomes Measurement Information System; SD, standard deviation; TESS, Toronto Extremity Salvage Score; UE, upper extremity.

The PROMIS Physical Function score demonstrated a very strong positive correlation with the LE TESS (r = 0.84; 95% CI, 0.72–0.91; p < 0.001) and a strong positive correlation with the UE TESS (r = 0.64; 95% CI, 0.34–0.83; p = 0.055) (Fig. 1A). The PROMIS Depression scores demonstrated a weak negative correlation with both the LE TESS (r = −0.38; 95% CI, –0.61 to −0.10; p = 0.010) and UE TESS (r = −0.38; 95% CI, –0.67 to −0.01; p = 0.055) (Fig. 1B). The PROMIS Pain Interference scores demonstrated a strong negative correlation with the LE TESS (r = −0.71; 95% CI, –0.83 to −0.52; p < 0.001) and a moderate negative correlation with the UE TESS (r = −0.62; 95% CI, –0.81 to –0.30; p = 0.001) (Fig. 1C).

Figure 1.

Pearson correlation plots demonstrating the relationship between TESS and PROMIS scores. (A) PROMIS Physical Function scores had a strong positive correlation with the LE TESS (r = 0.84, p < 0.001) and a moderate positive correlation with the UE TESS (r = 0.64, p = 0.055). (B) PROMIS Pain Interference scores had a strong negative correlation with the LE TESS (r = −0.71, p < 0.001) and a moderate negative correlation with the UE TESS (r = −0.62, p = 0.001). (C) PROMIS Depression scores had a weak negative correlation with the LE TESS (r = −0.38, p = 0.010) and a weak correlation with the UE TESS (r = −0.38, p = 0.055).

LE, lower extremity; PROMIS, Patient-Reported Outcomes Measurement Information System; TESS, Toronto Extremity Salvage Score; UE, upper extremity.

The UE TESS had a range of scores from 16 to 100 with a 27% ceiling effect and no floor effect, and the LE TESS had a range from 10 to 98 with no floor or ceiling effect. PROMIS Depression, Pain Interference and Physical Function scores ranged from 34 to 78, 39 to 78 and 20 to 73 for LE patients and from 34 to 76, 37 to 80 and 24 to 70 for UE patients. There was no floor or ceiling effect for any PROMIS measures.

Discussion

This study included a heterogeneous patient sample, with various ages, diagnoses and treatments, demonstrating the ability of the PROMIS to accurately score function in patients with substantial disability and those with very high function. On an average, completion of the three PROMIS domains combined required half as many questions as the UE or LE TESS survey. The PROMIS Physical Function scores demonstrated a strong positive correlation with the LE TESS and a moderate positive correlation with the UE TESS; the Pain Interference scores demonstrated a strong negative correlation with the LE TESS and a moderate correlation with the UE TESS; the Depression scores demonstrated a moderate negative correlation with both the LE TESS and UE TESS. The UE TESS had a 27% ceiling effect, whereas there was no floor or ceiling effect in the LE TESS or any of the PROMIS measures.

Our study has several limitations. Although our LE sample size was adequate to detect a medium effect size, there were only enough UE patients to detect a large effect size. Thus, a larger cohort may have demonstrated stronger relationships between the TESS and PROMIS, particularly in UE measures and Depression scores. In addition, a diverse group of patients was included, and our cohort was not large enough for subgroup analysis other than UE versus LE tumours. Therefore, although our results are generalizable, additional information may be gained from analysis by diagnosis and other variables in a larger study. Finally, our study was based only on preoperative scores, and therefore, the results may not apply to a postoperative assessment. It is possible that the TESS and PROMIS measures may differ in how they measure tumour-related symptoms versus surgical outcomes or additional floor/ceiling effects may be present in the postoperative population.

Responder burden

The PROMIS physical function (PF) CAT questionnaire required a mean of only 4.4 ± 1.3 questions for completion, compared with the standard 30-question item TESS questionnaire. In general, longer surveys provide more information regarding outcomes but at the expense of attrition and survey fatigue. CAT can optimize reliability while minimizing the number of items needed to capture a precise outcome score. In traditional outcome measures, scores at the limits of surveys tend to have more error than that in the middle range, whereas item response in CAT is able to achieve measurement precision over a range of ability scores [3], as demonstrated by our data. PROMIS PF correlates most strongly with TESS because both are measures of physical function. The algorithm for the PROMIS Physical Function domain selects from a total of 124 questions garnered from a variety of legacy outcome measures, representing a broader set of questions than represented in non-CAT questionnaires, such as the - TESS. This explains the high accuracy and precision of the algorithm of the PROMIS Physical Function, despite the smaller number of questions answered.

Obtaining PROs in an efficient and timely manner is key to reducing patient burden and attrition, as well as in providing the clinician with key information to help guide treatment decisions. Our study did not evaluate time of completion of surveys; however, previous studies have well documented that PROMIS CAT is faster to complete than the standard accepted PRO measures. In a previous study, comparing TESS versus PROMIS physical function in patients with LE bone metastases, 73% of the participants completed the PROMIS Physical Function survey within 1 min compared with an average completion time of more than 4 min for the TESS survey [23]. In a study comparing PROMIS scores with the disabilities of the arm, shoulder and hand (DASH) scores in patients with UE conditions, the participants completed the PROMIS PF survey in an average of 57 s compared with 262 s for the DASH survey [33]. Similarly, in a study of the validation of PROMIS PF in foot and ankle patients, the participants completed the survey with an average time of only 47 s [18].

Correlation

In our study population, correlation between the PROMIS CAT survey and TESS in an orthopaedic oncology clinic varied based on the anatomic location and outcome measure. In our LE cohort, the TESS had a strong positive correlation with the PROMIS Physical Function scores, a strong negative correlation with the PROMIS Pain Interference scores and a weak negative correlation with the PROMIS Depression scores in patients with LE tumours. In UE patients, the TESS had a moderate positive correlation with the PROMIS Physical Function scores, a moderate negative correlation with the PROMIS Pain Interference scores and a weak negative correlation with the PROMIS Depression scores. Of note, higher PROMIS Depression and Pain Interference scores indicate greater depression and pain, explaining the inverse relationship to the TESS, in which higher scores indicate superior outcomes. The greater strength of the LE correlations is likely attributable, at least in part, to the larger number of patients in that group and the ceiling effect observed with the UE TESS in the following section. It is also possible that functional compromise has a more profound effect on disability when it occurs in the LE compared with the UE. The PROMIS Physical Function survey has demonstrated good correlation with the DASH survey, a common PRO for UE function [33], [22]. This suggests that the weaker correlation between the UE TESS and PROMIS PF scores may be related to limitations of the UE TESS than the PROMIS.

Finally, the weak correlation with the PROMIS Depression scores in both UE and LE tumours confirms that the TESS was designed to measure function and pain but not depression. However, the Depression scores still demonstrated some negative association with the TESS scores suggesting an interrelationship between depressive symptoms and perceived function. A PROMIS Depression score ≥59.9 has been shown to correspond to a score of ≥10 in the Patient Health Questionnaire-9, which indicates moderate depression [34], [35]. Eighteen percent of our patient population reported scores above this threshold at some point during the time frame of the study. Additional research will be necessary to better understand the implications of depressive symptoms in the practice of orthopaedic oncology.

Floor/ceiling effects

A substantial ceiling effect was observed for the UE TESS score, which contributed to the weaker correlation between TESS and PROMIS scores in patients with UE tumours. There was no appreciable ceiling or floor effect for any of the other measures. The elimination of this UE TESS ceiling effect would be a substantial improvement in the assessment of patients with UE tumours. Importantly, the PROMIS Physical Function CAT does not appear to suffer the same ceiling effect. Of note, the PROMIS Physical Function CAT is designed to assess overall function, rather than extremity-specific function. Although the PROMIS Physical Function CAT has been criticized for disproportionally providing questions related to LE function, previous studies have shown a strong correlation with traditional UE patient–reported outcomes [33], [26], [30]. The more recently developed PROMIS UE CAT, which was designed to specifically assess UE function, may have increased sensitivity to capture UE impairment. However, similar to the UE TESS, the PROMIS UE CAT has demonstrated a ceiling effect [7], [8].

Conclusions

The patients in our study cohort completed all three PROMIS domains combined in half as many questions as the TESS survey. The PROMIS Physical Function and Pain Interference scores correlated strongly with the TESS for LE; the correlation was weaker for UE tumours, likely related to the substantial ceiling effect observed for the UE TESS. The PROMIS Depression domain is not measured by the TESS. Potential advantages of the PROMIS over TESS include assessing functional outcomes with greater efficiency (fewer questions) and accuracy (reduced ceiling effect), as well as capturing additional quality of life and mental health outcomes such as Pain Interference and Depression. Larger, long-term studies are needed to further validate PROMIS measures and their applications in orthopaedic oncology.

Conflict of interest

The authors have no conflicts of interest to disclose in relation to this article.

Funding

The authors have no acknowledgements to disclose and they received no funding for the work described in this article.

Appendix A. Supplementary data

The following is/are the supplementary data to this article: