Streamlining Trauma Research Evaluation with Advanced Measurement (STREAM) Study: Implementation of the PROMIS® Toolbox within an Orthopaedic Trauma Clinical Trials Consortium

Summary

The Patient-Reported Outcomes Measurement Information System (PROMIS) has developed item banks, short forms and computer-adaptive tests (CATs) to help standardize measurement for important patient reported outcome (PRO) domains. These tools have the potential to revolutionize outcome measurement in clinical research through greater assessment precision while reducing response burden. Perceived implementation challenges include the need for CAT software, mobile technology and internet access. Here, we present preliminary results examining the feasibility of using PROMIS tools within a large, multi-center clinical trials consortium. The assessment of 10 PROMIS domains was incorporated into the longitudinal data collection of six ongoing orthopaedic trauma clinical trials for participants being evaluated at 3, 6 and 12 months following an orthopaedic injury. Twelve-month assessments included both CAT assessments as well as completion of full item banks for a subset of domains. Data were collected for 1,000 trauma patients at 43 trauma centers using a custom-built application which included an interface with our primary data capture system. Paper short forms were available as backup instruments and used infrequently. Six and 12-month study assessments were conducted for 83% and 77% of patients, respectively. It was feasible to use PROMIS tools in a large multi-center, trauma orthopaedics research setting. The ability to efficiently assess a wide spectrum of domains is critically important to the successful completion of future large-scale trials.

Background and Rationale

Injuries requiring limb trauma care often result in significant long-term consequences for individuals. In 2020, there were nearly 858,000 inpatient hospitalization visits for extremity fractures in the US.^{1, 2} In 2014, there were almost 350,000 upper extremity and 255,000 lower extremity injuries per 10,000 full-time workers.³ A 2012 report noted that 70% of self-reported lost workdays in the US, representing 216.5 million days, were associated with musculoskeletal injuries.⁴ Although orthopaedic trauma is an important public health issue, research has been limited by 3 key barriers in the measurement of injury outcomes. The multi-factorial nature of trauma outcomes often requires the use of multiple measures, which can be costly to collect and burdensome to patients. Second, outcomes are time dependent and measures must be responsive to a wide range of outcome trajectories. Third, “lost to follow up” is common amongst this population and can potentially skew results.

More recently developed Patient-Reported Outcomes Measurement Information System (PROMIS) outcome instruments offer a potential solution to the main problems in orthopaedic trauma outcomes research. PROMIS instruments are available as a static short form or as a dynamic computerized adaptive test (CAT), where subsequent questions delivered are dependent upon respondent answers to previous items. This feature enhances precision and reduces patient burden.^5–12 CATs offer short, valid, precise measures, making measurement across multiple domains more feasible. Scaling is done using T-scores, which can help reduce floor and ceiling effects found in many patient-reported outcome measures (PROMs).^{7, 10, 13} The publicly available PROMIS item banks can form the basis for a standardized measurement strategy in orthopaedic trauma, which can facilitate the development of studies and the synthesis of research findings.

The STREAM Study examines the relevance and measurement properties of PROMIS tools in an orthopaedic trauma population. The overall strategy was to leverage the existing infrastructure of a large orthopaedic trauma clinical trials consortium, the Major Extremity Trauma and Rehabilitation Consortium (METRC).¹⁴ The study objectives are to evaluate reliability and construct validity of the PROMIS CATs, compare measurement precision of the six existing item banks (Supplemental Digital Content Figure 1) when applied in an orthopaedic trauma population versus the general population, and identify items from existing PROMIS item banks that function differently based on the population. The anticipated output of this project will include: 1) empirical evidence of the temporal stability, internal consistency, reliability, validity and responsiveness of PROMIS CATs and short forms in an orthopaedic trauma population; 2) a demonstration of the feasibility of the use of the PROMIS tools within the framework of a large clinical trials consortium; and 3) validation of the applicability of items in the current PROMIS item banks to this patient population, determined through differential item functioning analyses. We expect this study will yield data about outcome measurement tools that will significantly advance the field of trauma outcomes research by facilitating the use of a common measurement strategy for future clinical trials and outcomes research in trauma populations.

Methods

I. Study Participants

This is a prospective longitudinal cohort study of PROMIS within an orthopaedic trauma consortium. Those eligible to participate in the STREAM Study included all participants enrolled in one of six prospective studies being conducted by METRC. These “parent” studies included three studies of very severe, limb threatening tibia fractures and amputations (FIXIT, OUTLET, TAOS), two studies of slightly less severe lower limb fractures (OXYGEN, VANCO), and one study of isolated, low severity fractures (Pain) all of which have been completed and previously described.^15–20 The key characteristics of these studies are shown in Supplemental Digital Content Table 1. Participants in the STREAM study were recruited and informed consent was obtained during the 3-month study visit of the parent studies. Anyone unable to provide informed consent or requiring a proxy at the time of the 3-month parent study visit was excluded.

Enrolled participants were asked to complete assessments in conjunction with the measures collected at the parent study follow up visits occurring at 3, 6, and 12 months using the PROMIS measures and item banks (Supplemental Digital Content Figure 1). The single exception to this was the VANCO study which did not have a 12 month follow up visit. These assessments focused on six core domains of the PROMIS battery: (1) ability to participate in social roles and activities, (2) depression, (3) anxiety, (4) psychosocial illness impact (positive), (5) pain interference, and (6) physical function. The STREAM study also collected data pertaining to four exploratory domains: (1) sleep disturbance, (2) satisfaction with participation in social roles, (3) applied cognition, and (4) emotional support. The domains were considered exploratory in that very limited data exists about their role in orthopaedic trauma. As such, it was believed that limited data collection would impose a minimum burden to the patients but might yield important future data for the field.

II. Study Procedures

Study data for the parent studies were captured using research electronic data capture (REDCap)²¹ and study identification numbers assigned for those studies were cross referenced with a separate project for the STREAM Study to document co-enrollment. For consistency with other METRC studies, consent and other administrative events were documented in REDCap for all study participants. A computer application was created to collect study assessment data via a tablet or other smart device and built around a code engine provided by the developers of the PROMIS platform specifically for this study. This application allowed the use of CATs and short forms to assess the outcome domains, limiting respondent burden and improving ease of data collection.

The application included an interface connected with the STREAM REDCap database to link the study identification numbers across data environments. The system also included a custom designed administrative area to facilitate user management and data completeness monitoring. The system collected time-stamped responses, allowing survey burden to be calculated, and captured individual item responses, domain score calculations and assessment metadata. Users could pause and resume or restart assessments if needed and a system of integrated alerts helped site research staff ensure assessment completion for all participants.

If the assessment could not be conducted using the tablet application (e.g. due to a connectivity problem), short forms (non-adaptive instruments) for each domain were made available as hard-copy documents and through REDCap. Computer adaptive tests were only available in English, so Spanish-language participants also had to use short forms to provide assessment data. The computer application was the preferred data collection method and while it was designed to be used during an in-person assessment, it was possible for site research coordinators to email study participants a uniform resource locator (URL) to complete the assessment remotely if in-person collection was not possible.

III. Outcome Measures

At the 3-month and 6-month parent study visits, participants completed a survey which included the CATs for the six core and four exploratory outcome domains, designed to last approximately 10 to 12 minutes. At the 12-month assessment, participants completed the CATs for the six core domains and a subset of questions from the item banks associated with these domains. Because the psychometric goals of this study required complete item banks for these domains, participants were randomly allocated to one of three groups, each consisting of a subset of questions from specific domain item banks. In order to prevent the overall response burden from becoming unreasonable, the 12-month assessment collected approximately 100 item bank questions per respondent and was designed to take about 20 to 30 minutes to complete. The three groups were constructed as follows, based simply on a breakdown that would approximate 100 items per group:

• Group A: psychosocial illness impact item bank (39 items) + pain interference item bank (40 items) + depression item bank (28 items) = 107 items;

• Group B: physical function short form (12 items) + one half of remaining physical function item bank (55 items) + ability to participate in social roles and activities item bank (35 items) = 102 items;

• Group C: physical function short form (12 items) + one half of remaining physical function item bank (55 items) + anxiety item bank (29 items) = 96 items.

If a participant missed a follow up visit for the parent study or if there was no parent study visit scheduled (e.g. 12-month follow up visit for the VANCO Study), sites were instructed to attempt to contact the participant and conduct the STREAM Study assessment using other available means during the follow up timeline, such as emailing a URL to the survey or completing the assessment via a telephone interview where questions would be read to the participant and data entered into the database in real time by the coordinator.

Results

Between March 2014 and August 2017, 40 METRC civilian trauma centers and 3 military treatment facilities screened 1368 patients for eligibility. Of those, 201 eligible patients refused participation and an additional 102 were not enrolled due to other administrative reasons, leaving 1065 patients who provided informed consent to participate in the study. Because the screening and consent processes could begin prior to the first STREAM study assessment, some patients that consented were not fully enrolled due to withdrawing early, leaving 1000 participants as our final study sample. While STREAM had goals independent of the parent studies, study assessments were often conducted in conjunction with the clinical follow up visit of the parent study. Because of this inherent link between studies, individual withdrawals from STREAM were driven by patients withdrawing from parent studies (n=75) either before or after providing data for the current study.

The majority of participants enrolled were recruited from parent studies enrolling a larger number of patients (OUTLET, OXYGEN, and VANCO). Participants were primarily non-Hispanic white (73%) and male (69%) with a mean age of 41. Additional characteristics of the study participants are reported in Table 1. STREAM PROMIS assessments were completed at 3-, 6-, and 12-month follow up visits for 981 (98%), 825 (83%) and 769 (77%) individuals, respectively. The majority of these assessments were completed electronically through the computer application described above. A small proportion of the assessments at each time point were administered using back up paper short forms (ranging from 2 to 4%) or partially complete (missing at least 1 domain, ranging 2 to 5%). For assessments conducted electronically, the median number of items asked of participants at 3 and 6 months (CATs only, 10 domains) was 52 with the time required being roughly 12 minutes (3-month median (IQR): 12.7 (9.0–19.0); 6-month median (IQR): 11.7 (8.8–17.1). The 12-month assessment, which included 6 CATs supplemented by item bank questions, had a median of 120 questions and assessment time of 24 minutes (Table 2). In total, patients completed 21846 CAT administered domain items (Table 3).

Table 1.

Cohort Characteristics

	Number	Percent
Enrolled	1000
Parent study
FIXIT	94	9
OUTLET	244	24
OXYGEN	195	20
PAIN	180	18
TAOS	45	5
VANCO	242	24
Age, mean (SD)	41.2 (13.5)
Sex, Male	687	69
Race/Ethnicity
Hispanic	66	7
Non-Hispanic, black	163	16
Non-Hispanic, white	732	73
Other	36	4
Refused/Unknown	3	0
Education
Less than high school	142	14
High school or GED	326	33
Some college or higher	522	52
Refused/Unknown	10	1
Usual major activity (pre-injury)
Working/military active duty	758	76
Laid off/looking for work	47	5
Going to school	33	3
Taking care of your house	71	7
Something else	88	9
Refused/Unknown	3	0
Smoking history
Never smoked	412	41
Former smoker	215	22
Current smoker	367	37
Refused/Unknown	6	1

Table 2.

Assessment Breakdown and Characteristics

				Item Banks @ 12 Month
Assessment	Baseline (3-month)	6 Month	12 Month	A	B	C	None*
Number	981	825	769	254	256	249	10
% Paper Admin	2%	3%	4%	3%	3%	4%	60%
% Partial Complete	2%	2%	5%	4%	6%	3%	70%
Electronic Assessments Only, Median (IQR)†
Items per Assessment	52.0 (46.0 – 63.0)	52.0 (45.0 – 62.0)	120.0 (119.0 – 127.0)
Total Time (minutes)	12.7 (9.4 – 19.0)	11.7 (8.8 – 17.1)	24.2 (17.9 – 33.1)
Time per Item (seconds)	13.6 (10.5 – 19.6)	13.0 (10.1 – 18.3)	11.9 (8.7 – 16.3)

^*10 individuals had no item bank group assigned due to administrative errors.

^†Timing data was not collected for short form/paper administered assessments.

Table 3.

Computer Adaptive Test (CAT) Assessments Completed by PROMIS Domain and Study Time Point

CAT Item Bank	Baseline	6 month	12 month	Total
01. Physical Function Bank	960	799	740	2499
02. Pain Interference Bank	958	797	739	2494
03. Anxiety Bank	954	797	738	2489
04. Depression Bank	953	794	738	2485
05. Ability to Participate Social bank Version 2	952	793	737	2482
06. Psychosocial Illness Impact Pos Bank	947	791	734	2472
07. Applied Cognition Gen Concerns Bank	945	791	-	1736
08. Satisfaction with Roles and Activities Bank Version 2	943	790	-	1733
09. Sleep Disturbance Bank	941	788	-	1729
10. Emotional Support Bank	940	787	-	1727
Total	9493	7927	4426	21846

Across all timepoints, only 127 of 2575 (5%) assessments required multiple attempts to complete the evaluation. This occurred at 20 (47%) of the study sites and over half of these instances occurred at 3 of the highest enrolling centers. The number of attempts ranged from 2 to 6 when multiple attempts were needed. Reasons for multiple survey completion attempts were not recorded in a standardized manner. However, the following reasons for multiple attempts were most commonly reported by research coordinators: pausing and resuming the survey assessment to fit within the various elements of the clinical visit, scheduling constraints requiring participants to end the survey prematurely, and technical problems related to poor internet connectivity.

Discussion

Patient reported outcome measures are used in orthopaedic trauma to determine the health status of patients in research, clinical care, quality assessment, and cost-effectiveness analysis.^{5, 6} Some PROs are lengthy, which could lead to low compliance rates²² or are narrow in scope. Further, many were developed based on classical test theory, which relies on validation of entire scales rather than item level validation.^{5, 6} This led the National Institutes of Health (NIH) to fund development of the Patient-Reported Outcomes Measurement Information System.^{6–8, 10, 12, 13, 23} PROMIS was developed using item response theory, which assumes unidimensionality (each item in an item bank tests a single trait) and local independence (each item has a distinct function for estimating the trait).^{5, 7} PROMIS comprises item banks that are organized into domains of health (e.g., physical health, mental health, and social function)^{6, 7, 10, 13} and have been found to reliably report health and functional outcomes for patients with orthopaedic foot, ankle, upper extremity, and spine conditions.⁶

In a review of 88 studies published between 2013 and 2018, within the orthopaedic field, the PROMIS CAT approach was used most frequently, and the lower extremity body region was the most commonly examined. The majority (82%) of these studies reported on 1 to 3 PROMIS domains, with physical function (PF) and pain interference (PI) being the most frequently reported.¹⁰ In a general overview of the orthopaedic literature, PROMIS PF was strongly correlated with legacy PROM scores (range: 0.59–0.83) when evaluating upper and lower extremity as well as spine patients.⁸ PROMIS PF also had fewer questions and took less time to complete for lower extremity studies compared to legacy forms.

PROMSs are increasingly utilized in orthopaedics to capture health status indicators for clinical care, research, and cost-effective analyses.⁶ As PROMs take hold in these settings, clinicians must incorporate these practices into their clinical care and explain to patients the meaning of their T-scores. Improvements in both measurement techniques and computer technology have accelerated the pace of acceptance.^{5, 10} However, the comparison of PROMIS to legacy PROMs in the orthopaedic trauma field remains an issue.⁷ The PROMIS item banks and tools have not been fully tested among orthopaedic trauma patients, especially in the context of clinical trials. There have been limited studies comparing PROMIS to legacy measures, which has contributed to a lack of widespread acceptance.^{6, 7, 10, 23, 24} Additionally, specific infrastructure is needed to administer PROMIS CATs using a computer (i.e. survey functionality that can interact with the CAT engine).¹³ This may reduce accessibility, but solutions are emerging for implementing CATs with minimal IT resources.⁵ If properly developed, crosswalk tools may help improve the comparability of PROMIS with other PROMs, and the accelerated pace of research may increase the adoption of PROMIS instruments.^{6, 7, 12, 25} The PROsetta Stone® project is leading the effort to develop such crosswalks²⁶. There has already been an increase in literature reporting PROMIS measures over recent years¹⁰ and the release of new versions of PROMIS may accelerate the transition away from older versions.¹³

Large clinical studies conducted through networks such as METRC have set the stage to solidify the use of PROMs and improved measurement methods. Despite different approaches to measurement, it is possible to compare outcomes across studies using traditional (legacy) PROMs or PROMIS tools by applying crosswalks developed for this purpose. Data from the STREAM study will be used to develop additional crosswalk procedures and use these linkages to evaluate and validate the minimal clinically important differences (MCIDs) across measures through both distributional and anchor-based methods. The STREAM study has paired these measurement approaches with six large research studies which include a variety of legacy instruments typically used to assess orthopaedic trauma outcomes (Supplemental Digital Content Table 2). This study of 1000 patients having a range of lower limb injuries and concomitant trauma evaluates several clinically important outcomes across multiple assessments. These data, along with those from its six parent studies, are from surgical cases at level one trauma centers and offer a unique opportunity to validate PROMIS measures for use in the orthopaedic trauma patient population.

Substantial effort was required to develop the computer application that was the primary method of data collection. At the outset of the study, the options for administering PROMIS CATs were limited and the NIH Assessment Center (AC) was the primary resource to accomplish this.²⁷ We decided not to use the AC portal instead preferring custom application development which offered a tool that could easily connect to our REDCap installation. While offline data collection (locally stored on a device until an internet connection allows synching with a central database) was considered for this study, it had an additional set of technical challenges that would require more development than our timelines and budget would permit.

The STREAM study experienced several challenges and limitations. Successful execution of the study protocol relied on several items that were beyond the direct control of the study team. While we attempted to standardize the computing environment by centrally managing an inventory of tablet devices issued to sites, connectivity was sometimes a challenge despite the application not reporting any downtime. The activity of a typical clinical follow up visit was not entirely predictable, leading to occasions where completion of assessments started and stopped or required multiple attempts. The study design necessitated an attachment to six other studies, each with its own aims, processes, and challenges. By co-enrolling into the STREAM study, participants provided data beyond the scope of the parent study, with compensation for their time and only modest added burden to the overall clinical follow up visits. While the study benefited from its connection to the parent studies by using their regularly scheduled clinical follow up visits and achieving encouraging completion rates, this co-enrollment also tied it to attrition from the parent study. Finally, our study included few non-English speakers due to limited data collection options in their native languages, which may have disproportionately affected some participating sites. While not fully available during the study period, PROMIS is actively developing and validating item banks in languages other than English.

The results suggest that respondent fatigue was not a problem, as evidenced by the assessment completion times and by the overall follow up rate. Among the predominantly non-Hispanic white and male study population, most assessments were completed electronically using CATs. When 10 CAT domains were assessed for the 3- and 6-months assessments, a median of 52 questions were asked with a roughly 12-minute completion time required. The 6 CATs assessed at the 12-month assessment had a median of 120 questions with an assessment time of 24 minutes. We are also encouraged by the continued integration of PROM data collection tools, including CATs, into many other applications as a result of technological advances over the lifespan of this study. PROMIS tools are no longer unusual in the clinical and research settings and they are now natively built into REDCap and electronic medical records (EMR) systems. These improvements have resulted in METRC incorporating PROMIS CAT assessments into new and ongoing studies.

Conclusion

The STREAM Study used a multi-center, longitudinal observational approach to amass a valuable database that can be used to evaluate the reliability, validity, and responsiveness of PROMIS tools in orthopaedic trauma patients. The current study integrated data collection tools within a large multi-center, trauma orthopaedics research setting with few barriers encountered. Perhaps most importantly, the longitudinal nature of the study allows us to better measure clinically important change following orthopaedic trauma. The ability to assess multiple PRO domains using PROMIS CATs in the time typically required for a single legacy measure is critically important to the successful completion of future large-scale trials.

Supplementary Material

Supplemental Digital Content_3

Supplemental Digital Content Figure 1. Study Enrollment and Data Collection Flowchart