Feasibility of PROMIS CAT Administration in the Ambulatory Sports Medicine Clinic With Respect to Cost and Patient Compliance: A Single-Surgeon Experience

Abstract

Background:

Pay-for-performance reimbursement models are becoming increasingly popular, but the implementation of a routine patient-reported outcome (PRO) collection system places additional burden on both the patient and the provider. The Patient-Reported Outcomes Measurement Information System (PROMIS) was developed in an effort to make PRO collection more practical and efficient, but providers may be reluctant to embrace a transition to a PROMIS-based clinical outcome registry.

Hypothesis:

PROMIS can be successfully incorporated into daily clinical practice, with an overall patient compliance rate of 80%.

Study Design:

Cross-sectional study; Level of evidence, 3.

Methods:

As part of routine practice, all patients presenting to a single surgeon’s sports medicine clinic for an appointment were asked to complete a series of PROMIS computerized adaptive tests (CATs), including PROMIS Physical Function, Physical Function–Upper Extremity, Pain Interference, and Depression subscales. Overall compliance was calculated by dividing the number of survey sets completed by the number of eligible clinic visits. Compliance rates were further assessed by patient age, type of clinic visit, and location of injury. Costs associated with this system of routine PRO collection were categorized as start-up or maintenance costs.

Results:

From August 7, 2017, to December 8, 2017, there were 581 patients (1109 clinic encounters) who met inclusion criteria for the study. Of the 1109 clinic encounters, there was an overall compliance rate of 91.3% (1013/1109 visits during which the patient completed the entire PROMIS survey set). Overall, the full survey set consisted of a mean 15.3 questions and took a mean of 2.6 minutes to complete. Patients who were aged ≥62 years had a significantly lower compliance rate (81.8%; P < .0001) than each of the younger patient quartiles. When analyzing patients by the most common locations of injury (elbow, shoulder, hip, knee), the compliance rate for completing PROMIS was significantly higher for the hip than for the shoulder (95.1% vs 88.9%, respectively; P = .02). The cost of establishing a PROMIS-based registry using our project design and workflow was estimated at $2045, whereas the monthly maintenance cost was $1000.

Conclusion:

The routine electronic collection of PROMIS scores in the ambulatory orthopaedic clinic resulted in a compliance rate of over 90%, although older patients were generally less compliant than younger patients. Our system of data collection is practical and efficient in a high-volume orthopaedic clinic and places minimal financial burden on the provider.

As health care continues to transition toward value-based medicine, there is an increasing emphasis placed on the collection of patient-reported outcomes (PROs). In fact, many insurance companies are now utilizing pay-for-performance reimbursement models in lieu of fee-for-service models.²⁸ These models require the routine collection of PROs and offer financial incentives for high performance. Unfortunately, the implementation of a PRO collection system places a burden on both the patient and the provider. These challenges include increased clinic encounter times, added costs for providers, and the potential for survey fatigue in patients (which may result in unreliable responses and decreased patient compliance^5,10–12).

In an effort to make PRO collection more practical and efficient, the US National Institutes of Health created the Patient-Reported Outcomes Measurement Information System (PROMIS).¹ PROMIS was developed to standardize PRO collection across numerous health domains and conditions while minimizing the administration burden to patients and providers. PROMIS consists of traditional, short-form surveys that have a predetermined number of questions. However, the computerized adaptive test (CAT) versions of PROMIS allow for improved efficiency compared with traditional PRO measures, requiring decreased time for completion.^{3,4,7,9,16–19,29} A number of different PROMIS CAT forms have been utilized for clinical and research applications in orthopaedics, including not only physical function but also pain interference (the impact of pain on the patient’s life) and depression.^{1,2,6,15,23–25}

Despite this improved efficiency of PROMIS CATs, providers may be reluctant to embrace a transition to a PROMIS-based clinical outcome registry. This may be for a number of reasons, including already established PRO registries as well as concerns regarding the integration of PROMIS into existing clinical workflow. Additionally, providers may not be familiar with the technical and cost considerations of incorporating PROMIS into daily clinical and research applications.

The goal of this study was to provide a detailed account of a single-surgeon experience in the incorporation of PROMIS CATs into daily clinical practice. We hypothesized that PROMIS can be successfully incorporated into daily clinical practice, with an overall patient compliance rate of 80%. Moreover, we outlined the costs associated with this incorporation.

Methods

This study was performed with approval from an institutional review board. As part of routine practice, all patients presenting to a single surgeon’s (E.C.M.) sports medicine clinic for an appointment were asked to complete a series of PROMIS CAT questionnaires. PROMIS registry, which used an electronic platform designed for administration on tablets, was established before commencement of this feasibility study. The registry consisted of a set of 3 different PROMIS domains (CAT forms) and an intake form regarding the location of injury. The registry was then pilot tested for 3 weeks to ensure that there were no technological or administrative deficiencies.

Upon check-in, a member of the clinical (or research) team created an electronic registry record for the patient consisting of the medical record number and name. The patient then began the registry by answering a short intake form regarding the location of injury. For the first survey, patients with lower extremity injuries were prompted to complete PROMIS Physical Function (PF) v 2.0 CAT, whereas patients with upper extremity injuries were prompted to complete PROMIS Physical Function–Upper Extremity (UE) v 2.0 CAT. Patients who presented with both upper and lower extremity injuries received both surveys. All patients were then prompted to complete PROMIS Pain Interference (PI) v 1.1 CAT and PROMIS Depression (D) v 1.0 CAT, regardless of their injury. These surveys were administered on iPad tablets (Apple) using a secure web-based application (REDCap) that was designed to support data capture for research studies hosted at our institution.¹⁴

Patients who were administered PROMIS questionnaires during a clinical encounter from August 7, 2017, to December 8, 2017, were retrospectively screened for study inclusion. Patients were included for the analysis of compliance if they declined to participate or otherwise did not respond to survey questions for unknown reasons. Patient encounters were excluded from the analysis of compliance for any of the following reasons: non–English speaking patients, technical glitches interrupting the administration of PROMIS (server maintenance, loss of internet connection, tablet malfunction), appointment cancelations or no-shows, and brief clinic visits for only magnetic resonance imaging interpretation or injections (not routinely provided surveys).

PROMIS surveys were defined as complete if final scores were calculated for all distributed questionnaires and were defined as incomplete if patients never started or only partially responded to questionnaires. Clinic visits were individually categorized as new, returning, or postoperative. Demographic information, including age, sex, race, ethnicity, and employment status, was retrospectively collected using electronic medical records. The time to completion (as measured by the internal mechanism of the software during survey completion), total number of questions, and final scores for each survey were electronically recorded and extracted for analysis. Additionally, during a 14-week sample period, reasons for noncompliance were tracked and itemized. Examples of reasons for noncompliance included patient refusal, patient comprehension barriers, or logistical constraints (ie, patient brought back to the examination room before adequate time needed for survey completion).

Costs

Start-up cost estimates assumed the presence of an existing REDCap collaborator at the host institution and included the costs of electronic tablets, computers, and host institution server fees as well as the estimated cost for the development and routine testing of the PRO collection system before launch. Estimates for maintenance costs included the salary of the research assistant, who was required to supervise administration of PROMIS in the clinic for 2 days each week. Costs are reported as 2018 US dollars.

Statistical Analysis

Summary statistics were calculated for demographic data. Psychometric data were calculated for each PROMIS questionnaire, including the total number of questions and time to completion. Overall compliance was calculated by dividing the number of survey sets completed by the number of eligible clinic visits. Patients were grouped into quartiles based on age (0-30, 31-48, 49-61, and ≥62 years), and Fisher exact tests were used to detect significant differences in compliance between age quartile, location of injury, and visit type. P values <.05 were considered statistically significant.

Results

From August 7, 2017, to December 8, 2017, there were 581 patients (1109 clinic encounters) who met inclusion criteria for the study. Patients were typically seen on the 2 days per week that were allotted to clinic time for the senior author (E.C.M.). The ages of these patients ranged from 11 to 95 years, and the patients were evenly split between male and female. The most commonly reported injuries were those to the knee (38%), shoulder (32%), hip (17%), and elbow (5%). Further demographic information regarding race, ethnicity, employment status, and mean PROMIS scores is shown in Table 1.

TABLE 1.

Patient Demographics (N = 581)^a

Age, mean ± SD, y	46 ± 19
Sex, n (%)
Male	292 (50)
Female	289 (50)
Race, n (%)
White	356 (61)
Black	128 (22)
Asian	25 (4)
American Indian	4 (1)
Other/unknown	68 (12)
Ethnicity, n (%)
Not Hispanic or Latino	458 (79)
Hispanic or Latino	5 (1)
Unknown	118 (20)
Employment, n (%)
Employed	103 (18)
Other	105 (18)
Unknown	373 (64)
Reason for clinic visit, n (%)
Hand/wrist	5 (1)
Elbow	31 (5)
Shoulder	188 (32)
Hip	98 (17)
Knee	220 (38)
Foot/ankle	7 (1)
Multiple	30 (5)
Unknown	2 (1)
PROMIS score, mean ± SD
Physical Function–Upper Extremity	32.3 ± 8.6
Physical Function	39.7 ± 8.2
Pain Interference	60.5 ± 7.4
Depression	48.2 ± 9.6

^aPROMIS, Patient-Reported Outcomes Measurement Information System.

Overall, the full survey set consisted of a mean 15.3 questions and took a mean of 2.6 minutes to complete (Table 2). PROMIS UE and PF had the longest times to completion and averaged 4.8 and 4.2 questions, respectively, whereas PROMIS D took less time to complete and averaged 6.4 questions.

TABLE 2.

Psychometric Properties of PROMIS^a

	Physical Function–Upper Extremity	Physical Function	Pain Interference	Depression	Full Set
No. of completed surveys	433	606	1013	1013	1013
Mean No. of questions	4.8	4.2	4.3	6.4	15.3
Mean time to completion, s	69	66	43	46	158

^aPROMIS, Patient-Reported Outcomes Measurement Information System.

Of the 1109 clinic encounters, there was an overall compliance rate of 91.3% (1013/1109 visits during which the patient completed the entire PROMIS survey set) (Table 3). When stratified into quartiles by age, there were no significant differences in compliance for patients aged 0-30 years (96.1%), 31-48 years (94.3%), and 49-61 years (93.0%). However, patients who were aged ≥62 years had a significantly lower compliance rate than each of the younger patient quartiles (81.8%; P < .0001) (Figure 1). There were no significant differences for compliance among different types of clinic visits (P > .05). When analyzing patients by the most common locations of injury (elbow, shoulder, hip, knee), the compliance rate for completing PROMIS was significantly higher for the hip than for the shoulder (95.1% vs 88.9%, respectively; P = .02).

TABLE 3.

Patient Compliance in Responding to PROMIS^a

	No. of Surveys Completed	No. of Surveys Distributed	Compliance Rate, %
Overall compliance	1013	1109	91.3
Compliance by age quartile
Quartile 1 (0-30 y)	270	281	96.1
Quartile 2 (31-48 y)	265	281	94.3
Quartile 3 (49-61 y)	253	272	93.0
Quartile 4 (≥62 y)	225	275	81.8
Compliance by visit type
New	381	417	91.4
Returning	412	450	91.6
Postoperative	220	242	90.9
Compliance by location of injury
Hand/wrist	6	6	100.0
Elbow	53	57	93.0
Shoulder	352	396	88.9
Hip	173	182	95.1
Knee	382	420	91.0
Foot/ankle	7	7	100.0
Multiple	34	34	100.0
Unknown	6	7	85.7

^aPROMIS, Patient-Reported Outcomes Measurement Information System.

Figure 1.

Compliance by age quartile. *Statistically significant difference compared with quartiles 1 to 3 (P < .05).

Reasons for noncompliance were documented in a sample test range spanning 14 weeks, during which there were 658 patient encounters. In 622 of these encounters, the patient completed the entire PROMIS survey set (Figure 2). Seventeen patients (of the 36 who were noncompliant, or 47%) verbally declined participation for a variety of subjective reasons (eg, lack of interest, unfamiliarity with electronic tablets, or not having appropriate reading glasses). Twelve of 36 patients (33%) did not have time to complete the survey set prior to the consultation with the senior author, while 5 patients (14%) partially completed the forms, and 2 patients (6%) cited advanced age as being a factor for not completing the PROMIS questionnaires.

Figure 2.

Reasons for patient noncompliance in a sample of clinical encounters. PROMIS, Patient-Reported Outcomes Measurement Information System.

The cost of establishing a PROMIS-based registry using our project design and workflow was estimated at $2045, whereas the monthly maintenance cost was $1000. The estimated costs of the research assistant included 20 hours of start-up costs for building and testing the registry ($250) as well as the monthly maintenance cost of supervising PROMIS administration during 2 days of clinic each week ($1000). An itemized breakdown of costs is described in Table 4.

TABLE 4.

Costs of Creating and Maintaining a PROMIS-Based Registry^a

	Initial Start-up Cost, US$	Monthly Maintenance Cost, US$
Institutional REDCap server fee	300	—
Electronic tablets (n = 4)	1315	—
Tablet cases (n = 4)	180	—
Research assistant salaryb ($12.50/h)	250	1000
Total cost	2045	1000

^aPROMIS, Patient-Reported Outcomes Measurement Information System.

^bResearch assistant is a part-time employee and does not receive benefits.

Discussion

In this study, we found that patients were over 90% compliant in responding to PROMIS CAT questionnaires during routine collection in the ambulatory sports medicine clinic. However, older patients were significantly less likely to complete PROMIS questionnaires when compared with younger patients, and patients with shoulder injuries were less likely to complete forms than those with hip injuries. This protocol for the routine collection of PROMIS scores is both practical and efficient, placing only a small financial burden on the provider and minimizing the time and effort required from patients for survey completion. Higher cost savings could be projected if utilizing a similar protocol for PRO collection with multiple providers (only 1 provider was utilized in this study).

Our primary finding is that a series of PROMIS questionnaires can be completed in an ambulatory orthopaedic clinic with high rates of patient compliance. In our registry, 3 different PROMIS CAT domains were recorded: physical function, pain interference, and depression. Therefore, the provider can collect a valuable amount of patient-centric information in an extremely effective manner. The total survey time for the PRO collection of all 3 domains averaged 2.6 minutes across all patients.

Our system of having a research assistant administer surveys on electronic tablets to patients in the waiting area before going back to the examination room yielded an overall compliance rate of 91.3%. This finding is comparable with other studies in the literature that have evaluated electronic PRO completion rates in the clinical setting. For example, a study by Slover et al²⁷ examined the in-clinic response rate for the 5-item EuroQol and the Knee injury and Osteoarthritis Outcome Score (KOOS) using a similar methodology and calculated an overall compliance rate between 93% and 95%. Another recent study assessed the collection of traditional shoulder PROs from surgical patients using a combination of email notifications with in-clinic data collection and found a compliance rate of 94%, although the response rate decreased significantly throughout postoperative follow-up.²² These studies show that the routine collection of PROMIS scores exhibits similarly high rates of compliance when compared with the routine electronic collection of legacy PROs.

It is also important to review the response rates of electronic PRO administration when compared with the response rates of traditional pencil-and-paper administration in the literature. Of particular note, a study by Schamber et al²⁶ directly compared the response rates for a single surgical registry involving traditional hip and knee PROs and found that electronic PRO collection resulted in a significantly higher completion rate (100%) when compared with paper collection (50.6%).

The PROMIS completion rates in our study were high, regardless of the clinic visit type (including returning and postoperative patients). This indicates that there was no significant dissatisfaction for returning or postoperative patients that caused them to refuse the PROMIS surveys. However, it is possible that there may be “survey fatigue” that occurs if PROMIS scores are obtained at multiple times during the postoperative course.

We are not sure why there was a discrepancy in compliance based on the anatomic location of injury. However, there is a reasonable explanation for the differences in compliance rates across age groups. It is possible that the decreased response rates for older patients is due to an unfamiliarity with using electronic tablets; this can in turn lead to a longer time to completion.^8,21 A longer time to completion would increase the likelihood for patients to be interrupted by clinical staff and taken to the examination room before finishing the PROMIS forms, a common complaint among our noncompliant patients.

The value in PROMIS CAT forms derives from the ability to obtain a large amount of information without overburdening either the patient or the provider. The full PROMIS survey set, which includes scores for a patient’s physical function, pain, and emotional health, required an average of only 15.3 questions and 2.6 minutes to complete. This is significantly less than traditional legacy PRO measures that attempt to capture a similar amount of information.^{4,7,16–19,29} By keeping the number of questions and time to completion at a minimum, it is possible to mitigate the effects of survey fatigue and prevent delays in the clinic.^11,13

We are not sure why PROMIS D required, on average, about 2 more questions than the PROMIS PF, UE, or PI questionnaires while also being completed in less time. With regard to the differences in the time to completion per question, it is possible that patients spend more time responding to questions relevant to their specific health concerns in an effort to most accurately characterize their primary ailments (in general, our orthopaedic population is skewed toward limitations in physical function and pain and less so for emotional health).

The financial burden of using this system of routine PROMIS administration is relatively low compared with other outcome collection systems on the market. Even without an existing REDCap collaborator, previous estimates for instituting REDCap from scratch have been shown to be less than $9000, although this cost is significantly mitigated depending on the current infrastructure and information technology capabilities of the host institution.²⁰ With an existing REDCap collaborator at the host institution, the initial start-up costs included 1-time fees that totaled $2045, whereas the monthly maintenance costs for high-volume outcome collection were almost entirely dependent on the salary of a research assistant or coordinator. Furthermore, because these resources can be shared among multiple providers, expanding this data collection system at a department level would result in diminishing costs per provider. With value-based reimbursement models becoming increasingly popular in modern health care, the potential increase in revenue is likely to offset the minimal financial burden required for this collection system.

This study has several limitations. We only included the experience of a single provider, and the research would benefit from a larger, more diverse group of providers from different specialties in orthopaedics employing the same collection techniques. Our population does not include a large percentage of Medicaid or indigent patients who may have lower response rates. Non-English speakers may also have lower response rates because our questionnaires were only available in English. Although we kept a record of which patients declined to participate in PROMIS data collection, the actual reasons for declining or not completing the PROMIS forms were not always clear. Thus, a more thorough investigation into the causes for noncompliance could result in a better understanding for areas of improvement in routine PROMIS collection. Finally, although minimizing the time to completion theoretically results in fewer clinic delays, it is extremely difficult to actually quantify this benefit in a clinical practice.

Conclusion

The routine electronic collection of PROMIS scores in an ambulatory orthopaedic clinic resulted in a compliance rate of over 90%, although older patients were generally less compliant than younger patients. Our system of data collection is practical and efficient in a high-volume orthopaedic clinic and places a minimal financial burden on the provider.