Collection and Utilization of Patient-Reported Outcome Measures in a Colorectal Surgery Clinic

Abstract

Introduction:

The routine collection of patient-reported outcome measures (PROMs) promises to improve patient care. However, in colorectal surgery, PROMs are uncommonly collected outside of clinical research studies and rarely used in clinical care. We designed and implemented a quality improvement project with the goals of routinely collecting PROMs and increasing the frequency that PROMs are utilized by colorectal surgeons in clinical practice.

Methods:

This mixed-methods, quality improvement project was conducted in the colorectal surgery clinic of a tertiary academic medical center. Patients were administered up to five PROMs before each appointment. PROM completion rates were measured. Additionally, we performed two educational interventions to increase utilization of our electronic health record’s PROM dashboard by colorectal surgeons. Utilization rates and attitudes toward the PROM dashboard were measured.

Results:

Overall, patients completed 3600 of 3977 (90.9%) administered PROMs during the study period. At baseline, colorectal surgeons reviewed 6.7% of completed PROMs. After two educational interventions, this increased to 39.3% (P = 0.004). Colorectal surgeons also felt that the PROM dashboard was easier to use. Barriers to greater PROM dashboard utilization included poor user interface/user experience and a perceived lack of knowledge, time, and relevance.

Conclusions:

The collection of PROMs in colorectal surgery clinics is feasible and can result in high PROM completion rates. Educational interventions can improve the utilization of PROMs by colorectal surgeons in clinical practice. Our experience collecting PROMs through this quality improvement initiative can serve as a template for other colorectal surgery clinics interested in collecting and utilizing data from PROMs.

Introduction

Patient-reported outcome measures (PROMs) are validated tools that are used to assess the patient’s perspective of their health, function, and quality of life.¹ This perspective can provide a more well-rounded and complete evaluation of clinical treatments and outcomes, improving both patient-provider communication and patient satisfaction.^2–7 Moreover, the use of PROMs can help physicians in counseling patients about anticipated postoperative outcomes, including those that are patient reported. Therefore, clinicians and researchers are becoming increasingly interested in incorporating the collection and utilization of PROMs into routine clinical practice.^8–12

There are various technical and logistical barriers in the way of this goal. These barriers include lack of patient/provider time, poor information technology infrastructure for PROM collection/interpretation, difficulty using electronic devices, and uncertainty about how to best integrate PROMs into the clinical workflow.^13,14 To overcome these barriers, guidelines have been published suggesting best practices for the routine collection of PROMs in a generic clinical environment.¹⁵ Among other recommendations, these guidelines support electronically collecting only validated PROMs. Guidelines published in the field of orthopedic surgery also suggest collecting PROMs that are commonly used within the specialty and integrating the results into the electronic health record (EHR).^16,17 Although a few (primarily orthopedic) groups have attempted to implement routine PROM collection and utilization into their specific clinical environments, we identified none that have done so in the field of colorectal surgery.^18–23

We conducted this quality improvement (QI) project with the goal of routinely collecting and utilizing PROMs in the colorectal surgery clinic of a tertiary academic medical center. We hope that our experience implementing it can serve as a template for other colorectal and general surgery clinics interested in collecting and utilizing PROMs in their own practice.

Materials and Methods

Institutional review board

The authors submitted an application to the Vanderbilt University Medical Center (VUMC) Institutional Review Board (IRB). Upon review, the VUMC IRB determined this work to be a QI project and subsequently withdrew our IRB application. Being a QI project, this work required neither IRB review nor approval to perform.

Study setting

This QI project was conducted in the VUMC colorectal surgery clinic. The VUMC colorectal surgery clinic functions as the main colorectal surgery clinic of a tertiary academic medical center. The clinic has a staff of seven attending colorectal surgeons, who together treat the full spectrum of diseases of the colon, rectum, and anus. PROM collection in the VUMC colorectal surgery clinic was implemented under the auspices of the Vanderbilt Patient-Reported Outcomes Measurement System (VPROMS). VPROMS is an enterprise-level initiative, begun by VUMC leadership in 2018, to integrate PROMs into routine clinical practice across the academic health center as part of VUMC’s ongoing focus on personalized medicine.²⁴

Stage 1: routine collection of patient-reported outcome measures

A multidisciplinary group of stakeholders including clinicians, nurse practitioners, nurses, and researchers with experience in PROMs was convened to select a parsimonious group of measures. Although there was no formal qualitative system for PROM selection, the group prioritized the selection of PROMs that had strong psychometric properties, were commonly used in both research and in the clinical setting, and represented the least amount of burden to the patients. A wide variety of PROMs were considered and the following five were chosen by consensus: the Patient-Reported Outcome Measurement Information System (PROMIS) Global-10, Quality of Recovery (QoR)-9, the European Organization for Research and Treatment of Cancer Quality of Life in Colorectal Cancer (EORTC-QLQ-CR29), the Short Inflammatory Bowel Disease Questionnaire (SIBDQ), and the Diverticulitis Quality of Life (DV-QOL) instruments (Table 1).^25–31 These PROMs were built into VUMC’s EHR and online patient portal (My Health at Vanderbilt, MHAV) by VPROMS.

Table 1 –

Descriptions of patient-reported outcome measures we routinely collected in the VUMC colorectal surgery clinic.

Patient-reported outcome measure	Measurement	Number of questions	Output	Target patients
PROMIS Global-10	Patient’s perception of their overall health; overall quality of life	10	Three health status scores: a physical health score, a mental health score, and an estimated EQ-5D score	Every patient was asked to complete the PROMIS Global-10 before their appointment
QoR-9	Recovery after anesthesia	9	Scored from 0 to 18, with a higher score indicating a better recovery	All patients within 6 wk of a colorectal surgery were asked to complete the QoR-9 before their appointment
SIBDQ	Health-related quality of life in IBD	10	Scored from 1 to 7, with a higher score indicating a higher health-related quality of life	All patients with a diagnosis of IBD (Crohn’s or UC) were asked to complete the SIBDQ before their appointment
DV-QOL	Health-related quality of life in symptomatic uncomplicated diverticular disease	17	Four outputs measuring health-related quality of life: diverticulitis concerns, emotions, behavior changes, and overall	All patients with a diagnosis of diverticulitis were asked to complete the DV-QOL before their appointment
EORTC-QLQ-CR29	Health-related quality of life in colorectal cancer	29	Five functional and 18 symptom scales. Scores for each are reported from 0 to 100 with higher scores representing better function on the functional scales and worse symptoms on the symptom scales	All patients with a diagnosis of colorectal cancer were asked to complete the EORTC-QLQ-CR29 before their appointment

IBD = inflammatory bowel disease; UC = ulcerative colitis.

Stage 1: patient-reported outcome measure collection and completion rates

Patients were asked to fill out their assigned PROMs before being seen by their attending colorectal surgeon. If the patient had an account with MHAV before their appointment, MHAV would ask them to fill out their assigned PROMs while completing the rest of their preappointment paperwork. If the patient did not have an MHAV account or failed to fill out their assigned PROMs in MHAV before arriving at clinic, they would be asked to fill out their assigned PROMs upon checking into the clinic. The receptionist would load the assigned PROM(s) on one of 10 electronic tablets that they would then hand to the patient to complete. PROMs were loaded by scanning a quick response code in the receptionist’s patient check-in workflow. The patient could complete their assigned PROM(s) on the tablet while in the waiting room or examination room before being seen by their attending surgeon.

PROMs were implemented in the VUMC colorectal surgery clinic in April 2021, and the PROM completion rate was tracked through VUMC’s EHR through August 2021. Results were analyzed using Tableau, in a dashboard synced to the EHR.

Stage 2: increasing clinical utilization of patient-reported outcome measures

Attending colorectal surgeons were able to access the results of a patient’s PROM(s) through a separate dashboard nested in VUMC’s EHR (the PROM dashboard). The PROM dashboard displays each PROM in table format. In addition to the patient’s score on each PROM, the PROM dashboard also displays the patient’s responses to each question of each completed PROM. The PROM dashboard allows the attending surgeon to display a patient’s results graphically over time.

We created a survey to assess the frequency with which attending colorectal surgeons utilized the PROM dashboard in the course of patient care, as well as their attitudes toward using the PROM dashboard (Supplementary Appendix A). In addition to recording PROM dashboard utilization rates, our survey included five user experience surveys that are standard in the health technology sector, to assess surgeon attitudes toward the PROM dashboard on a one-to-five scale in the following domains: ease of use, usefulness, ease of finding relevant information, the look and feel of the PROM dashboard, and the ease of accomplishing one’s goal when using the PROM dashboard.

In this second stage of this QI project, we conducted two Plan Do Study Act (PDSA) cycles from October to November 2021, with the primary aim of increasing the frequency that attending physicians review the PROM dashboard during colorectal surgery clinic visits to over 30% of visits (Fig. 1).^14,32 We first obtained baseline measurements of PROM dashboard utilization during patient care encounters and physician attitudes toward the PROM dashboard (phase 0). After reviewing baseline usage and attitudes, two authors (N.J.H. and A.T.H.) designed and gave a 10-min presentation during the monthly colorectal surgery section meeting on the clinical value of PROMs, as well as how to access and use the PROM dashboard (intervention 1). We then administered the same survey to the colorectal surgeons to assess changes in PROM dashboard utilization rates following the first intervention (phase 1). This concluded our first PDSA cycle.

Fig. 1 –

Schematic that details the overall design of our quality improvement project to increase PROM dashboard utilization rates among attending colorectal surgeons.

After reviewing PROM dashboard usage and surgeon attitudes in phase 1, we then designed and performed a second intervention, in which one of the study authors (N.J.H.), held a 10-min one-on-one training session with each surgeon (intervention 2). During the session, the surgeon practiced accessing the PROM dashboard under supervision and had the opportunity to ask any questions they had about accessing the PROM dashboard, utilizing the PROM dashboard in their clinical practice, or the value of PROMs or the PROM dashboard. We then administered the survey a third time to obtain a final set of utilization rates and surgeon attitudes regarding using the PROM dashboard (phase 2). This concluded our second PDSA cycle (Fig. 1).

Data analysis

Fisher’s exact test was performed to assess for changes in PROM dashboard utilization rates between phase 0 and phase 2. The Mann-Whitney U-test was used to test for differences in surgeon attitudes toward the PROM dashboard between phases of the study. Results were considered significant if the P-value was <0.05. GraphPad Prism 9 software was used to perform all statistical analyses, as well as generate all graphs.

Qualitative analysis

A series of open-ended questions were asked in the survey to further identify barriers to PROM implementation in the clinical setting. Answers were analyzed thematically using a constant comparative approach.³³ Coding for major themes was conducted independently by investigators (N.J.H. and A.T.H.) in Microsoft Excel and discussed to consensus iteratively using thematic analysis.³⁴

Results

Patient-reported outcome measure completion rates

Over 5 mo of data collection, 1827 patients, comprising 2682 encounters, were seen in the VUMC colorectal surgery clinic (Table 2). In 2650 encounters (99.8%), the patient was asked to complete one or more PROMs. In 2433 encounters (90.7%), the patient completed at least 1 PROM (Table 2).

Table 2 –

Table describing PROM completion in the VUMC colorectal surgery clinic by patients and encounters.

Totals
Distinct patient count	1827
Distinct patients with completed PROMs	1740
Distinct encounter count	2682
Encounters with PROMs distinct count	2650
% Encounters with PROMs	99.8%
Encounters with completed PROMs distinct count	2433
% Encounters with completed PROMs	90.7%

Overall, 3977 PROM assessments were distributed to patients; 3600 (90.5%) were completed (Table 3). Of these, 1702 PROMs (47.3%) were completed via MHAV while 1898 PROMs (52.7%) were completed on a tablet after checking in at the VUMC colorectal surgery clinic. The overall PROM completion rate decreased slightly over the 5 mo of the study (Fig. 2). The PROM with the highest completion rate was the PROMIS Global-10, with 91.7% of assessments completed. Completion rates of the QoR-9, EORTC-QLQ-CR29, SIBDQ, and DV-QOL PROMs ranged from 86.3% to 89.5% (Table 3).

Table 3 –

Table describing the PROM completion rate in the VUMC colorectal surgery clinic for each PROM and overall.

PROM	Total administered	Total completed	% Completed
PROMIS Global-10	2650	2430	91.7
SIBDQ	419	375	89.5
QoR-9	629	553	87.9
DV-QOL	111	97	87.4
EORTC-QLQ-CR29	168	145	86.3
Overall	3977	3600	90.5

Fig. 2 –

Run chart displaying PROM completion rates in the colorectal surgery clinic by month. Month 1 = April 2021; Month 5 = August 2021.

Patient-reported outcome measure dashboard utilization by colorectal surgeons

At baseline, attending surgeons in the colorectal surgery clinic reviewed the PROMs of 6.7% of patients in the course of routine patient care (Fig. 3A). Following a 10-min presentation on how to access and use the PROM dashboard (intervention 1), this utilization rate increased to 36.4%. After the second intervention, a one-on-one educational session, PROM dashboard utilization by colorectal surgeons again increased, to 39.3% (P = 0.004) (Fig. 3A).

Fig. 3 –

Graphs detailing PROM dashboard utilization rates by colorectal surgeons (A) and their attitudes toward the PROM dashboard (B-F). A value of five represents the most positive attitude in a particular domain, while a value of one represents the most negative attitude in a particular domain. A value of three represents a neutral attitude. Graphs B-F display surgeon attitudes about the PROM dashboard’s ease of use (B), usefulness (C), ease of finding information (D), and look and feel (E) from baseline to phase 2. Finally, the perceived ability of colorectal surgeons to accomplish their goal after navigating to the PROM dashboard is shown (F). Horizontal bars in graphs B-F represent the median. *P < 0.05. **P < 0.01.

Attending surgeons were asked, “What are some reasons why you didn’t review a patient’s new PROM scores?” Prior to the first intervention, the most commonly cited factor was a perceived lack of knowledge that the PROM dashboard existed as a resource. Following the first intervention, the most commonly cited factor was a perceived lack of relevance to the patient visit.

“Not needed. Patient well known to me.”

Attending surgeons were also asked, “What are some reasons why you didn’t modify a patient’s care plan based on their PROM scores?” After completion of our two PDSA cycles, the most commonly cited factor was a belief that the PROM dashboard did not provide information that would contribute to decision-making. Some surgeons also cited a lack of comfort with the PROM dashboard, having only recently been introduced to it.

“Light clinic, no large care plan changes made today.”

Physician attitudes toward the patient-reported outcome measure dashboard

After two educational interventions, surgeons felt that it was easier to find what they were looking for in the PROM dashboard (Fig. 3D). However, there was no significant change in surgeons’ perceptions of the PROM dashboard’s ease of use, usefulness, the look and feel of the PROM dashboard, or their ability to accomplish their goal after navigating to the PROM dashboard (Fig. 3B, C, E, and F).

Finally, surgeons were asked, “What would make you use the PROM dashboard more frequently?” After both our educational interventions, the most common respsonse was the ability to pull the data from the PROM dashboard into patient notes using smart text, such as a dot phrase. Surgeons also stated that improving the PROM dashboard’s user interface/user experience (i.e., the graphical layout of the PROM dashboard and how the colorectal surgeon interacts with it) would increase utilization.

“Ability to pull data from the PROM [dashboard] into the note using smart text.”

Discussion

We performed a two-stage QI project in which we first successfully systematized the collection of PROMs from patients in an academic colorectal surgery clinic, maintaining a >90% completion rate across the 5-mo study period. In the second stage, we performed two PDSA cycles that significantly increased rates of PROM utilization by colorectal surgeons. We also assessed surgeons’ attitudes about using PROMs during patient encounters in clinic.

Best practices for patient-reported outcome measure collection in colorectal surgery

The routine collection of PROMs promises to inform clinical practice, enhance patient management, improve the value of delivered care, monitor health outcomes over time, and evaluate health system performance.³⁵ Within the field of colorectal surgery, PROMs have been widely used in surgical outcomes research over the last decade.³⁶ Yet, the routine collection of PROMs remains limited, despite calls to normalize it.^8–12 To the best of our knowledge, we are the first to describe the implementation of a program to routinely collect PROMs in a colorectal surgery clinic.

We achieved a high PROM completion rate by effectively applying guidelines for the electronic collection of PROMs to our colorectal surgery clinic and, when possible, minimizing barriers to PROM completion. Snyder and Wu previously outlined 3 phases and 11 key questions to help guide the effective collection of PROMs.^15,37 Using these questions, previous literature, and the results of our QI project as a guide, we have summarized what we view as best practices for PROM collection in colorectal surgery clinics in Table 4^14,38–41

Table 4 –

Suggested best practices for PROM collection in colorectal surgery clinics using Snyder and Wu’s questions as a I framework.^15,37

Question	Suggested best practice in colorectal surgery
Planning phase
1. What strategy will be used for integrating PROs in EHRs?	Whenever possible, PROMs and PROs should be fully integrated into the EHR. This allows for PROM collection through both the patient portal and EHR. Moreover, it facilitates real-time review by surgeons and the ability to integrate PROMs into clinical care
2. How will the PRO-EHR system be governed?	The specifics of who in the organization has control of routine PROM collection and integration with the EHR is a matter that will often be decided locally. We pursued a hybrid model of PRO-EHR system governance, where a committee comprising clinicians, nurse practitioners, nurses, and researchers with experience in PROMs selected desired PROMs and then oversaw implementation in collaboration with an institutional project manager. Regardless of operational control, surgeons should ensure that the necessary PROMs to track global patient health, as well as health in common colorectal diseases, are recorded in their PRO-EHR system
3. What are the ethical and legal issues?	Patient privacy in PROM collection is paramount. Collection of PROMs should be done in the EHR and through an HIPPA-compliant patient portal. We informed patients about their PROM surveys, going into detail when asked, but did not specifically provide opt-out information. We believe this approach improved our response rate
4. How can PRO data from multiple EHRs be pooled?	Our institution functions under a single EHR. Therefore, we do not have best practice recommendations for this question
Selection phase
5. Which populations and patients are most suitable for the collection and use of PRO data, and how can EHRs support the identification of suitable patients?	The vast majority of patients in the colorectal surgery clinic are suitable for the collection and use of PRO data. The EHR should be lever-aged to automatically identify postoperative and disease status, administering or withholding surgery-specific or disease-specific PROMs accordingly There are important limitations to PROM collection surrounding spoken language, disability, cognitive decline, and unfamiliarity with electronics that are present in the colorectal surgery clinic population. Such factors may result in inequitable PROM completion rates. Efforts, such as choosing PROMs available in multiple languages, should be made to reduce these disparities
6. Which outcomes are important to measure for a given population?	A mixture of generic and disease-specific PROMs should be collected from patients in the colorectal surgery clinic. We recommend collecting, at a minimum the PROMIS Global-10, QoR-9, EORTC-QLQ-CR29, SIBDQ, and DV-QOL instruments (see Discussion). If a particular clinic focuses on a less common colorectal pathology, then that clinic should consider collecting a validated, disease-specific PROM for patients with that condition. It is important to standardize the selection of PROMs across colorectal surgery clinics to improve the comparability of research and clinical performance
7. How should candidate PRO measures be evaluated?	All PROMs collected should be validated and commonly used in the colorectal surgery literature
Engagement phase
8. How, where, and with what frequency will PROs be administered?	We strongly recommend that PROMs be routinely administered to each colorectal surgery clinic patient before every appointment. This provides several advantages, including the ability to track disease states and severities across time, perform high-powered prospective and retrospective outcomes research, and monitor the quality of care that is provided. The specific PROMs collected at each appointment should vary depending on the patient’s disease state and postoperative status
9. How will PRO data be displayed and accessed in the EHR?	PRO data should be accessed on a user-friendly dashboard within the EHR and end-user testing should be performed before operationalizing the dashboard. Overall scores should be readily visualized, and individual responses to questions should be available by clicking on the PROM. Responses should graphically display the directionality and meaning of PRO scores for easy understanding of change over time. If possible, dot phrases or other code should be available that readily pulls PRO data into the provider’s note for easy access
10. How will PRO data be acted upon?	Routinely collected PRO data in colorectal surgery clinics can be utilized for outcomes research, quality improvement, and clinical care. Further research is critical to enhance our understanding of how to apply PROMs to improve clinical practice
11. How can users be trained and engaged?	PROMs can be collected electronically on tablets following arrival at the colorectal surgery clinic. The receptionist should be able to load the PROM(s) on a tablet with minimal disruption to their workflow. If the clinic has a patient portal linked to their EHR, such as MyChart or MyHealth, patients should have the opportunity to complete their PROMs through the portal in the days leading up to their appointment

PRO = patient-reported outcome; HIPAA = Health Insurance Portability and Accountability Act.

Some have suggested, to improve PROM completion rates, that clinicians and office staff should emphasize the importance of PROM collection to patients.^37,40,42 We disagree that this is necessary, and feel that it may instead limit PROM completion by patients because it conveys a feeling of optionality. In our clinic, clinic staff asked patients to complete the PROM questionnaires the same way they would ask patients to complete the rest of their standard preappointment paperwork. If we are to routinely collect PROMs, we suggest that we should view the process as a standard, expected action by the patient before being seen by their provider, the same way we would treat the collection of any other preappointment questionnaire.

A common challenge when standardizing the collection of PROMs is choosing which PROMs to use. This challenge is particularly germane to colorectal surgery as, in the last 15 y, at least 150 unique PROMs have been utilized in published colorectal research.³⁶ The heterogeneity of PROMs in colorectal surgery limits the meaningful comparison of study results and the ability to compare clinical performance across sites.¹¹ With this in mind, we chose to routinely collect five of the most common PROMs utilized in colorectal research: the PROMIS Global-10, QoR-9, EORTC-QLQ-CR29, SIBDQ, and DV-QOL instruments. This group contains a mixture of generic PROMs and colorectal disease-specific PROMs. We chose disease-specific PROMs for colorectal cancer, inflammatory bowel disease, and diverticulitis because of the high frequency with which these diseases are seen in colorectal surgery clinics. We limited the number of PROMs routinely collected to five, to prevent patients from developing survey fatigue. We encourage other colorectal clinics planning to routinely collect PROMs to model this approach, to improve standardization and the comparability of PROMs within the specialty.

Finally, to implement these best practices in the colorectal surgery clinic, it is essential that a physician champion defines goals and objectives, drives the process forward, and represents the interests of surgeons within the health system.^{22,38,39,43,44} In our implementation program, one of the authors (A.T.H.) served as the physician champion. He worked with our health system’s administration to define goals and priorities, choose the particular PROMs that would be collected, and orient clinical staff to the PROM collection process. Physician champions can also generate enthusiasm for PROM collection and utilization among their partners and provide suggestions for how to use PROMs to enhance colorectal surgery practice.

Clinical use of patient-reported outcome measures by colorectal surgeons

Our work is also the first to assess the rates with which colorectal surgeons utilize PROMs in clinical practice. We found that, prior to educational interventions, colorectal surgeons rarely used our EHR’s PROM dashboard in clinical care.⁴⁵ However, our two educational interventions successfully raised the PROM utilization rate by surgeons in our clinic nearly six-fold, from 6.7% to 39.3%. Our results suggest that targeted interventions aimed at teaching colorectal surgeons about how and where to access PROMs in the EHR can significantly increase PROM utilization by surgeons in clinical practice.

Only one group before ours has documented the attitudes and behaviors of colorectal surgeons with respect to the clinical utilization of PROMs.⁴⁶ As in their study, colorectal surgeons in our clinical setting were open to the idea of using PROMs in their practice but felt limited by a lack of knowledge about their existence within the EHR and how to use them in clinical care. Many also expressed concern about potential disruptions to their clinical workflow. Following our educational interventions, surgeons said that they would use PROMs more often but again felt limited by time, EHR design, and a feeling that certain PROMs were unnecessary for patient care, particularly among well-known patients. These responses indicate that the collection of PROMs and their use in clinical practice pose separate challenges.

Assessing surgeon attitudes toward the PROM dashboard over time allowed us to measure changes in attitude following our educational interventions. This allowed us to assess the effectiveness of our interventions over multiple domains. Moreover, it demonstrated to us areas of strength and weakness in the user interface/user experience of the PROM dashboard, providing future directions for our QI work.

The largest barrier to PROM utilization in clinical practice is a lack of knowledge about how to effectively use them to improve patient care.^13,46,47 Although some recommendations exist for generic PROM interpretation and application in clinical practice, we only found two papers (both focused on colorectal cancer) that discussed how to specifically apply PROMs to patient care in colorectal surgery.^46,48–52 As the routine collection of PROMs becomes more common in the field of colorectal surgery, further research will be crucial to inform the use of PROMs in clinical decision-making and patient care across the full breadth of colorectal pathology.

Prior studies have consistently indicated that the manner in which PROMs are integrated into the EHR can facilitate or hinder clinical utilization.^1,47,53 Our results support this hypothesis. Although colorectal surgeons reported that the PROM dashboard became easier to use following educational interventions, they continued to describe its user interface/user experience as a barrier to more frequent use. This is perhaps unsurprising because our PROM dashboard did not convey PROM score directionality (whether higher scores are better/worse) or meaning, which are considered best practices for PROM dashboard design.⁵⁴ Moreover, surgeons in our study expressed frustration that summary scores were often buried among responses to individual PROM questions and that the PROMs in the PROM dashboard could not be pulled into their clinic notes using a smart phrase.

Ultimately, if an EHR dashboard is not intuitive to use, surgeons will not use it. Designers and project managers need to be aware that when building for surgeons, they are building for a user that has multiple competing demands for their attention, is already battling click fatigue, and whose clinical workflow is moderately habitual. In our work, performing usability testing with colorectal surgeons as end-users before building the PROM dashboard could have identified some of the aforementioned feature requests earlier, which would have allowed us to incorporate them into the initial build.⁵⁵ This would have likely increased PROM dashboard utilization in our clinic. We are currently working with VPROMS to incorporate these feature requests into our PROM dashboard.

Limitations

After implementation, we monitored the PROM completion rate for 5 mo, suggesting that the best practices we followed were sustainable. Still, it is possible that our PROM completion rate will decrease over time. Similarly, the second stage of our study occurred over 2 mo. Therefore, we cannot know whether the observed increase in PROM utilization among colorectal surgeons following our two PDSA cycles will be sustained.

As a colorectal surgery clinic at a tertiary academic medical center, the ability to implement best practices for the routine collection of PROMs in our clinic may differ from that of other practice settings. For example, group or individual practices that are not affiliated with larger health systems may not have an institutional project manager or information technology department that can help implement ideal PROM collection practices. Although we made our best practice recommendations (Table 4) for the routine collection of PROMs with these limitations in mind, we recognize that practical constraints may prevent some colorectal clinics from achieving best practices. In such cases, we feel that priority should be given to collecting commonly used PROMs, like the ones collected in our clinic, to ensure comparability. This will allow less-resourced practices to take advantage of the coming literature focused on how to use PROMs to improve the quality of care provided to patients.

A final limitation of our study is that we did not have a control group in the second stage of our implementation program, focused on improving the PROM utilization rate among colorectal surgeons. However, the small number of surgeons in our practice would have limited the power of such a controlled design. Although the increase in PROM utilization following our educational interventions is correlational, our results still suggest that they effectively increased PROM utilization.

Conclusions

By applying best-practice guidelines for the routine collection of PROMs in our colorectal surgery clinic, we successfully implemented a QI initiative that routinely collects PROMs, maintaining a high completion rate. From this experience, we suggest best practices for the routine collection of PROMs in colorectal surgery clinics (Table 4). Despite our high PROM completion rate, however, colorectal surgeons did not review PROMs until after educational interventions were performed. This stresses the need for physician champions to lead educational efforts when implementing PROM collection programs, to promote their clinical usage. Ultimately, routine PROM collection in colorectal surgery clinics is feasible, but further investigation is necessary to understand how PROMs can be effectively utilized in routine patient care.