Defining a Research Agenda for Patient-Reported Outcomes in Surgery: A Delphi Survey of Stakeholders

Abstract

Background

Identifying timely and important research questions using relevant patient-reported outcomes (PROs) in surgery remains paramount in the current medical climate. The inaugural Patient-Reported Outcomes in Surgery Conference brought together stakeholders in PROs research in surgery, with the aim of creating a research agenda to help determine future directions and advance cross-disciplinary collaboration.

Study Design

An iterative Web-based interface was used to create a modified Delphi survey. Participation was limited to conference registrants, which included surgeons, PROs researchers, payers, and other stakeholders. In the first round, research items were generated from qualitative review of responses to open-ended prompts. In the second round, items were ranked using a 5-point Likert scale; attendees were also asked to submit any new items. In the final round, the top 30 items and newly submitted items were redistributed for final ranking using a 3-point Likert scale. The top 20 items by mean rating were selected for the research agenda.

Results

In round one, participants submitted 459 items, which were reduced to 53 distinct items within seven themes of PROs research. A research agenda was formulated after two successive rounds of ranking. The research agenda identified three themes important for future PROs research in surgery: (1) PROs in the decision-making process, (2) integrating PROs into the EHR and, (3) measuring quality in surgery with PROs.

Conclusions

The PROS Conference research agenda was created using a modified Delphi survey of stakeholders that will help researchers, surgeons, and funders identify crucial areas of future PROs research in surgery.

Patient-reported outcomes (PROs), as defined by the FDA, are “any report of the status of a patient's health condition that comes directly from the patient, without interpretation of the patient's response by a clinician or anyone else.”¹ In practice, this entails measuring a subjective outcome such as function after knee arthroplasty with a questionnaire. Publications on PROs in the surgical literature have tripled during the last decade. The growing recognition among surgeons that many postoperative outcomes, such as functional improvement or symptom severity, are best measured by the patient and the increased availability of measures for historically subjective topics have undoubtedly contributed to this surge in publications.^2–4 PROs now play an important role in the planning of comparative effectiveness research and are currently used as primary outcomes in surgical trials.⁵ The Institute of Medicine and the National Quality Forum have long advocated that patient-centered care should be a requirement in modern health care, and this perspective has influenced health care deliberations during the last decade.⁶ Furthermore, the Affordable Care Act has encouraged patient-reported data collection in surgery by creating new payment models through the Centers for Medicaid and Medicare Services (CMS) and by targeted funding of research from the Patient-Centered Outcomes Research Institute.^7,8 These new models underscore the growing importance of PROs in surgery.

Despite the potential value of PROs in surgical care, many methodological and logistical concerns remain.⁹ Addressing these issues is difficult in a research climate of increasingly scarce funding.¹⁰ Thus, researchers and funding agencies need input and clarity from stakeholders in order to better prioritize research and improve collaboration between institutions. Furthermore, options for surgical techniques and technologies are rapidly expanding while health care resources are shrinking. To better understand the outcomes of surgery from the patient perspective, there is no better time than now for the surgical and research communities to come together and reach consensus regarding the most important and timely issues with PROs research in surgery.

To address these issues, the Patient-Reported Outcomes in Surgery Conference was formed, with sponsorship from The Plastic Surgery Foundation and the Agency for Healthcare Research and Quality. The conference brought together stakeholders from diverse fields, including payers, patient advocates, surgeons, researchers, industry representatives, regulators, and health information technology vendors. The two-day conference was held in Washington, D.C., (January 29-30, 2015) and included panel discussions on current PROs research in surgery within the areas of clinical care, comparative effectiveness, patient access, psychometric development, surgical trials, and quality, as well mapping out the future directions for each field. The specific goals of the conference were to improve the accessibility and interpretability of PROs data for patients and providers, to develop a consensus around methodological issues of PROs measurement, and to develop a research agenda for PROs measurement in surgery. The formation of a research agenda should prioritize research questions deemed to be timely and important by stakeholders to guide future collaboration and funding. To meet this aim, the conference leadership developed an agenda on future PROs research in surgery by use of a formalized group-consensus process.

METHODS

This study used a modified Delphi survey to achieve formal group consensus, maximizing dialogue through anonymous, structured feedback.^11,12 To facilitate the use of the Delphi model, an expert panel was assembled before the opening of the conference. The panel, composed of conference leaders and methodological experts, developed the study schema and consensus criteria on the basis of a three-round, Web-based survey (Figure 1). The study was approved by the Memorial Sloan Kettering Cancer Center Institutional Review Board (prospective exemption waiver X15-004).

Figure 1.

Flow chart for the three rounds of the modified Delphi survey

Emails containing a link to a Web-based survey were distributed to all conference registrants one week before the start of the conference (round one), which was held January 29–30, 2015. Participants were asked to anonymously identify timely and important PROs research topics through seven open-ended questions related to clinical care, comparative effectiveness, patient satisfaction, and quality metrics. There was no limit to the length of the responses. Item submissions were reviewed in parallel by two researchers (M.P. and W.C.). This involved separating compound responses into individual items, summarizing submissions into concise items, and categorizing items into generic themes. After all submissions were reviewed, identical items were deleted and similar items with overlapping content were consolidated into broader concepts. The expert panel then evaluated the consistency of each researcher's item reduction and endorsed the selection of research items for round two.

All conference attendees were invited to participate in a Web-based survey during the conference (round two). Participants were given the final research items from round one, in randomized order, and were asked to rank the items by research importance using a 5-point Likert scale, with high and low research priority as anchors. In addition, an open-ended question prompt at the conclusion of the survey allowed participants to submit additional research questions. Mirroring the process in round one, all new research questions submitted in round two were subjected to qualitative review and consolidation into final research items. Round two concluded with the completion of the conference.

Following the conference, all attendees were asked, via email, to participate in a Web-based survey, regardless of whether they participated in previous rounds. Email reminders to complete the survey were distributed weekly, and each registrant was limited to one survey response each. The top 30 items by mean priority score from round two, along with new item submissions, were distributed for final ranking by research priority, in order to reach the a priori goal of a 20 item research agenda. To reduce the possibility of a ceiling effect, a 3-point Likert scale was used in the final round and participants were encouraged to rate only 10 items as “high research priority.” As in round two, item order was randomized. The top 20 items by mean Likert score were selected as the consensus research agenda.

RESULTS

Of the 143 people registered for the conference, 137 provided valid email addresses. Potential subjects were invited to participate during each round. In the first round, 83 participants (61% of conference registrants with email addresses) submitted 356 responses to open-ended questions. Table 1 presents the demographic characteristics of respondents by round. After review, a total of 459 research items were submitted, with a mean of 5.5 items submitted per participant.

Table 1.

Characteristics of survey responders by round

	Round
Characteristic	One	Two	Three
Age
<31	6	9	4
31–40	43	30	37
41–50	24	26	29
51–60	14	19	20
≥61	12	17	7
Conference role
Speaker	18	24	26
Registrant	82	76	74
PROs experience
<1 year	7	9	11
1–3 years	33	26	33
4–6 years	22	17	17
7–10 years	12	9	13
>10 years	27	39	26
Total, no. (%)	83 (61)	53 (39)	57 (42)

Data are % of conference registrants with a valid email address (N=137). Percentage totals may not equal 100 due to rounding. PROs, patient-reported outcomes.

After item reduction was performed and the expert panel reviewed the results, there were 53 items within seven themes: clinical care, comparative effectiveness, data management, ethics, performance measurement, education, and other. In the second round, 53 participants (39% of emailed registrants) responded to email invitations during the conference. Responders ranked the 53 items by research priority and submitted 9 new items, which were reviewed and consolidated to three new items. In the third round, 57 participants (42% of emailed registrants) responded to survey invitations. The top 20 items from round three were selected as the consensus research agenda for future PROs research (Table 2). Ranking of item importance remained stable between rounds, with only two items from the top 20 of round two failing to make the final research agenda.

Table 2.

The final research agenda, from 459 items initially submitted

Rank	Mean	Item
1	2.52	Impact of PROs on patient and/or provider decision making.
2	2.49	Accuracy of measuring quality in surgery with PROs versus clinical quality metrics.
3	2.46	Efficacy of patient-reported performance measures to reduce costs and improve quality.
4	2.41	Improve PROs data collection, integration, and presentation into the EHR.
5	2.36	Impact of patient expectations on their satisfaction with surgery.
6	2.35	Optimize presentation of PROs data to providers for rapid interpretation and action.
6	2.35	Efficient integration of PROs data collection and reporting into the clinical workflow.
8	2.33	Determine optimal method for transitioning PROMs from research tools to performance measures.
9	2.29	Create systems that use PROs data to alert providers to patient needs and flag actionable items.
10	2.27	Influence of patient-reported data on patient satisfaction with decision-making.
11	2.23	Establish PROs benchmarks in surgical care.
12	2.22	Effect of preoperative education on patient satisfaction with surgery.
13	2.21	Improve utilization of PROs among nonacademic providers and institutions.
14	2.19	Influence of patient-reported data on patient satisfaction with clinical care.
14	2.19	Develop strategies for better patient engagement and improved response rates.
16	2.17	Accuracy of patient-reported data as a primary outcome in surgical trials.
17	2.15	Role of PROs data in patient education.
18	2.13	Explore if patient access to PROs data improves quality.
18	2.13	Risk adjust and standardize PROs data.
20	2.11	Identify barriers to successful implementation of PROs measures in clinical trials.

EHR, electronic health record; PROs, patient reported outcomes; PROMs, patient-reported outcome measures.

DISCUSSION

Using a modified Delphi approach, we engaged an international stakeholder group of surgeons, researchers, patient advocates, funding agency representatives, health information technology vendors, and regulators to reach consensus on future PROs research priorities in surgery. The final results of the survey contain the top 20 items from over 450 topics initially submitted by participants. Our results represent the first consensus-driven, surgery-focused PROs research agenda to date. Within the research agenda, three themes emerged as priorities for future research in surgery: (1) PROs in the surgical decision-making process, (2) challenges to integration of PROs in the electronic health record (EHR), and (3) PROs and the measurement of quality (Figure 2). Among all items, the decision-making process was rated of high importance, with “Impact of PROs on patient and/or provider decision-making” the top-ranked item for both rounds.

Figure 2.

Most important themes for PROs research in surgery identified from the Delphi survey

Priority 1: Incorporating PROs Data into the Decision-Making Process

It is not surprising that the use of PROs in the clinical decision-making process had the highest importance. Decision-making in surgery has traditionally relied on surgeon experience and established objective measures, such as 30-day mortality and hospital length of stay. Although currently underutilized, PROs have proven to be effective at measuring subjective outcomes after surgery.^13,14 In this respect, PROs data provide an added dimension to the evaluation of new surgical techniques and technology, which may enable surgeons to better understand subjective outcomes. For example, outcomes in randomized controlled trials comparing open and laparoscopic hernia repair techniques have previously focused on visual analogue scores of pain, hernia recurrence, complications, and operative time.^15,16 As the differences between emerging techniques become more nuanced (e.g., robotic vs. laparoscopic), traditional measures may be insensitive to improvements in patient disability and well-being, both of which are central to arguments for minimally invasive surgery. Additionally, the inclusion of patients in shared decision-making requires health care that aligns with patient preferences and values.¹⁷ Routine and accurate measurement of PROs in surgical trials and clinical care provides a valued outcome for surgeons and expands patient involvement in decision-making.

Improving decision-making in surgery will require more than just greater implementation of PROs in surgical trials and clinical care. This study identified problematic aspects within the theoretical framework for decision-making—specifically, the relationship between decision-making and patient expectations, education, and satisfaction. Understanding the effects of patient expectations on the decision-making process requires accurate measurement of expectations, as well as honest assessment of the ability to recalibrate these expectations through preoperative education.¹⁸ A randomized trial of hip and knee arthroplasties showed that preoperative education can influence patient expectations of postoperative recovery.¹⁹ Furthermore, Ho et al. reported that patient satisfaction with preoperative information was the strongest predictor of satisfaction with the overall outcome—stronger even than the method of surgery and whether complications occurred.²⁰ Yet, thus far, patient expectations have inconsistently correlated with patient satisfaction after surgery, and there is no accepted method for capture of perioperative expectations.²¹ Studying the relationship between treatment decision-making and patient education, expectations, and satisfaction has become paramount, given that patient satisfaction, which has been measured for the past decade, is now being used to calibrate surgeon and institutional reimbursement.²² Future research will need to further explore the theoretical framework for the decision-making process and identify measurable factors that surgeons and institutions can use to improve care.

Priority 2: Integrating PROs Data into the Electronic Health Record

Concerns remain regarding the integration of PROs into the EHR. Paper administration and processing of PROs can be time-consuming, costly, and too burdensome for a busy surgical practice. EHR integration improves the logistics of administration, although it raises additional concerns related to the security of patient and provider information. Existing research has focused primarily on PROs measurement, rather than on the EHR interface with patients and providers.²³ Effective systems must optimize the presentation of PROs data to enable surgeons to effectively interpret this information for decision-making. Likewise, enhanced feedback to patients may help validate the time commitment required to complete patient-reported measures and may potentially improve patient response rates. Without significant collaborative efforts to develop and improve EHR platforms, the effective use of patient-reported data by providers is unlikely to increase.

The NIH Patient Reported Outcomes Measurement Information System (PROMIS) may provide some insight for institutions collaborating on the electronic integration of PROs data collection and presentation. As a collective effort between institutions, PROMIS uses a centralized, Web-based system for PROs data collection and features immediate, standardized scoring using a shared item bank.^24,25 Despite the success of PROMIS with electronic administration and scoring across multiple institutions, they are not uniformly calibrated or validated for measuring the impact of surgical procedures. Future systems that measure PROs in surgery should incorporate the advances made by PROMIS, as well as address the issues identified by our Delphi survey, including presentation of PROs data to patients that is responsive to education level and language abilities, enabling easy interpretation by surgeons for immediate action and improving the integration of PROs systems into the EHR.

Priority 3: Patient-Reported Outcomes and Quality Assessment

PROs and the measurement of quality emerged as the final theme from the Delphi survey and raised concerns about the validity, efficacy, and risk adjustment of PROs measures in surgery. Foremost, the survey identified the validity of measuring quality with PROs versus traditional clinical outcomes measures as a significant consideration for stakeholders. Moreover, the efficacy of PROs instruments to improve quality and lower costs has not been extensively studied. System-wide introduction of PROs instruments should follow thoughtful research initiatives that demonstrate their effectiveness. To address this need for validation, the CMS Innovation Center could be a potential resource in evaluating the efficacy of selected PROs measures to improve quality, as it has already seen success in assessing new reimbursement strategies for Accountable Care Organizations.²² Emerging evidence has begun to show a correlation between patient satisfaction and surgical outcomes; however, comparison of selected PROs measures between providers, surgical groups, and institutions will require thorough risk and case adjustment.²⁶ Furthermore, to alleviate potential skepticism within the surgical community, the establishment of benchmarks will necessitate adequate transparency with regard to reasoning and methodologies. Successful use of PROs in performance measurement of surgery will require a thoughtful and open collaboration among stakeholders.

Measuring performance with PROs, however, is not a new concept. During the past decade, the UK National Health Service Patient Reported Outcome Measures initiative has collected health-related quality of life (HR-QOL), patient satisfaction, and functional data after inguinal hernia repair, hip and knee arthroplasty, and varicose vein ablation.²⁷ Internationally, the Patient Reported Outcome Measures initiative is perhaps the most ambitious quality improvement project to date and has started to evaluate the changes in HR-QOL and patient satisfaction after surgery, at the provider and institution level.²⁸ In a comparable move, CMS plans to encourage providers and institutions to routinely collect PROs data through funding models, such as the Meaningful Use and Physician Quality and Reporting System. The funding models initially incentivize PROs data collection through physician and institution reimbursements; however, after an introductory period, the models penalize participants who do not meet reporting/collection requirements. It remains to be determined whether routine measurement and comparison of PROs data will improve outcomes and lower costs in the long run.

Limitations

There are several limitations to this study, many of which are inherent to the Delphi process. The qualitative round of the survey was susceptible to influence from both the expert panel and the reviewers. To address this, the independent reviewers worked separately, without interaction with the expert panel. Final review by the expert panel looked for differences between reviewers, which is an accepted method for item generation and review and has been implemented by other groups in creating a research agenda.^11,29 In addition, the survey was susceptible to nonresponder bias. Studies have demonstrated that the demographic characteristics and survey results of responders are not equivalent to those of nonresponders.³⁰ Unfortunately, demographic data were not collected during registration, so we were unable to compare responders and nonresponders. This represents a serious limitation to this study and merits consideration in conducting future group-consensus studies. Despite this limitation, the survey did receive robust participation for all three rounds, and the sample size was comparable to or greater than that of similar conference-based surveys.^31,32

CONCLUSIONS

In an era of patient-centered care, PROs can serve as a useful complement to ongoing discussions on health care expenditures by including the patient voice, and they have considerable potential in the determination of quality in an evolving health care system. The incredible growth of PROs in clinical care and surgical trials has led to many potential research endeavors and collaborations. The PROS Conference developed a research agenda for researchers, surgeons, and funding agencies, to help prioritize research on PROs measurement in surgery, in order to direct funding and institutional collaboration. Future research initiatives should address PROs in the decision-making process, challenges to integrating PROs into the EHR, and PROs and the measurement of quality.

Abbreviations

CMS

Centers for Medicaid and Medicare Services

EHR

electronic health record

HR-QOL

health-related quality of life

PROs

patient-reported outcomes

PROMIS

Patient Reported Outcomes Measurement Information System