What Do Orthopaedists Believe is Needed for Incorporating Patient-reported Outcome Measures into Clinical Care? A Qualitative Study

Abstract

Background

Patient-reported outcome measures (PROMs) are increasingly promoted for use in routine orthopaedic care with the expectation that if they are made available during encounters, they will be incorporated into clinical practice. We investigated an initiative in which PROMs were systematically collected and provided via the electronic health record but were infrequently used.

Questions/purposes

In a qualitative study, we asked: (1) Why are PROM results not being used in clinical care when they are available to surgeons? (2) What aspects of PROMs are seen as useful for clinical care? (3) How are PROMs generally perceived by surgeons and orthopaedic leaders?

Methods

A cross-sectional qualitative study was conducted in a single health system in an urban setting using semistructured interviews with a purposive sample of orthopaedic surgeons and leaders who would have substantial knowledge of and experience with the organization’s PROM system, which was embedded in the electronic health record and developed for use in clinical care but was not being used. We included surgeons whose practices consisted of at least 90% patients with osteoarthritis, including surgical and nonsurgical management, and thus their patients would be completing PROMs surveys, or surgeons who were leaders in one of the three orthopaedic divisions in the health plan. The senior research manager for orthopaedics identified 14 potential participants meeting these criteria, 11 of whom agreed to study participation. Participants included nine surgeons and two orthopaedic leaders; the majority were men, with a median of 13 years of clinical practice. Study interviews were conducted by an experienced interviewer not known to participants, in private conference rooms in the healthcare setting, and a median (range) of 27 minutes (16 to 40) in length. A content analysis approach was employed for data analysis, with thematic inductive saturation reached in the analysis and attention to trustworthiness and rigor during the analytic process.

Results

Interviewees reported that PROM scores are not being used in patient clinical care because of logistical barriers, such as access and display issues and the time required, and perceptual barriers, such as concerns about patient understanding and the validity and reliability of measures. Surgeons preferred talking with patients about the personal outcomes patients had identified as important; most patients preferred to assess progress toward their own goals than PROMs scores for other people. Surgeons also identified changes that could facilitate PROM use and reduce barriers in clinical care, including pushing PROM scores to physicians’ inboxes, developing inserts for physician notes, using easy-to-understand graphical displays, and engaging patients about PROMs earlier in the care process. Participants all agreed that PROMs in aggregate use are valuable for the organization, department, and individual surgeons, but individual patient scores are not.

Conclusion

Despite the availability of PROMs, there are important barriers to incorporating and using PROMs in clinical care. Providing access to PROM scores without clearly understanding how and why surgeons may consider using or incorporating them into their clinical practice can result in expensive and underused systems that add little value for the clinician, patient, or organization.

Clinical Relevance

Involving front-line orthopaedic surgeons and leaders in shaping the design and structure of PROM systems is important for use in clinical care, but these interviewees seemed to see aggregate data as more valuable than individual patient scores.

Introduction

Orthopaedic departments have been leaders in appreciating the value of patient-reported outcome measures (PROMs) and developing systems to collect these data for use in care, quality assessment, and research [11, 21, 29, 38]. PROMs are increasingly used for care improvement and as a component of value-based reimbursement systems, facilitating their use in routine clinical practice [2, 5, 10, 31, 39]. PROMs are collected using multi-item questionnaires assessing patients’ perspectives on their symptoms, function, overall health, and quality of life; most are reported as numerical summary scores [13, 21, 31, 34]. A wide variety of PROMs focus on specific medical conditions or procedures, with 42 unique PROMs available for use in total joint arthroplasty alone [22, 38]. Incorporating PROMs into routine clinical practice could have many benefits, including obtaining key patient information, supporting patient-centeredness and shared decision-making, monitoring symptom change, and facilitating clinician-patient interaction and communication [3, 17, 23, 26]. Authors have focused on implementation issues, including identifying appropriate PROMs with good psychometric properties, providing organizational systems and resources to support data collection, facilitating patient completion in a timely manner, and displaying PROMs with easy-to-interpret formatting during the encounter [15, 18, 24, 28, 32].

Underlying the movement toward incorporating PROMs in clinical settings is the assumption that if PROMs are made available, they will be used in clinical care. Studies examining the use of PROMs in clinical settings have found little use of these tools when PROMs are available; surgeons have suggested that a lack of knowledge about how to use PROMs in clinical care, the amount of time they take to use, the perception that they do not provide actionable information, and the fact that gathering and handling of PROMs data adds work to an already busy schedule are reasons why they are not more widely used in practice [1, 32]. This was made clear in our local initiative on the use of PROMs in clinical care that provided PROM data to orthopaedic surgeons via an electronic health record. Physician use of PROMs was then tracked across 6 months showing limited access and use (< 3%). Although surgeons have expressed interest in the potential of PROMs for clinical care, effective use of PROMs in this setting will require an understanding of the surgeon’s perspective [33]. A qualitative approach that provides an investigation into how surgeons perceive the use of PROMs in clinical care is a first step in addressing this complex issue and providing insight for developing effective PROM systems and their use.

In this qualitative study, we asked: (1) Why are PROM results not being used in clinical care when they are available to surgeons? (2) What aspects of PROMs are seen as useful for clinical care? (3) How are PROMs generally perceived by surgeons and orthopaedic leaders?

Patients and Methods

Study Design and Setting

This qualitative cross-sectional study is a component of the PROM Optimization Through Technology and Engagement (PROMOTE) study, which was funded by the Agency for Healthcare Research and Quality. The goal of the study was to learn how to enhance the collection and use of patient-reported outcomes in clinical care. The study was conducted at a nonprofit research center affiliated with a mixed-model health system in the upper Midwest that includes 50 clinics and six hospitals that care for 1 million patients. The health system had prioritized building a health information technology PROMs system within the EPIC medical record to expand the use of PROMs in clinical care and for care improvement. The initiative started in orthopaedics because the division had been focusing on the use of PROMs in the care of patients undergoing knee or hip replacement surgery. PROMs in orthopaedics are completed by patients 1 week before surgery and again at 3 and 12 months after surgery; completed PROMs then are entered into the EMR PROMs system. Consistent with a qualitative approach, the study partnered with operational leaders 6 months after the EMR PROMs system rollout to explore the perceptions of those who would have the most experience with the PROMs system. Interviews for the study were conducted with orthopaedic surgeons and orthopaedic leaders from November 2019 to January 2020.

Participants

We used a purposive sampling strategy [33] to identify a sample of participants. This type of sampling is used in qualitative research to select respondents for their ability to provide the needed information. In this case, we sought information about users’ experiences with the PROM system that focused on hip and knee replacement surgery, and so we looked for individuals who could address its use in clinical care. For our sample, we included orthopaedic surgeons whose practices consisted of at least 90% patients with osteoarthritis, including surgical and nonsurgical management (which meant their patients would be completing PROM surveys), or individuals who were leaders in one of the three orthopaedic divisions in the health plan who would also be familiar with the PROMs system and its use in clinical care. The surgeon sample was limited by the practice composition of most surgeons in the health plan, which more often serves a broader patient profile. Eligible participants were identified by the senior research manager for orthopaedics (MR) in collaboration with orthopaedic leadership. Fourteen eligible participants meeting the inclusion criteria were sent an email from the senior medical director of orthopaedics explaining the study and inviting their participation; a follow-up email was sent 1 week later. Eleven of 14 eligible participants consented to study participation.

The sample consisted of nine orthopaedic surgeons and two leaders who are program directors with responsibility for managing and providing leadership within their orthopaedic divisions. Participants had a median (range) age of 46 years (36 to 68), 8 of 9 surgeons and 1 of 2 orthopaedic leaders were men, and 8 of 9 surgeons specialized in joint replacement and 1 was a sports medicine specialist. Surgeons obtained their medical degrees between 1984 and 2016 and had a median (range) of 13 years (4 to 36) of experience.

Data Collection

Study interviews were scheduled based on participant preference; interviews were conducted in private conference rooms in the healthcare setting. The study purpose was reviewed, and informed consent was obtained at the start of each interview. Semistructured interviews were conducted, focusing on the interviewees’ perception and use of PROMs in clinical care. The interview guide contained nine questions, seven of which were used for this study (Appendix 1; http://links.lww.com/CORR/A674), and these questions explored the use of PROMs via the electronic health record during clinical care, barriers to using PROMs during patient visits, perceptions of the benefits and challenges of using PROMs in clinical care, and general perceptions of the importance and role of PROMs in the participant’s department. A set of structured probes was used to explore participant responses in greater depth. A single interviewer (RRW) with extensive experience in face-to-face research interviewing with physicians, specialists, and healthcare leaders; in-depth knowledge of the study; and experience in healthcare conducted the interviews over 12 weeks. The interviewer was not known to study participants. Interviews were a median (range) of 27 minutes (16 to 40) in length (16 to 40) and were audio recorded, deidentified, and professionally transcribed for data analysis.

Ethical Approval

The study was reviewed, approved, and monitored by the local institutional review board.

Data Analysis

Interview data were analyzed in NVivo qualitative analysis software (version 12.6.0 for Mac) using a conventional content analysis approach [25]. The analysis team comprised five authors (RRW, LIS, JYZ, CKN, EAC) with multidisciplinary experience including qualitative methods, patient care, patient perspectives, and healthcare data. This multidisciplinary perspective presents the opportunity for fresh ideas and data interpretations with the potential for insights from an array of perspectives. The team first independently reviewed each transcript, identifying initial codes and emerging patterns, then met regularly to systematically code and structure the data, as well as identify emergent patterns, categories, and themes in the data. Coding issues were discussed until inductive thematic saturation was reached, where no new codes or themes emerged in the analysis [37]. A final coding framework was established, which one analyst (RRW) then applied to all data. A constant comparative method using an iterative process and addressing issues of reflexivity in the analysis [35] was employed throughout the process, with team members actively reflecting on the meaning of the topic from their personal perspective and to orthopaedic care, as the evolving thematic framework was developed. The data were considered and discussed from a variety of viewpoints with individual assumptions addressed and personal biases identified and challenged, as well as alternative viewpoints considered. A study codebook was maintained to compile an audit trail, address coding decisions, and to document the data framework to enhance trustworthiness and analytic rigor in the analysis. Four primary themes emerged in the analysis addressing the three study questions. All quotes cited in the results section are from surgeons denoted by “(S)” unless otherwise indicated as from an orthopaedic leader, denoted by “(OL)”; a participant study number is also designated for each quote (01-11).

Results

Why Are PROMs Results Not Being Used in Clinical Care When They Are Available to Surgeons?

There were two themes identified regarding why PROM results are not being used when provided to surgeons. The first is that PROM scores are not being used in the care of individual patients because of logistical and perceptual barriers. All respondents identified barriers to the use of PROMs in clinical care. Logistical barriers focused on structural issues concerning the access and display of PROM scores in the electronic health record and the time involved during the patient visit to address them. PROMs are available in our electronic health record but viewing them requires surgeons to move through a series of screens to access them, prompting comments such as, “if it’s five clicks deep nobody’s ever looking at it” (S06) and “it’s onerous to access it during the clinic visit” (S10). Respondents noted they rarely use the electronic health record during patient visits except for accessing radiographs or other imaging files: “I don’t usually look at the computer while I’m talking to the patient, it’s always before or after the visit” (S09) and “anything that takes away from that short interaction with my patient is a detriment to me” (S11). Respondents also did not like the way PROM scores were displayed, saying: “I don’t find the electronic health record presentation of the information very helpful” (S07). Many expressed concerns regarding missing data. PROM surveys are completed voluntarily preoperatively and at 3 and 12 months postoperatively, and respondents noted that if there are missing data because of a lack of completion or timeliness in posting data, it is difficult to interpret or effectively use PROMs in patient care. Respondents also said there was not enough time during a patient visit to add a conversation about PROMs, saying: “Time is always a big issue” (S08), “visits are short” (OL01), and “I worry that they’re going to prolong the clinic appointment even longer” (S03), with one respondent noting: “Most of us already run late in our clinics and you’re adding another layer of information that needs to be discussed … without clear clinical benefit” (S03).

Perceptual barriers focused on the difficulty of helping patients understand PROMs and surgeon concerns regarding PROM measurement and measurement-driven care. Most surgeons stated that patients do not understand PROM scores: “Patients have no idea what to compare their numbers to” (S07) and “I think it would be more confusing for them and it would generate a lot of questions” (S09). Respondents expressed concern that PROM discussions could open a Pandora’s box, with some patients misinterpreting the scores or not seeing scores they expect, requiring additional time and effort to clarify and explain measures and their scoring. Some providers also expressed concern about the validity of PROM measures, saying: “Scores can vary week to week” (S03), and that while there are a variety of PROMs, “there are problems with all scores. There isn’t a perfect score” (S06). Respondents expressed concerns about poorly worded questions, lack of coverage of certain topics in established measures, and confounding factors that are not accounted for in measures: “I take all this with a grain of salt because there's so many overlapping things that could influence the scores” (S07) and “there’s just too many confounding variables involved to really tell patients what it really means” (S08). Respondents also voiced concerns about the concept of measurement-driven care saying, “I think you lose a little bit of the art of medicine when you start looking at pure metrics” (S03) and “it’s more data in a slew of data” (OL05) for surgeons to use and try to incorporate into their practice. Surgeons instead wanted to focus on their interactions with patients for assessment, perceiving PROMs as not useful in a clinical context: “I don’t think it really has any clinical significance during the clinical visits to go over these exam scores” (S03) and “I care less about what their number is as opposed to the … How is your pain? Does it hurt? Does it hurt less than before? Is it the same pain?” (S07).

The second theme we identified in our study is that patient-identified outcomes are seen as more valuable in patient care than PROM scores. Respondents noted it was more beneficial to engage patients about their individual patient-reported or desired outcomes than to discuss PROM scores. Respondents also talked about the interpretation and length of PROMs, noting that “having a couple of key questions” focusing on what is most important to the patient as an outcome of their surgery is more beneficial than a PROM score. As one respondent noted, “That’s probably more meaningful than any of the score data. Are you sleeping through the night? Are you playing with your grandkids? Are you going on a hike? Those are the questions that I think are more meaningful … more useful for the patients than your Oxford score was 45 or 34, or whatever it was” (S03). Another respondent expressed concern about the use of score data, saying: “I think patients want to feel like they’re being addressed not as a number … they want to know that they’re making progress on their own individual basis” (S09).

What Aspects of PROMs Are Seen as Useful in Clinical Care?

The main theme regarding the usefulness of PROMs for clinical care is that changes in approaches to using PROMs in clinical care could enhance use and reduce barriers. Respondents noted that patients needed to be engaged earlier in the care process about PROMs: “Include it so we can have this information before the patients sign-up for surgery” (S02), and they encouraged “a patient-facing version of this … so if a patient could log in and access their outcomes and understand visually how they are doing and have a descriptive of what that means” (OL05). They also stated that patient engagement with PROMs needs to be “straightforward and simple for them to understand.” Respondents talked about how PROMs could assist in patient communication and interaction regarding decision-making and the outcomes of surgery, saying they “could be useful in the preoperative setting … as a tool to help guide the decision to consider surgery more strongly” (S02), and “I think that would be good from an expectation standpoint to counsel patients in terms of appropriate expectations after the joint replacement” (S03). They also noted that presenting PROMs as “graphical information” could further facilitate patient understanding and conversation. Some respondents also thought PROMs could be useful for continuity of care: “If someone else is caring for my patients … having something there for one of the nurses who gets a call about the patients or one of my partners” (S11) and “there would be some consistency with that … to incorporate that in whatever type of interaction they’re having with the patient” (OL01). Respondents also discussed changes to the display of PROMs in the electronic health record that could expedite use: “Put it in the medical record in a way that is visually easy to use, and that you could import into your own (patient) note easily” (S06) and have the information “pushed to our in-basket when it came through in real time” (S02). One respondent echoing many said, “it should be super, super easy and super, super fast” (S06).

How Are PROMs Generally Perceived by Surgeons and Orthopaedic Leaders?

The primary theme regarding how surgeons and orthopaedic leaders view PROMs is that PROMS are widely perceived as valuable in aggregate use for the organization or department and individual surgeons. Overall, respondents perceived considerable value in the use of PROMs in aggregate for the organization or department and individual surgeons. They noted, “for the organization, it certainly speaks to the level of care that you deliver … and that patients have great outcomes” (OL05), and “it can be used to determine how well the organization is caring [for] and improving these patient-reported outcomes for patients with hip and knee arthritis” (S02), and finally, “in a competitive market … having access to that and being able to display our outcomes and patient satisfaction is probably very important to the payers” (S11). Respondents also talked about using the data to address questions such as, “how are we doing as a group … can we do things better?” (S04), using the data to “determine averages and trends,” as well as “research purposes” and “marketing data.” Respondents overall perceived there was clear value for “the big picture” of how aggregate PROM data could be used to evaluate and promote the organization or department. Respondents also discussed how aggregate PROMs could be useful for individual surgeons “as a tool to look at their practice,” (S04) saying, “I think it, used in aggregate, can be useful for a surgeon in terms of knowing how their patients are doing in general” (S02). Other respondents noted, “if you don’t measure something, you have no idea how well you’re really doing” (S07) and “you want to be able to compare the success of your patients to that of others, which is why you need more of a larger perspective” (S10). Respondents also noted that PROMs could be useful as an “improvement tool” to facilitate individual improvement or “if you’re switching implants or … to compare yourself to some sort of mean and see how you’re performing” (S03).

Discussion

PROMs are increasingly being collected for use in orthopaedic clinical care, stimulated in part by care improvement needs and value-based reimbursement systems. We sought to understand why surgeons are not using available PROMs and how surgeons perceive the potential utility of PROMs. We identified four primary logistical and perceptual barriers: difficulties in access of PROMs in the electronic health record, the display of PROMs in the electronic health record, the time involved, and the perception that patients do not understand them. Surgeons instead thought that outcomes identified by patients were more valuable than PROM scores. PROMs were seen as more valuable in their aggregate form for the organization, department, and individual surgeon. To improve the use of PROMs in patient care, surgeons need to be involved in the planning and development of PROM systems to assure their effectiveness.

Limitations

This study has limitations. This was a small, single-site, exploratory study with a purposive sample composed of surgeons who had a majority of patients completing the PROMs; consequently, surgeons used the PROMs system frequently. This narrowed the number of potential participants substantially, but we targeted study interviews to those with the most experience with the PROMs system, including orthopaedic leaders and surgeons. Although these two groups may not seem cohesive, in this organization, both were tasked with implementing PROMs in clinical practice, highlighting the importance of both views to the investigation. In qualitative research, the focus in sampling is on depth of experience with a topic rather than breadth of participants for the sake of inclusion. Thematic analysis depends on depth of experience, and although there are many factors, including personal reflexive perceptions that can influence the analytic approach, we spent time addressing these in a variety of ways until inductive thematic saturation was reached, when no new themes emerged and all alternatives had been considered and weighed in the analysis. This was also facilitated by the multidisciplinary analytic team, which brought a wide variety of views and perspectives to the analysis. We do note, however, as a qualitative cross-sectional study at a single organization, there may be other factors not mentioned in the interviews that could impact PROM use, for example, the relative value unit (RVU)-driven surgeon remuneration system used by Medicare to determine reimbursement to providers, encouraging high clinic and surgeon volumes that add to clinic time pressures. Although there was no mention of RVUs during any interview, we note it as an example of an underlying issue that can have subtle influence on provider behavior.

Why Are PROMs Results Not Being Used in Clinical Care When They Are Available to Surgeons?

Improving the use of PROMs in clinical care requires addressing both the logistical and perceptual barriers identified by study participants. Logistical barriers included challenges accessing PROMs in the electronic health record, the poor display of PROMs, and the frequency of missing data, as well as finding the time during clinical encounters to address PROMs. Perceptual barriers focused on a lack of patient understanding about PROM scores and surgeon concerns regarding the validity and reliability of measures. Surgeons still perceive limited utility for PROMs in patient care, noting that they preferred talking with patients about outcomes the patient had identified as important, seeing these personalized outcomes as more meaningful rather than a numbered score that can be difficult to explain. Other studies examining perceptions of PROMs have also identified these issues [1, 7, 11, 14, 21, 23, 26, 32, 40]. To fully incorporate PROM scores into clinical care, well-designed PROM systems will need to be developed with measures that not only have good psychometric properties and strong evidence to support their use for comparisons, but also PROMs that reflect the unique outcomes desired by individual patients. The interviewees also note the importance of including an easy-to-understand patient-facing version that could promote communication, shared decision-making, and patient engagement in their care. Substantial investments of time and resources by health systems into developing PROM systems that surgeons find easy to use, incorporate into clinical care, and promote surgeon-patient interaction will need to be made if PROMs are to be effectively incorporated and used in clinical settings with individual patients.

What Aspects of PROMs Are Seen as Useful in Clinical Care?

The orthopaedic surgeons and leaders in this study suggested changes in the approach to using PROMs in clinical care that could reduce barriers and enhance their usability. Several specific suggestions arose that may result in more effective incorporation of PROMs in clinical settings (Fig. 1). Importantly, participants noted the need to engage patients about PROMs earlier in the care process, perhaps by creating a patient-facing version of PROMs with easy-to-understand graphical displays. They also noted that using PROMs in shared decision-making to guide decisions about surgery or to counsel patients on expectations regarding postsurgical outcomes could enhance PROM use and improve patient communication and interaction about these data. A recent study of surgeons’ perceptions of PROMs found surgeons endorsed the use of PROMs for counseling patients in preoperative and postoperative settings [32]. Researchers have also recommended using PROMs to enhance engagement and communication with patients, to help guide clinical and shared decision-making, and to improve patients’ experiences [5, 6, 12, 17, 27, 30]. Integrating PROMs into clinical care, however, will require PROM systems to be more flexible and accessible with interactive capability, such as pushing score data to physician inboxes and developing smart sets for use in physician notes. Both in this study and across the evidence, issues of resources, technology, and infrastructure for incorporating PROMs into clinical care are emphasized as important for successful use [3, 7, 8, 18, 28]. A survey of surgeons found that efficient tools and technologies were seen as a prerequisite for use in clinical care [26].

Fig. 1.

Recommendations for improving the use of PROMs in clinical care.

How Are PROMs Generally Perceived by Surgeons and Orthopaedic Leaders?

Although there were many barriers identified for using PROMS with individual patients, there was wide agreement regarding PROM usefulness in aggregate form for the organization, department, and individual surgeons. PROMs have long been used in evaluating, promoting, and displaying aggregate patient outcomes and satisfaction for health insurers and public reporting, as well as for research [4, 36, 41]. With the development of orthopaedic registries, aggregate PROM data are also being used for improvement initiatives and as a component of value-based payment systems and quality assurance programs [19, 20, 31, 39]. The use of aggregate individual data by surgeons, with its ability to anonymously compare to others, may prove especially useful for surgeons looking to improve their skill and practice. The use of PROM data in aggregate has increased steadily as the value and potential use of these data become clearer and PROMs become more widely collected [7, 9, 16]. It may be that the use of PROMs in clinical care will also increase as the value potential of PROMs is realized, not just in their aggregate form, but with individual surgeons and patients.

Conclusion

The barriers to the use of PROMs in clinical care identified through these interviews and the proposed changes that address them could facilitate PROM use in clinical settings and are important considerations for the many orthopaedic groups that are collecting these data. These changes will require a robust and well-resourced PROMs system, with agreed upon measures that have good psychometric properties, is easy to use, incorporates displays for both providers and patients, and that can be easily accessed and used during the clinic visit. Understanding the surgeon’s perspective on the use of PROMs in clinical care is an important first step toward achieving this goal. It could also help forestall the development of expensive and poorly used systems that bring little value to the clinician, patient, or organization. Involving end users in the development and application of new concepts has long been a tenet of improvement work in a variety of fields. Involving frontline orthopaedic surgeons and leaders in shaping the design and structure of PROM systems is needed to address existing barriers and to enhance their use in clinical care.