Abstract
Purpose
The use of patient-reported outcome (PRO) measures in clinical practice is increasing. Following the creation of a ‘User’s Guide to Implementing PRO Assessment in Clinical Practice’ by the International Society for Quality of Life Research (ISOQOL), volunteers from ISOQOL sought to create a Companion Guide to assist health care providers with the scientific and practical considerations involved in implementing and using PRO measures in clinical care by using information from real-world case studies. This paper summarizes the key issues presented in the Companion Guide.
Methods
Ten respondents, who were members of the ISOQOL’s CP-SIG and worked in various clinical areas, participated in a survey or telephone interview. Participants were from Canada (n = 2), Denmark (n = 1), England (n = 2), Holland (n = 1), and the United States (n = 4).
Results
Based on the information provided by respondents, a Companion Guide was produced, organized according to the nine questions presented in the User’s Guide. An additional section for key take-home messages was also provided. This guide provides examples of issues and considerations related to the implementation of PRO measures in clinical practice.
Conclusions
Respondents provided insight into their experiences and emphasized that PRO initiatives were likely to be more successful if there is purposeful, designed integration into clinical practice, meaningful substantive engagement with all stakeholders and access to necessary organizational resources. The ability to leverage existing technology as well as realistic and stakeholder consensus-driven expectations for planning and timing were also key to the successful implementation of PRO measures.
Keywords: Clinical practice, Patient-reported outcome, PRO, Guidelines, ISOQOL, Implementation
Background
Patient reports on self-defined daily functioning and wellbeing can be used in conjunction with other clinical information to inform management of patient care. Patient-reported outcome (PRO) measures are increasingly being used in clinical practice to support patient care by helping providers monitor health outcomes and health-related quality of life (HRQOL), track patient progress, enhance communication with patients, and improve patient satisfaction with care received [1, 2]. However, previous literature suggests that healthcare providers often experience significant practical challenges—such as administrative burden, cost of use, problems with PRO interpretation, and skepticism around clinical meaning—to successfully integrating PRO assessment into clinical practice workflows [3–6]. Following discussions by ISOQOL members about these challenges, a ‘User’s Guide to Implementing Patient-Reported Outcomes Assessment in Clinical Practice (the “User’s Guide”)’ was created in 2011 by the International Society for Quality of Life Research (ISOQOL) [7]. The User’s Guide poses nine questions that should ideally be addressed when implementing PROs in clinical settings (see Table 1 below).
Table 1.
Nine core questions from ISOQOL’s user’s guide
|
Following this, in 2013, ISOQOL members attending the clinical practice special interest group (CP-SIG) meeting discussed the need for further aid in understanding the realities, challenges and opportunities associated with using PRO measures in clinical practice. They identified that assistance could be provided through the sharing of real-world experiences. Despite the already existing User’s Guide, there was a clear need for this guidance to be enhanced through the provision of case studies detailing current PRO implementations in clinical practice. This would thus assist health care providers with the operational issues involved in implementing and using PRO measures in clinical care.
In response to this need, members of the CP-SIG were invited to partake in either a survey or an interview. As part of this, members were encouraged to share their own experiences of employing PRO measures within clinical practice settings. The findings were then presented in a Companion Guide, compiling information from real-world PRO implementation that ultimately enabled the User’s Guide to be brought “to life.” This Companion Guide can be accessed on the ISOQOL website at http://www.isoqol.org/research-publications-landing-page/isoqol-publications. This is a key resource that accompanies the User’s Guide to assist in optimal use of PRO measures in clinical care. This paper provides a brief overview of the experiences and challenges discussed by participants when implementing PRO measures, alongside the key lessons that were learned.
Methods
Study design and participants
The core working group of ISOQOL CP-SIG volunteers (KH, LN, TE, EC, SPM) initially developed a template based on the nine questions in the User’s Guide. This template sought to inform the collection of real-life case studies with which to illuminate pragmatic experience and guidance for PRO implementation and use. In early 2014, the working group sent an open invitation to members of ISOQOL’s CP-SIG to share their experiences of using PRO measures in clinical practice by participating in either a survey or telephone interview.
Ten CP-SIG members agreed to take part in the survey or interview and provided written consent to participate. Six of the respondents in the sample worked within academic hospital settings and four worked within a university. Respondents were located in a range of countries including, Canada (n = 2), Denmark (n = 1), England (n = 2), Holland (n = 1), and the United States (n = 4). The participants worked with a range of patient populations and specialties, including General Practice/Pediatrics (n = 2), Occupational Medicine (n = 1), Oncology (n = 2), Lung/Heart Transplantation (n = 1), Orthopedics (n = 1), Rheumatology (n = 1), and Chronic Pain (n = 1). Below is a list of the participating respondent sites (Table 2).
Table 2.
Participant responding sites
| Case study # | Population | Country |
|---|---|---|
| #1 | Chronic lower back pain | Canada |
| #2 | Rheumatoid arthritis | United States |
| #3 | Prostate cancer | United States |
| #4 | General oncology | United States |
| #5 | General pediatrics | Holland |
| #6 | Occupational medicine | Denmark |
| #7 | Orthopedics | United States |
| #8 | General practice, idiopathic pulmonary fibrosis | England |
| #9 | Lung and heart transplant | Canada |
| #10 | General oncology | England |
Surveys and interviews
Out of the ten participants, six members completed the survey and four participated in 60-min telephone interviews. The survey centered around the nine key areas identified in the User’s Guide (see Table 1). Respondents were encouraged to describe experiences, challenges, and lessons learned in each of these areas. Additional questions included consideration of the timeframe for implementing PRO measures and key take-home messages. This survey was open ended, for example: ‘what were your purposes and goals for collecting PROs in clinical practice and what resources were available?’. Additional questions to consider for each main area were also presented beneath, for instance: ‘A. Was PRO data collection planned as a specific project or as part of daily routine?’ and ‘B. What additional resources would have been particularly helpful?’. These provided probes for participants to consider when providing their answers. In the telephone interviews, questions were asked using the survey template as a basis and in-depth notes or audio-recordings were taken by the interviewers. These telephone interviews adhered to the survey structure whereby the main question was asked, followed by probing using additional questions.
Thematic data analysis
Completed case study templates and telephone interview notes were entered into a web-based mixed methods analysis program and database for coding (http://www.Dedoose.com). Through Dedoose, data were excerpted and coded according to the nine main topic areas from the User’s Guide. The code applications were reviewed and harmonized by at least two team members to assure reliability. Excerpted text was then exported into separate Excel spreadsheets corresponding to each of the nine topic areas. Individual members of the core working group each analyzed up to three of the nine sub-sections to identify and describe key themes [8]. Themes were generated through reading and re-reading the text and actively looking for patterns and meanings within the data. Finally, the thematic results of each subsection were reviewed and harmonized by at least two team members to assure reliability. This approach was consistent for both the survey data and the telephone interview data.
Results
From the surveys and telephone interviews, results were analyzed and summarized below according to the nine main topic areas with an additional Section (10) focusing on key take-home messages.
Section 1: Goals, resources, and barriers to PRO collection
Multiple objectives can be achieved through successful PRO measure implementation. When discussing implementation within clinical practice, respondents felt it was crucial that careful, inclusive organizational planning was the starting point. Prior to implementation, three key requirements were described including (1) goal setting for PRO collection, (2) identifying needed implementation resources, and (3) clearly communicating justifications for a PRO implementation to essential stakeholders.
A range of PRO collection goals were described, including but not limited to, screening, monitoring, treatment evaluation and planning, aiding in treatment decision-making, and improving patient and provider satisfaction. One participant recalled that “The goal is to monitor and screen children with various chronic illnesses to be able to detect problems that arise at any early stage and to provide tailored interventions before the problems increase.” (#5).
The availability of resources varied greatly between sites, with some describing limited resources, and others describing substantial support. This needs to be taken into consideration when planning PRO implementation, alongside taking into account limitations that may exist. The importance of not underestimating the amount of time and human resources required to establish and sustain PRO projects was discussed, as was the importance of training both at the start and throughout the implementation process. Following this, various forms of training were noted by respondents, ranging from short presentations to the development of a training program utilizing a behavioral change framework.
Barriers existed in the form of limited resources, and attention is needed to ensure that these barriers do not prevent successful PRO implementation. For example, respondents highlighted that a lack of up-front funding, or reliance on inconsistent funding sources, both presented a challenge. Similarly, the time-consuming nature of new electronic system development acted as a barrier.
Other considerations in PRO planning were discussed, for instance the need to clearly communicate justifications for PRO implementation. Although the need for clear communication may seem self-evident, respondents noted that efforts must be made before initiating a PRO implementation to ensure that all concerned stakeholders would derive value from the initiative. To ensure successful PRO implementation clinicians should agree on its purpose, and the measures should fill information gaps and meet the needs of all relevant stakeholders.
Section 2: Groups of patients assessed
Various groups of patients may be involved in PRO assessments; thus attention to language, physical and mental abilities, and age-based considerations were reported as being essential to consider. Respondents highlighted the importance of PRO data collection systems allowing patients to participate irrespective of their language abilities, their manual dexterity, their age and their comfort with using different technologies.
One participant noted that although a previous study conducted had required that patients speak, read and write in English, that “it is extremely important to design future systems to allow participation among non-English speakers as well” (#3).
Support that could be provided according to the needs of the patient was discussed, for instance the mode of data collection, such as the use of electronic tablets, or through availability of team members to offer support.
Section 3: Selection of PRO measures
Determining the most appropriate PRO measure to collect the desired information was discussed by respondents. Four major themes emerged: (1) use of existing guidelines and conceptual models; (2) consideration of measurement properties; (3) measurement ease of use; and (4) engaging clinicians, patients, and other stakeholders in reaching consensus.
The use of existing guidelines and conceptual models to assist in the selection of most appropriate PRO measures was noted. Respondents reported referring to a range of sources, including, the ISOQOL User’s Guide to Implementing PROs [7], the US Food and Drug Administration (FDA) Industry Guidance document [9], the European Medicines Agency (EMA) Reflections Paper [10], the International Classification of Functioning (ICF) [11], the Wilson and Cleary model [12], and the Triple Aim Framework [13]. Other respondents also mentioned use of the online PROMIS system (https://www.assessmentcenter.net/), as it provided a useful mode of assessment for their patient population. Additional evidence to inform PRO measure selection was also sought, for instance through conducting literature reviews, alongside consideration of other drivers such as the validity of the PRO instrument, its recall period, and its availability in multiple languages.
Practical considerations were also discussed, such as ease of use of a measure. Measures were reportedly chosen by taking into account accessibility, length (i.e., number of items/time burden), response options, and ease of scoring. These factors are important to regard to prevent undue burden on patients and unnecessary data for clinicians.
Respondents identified that reaching consensus on PRO selection was achieved through meetings and focus groups involving various stakeholders and patients. Clinicians felt that it was consistent with their role to contribute to the selection and implementation of PRO measures.
Section 4: Timing of PRO administration
Data collection was reported as being linked with clinical and/or research visits in all cases. In addition, the length of time for the collection of follow-up data was influenced by a variety of factors, such as disease and/or treatment, the discretion of the care team, frequency of outpatient visits, and type of PRO measurement being used. For instance, one participant noted:
Completion of PROs depends on the purpose of data collection. For cancer studies, we typically collect for 2 weeks and then have a week off. If monitoring side effects then you may need to do it more frequently (weekly).
(#10)
Section 5: PRO administration and scoring
A myriad of alternative ways to administer and score PRO data exist. When considering this, three key themes emerged: (1) mode and format of data capture, (2) measure scoring, and (3) engagement with providers and patients.
Multiple workflows for collecting PRO data from patients were reported. These included collections via web-based systems that patients could access at home with email prompts. Electronic collection of PRO data was discussed, and respondents noted many benefits such as error reductions, automatic scoring calculations, and management of data security measures. Paper-and-pencil administration was also referred to, with one respondent reporting higher response rates with this format of collection. However, many issues were discussed such as “time to transfer the data into our system” (#1), scans of scoring sheets being unreadable, and delays in clinic schedule if forms were not completed before the visit. Challenges with patients completing measures in the clinician’s office/waiting room or outpatient clinic settings were discussed, for instance:
You have to remember to print and have a copy before the patient walks in the clinic room. This is why screen prompts and having the questionnaire embedded in the EHR is valuable.
(#8)
The involvement of multiple stakeholders to minimize the impact of PRO data collection on the clinician, e.g., medical assistants and nurse practitioners, was noted. The scoring of PRO measures was also considered, with discussions centering around electronic software scoring, and manual scoring. Challenges of manual scoring included:
When done manually on paper, they [PRO measures] are often scanned into MR [medical record] and lost in the letters section and therefore not acted upon.
(#8)
In relation to engagement with providers and patients, one respondent noted that PRO administration is most effective when the clinicians are ‘on board’ and consequently engaged in both the process and the initiative.
Discussions around necessary care being taken when presenting new PRO measures to patients also occurred. Care must be taken to ensure patients understand the purpose of the PRO assessment and results, and to ensure that the value of PRO collection is fully communicated to them. Concerns around relying on primary health providers to refer patients to PRO systems were mentioned, specifically that such referral often did not occur. For instance, one respondent noted:
It depends on the clinic as well, and the provider, because if they really bought into it, I mean if a patient refuses—I’ve got some providers that go in there and sit down with them and say, this is really important, and finish with them. Whereas others were like ‘I don’t care less if they do it.’
(#7)
Section 6: Score interpretation and follow‑up
Several challenges with interpretation of PRO measure results were discussed by respondents, for instance, the patient difficulty in interpreting scores and the lack of time for providers to review scores at every visit. Key themes relating to interpretation of PRO results emerged, including: (1) using standardized data, (2) representing data graphically, (3) using comparison data, (4) education and training, and (5) stakeholder consensus.
Discussions arose surrounding the use of standardized data, with respondents referring to meaningful change in scores, the use of relating clinical variables to PRO results, and providing written information linking PRO feedback. The Companion Guide also covers respondent discussion of graphical representation of data to communicate PRO results. The use of comparison data to inform interpretation was likewise discussed; this may include comparison to population scores, to others with the same disease, to expected scores, or to baseline data.
Throughout the survey and interviews the importance of training to support clinicians and patients was highlighted, for instance:
Most people believe they understand measurement far better than they actually do.
(#2)
Various PRO measures do not have standardized cut-points like a minimum clinically important difference (MCID) value defined in the literature, so respondents reported working with clinicians to reach consensus regarding key score changes. The importance of working together with different stakeholders was also considered, ensuring that PRO concepts are relevant to both patients and clinicians, to develop approaches to enhance the visual presentation of PRO data and its interpretation, and to determine clinical utility.
Section 7: Presentation of results
Following PRO collection and scoring, consideration turns to the presentation of the results with key plans needed for: (1) when and where, (2) how, (3) and to whom results are provided.
Suggestions for when to present PRO results depend upon the respondent’s framework for clinical management. Discussions centered around presenting results in real time, in advance of clinic consultations, in quarterly team meetings, through electronic systems, and even presenting them at a ‘research day’ with patients and relatives present. In terms of how results are presented respondents discussed a manner of formats including graphs, electronic patient lists with significant scores highlighted, and electronic summary reports.
PRO results were reportedly used by health providers to inform patient management. These results tended to be used by one specific clinician; however in some circumstances, multidisciplinary teams and wider groups of stakeholders were involved. In these situations, efforts were made to only show each team member scores relevant to their area of clinical expertise. Results were also reportedly presented to patients or to patients’ caregivers, with one respondent emphasizing the need for additional training to support clinicians in communicating these results. A further respondent asked:
How do we present PRO data to patients themselves, especially if they see their scores getting worse?
(#10)
Section 8: Responding to issues
The appropriate response to issues identified through PRO assessment was discussed by respondents. For example, the importance of linking a patient’s data to clinical decision-making. In one case, a respondent explained that their system aids in care management as it enables clinicians to determine whether the patient requires a visit or consultation, and what symptoms are a priority in terms of treatment and management. This is processed by an automatic algorithm and assigns patients by a color depending on whether contact is needed.
It was noted that following issues identified in PRO assessments, a referral can be made. For instance, “if necessary, the patient or parent can be referred to the psychosocial services department” (#5).
Section 9: Evaluation of PRO value
When evaluating the impact of PRO measure application in clinical practice, respondents discussed both formal and informal evaluations. Formal evaluations described by respondents included an exploration of providing standardized PRO information in oncology care, the use of tele-completed PRO measures within an occupational medicine clinic, routine, annual evaluations of a pediatric PRO system, and clinician interviews as part of a chronic pain management program. Informal evaluations were also described, for instance the reporting of positive responses from clinicians and patients. One respondent reported for a clinician:
He loves being able to have these simple easy reports and watch the trends of his patients.
(#7)
However, skepticism among some providers was also described, particularly those who are new to the concept of PRO application. The importance of future research to gain better understanding of PRO information use in informing decision-making was discussed.
Section 10: Key take‑home messages
Respondents were asked to provide key take-home messages for using PRO measures in clinical practice. Many benefits were reported, and crucial messages included the idea that PRO implementation needs ease of use, ease of access and a clinical determinant. PRO measures need to be regarded as the center of care, must address gaps in information needs, and the feedback needs to be actionable.
The importance of engaging stakeholders was emphasized, as support is needed from all those involved, including administrative support, clinicians, and patients. Respondents also considered the need for available adequate institutional resources to support PRO assessment. Three respondents discussed the importance of considering the way in which PRO data will be integrated into the EHR and the organizations technical infrastructure. The timeframe of implementation was also discussed, with one respondent estimating it took around 3 months to carry out a pilot. An additional respondent suggested that the entire process from conceptualization to implementation took about 3 years, and a further respondent estimated that the process took a number of years.
Conclusions
The use of PRO measures in clinical practice is constantly developing and evolving with time. Challenges faced by health care providers highlight the need for comprehensive guidance to support PRO assessment in clinical practice.
The Companion Guide was designed to allow those working in the field of quality of life research to gain insight and learn from the case studies presented. The intention was to provide a concise, user-friendly guide to contextualize the information provided within the User’s Guide using real-world examples from various clinical areas.
The ten case studies from these clinical sites allow the Companion Guide to outline some of the potential barriers and opportunities when embarking on a PRO program in clinical practice, covering a range of patient populations, specialties and countries. This paper provides a brief overview of the information presented in the Companion Guide, reflecting on the experiences of PRO implementation in practice. From this, researchers can access the full Companion Guide to gain further case study details.
A potential limitation with this work is that advances may have occurred since the case studies were presented; for instance, advances in technology may have provided opportunities for improving practice, solutions to current challenges, or potentially presented additional obstacles that were not known when this companion guide was initiated. Thus, the aspiration is that the Companion Guide will continue as a living repository for PRO researchers and practitioners and will be updated as new solutions become available, new challenges are faced, and new case studies are available to reflect upon.
The full Companion Guide, and User’s Guide, can be accessed at http://www.isoqol.org/research-publications-landing-page/isoqol-publications. ISOQOL also offers a number of other education courses, webinars and training—these can be accessed at http://www.isoqol.org/education-events.
Acknowledgements
This paper is produced on behalf of the International Society for Quality of Life Research (ISOQOL). All authors are members of ISOQOL. The manuscript was reviewed and approved by the ISOQOL Board of Directors as an ISOQOL publication and does not reflect an endorsement of the ISOQOL membership. We would like to acknowledge the time and effort of our case study contributors in supporting the creation of this Companion Guide.
Footnotes
Conflict of interest The authors declare that they have no conflict of interest.
Ethical approval All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
References
- 1.Chen J, Ou L, & Hollis SJ (2013). A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Services Research, 13, 211. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, et al. (2004). Measuring quality of life in routine oncology practice improves communication and patient wellbeing: A randomized controlled trial. Journal of Clinical Oncology, 22(4), 714–724. [DOI] [PubMed] [Google Scholar]
- 3.Boyce MB, Browne JP, & Greenhalgh J (2014). The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: A systematic review of qualitative research. BMJ Quality and Safety, 23(6), 508–518. [DOI] [PubMed] [Google Scholar]
- 4.Haywood KL, Garratt AM, Carrivick S, Magnall J, & Skevington SM (2009). Continence specialists use of quality of life information in routine practice: A national survey of practitioners. Quality of Life Research, 18(4), 423–433. [DOI] [PubMed] [Google Scholar]
- 5.Lohr KN, & Zebrack BJ (2009). Using patient-reported outcomes in clinical practice: Challenges and opportunities. Quality of Life Research, 18(1), 99–107. [DOI] [PubMed] [Google Scholar]
- 6.Valderas JM, Kotzeva A, Espallargues M, Guyatt G, Ferrans CE, Halyard MY, et al. (2008). The impact of measuring patient-reported outcomes in clinical practice: A systematic review of the literature. Quality of Life Research, 17, 179–193. [DOI] [PubMed] [Google Scholar]
- 7.International Society for Quality of Life Research. (2015). User’s guide to implementing patient-reported outcomes assessment in clinical practice, version 2. Retrieved March 27, 2018, from http://www.isoqol.org/UserFiles/2015UsersGuide-Version2.pdf.
- 8.Guest G, MacQueen KM, & Namey EE (2011). Applied thematic analysis. Thousand Oaks, CA: Sage. [Google Scholar]
- 9.U.S. Department of Health and Human Services FDA Center for Drug Evaluation and Research, U.S. Department of Health and Human Services FDA Center for Biologics Evaluation and Research, U.S. Department of Health and Human Services FDA Center for Devices and Radiological Health. (2006). Guidance for industry: patient-reported outcome measures: Use in medical product development to support labeling claims: Draft guidance. Health Quality of Life Outcomes, 4(1), 79. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Committee for Medicinal Products for Human Use. (2005). Reflection paper on the regulatory guidance for the use of health-related quality of life (HRQL) measures in the evaluation of medicinal products. London: European Medicines Agency. [Google Scholar]
- 11.World Health Organization WH. (2001). International classification of functioning. Disability: ICF. World Health Organization. [Google Scholar]
- 12.Wilson IB, & Cleary PD (1995). Linking clinical variables with health-related quality of life: A conceptual model of patient outcomes. Jama, 273(1), 59–65. [PubMed] [Google Scholar]
- 13.Berwick DM, Nolan TW, & Whittington J (2008). The triple aim: Care, health, and cost. Health Affairs, 27(3), 759–769. [DOI] [PubMed] [Google Scholar]