Table 2.

Characteristics of the included grey literature

Author/organisation, title, year	Web reference	Type of material	Date accessed	Brief summary/relevant findings identified PRO data barriers identified PRO data enablers
Aaronson et al. User’s Guide to Implementing Patient-Reported Outcomes Assessment in Clinical Practice. Version 2: January 2015.54	https://www.isoqol.org/UserFiles/2015UsersGuide-Version2.pdf	User’s guide	09 July 2019	A User’s Guide developed by a team from the International Society for Quality of Life Research to provide practical guidance for clinicians with an interest in using PROs data in clinical practice. A combination of different tools to facilitate PROs data interpretation were recommended, and their advantages and disadvantages were described. Recommended (eg, tools to aid PROs data interpretation vary depending on whether the patient’s current score or a change in score is fed back). Barriers: a lack of familiarity with PROs data among clinicians, time and resource constraints, fitting the use of PRO data within existing clinical workflows. Enablers: the provision of simple written guidance of PRO scoring meaning (eg, higher scores mean better functioning) can provide a general indication of the meaning of scores, but will not provide information about the clinical importance of results.
Batalden et al. Enabling uptake of a registry-supported care and learning system in the United States: A report to the Robert Wood Johnson Foundation from Karolinska Institutet and The Dartmouth Institute, 2014.44	https://srq.nu/en/welcome/	Technical report	10 July 2019	The authors outlined a synergistic, learning health system model based on a case study from the Swedish Rheumatology Quality Registry whereby several data feedback systems were involved. PRO data were fed forward in a shared information environment and combined with clinical data displayed on a dashboard for outcome evaluation and clinical decision-making Barriers: a lack of interoperability between health information systems coupled with administrative workloads for clinicians, time and resource constraints in clinical practice. Enablers: creating seamless exchange of PROs data across health information platforms, the creation of PROs terminology and data exchange standards to facilitate point-of-care data solutions.
Canadian Institute for Health Information (CIHI). Health outcomes of care: An idea whose time has come, 2012.1	https://secure.cihi.ca/free_products/HealthOutcomes2012_EN.pdf	Technical report	23 July 2019	A report produced by authors from Statistics Canada and the Canadian Institute for Health Information which presented PRO data development options (using several case studies) to address gaps related to health outcomes. The authors included information related to challenges involved with the use of PROs among healthcare professionals. Barriers: concerns related to the need for additional time and resources to facilitate uptake of PROs among clinicians. Enablers: engagement of clinicians, the implementation of incentives to encourage use of PROs data and the need for further research related to case-mix adjustment methods for PROs data.
CIHI. PROMs Background Document, 2015.47	https://www.cihi.ca/sites/default/files/document/proms_background_may21_en-web.pdf	Report	23 July 2019	The authors provided an overview of the coordinated approach to PROMs collection and reporting established in Canada, including the initial implementation steps and a review of the international PROMs landscape. Barriers: a lack of collection of risk adjustment variables and data linkage processes for PRO data. Enablers: implementation of a coordinated, timely reporting approach and the ability to produce comparable PRO data report formats (across jurisdictions) to drive system improvements.
CIHI. CIHI PROMs Forum Proceedings, 2015.53	https://www.cihi.ca/sites/default/files/document/proms_forum_proceedings_-_may_26_enweb.pdf	Forum proceedings	23 July 2019	An outline of the proceedings from a PROMs Forum hosted by the Canadian Institute for Health Information. In brief, the value of targeting PROs data initiates towards clinicians was outlined, including three clinical areas (eg, renal care) in which well-established PROs reporting mechanisms were determined to be most desirable. Barriers: a lack of timeliness for PRO data reporting, data capture delays, reporting biases and a lack of establishment of PRO outcome thresholds/performance targets were identified as a challenge for engaging clinicians. Enablers: leveraging existing infrastructure to facilitate collection and reporting of PROs data and the engagement of clinical champions which were identified as success factors for PROMs initiatives.
CIHI. Patient-centred measurement and reporting in Canada launching the discussion toward a future state, 2017.45	https://www.cihi.ca/sites/default/files/document/visioning-day-paper-en-web.pdf	Technical report	26 July 2019	The authors presented a summary report based on presentations delivered at an invitational visioning day hosted by the Canadian Institute for Health Information. In brief, a common set of priorities for measurement and reporting of PROs data were highlighted among 33 participants. Barriers: a lack of cross-country coordination of PRO data initiatives and limited capacity for clinicians/services/systems to compare results internationally. Enablers: provision of PRO data education, guidelines and work tools targeted towards clinicians to help with the interpretation of routinely reported PRO results and to understand how to improve care delivery.
Cappelleri et al. Patient-Reported Outcomes: Measurement, Implementation and Interpretation, 2014.50	https://www.crcpress.com/Patient-Reported-Outcomes -Measurement-Implementation-and-Interpretation/Cappelleri-Zou-Bushmakin-Alvir-Alemayehu-Symonds/p/book/9781138199590	Book/book chapter	17 July 2019	The authors provided a comprehensive overview of various PRO data elements (eg, measurement validity/reliability, missing data and statistical techniques) that can be used to advance the validation and use of these data. Barriers: issues associated with missing data and response-shift bias were highlighted for PRO datasets. The authors suggested the use of a statistical analysis plan to ensure analyses/reports are insensitive to missing data. Enablers: electronic data capture to minimise missing PROs data, the use of descriptive statistics for presenting PRO scores, exploring the distribution of PRO datasets as an essential elements of data summarisation.
Chen. Integrated Care: Patient reported outcome measures and patient reported experience measures - A rapid scoping review, 2015.42	https://www.aci.health.nsw.gov.au/__data/assets/pdf_file/0009/281979/ACI_Proms_Prems_Report.pdf	Technical report	08 July 2019	A report based on the outcomes of a scoping review that was undertaken to examine the issues of implementing a large-scale PROMs initiative, with a particular focus on patient-centre care in New South Wales, Australia. Barriers: issues related to the phenomenon of response shift for PROs data and a lack of established clinically meaningful cut-offs (eg, particularly for longitudinal data). The author included several analytical methods that can be used to identify these issues. Enablers: stakeholder engagement and generating clinical ‘buy-in’ may enable uptake and use of PROs data if clinicians are educated and trained to understand the relevance of these data and their use for quality improvement purposes.
Clinical Oncology Society of Australia (COSA). Implementing monitoring of patient-reported outcomes into cancer care in Australia - A COSA Think Tank Report, 2018.41	https://www.cosa.org.au/media/332504/cosa_pros_think_tank_report_final.pdf	Technical report	12 July 2019	A report based on the findings from a Think Tank that involved 32 participants and was focused on approaches to embed PRO assessment as part of routine cancer care in Australia. The authors highlighted effective methods for implementing PRO monitoring and discussed the benefits of using PRO data in clinical practice. Barriers: a lack of awareness of PROs and perceptions of risk among clinicians, system-level issues (eg, limited resources, variability of information technology systems), the alteration of clinical workflows to facilitate use of PROs. Enablers: education and training for clinicians (eg, why PRO data are important, how to use these data as part of clinical practice) and engaging clinical champions.
Desomer et al. Use of patient-reported outcome and experience measures in patient care and policy. Belgian Health Care Knowledge Centre, 2018.4	https://kce.fgov.be/en/use-of-patient-reported-outcome-and-experience-measures-in-patient-care-and-policy	Technical report	26 July 2019	A report based on an evaluation of the uses, benefits, barriers and facilitators of PRO and experience measures in clinical practice undertaken by a research team from the Belgian Health Care Knowledge Centre. The authors included an analysis of international initiatives and a review of the peer-reviewed literature along with a set of recommendations to facilitate the introduction of PROs. Barriers: PRO data selection bias (eg, due to cultural or language barriers), lack of interoperability between information technology systems, data reporting time delays, a lack of knowledge about the value of PROs and perceived administrative burden among clinicians. Enablers: using a bottom-up (clinically driven) approach combined with top-down guidance (policy driven) to improve use of PROs in clinical practice, accessible data infrastructure (eg, interactive tools for analyses and data visualisation) and easy-to-read reports linked to concrete actions for clinicians.
Duckett et al. Targeting zero: Supporting the Victorian hospital system to eliminate avoidable harm and strengthen quality of care - report of the Review of Hospital Safety and Quality Assurance in Victoria, 2016.40	https://www.dhhs.vic.gov.au/sites/default/files/documents/201610/Hospital_Safety_and_Quality_Assurance_in_Victoria.pdf	Technical report	26 July 2019	A report based on a review of the governance of quality and safety monitoring and data reporting throughout hospitals located in Victoria, Australia. The review process included stakeholder and expert consultation methods and the authors presented several recommendations, including the establishment of systematic collection of PROMs at a state-level. N/A: information related to barriers and enablers for PRO data was not included.
Duckett et al. Strengthening Safety Statistics: How to make hospital safety data more useful: The Grattan Institute, 2017.49	https://grattan.edu.au/wp-content/uploads/2017/11/893-strengthening-safety-statistics.pdf	Technical report	26 July 2019	A technical report focused on methods to use to enhance the presentation of hospital safety data (in general), which also included information related to PROs data. The author suggested that aggregated data must be presented in a meaningful and simple ways and directed towards appropriate audiences who can take action. Barriers: the inclusion of statistical information and the assumption that clinicians will confidently interpret data without an adequate explanation or tailored training. Enablers: the identification of the audiences’ needs and preferences and use of visual aids (eg, line or bar graphs) were recommended for data reporting purposes to reduce information overload and increase the utility of the data.
Franklin et al. Framework to guide the collection and use of Patient-Reported Outcome Measures in the learning healthcare system, 2017.43	https://egems.academyhealth.org/articles/10.5334/egems.227/	Technical report	09 July 2019	A report outlining the findings based on key informant interviews (conducted with 46 individuals who were actively engaged in the use of PROMs in diverse clinical settings), two interactive web-based discussions and an in-person workshop. The authors presented an implementation framework and included a toolkit of strategies to accelerate collection and use of PROMs. Barriers: altered clinical workflows; limited web-based tools to support real-time scoring and trending of data across clinical settings, lack of data visualisation tools. Enablers: the establishment and availability of population norms and benchmarks for PRO measures to compare data within and between specific clinical groups were outlined as enablers for use among clinicians, quality leaders and health system payers.
Nelson et al. Using Patient-Reported Information to Improve Health Outcomes and Health Care Value: Case studies from Dartmouth, KarolInska and Group Health. Lebanon, New Hampshire: The Dartmouth Institute for Health Policy and Clinical Practice, 2012.39	https://www.researchgate.net/publication/232607583_Using_Patient-Reported_Information_to_Improve_Health_Outcomes_and_Health_Care_Value_Case_studIes_fomm_Dartmouth_KarolInska_and_Group_Health	Technical report	11 July 2019	A peer-reviewed, technical report outlining the feasibility, utility and lessons related to PROs data collection systems. The authors presented three case studies from PRO initiatives based at the Dartmouth-Hitchcock Spine (Lebanon), the Swedish Rheumatoid Arthritis Registry and Group Health Cooperative (Seattle, Washington). Barriers: the need for high adoption, completion and follow-up rates for PROs data, some clinicians may not know what to do with the results and decision support resources (eg, clinical practice guidelines) need to be developed to guide responses to PROs results. Enablers: leveraging PROs by supplementing these data with other information sources (eg, diagnosis data, biometrics), the use of standardised training materials for clinicians and co-designing data displays with end-users to promote utility of PROs among clinicians.
NSW Agency for Clinical Innovation. Patient Reported Measures – Program overview, 2018.46	https://www.aci.health.nsw.gov.au/__data/assets/pdf_file/0004/415219/ACI18050_PRM_ProgOverview_Guide_v1.pdf	Programme overview and guide	05 July 2019	A guide and overview of the Agency for Clinical Innovation Patient Reported Outcome Measures program established in New South Wales, Australia. The document outlined implementation considerations related to PROs. Barriers: PRO data collection and use may be duplicative or burdensome for clinicians, there may be a perceived lack of relevancy or meaning to clinicians, response rate issues. Enablers: routine reporting of PRO data back to clinicians to encourage them to take action. (eg, with the ability to view data in real time and perform analytics of patient populations).
Paxton Partners, Patient-Reported Outcome Measures: Literature scan, personal communication, 2018.	N/A	Report	14 June 2019	A report based on the implementation considerations required for the establishment of a PROMs collection system in Victoria, Australia. The authors included a review of the literature and evidence from the experiences of early PRO data adopters located in other countries and jurisdictions. Barriers: variations in the approaches used to collect PROs data (eg, the PRO measure used, the patient population, format and timing of feedback and the level of aggregation of the data), resources and costs required. Enablers: clinician engagement and the collection of PROs data via integration with existing data collection systems (eg, clinical quality registries) to allow these data to be fed back and used at the micro-level, meso-level and macro-level.
Peterson. Learning and understanding for quality improvement under different conditions - An analysis of quality registry-based collaboratives in acute and chronic care, 2015.52	http://hj.diva-portal.org/smash/get/diva2:871675/FULLTEXT01.pdf	Dissertation	08 July 2019	A dissertation based on the use of Quality Improvement Collaboratives (QICs) in three national registries (which are also used for follow-up purposes) in Sweden. The author used an interactive approach to examine if, and how, QICs contributed to quality improvement in the provision of healthcare. Barriers: issues related to missing or incomplete data in healthcare, lack of motivation among clinical teams, time constraints, staff turnover in clinical settings, clinicians need to know how to extract data from different systems. Enablers: continuous monitoring of a clinical teams’ own data (in general) and ability to retrieve data in real time, learning from others and the formation of ‘Communities of Practice’ during quality improvement initiatives.
Raine et al. Patient-reported outcome measures and the evaluation of services. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health: National Institute for Health Research, 2016.51	https://www.ncbi.nlm.nih.gov/books/NBK361255/	Book/book chapter	16 July 2019	The authors provided an overview of the progress made in relation to PROs and outlined the main challenges that need to be addressed to further the field. Using the experiences and lessons learnt from several large-scale PROMs programs in different countries, the authors describe the role of PRO data and the need to engage clinicians to ensure uptake. Barriers: a lack of high-level evidence in the field of PROMs, a lack of integration of PROs data within health records and existing processes of care, missing or incomplete data (eg, for sensitive questions), score interpretation difficulties and response shift bias. Enablers: the establishment of essential training and education mechanisms for clinicians to strengthen their understanding of PRO data and interpretation of results.
Snyder et al. Testing Ways to Display Patient-Reported Outcomes Data for Patients and Clinicians, 2018.55	https://www.pcori.org/sites/default/files/Snyder054-Final-Research-Report.pdf	Research report	26 July 2019	A final research report produced by a research team from the Patient-Centred Outcomes Research Institute in the USA. Using a three-part mixed methods study, the authors identified and tested a range of approaches for presenting PRO data (individual and group level) to promote understanding among clinicians and patients from cancer treatment settings. Barriers: a lack of information is available to explain the methods used to determine clinical importance for PRO data, a lack of standardisation related to scoring of PRO data and how these data are presented for use in clinical practice. Enablers: the provision of guidelines, work tools and education/training targeted towards different audiences to assist with PRO data interpretation.
Thompson et al. Patient-reported Outcome Measures: An environmental scan of the Australian healthcare sector, 2016.3	https://www.safetyandquality.gov.au/sites/default/files/migrated/PROMs-Environmental-Scan-December-2016.pdf	Final report (environmental scan)	14 June 2019	A report based on an environmental scan of the literature undertaken by authors from the Australian Health Services Research Institute. The authors described status of the collection and use of PROMs initiatives in the Australian healthcare system. Barriers: a limited number of empirical examples of the application of aggregated PROs data for benchmarking purposes were found, patterns and case studies for PRO data collection were found to be variable. Enablers: the provision of timely reports, facilitation of benchmarking workshops for clinicians, a high level of clinician engagement and a willingness to integrate PROs within existing data collection systems.
Williams et al. Patient-reported outcome measures: Literature review, 2016.5	https://www.safetyandquality.gov.au/sites/default/files/migrated/PROMs-Literature-Review-December-2016.pdf	Final report (literature review)	14 June 2019	A report based on the findings from a literature review conducted by researchers from the Australian Health Services Research Institute. The authors describe the international evidence to support the rationale for PROs data collections and different mechanisms used to facilitate collection, data uses and the impact of these data. Barriers: the resources/costs required for PRO data collection, response burden/patient confidentiality concerns (eg, sensitive questions), and time and workload constraints for clinicians to implement PROs into routine practice. Enablers: the use of integrated information technologies to support electronic capture of PRO data and real-time feedback to clinicians, training and support for clinicians to effectively use PRO data (eg, increased familiarity with measures and interpretation of results).
World Economic Forum. Value in healthcare laying the foundation for health system transformation. Cologny/Geneva, Switzerland: World Economic Forum, 2017.48	http://www3.weforum.org/docs/WEF_Insight_Report_Value_Healthcare_Laying_Foundation.pdf	Report	05 July 2019	A report based on a collaborative project undertaken by authors from the World Economic Forum and The Boston Consulting Group whereby the foundational principles of value-based healthcare, including information related to PROs data were described. Barriers: there is a lack of development of enhanced benchmarking methodologies for data (eg, data collected through the use of clinical quality registry infrastructure). Enablers: interoperability issues and semantic interoperability issues related to linking data (in general) and for performing risk-adjustments to draw meaningful inferences from data (ie, which can also be extended to PROs data).

N/A, not available; PROM, patient-reported outcome measure; PROs, patient-reported outcomes.