. 2020 Nov 23;10(11):e038190. doi: 10.1136/bmjopen-2020-038190

Table 1.

Characteristics of the included academic literature

Author, year, country	Study method	Clinical area	Number of participants or included studies	Study aim/design	Relevant findings	Review question related to	JBI level of evidence	Strength of evidence
Aiyegbusi et al, 2019, UK³⁰	Semistructured interviews and focus groups	Chronic kidney disease	12 patients with chronic kidney disease and 22 healthcare professionals (nurses, psychologist, nephrologist, registrars and surgeons)	Thematic analysis of participants’ views on the use of a PROM system	Healthcare professionals suggested graphical representations of PROM feedback (rather than numeric), and to include ‘traffic light’ colour-coding for quick and easy review. Healthcare professionals believed that ‘alert fatigue’ from PROM feedback could be a barrier to use of PROM data, with the numerous alerts being provided to healthcare professionals encouraging them to ignore the PROM results.	Q1 Q3	Level 4	Moderate
Allwood et al, 2013, UK³¹	Structured focus groups	All healthcare areas	107 healthcare professionals (including consultants, junior doctors, nurses and allied health professionals)	Thematic analysis of participants’ comprehension and format preference for PROM data	Healthcare professionals were generally positive about the use of bar charts and caterpillar plots for the display of PROM results. Opinions were mixed for the use of tables, funnel plots and spider plots. Healthcare professionals found that tables with icons were insufficient.	Q1 Q2	Level 4	High
Arcia et al, 2018, USA¹²	Case study	Unspecified	2 case studies of PRO feedback projects	Explore methods affecting the design decisions of PRO feedback projects	Summarises considerations that must be understood for the visualisation of PROs data, including the range and direction of scoring.	Q1	Level 4	Very low
Bantun et al, 2016²⁴	Integrated literature review, dates: 1999–2014	Oncology	9 included studies	Exploring the interpretation of graphical presentations of PRO data in clinical practice	HRQOL PROs can be accurately interpreted by healthcare professionals and patients; line graphs and bar charts were the most preferred format for PROs; patients prefer simple graphs, while healthcare professionals prefer simple graphs with CIs.	Q1 Q2	Level 4	High
Boyce et al, 2014²⁵	Systematic review, dates: up to 2012	All healthcare areas	16 included studies	Summarise qualitative studies that explore the experience of healthcare professionals using PROMs	Healthcare professionals value PROMs if they can be used to aid decision making. They appreciate graphical presentations that clearly depict clinically important changes. However, they can question whether the PROM data produced are an accurate reflection of care. Attitude towards the use of PROMs may be improved by engaging the healthcare professionals in the planning stage of PROMs introduction.	Q1 Q3	Level 4	High
Brehaut et al, 2016, Canada²⁹	Opinion	All healthcare areas	68 included studies	Identify suggestions for designing and delivering effective feedback interventions	Barriers: the use of unnecessary three-dimensional graphical elements which can clutter the display and bias the interpretation of the underlying information. Enablers: closely linking visual displays with summary messages, minimisation of extraneous cognitive load for target audiences, the provision of short, actionable messages combined with optional detail and addressing the credibility of the data source used to produce the feedback.	Q1 Q3	Level 5	Low
Brundage et al, 2015, USA⁶	Survey followed by a semi-structed interview	Cancer	50 patients with cancer and 20 oncology healthcare professionals (doctors and nurses)	Explore interpretation accuracy, ratings of ease-of understanding and usefulness of graphical formats. The interview explored helpful and confusing format attributes.	Both patients and healthcare professionals prefer line graphs across group-level data and individual-level data formats (compared with bar charts and cumulative distributions), but healthcare professionals prefer greater detail (ie, statistical details) for group-level data.	Q1 Q2	Level 3	Moderate
Brundage et al, 2018, USA²	Survey followed by an interview with healthcare professionals	Cancer	233 healthcare professionals and 248 PRO researchers	Explore interpretation, accuracy and clarity ratings of graphical formats and difference score representations	Participants were accurate in their interpretation of PRO line graphs when the directionality of the score was indicated with a label ‘better’. Participants were more accurate in their interpretation of pie charts compared with bar graphs, for the display of proportions.	Q1 Q2	Level 3	Very high
Edbrooke-Childs et al, 2016, UK³²	Pre–post observational study	Child mental Health	48 healthcare professionals attended the 1-day training course, 17 healthcare professionals attended the 3-day training course	Evaluate the effect of the training courses on attitudes and self-efficacy towards PROMs and feedback	Increased time and duration of PROMS training showed greater improvement in attitudes towards PROMS, feedback attitudes and PROM self-efficacy.	Q1 Q3	Level 3	Moderate
Forsberg et al, 2015, USA and Sweden²⁶	Case study	Pain and spin conditions, rheumatology, and private healthcare	3 case studies of PRO feedback used in routine practice	Describe the principles and lessons learnt from using PROs in the 3 case studies	Healthcare professionals need to be able to place the PRO results within the context of the patient’s current clinical state, prognosis and attitudes (eg, a patient’s health status may be declining despite receiving best care). Healthcare professionals need to know what to do with the results, such as when the results are suggesting a significant health problem.	Q3	Level 4	High
Hartzler et al, 2016, USA³³	Preliminary focus groups and interviews, followed by a pre–post study	Prostate cancer	The focus group included 60 prostate cancer survivors. 50 patients and 50 providers completed the interviews. 12 patients completed the pre–post observation	The focus groups assessed the needs of patients in relation to PROM feedback. The interviews evaluated preferred feedback methods. The pre–post study evaluated self-efficacy, satisfaction, communication and compliance with the PRO dashboard.	Patients prioritised needs for dashboards to compare longitudinal trends and provide comparative groups. Patients and providers preferred bar charts and line graphs compared with tables and pictographs.	Q1 Q2	Level 2	Low
Hildon et al, 2012, UK³⁴	Focus groups	Knee surgery	45 patients who were planning or had undergone knee surgery	Thematic analysis of patients preferred PROM format	Patients were generally positive about the use of bar charts and caterpillar plots. Opinions were mixed for tables and tables with icons. Patients did not like funnel plots. Patients liked the use of ‘traffic-light’ colour scheme and did not like the use of CIs.	Q1 Q2	Level 4	High
Jensen et al, 2016, USA²⁸	Workshop proceedings	All healthcare areas	519 participants (including patients, healthcare professionals, researchers, healthcare system leaders and policymakers) attended the workshop, either in person or online	Summary of workshop outcomes	Healthcare professionals should be provided with guidance in interpreting PRO scores, as they may not know the meaning of just raw scores. Translate PROs into specific actions for healthcare professionals by establishing clear recommendations on how to respond to PRO scores in clinical settings.	Q3	Level 5	High
Kuijpers et al, 2016, UK, the Netherlands, Austria and Poland³⁵	Questionnaire	Cancer	548 patients with cancer and 227 healthcare professionals (doctors and nurses)	Understanding of PROM scores and preferences for different formats	Patients had no preference between non-coloured bar charts and non-coloured line graphs. Patients preferred coloured bar charts over coloured line graphs. Healthcare professionals showed a preference for line graphs with ‘traffic light’-coloured thresholds. Understanding did not differ between graphical formats for patients or healthcare professionals.	Q1 Q2	Level 4	High
Oliver et al, 2019, Australia, USA and Sweden²⁷	Case study	Multiple sclerosis, spinal care and rheumatology	3 case studies of PRO feedback used in routine practice	Features that aid in the interpretation of PROs in the 3 case studies	The use of colour coding and threshold indicators, linked decision support functions (such as predictive calculators) can aid interpretation of PRO scores.	Q1	Level 4	Very low
Snyder et al, 2019, USA¹¹	Consensus panel	Cancer	Participants included healthcare professionals, PRO researchers, patients and caregivers. 28 participants in meeting 1, and 27 participants in meeting 2 (participants were not mutually exclusive)	A modified Delphi process to develop recommendations for PRO data display	Recommendations for the display of PRO data include using labelling and thresholds, not mixing score direction in a single display, accommodating both normed and non-normed scoring, displaying CIs, indicating possibly concerning results.	Q1 Q2	Level 5	High
Talib et al, 2018, USA³⁶	Interview	Primary care	23 patients in primary care	Thematic analysis of the patient’s perception of the utility of PRO in primary care	The patients found the colour coding severe symptoms useful but recommended the addition of ‘traffic-light’ colour scheme.	Q1	Level 4	High
Van Overveld et al, 2017, the Netherlands³⁷	Semistructured interview	Head and neck	37 patients, healthcare professionals (doctors, nurses, speech pathologist, dietician, allied health) and health insurers.	Content analysis of participants preferred PRO feedback method	Patients want PROs feedback to include explanations of how to read the PRO graph, the inclusion of a comparison and the feedback delivered around once a year. Healthcare professionals want PROs feedback to be simple and include a comparison groups (such as national average, best and worst performer). Healthcare professionals want PROs feedback between 1 and 4 times a year and receive the feedback via email.	Q1 Q2 Q3	Level 4	High
Wu et al, 2016, USA³⁸	Semistructured interview	Cancer	42 patients with cancer and 12 healthcare professionals (doctors and nurses)	Evaluate participants’ views of a webtool that was designed to allow PRO use in clinical practice	Patients and healthcare professionals recommended having PROs score directionality be consistent, and more explanation of the score meaning. Healthcare professionals also recommended including if the score indicates better or worse health.	Q1	Level 4	High

AuSCR, Australian Stroke Clinical Registry; HRQOL, Health-related quality of life; JBI, Joanna Briggs Institute; PRO, patient-reported outcome; PROM, patient-reported outcome measure.