Table 2.

Summary of data extraction: visualization strategies and preferences, interpretation accuracy, comparators; use of PRO data on individual and group level, in clinicians

Author	Primary study goal	Study population	Presenting PROMs data		Visualization		Outcomes of included studies
			Type of PROMs	What is being presented?	Graphic visualisation format	Comparator	Preferences	Interpretation accuracy
Individual level PROMs data visualization, clinicians
Brundage [16]	To investigate the interpretability of current PRO data presentation formats	N = 50 patients with variety of cancer diagnoses; N = 20 clinicians in active practice, from Johns Hopkins Clinical Research Network (JHCRN)*	EORTC-QLQ-C30 scores	Individual scores and group means	Two line graphs:	Previous scores	Preference for line graphs: overall ease-of-understanding and usefulness	Interpretation accuracy was high across formats
					Higher = better functioning		90% of clinicians preferred formats displaying multiple time-points vs single time-points	Inconsistency between whether higher scores were better or worse contributes to incorrect accuracy (Uptrend lines intuitively signify improvement of understanding)
					Higher = more symptoms		Tabulated scores were considered boring but straightforward
					Bubble plot
					Heat maps
Brundage [37]	To evaluate the interpretation accuracy and perceived clarity of various strategies for displaying clinical trial PRO findings	Oncology clinicians (N = 233) and PRO researchers (N = 248), from JHCRN*	PRO results from hypothetical clinical trial (cross-sectional, longitudinal)	Longitudinal individual data, proportions	Bar chart	Line graph also normed against general population	Not one approach for either longitudinal data or proportions changed is universally appealing, nor is free of misinterpretation errors	Line graph:
					Pie chart			More likely to be interpreted correctly “better” vs “normed” graphs (p = 0.04)
					3 Line graphs:			No differences between “better” and “more”
					1) Higher = “better” functioning;			Regardless of graph type and version, adding asterisks for clinical significance and confidence limits did not contribute to better interpretation accuracy
					2) Higher = “more” symptoms;			Bar chart/pie chart:
					3) “Normed” against general population			Respondents less likely to make interpretation errors with pie vs bar charts (p < 0.001)
								Odds of selecting an “incorrect” treatment significantly lower in pie charts compared to bar charts
								Clarity ratings did not differ between formats
Damman [14]	To investigate:	Interviews: patients with Parkinson's disease (N = 13) and clinicians (N = 14)	Not specified	Individual scores	Line graph	Patients with the same age, gender and disease duration	Strong preference for individual PROMs data over time	Individual PROMs scores with comparative data of similar patients were found useful, some expressed doubts
	(a) How patients and clinicians think about using PROMs during consultations;	Survey: patients (N = 115), the Netherlands			Bar graph		Line and bar graphs
	(b) For which purpose patients and clinicians use PROMs during consultations;				Line graph with comparative data over time (i.e. average scores of similar patients)		Scores from repeated measurements over time
	(c) How patients interpret PROMs information presented in various formats						Multiple individual quality of life, rather than one overall quality of life score
							Identified the possibility to use aggregated PROMs scores as evidence for treatment options
Grossman [27]	To identify the design requirements for an interface that assists patients with PRO survey completion and interpretation; to build and evaluate the interface of PROMs feedback	Interview: N = 13 patients with heart failure and N = 11 clinicians, study location or country was not described	Health IT Usability Evaluation Scale (Health-ITUES)	Individual scores	Small cards: Contains a short sentence to describe a severe symptom, which when clicked on provides textual educational information	N/A	Perceiving the mockup as useful and easy-to-use	Two providers reported that PROs might reduce their cognitive load
		Usability testing: N = 12 patients with heart failure			Graph:		Preference for tracking symptoms over time
					Bar chart that lists the patient’s symptoms from most to least severe and displays each symptom’s severity score
					Large cards:
					Displays a symptom name, its two-sentence description, a visual representation of its severity, and a link to textual educational information
Hartzler [36]	To share lessons learned from engaging clinicians to inform design of visual dashboards	Clinicians: N = 12 for interviews, N = 40 for surveys and consensus meeting, N = 9 for user testing, study location or country was not described	Not specified	Individual scores	PRO data needs appear to differ for health care providers and administrative staff as key target users	N/A	Participants liked viewing trends over time	Value was found in developing meaningful ways to report on this new source of data
					The functional prototype has 3 components:		Participants found the views to provide a useful basis for comparison	In addition to the information buttons provided on “how to interpret this chart,” clear labels are needed, such as on chart axes
					(1) An “At a glance” screen providing a simple data overview of PROs data
					(2) An “Analyze” screen providing a data view the user can filter
					(3) A “Data quality” screen
Hartzler [6]	To conduct a HCD to engage patients, providers, and interaction design experts in the development of visual “PRO dashboards” that illustrate personalized trends in patients’ HRQoL following prostate cancer treatment	Focus groups (N = 60 patients)	Not specified	Individual scores	Pictographs	The dashboard compares patients’ trends with trends from “men like you” matched by default by age and treatment derived from a previously published prostate cancer cohort	Pictographs less helpful than bar charts, line graphs, or tables (P < .001)	Pictographs might reach patients with limited literacy
		N = 50 prostate cancer patients and N = 50 clinicians, study location or country was not described			Bar charts		Preferred bar charts and line graphs	Some participants, both patients and providers, expressed concern over inclusion of comparison scores without representation of data variability (e.g., confidence intervals, error bars), while others preferred simpler charts and graphs
					Line graphs
Izard [3]	To develop graphic dashboards of questionnaire responses from patients with prostate cancer to facilitate clinical integration of HRQoL measurement	N = 50 prostate cancer patients and N = 50 providers from Seattle, USA	Expanded Prostate Cancer Index	Individual scores	Bar chart	Previous scores; ‘patients like me’	No universally preferred dashboard format: 30% preferred tables, 34% preferred bar charts, and 34% preferred line graphs	Helpfulness and confidence ratings varied among dashboard format. Pictographs had the lowest helpfulness compared with table, bar, and line graph formats
					Line graph
					Table that display HRQOL data in raw form
					Facial expression pictograph
Jagsi [35]	To investigate practicing oncologists view on incorporating routine collection of PROs into cancer care	N = 17 oncologists, USA	Edmonton Symptom Assessment System	Individual scores	Bar chart	Previous scores	Ability to track symptoms over time and effect of intervention	Keep it simple: limit number of symptoms
								Link number scale to narrative
Kuijpers [4]	To investigate patients’ and clinicians’ understanding of and preferences for different graphical presentation styles for individual-level EORTC QLQC30 scores	N = 548 cancer patients in four European countries and N = 227 clinicians, the Netherlands	EORTC QLQ-C30	Individual scores	Bar chart with color	The preferred comparison group was one’s own previous results (40.9%)	Medical specialist:	Medical specialist:
					Bar chart without color		Heat map (46%)	Objective understanding of 78%
					Line graph with color		Nurses:	Nurses:
					Line graph without color		Bar chart (non-colored) and heat map (32%)	Objective understanding of 74%
					Heat map			85% of all HCP’s indicated that the graphs were (easy) to understand, not differing between professions or graphical presentation styles
								Uniformity in scoring direction would be preferred
								A significant difference for overall change scores, with the non-colored bar charts being interpreted correctly more often than the other graphical displays
Ragouzeos [25]	To develop a “dashboard” for RA patients to display relevant PRO measures for discussion during a routine RA clinical visit	Patients with rheumatology (N = 45) and providers (N = 12), USA	Not specified	Individual scores	Prototype PRO dashboard (on paper)	Previous scores	N/A	Information needs to be clearly focused on what is most essential
								Important to show progress over time
Santana [33]	To describe the process, feasibility and acceptability of use of the Health Utilities Index (HUI) in routine clinical care	Pre- and post-heart and -lung transplant patients (N = 151), Canada	Health Utilities Index (HUI)	Individual scores	HUI score card, using a color-coded system	Pre- and post-treatment scores	N/A	Clinicians did not need much time to understand the use of the HUI score card
								Clinicians developed their own way of using the information over time
Smith [18]	To improve formats for presenting individual-level PRO data (for patient monitoring) and group-level PRO data (for reporting comparative clinical studies)	N = 40 clinicians in active practice and N = 39 patients diagnosed with cancer ≥ 6 months previously, not currently receiving chemotherapy/radiation or within 6 months of surgery, from JHCRN*	Not specified	Individual scores, proportional data	Line graphs	Previous scores	75% preferred the line graph	Ease-of-understanding ratings were high for all formats
					Pie charts			Directional inconsistency emerged as an interpretation challenge
					Bar charts			Format interpretation challenges included explaining the meaning of scores (i.e., whether scores are good/bad, what normal is)
					Icon array
Snyder [34]	To test approaches for presenting PRO data to improve interpretability	N = 627 cancer patients/survivors, N = 236 oncology clinicians, and N = 250 PRO researchers for survey, from JHCRN*	Not specified	Individual scores	3 line-graphs:	Previous scores	N/A	The proportion responding correctly across the 4 directionality items ranged from 80 to 100%
		N = 10 patients and N = 10 clinicians for interviews			1) Green-shaded normal range			Red circles were interpreted more accurately than green shading
					2) Red-circled possibly concerning scores			Higher = better were interpreted more accurately versus higher = more
					3) Red threshold-lines between normal and concerning scores			Threshold-line significantly more likely to be rated “very” clear and most useful compared with green shading and red circles
Group level/aggregated PROMs data visualization, clinicians
Brundage [16]	To investigate the interpretability of current PRO data presentation formats	N = 50 patients with variety of cancer diagnoses; N = 20 clinicians in active practice, from JHCRN*	EORTC-QLQ-C30 scores	Individual scores and group means	Line graph means over time	Two treatments (study arms)	90% of clinicians preferred formats displaying multiple time-points	Line graphs contributed to overall ease-of-understanding and usefulness
					Line graph with norms		Preference for line graphs of normed sores or with confidence intervals	Normed scores provided basis for comparison beyond two treatments, p-values and confidence intervals were particularly important for publications
					Line graph with confidence intervals		Some preference for bar charts to compare treatments	Cumulative distributing function confusing and difficult to interpret
					Bar chart of average changes			Inconsistency between whether higher scores were better or worse contributes to incorrect accuracy
					Bar chart with definition (improved, stable, worsened)
					•Cumulative distribution functions
Damman [14]	To investigate:	Interviews: patients with Parkinson's disease (N = 13) and clinicians (N = 14)	Not specified	Individual scores	Line graph with results of 2 treatment options	Patients with the same age, gender and disease duration	Identified the possibility to use aggregated PROMs scores as evidence for treatment options	N/A
	(a) How patients and clinicians think about using PROMs during consultations;	Survey: patients (N = 115), the Netherlands			Bar chart with results of 2 treatment options		Aggregated PROMs scores for provider options could be useful, but would not be used much in clinical practice
	(b) For which purpose patients and clinicians use PROMs during consultations;				Bar chart with performance of 2 providers
	(c) How patients interpret PROMs information presented in various formats
Liu [28]	To develop Rheumatoid Arthritis (RA) ‘dashboard’ that could facilitate conversations about PROs and is acceptable to a wide range of patients, including English and Spanish speakers, with adequate or limited health literacy	N = 25 RA patients and N = 11 clinicians from two academic rheumatology clinics, California (USA)	(1) Clinical Disease Activity Index (CDAI)	Individual scores	Line graph	Aggregated clinical data	A dashboard is a potential method for aggregating data from various sources	A ‘snapshot’ of relevant information for a particular patient would make HCP’s own medical decisions easier
			(2) Patient-Reported Outcomes Measurement Information System (PROMIS)-physical function scale				Clinicians were very interested in customizing the dashboard to their own needs and recommended that it can be designed to present information that is more detailed
			(3) Pain score
Smith [18]	To improve formats for presenting individual-level PRO data (for patient monitoring) and group-level PRO data (for reporting comparative clinical studies)	N = 40 clinicians in active practice and N = 39 patients diagnosed with cancer ≥ 6 months previously, not currently receiving chemotherapy/radiation or within 6 months of surgery, from JHCRN*	Not specified	Individual scores, proportional data	Line graphs	Average changes	For proportional data formats: pie charts (70%)	Median ease-of-understanding ranged from 6.5 to 8
					Pie charts		Few clinicians (10%) preferred bar charts	Mixed feelings about indications of clinical significance in terms of having p-values in addition to confidence intervals and asterisks indicating important differences
					Bar charts		75% preferred the line graph	Directional inconsistency emerged as an interpretation challenge
					Icon array			Format interpretation challenges included explaining the meaning of scores (i.e., whether scores are good/bad, what normal is)
Tolbert [29]	To identify the association of PRO score directionality and score norming on a) how accurately PRO scores are interpreted and b) how clearly they are rated by patients, clinicians, and PRO researchers	N = 629 patients (various oncologic diagnoses), N = 139 oncology clinicians, and N = 249 PRO researchers, conducted at the Johns Hopkins Clinical Research Network (JHCRN)*	Two treatments	Mean scores	Line graph 3 versions:	Two treatments	“Better” formatted line graphs were rated most often as “very clear” or “somewhat clear” by all three groups (84% by patients, 81% by clinicians, and 85% by researchers)	Answers correct for “better”: 70%; “more”: 65%; “normed”: 65%
					(1) Lines going up indicating “better” outcomes		However, the range in the proportion rating each format “very clear” or “somewhat clear” was narrow: 77% to 85%	“More” line graph comments noted that up could mean different things, which could lead to errors
					(2) Lines going up indicating “more” (better for function domains, worse for symptoms). 3) Lines “normed” to a general population average of 50			“Better” line graph comments pointed out how changing the scale could result in interpretation errors as one must orient to the direction of the scale each time
Tolbert [20]	To identify best practices for presenting PRO results expressed as proportions of patients with changes from baseline (improved/ stable/ worsened) for use in patient educational materials and decision aids	N = 629 patients (various oncologic diagnoses, treated), N = 139 oncology clinicians, and N = 249 PRO researchers, conducted at the Johns Hopkins Clinical Research Network (JHCRN)*	Two treatments	Proportions changed	Pie chart	Two treatments	Preferred pie charts: these were easy to read and enabled obtaining information quickly	Clinician and researchers scored pie charts as the most accurately interpreted
					Bar chart		43% had positive feedback on icon arrays	In general, bar graphs were less accurately interpreted than pie charts and icon arrays
					Icon array		38% had positive feedback on bar charts	Noted helpful aspects of bar charts: “Side
								by side comparisons are much easier to read and comprehend”
van Overveld [19]	To investigate the preferences of receiving feedback between stakeholders	N = 37 patients, medical specialists, allied health professionals and health insurers in the Netherlands	Audit data on professional practice and health care outcomes	National average scores	Bar graph	National average scores	Preference for bar charts since they are easier to read	Box plots, Kaplan–Meier graphs and funnel plots give a less clear overview and are more difficult to interpret
					Pie chart		For survival and process indicators: Kaplan–Meier graphs and box plots	Find a balance between giving feedback and giving too much information
					Line graph			Give an overview of the results first, followed by the details
					Point graph			Present it that one can easily understand without explanation
					Area graph
					Box plot
					Kaplan- Meier graph
					Funnel plot

Definitions: individual level PROMs data—The patient’s perspective on their health status; Group level PROMs data—Aggregated PROM scores collected in clinical studies or trials

Abbreviations: CDAI—Clinical Disease Activity Index; EHR—Electronic Health Record; EORTC-QLQ-C30—European Organization for Research and Treatment of Cancer Quality of life questionnaire Core 30; HADS—Hospital Anxiety and Depression Scale; HCD—Human Centered Design; HRQoL—Health-Related Quality of Life; HUI—Health Utility Index; Health-ITUES—Health IT Usability Evaluation Scale; JHCRN—Johns Hopkins Clinical Research Network; N/A—Not Applicable; PRO(s) —Patient Reported Outcome(s); PROMs—Patient Reported Outcome Measurements; PROMIS—Patient-Reported Outcomes Measurement Information System; QoL—Quality of Life; REALM-SF—Rapid Estimate of Adult Literacy in Medicine Short Form; SD—Standard Deviation; WHOQOL—World Health Organization Quality of Life

^*JHCRN—Johns Hopkins Clinical Research Network: A consortium of academic and community health systems in the US mid-Atlantic with clinics outside the USA as well