Healthcare Providers’ Understanding of Data Displays of Clinical Trial Information: A Scoping Review of the Literature

Abstract

Background:

Healthcare providers often encounter clinical trial results in the form of visual data displays. Although there is a robust literature on patient responses to data displays in medical settings, less is known about how providers comprehend and apply this information. Our study provides a scoping review of the literature on providers’ reactions to and perceptions of data displays.

Methods:

We searched article databases (PubMed, PsycINFO, Web of Science, Cumulative Index to Nursing and Allied Health Literature, the Cochrane Library) supplemented by handsearching. Eligible articles were published in English from 1990–2020.

Results:

We identified 15 articles meeting our criteria. Studies with physicians were more prevalent (13/15) than those with other healthcare providers (6/15). Commonly assessed outcomes included objective (10/15) and subjective comprehension (4/15), preference for certain data display formats (6/15), and hypothetical decision-making around prescribing (4/15). In studies that assessed comprehension of clinical trial concepts, scores were average or below what would be considered mastery of the information). Data display formats that were preferred did not always correlate with better comprehension of information; lesser preferred formats (e.g., icon array) often resulted in better comprehension.

Conclusion:

Our findings suggest that healthcare providers may not accurately interpret complex types of data displays, and it is unknown if such limitations affect actual decision-making. Interventions are needed to enhance comprehension of complex data displays within the context of prescription drug professional promotion.

INTRODUCTION

A number of factors can influence the prescribing recommendations and behaviors of healthcare providers. The ability to understand and interpret clinical research to guide therapeutic decision-making is one such factor and a critical skill for healthcare providers who prescribe medications. A systematic review of prescribers’ knowledge and skills in evaluating clinical research results suggested that familiarity with evidence-based medicine concepts and skills, such as understanding statistical concepts or identifying study bias, ranged from low to moderate.¹ A recent qualitative study with 77 U.S. physicians found that understanding of clinical trial concepts and terms displayed in prescription drug promotion was less than optimal.² Further, prescribers may be unduly influenced by different statistical measures used to represent study outcomes.¹ For example, in one large study, physicians rated a cholesterol-lowering drug as more effective and were more likely to prescribe it when the same trial data were presented as a relative risk reduction compared with an absolute risk reduction.³

Visual data displays are commonly used to communicate clinical trial results in clinical trial summaries, medical journals, and prescription drug promotional materials. Data displays vary widely in their presentation formats and complexity,⁴ from basic tables and charts to more complex presentations such as survival curves and forest plots of meta-analysis findings. Although there is a robust literature on patient responses to data displays in medical settings,^5–9 there has been less research attention on how healthcare providers understand and use this information, and more specifically, whether certain visual elements of the data display influence healthcare providers’ perceptions of the drug’s efficacy and prescribing related decision-making. For example, research with patients suggests that poorly or inaccurately designed data displays can increase the potential for denominator neglect,⁹ which is the tendency to focus only on the numerator without placing that information in the context of the larger study population. However, it is unclear if healthcare providers are susceptible to the same misperceptions, and if these misperceptions in turn influence prescribing related decisions or recommendations.

Our scoping review sought to answer the following questions: (1) what is healthcare providers’ understanding and processing of information in data displays used in medical settings; and (2) what characteristics of display design and presentation influence prescribing related decisions? By definition, scoping reviews are a type of evidence synthesis that use systematic methods to identify and map the breath of evidence on a topic.¹⁰As part of this effort, a related goal was to identify evidence gaps in the literature and suggest areas for future research. An understanding of how healthcare providers comprehend, process, and use this information can inform future best practices in developing data displays that can accurately and effectively communicate clinical trial information to those who prescribe medications. This information can be useful for industries that create and disseminate complex data displays of clinical trial information and for agencies that provide evidence-based guidance for their development. Further, gaps in understanding around data displays may inform future education opportunities for healthcare providers who rely on clinical trial results to make prescribing decisions.

METHODS

Our scoping review follows standard methods for the identification, selection, and synthesis of relevant research, described below.¹⁰ While we comment on the limitations to the evidence more broadly, we did not conduct a formal risk of bias assessment. We registered our protocol prior to data collection on Open Science Framework (osf.io/pm8×7).

Data Sources

The authors worked with a health sciences library specialist to develop search strategies in PubMed, American Psychological Association (APA) PsycINFO, Web of Science, Cumulative Index to Nursing and Allied Health Literature (CINAHL), and the Cochrane Library. We developed search terms to identify relevant populations (i.e., physicians, healthcare providers, nurse practitioners, physician assistants), interventions (i.e., visual data displays pertaining to medical interventions), and outcomes of interest, which included comprehension, perceptions, preferences, and decision-making. We conducted our database searches in October of 2020. The full search strategy is available in Supplementary Appendix A. Hand searches for additional publications were also conducted using reference lists from key publications and web tools such as PubMed’s Cited by and Google Scholar. The search period spanned peer-reviewed literature published between January 1990 and October 13, 2020. We used DistillerSR software to manage the article search and selection and for identification of duplicates.

Study Selection

Two reviewers screened titles and abstracts (CW, MB), and any records excluded by one of these reviewers was reviewed by a third member of the team (JT). Titles and abstracts selected for full review were dually reviewed for eligibility. Conflicts regarding inclusion were adjudicated by the lead investigator (JT). We included studies meeting the following criteria: original research, inclusion of healthcare providers who prescribe medication (e.g., physicians, nurse practitioners, physician assistants), and outcomes that assessed comprehension, attitudes, preferences or decisions in the context of exposure to a medical data display. We accepted eligible studies published after 1990, in English, and conducted in the United States or another very high or high human development index nation (http://hdr.undp.org/en/composite/HDI). We excluded systematic or narrative reviews and meta-analyses and primary studies published only in abstract or poster format or as letters.

Data Extraction

One reviewer extracted data, and the lead investigator checked the data for quality and accuracy. We extracted the following: study characteristics and design; intervention and comparison information; and eligible results from each study. We also noted the strengths and limitations of each study.

Data Synthesis

Outcomes from the included studies varied widely in terms of how they were operationalized and measured. This precluded any quantitative or meta-analytic synthesis of data. Therefore, we qualitatively synthesize findings and characteristics across studies. As part of this synthesis, we also identified gaps in the evidence.

RESULTS

Searches of five electronic databases yielded 722 unique citations, and an additional 11 articles were identified from hand searching. We screened 733 titles and abstracts and identified 68 articles for full-text review, of which 53 were excluded. We extracted and synthesized data from 15 articles meeting our inclusion criteria. Figure 1 shows the disposition of the reviewed articles in our study. Supplementary Appendix B provides details for each individual study.

Figure 1:

Article Flow Diagram

Table 1 synthesizes grouped characteristics of the 15 included articles. Studies with physicians were more prevalent (13/15) than those that included other medical providers (6/15), which included physician assistants, nurse practitioners, and medical students or residents. Four of the studies with non-physicians (4/6) also included physicians and made comparisons by provider type. Sample sizes ranged from 33 to 968. Types of data in the displays were varied and included risk of adverse events, survival, health-related quality of life, and treatment benefit; only one study assessed reactions to data from a meta-analysis display. Common study comparisons were type of visual display (e.g., icon array versus other format) and visual display compared with narrative text; less common comparisons included varied length of time, framing (positive versus negative), additive effects (one display versus two), and stylistic data display elements (e.g., shading, colors). Studies also varied regarding the medical condition that was the subject of the study, with a slight focus (5 of 15) on cancer. Other medical conditions were varied and included general surgery, anesthesia, diabetes, heart disease, and stroke. Studies used a variety of designs including a between-subject experimental design (4/15), within-subjects design (4/15), both a between and within-subject design (3/15), cross-sectional surveys (2/15), and qualitative interviews (2/15).

Table 1:

Characteristics of Studies Assessing Prescribers’ Perceptions of Data Displays (N=15)

Characteristic	Number of Studies (%)	Author
Population a
Physicians (MDs, DOs)	13 (87%)	Baicus et al., 2017; Brundage et al., 2011; Elting and Bodey, 1991; Elting et al., 1999; Garcia-Retamero et al., 2016; Garcia-Retamero et al., 2020; Kujipers et al., 2016; Mazur and Hickam, 1993; Moynihan et al., 2018; Petit-Moneger et al., 2017; Raina et al., 2005; Snyder et al., 2017; Zikmund-Fisher et al., 2019
Other healthcare providers (NPs, PAs)	3 (20%)	Elting and Bodey, 1991; Kujipers et al., 2016, Snyder et al., 2017;
Medical students/residents	3 (20%)	Friederichs et al., 2014; Mazur and Hickam, 1993; Windish et al., 2007
Type of data in display a
Risk of disease/adverse effects	3 (20%)	Baicus et al., 2017; Garcia-Retamero et al., 2016; Garcia-Retamero et al., 2020;
Health-related quality of life/PROs	3 (20%)	Brundage et al., 2011; Kujipers et al., 2016; Snyder et al., 2017;
Mortality/survival	3 (20%)	Elting and Bodey, 1991; Mazur and Hickam, 1993; Windish et al., 2007
Treatment benefit/effectiveness	2 (13%)	Elting et al., 1999; Zikmund-Fisher et al., 2019
Changes over time/presentation of multiple time periods	2 (13%)	Baicus et al., 2017; Brundage et al., 2011
Disease or infection rates	2 (13%)	Elting and Bodey, 1991; Friederichs et al., 2014
Meta-analysis outcomes	1 (7%)	Raina et al., 2005
Other (test sensitivity, unspecified clinical trial data, clinical practice feedback data)	3 (20%)	Friederichs et al., 2014; Moynihan et al., 2018; Petit-Moneger et al., 2017
Data display comparisons a
Types of visual display (e.g., bar chart, line graph, icon array, table)	6 (40%)	Brundage et al., 2011; Elting and Bodey, 1991; Elting et al., 1999; Kujipers et al., 2016; Petit-Moneger et al., 2017; Zikmund-Fisher et al., 2019
Visual vs. no-visual (narrative, numbers)	4 (27%)	Elting and Bodey, 1991; Friederichs et al., 2014; Garcia-Retamero et al., 2016; Garcia-Retamero et al., 2020
Display formatting or presentation (e.g., color, shading, gradient, threshold lines)	3 (20%)	Kujipers et al., 2016; Snyder et al., 2017; Zikmund-Fisher et al., 2019
Framing (positive vs. negative response)	1 (7%)	Elting et al., 1999
Length of time (e.g., 1-year vs. 5-year risk)	1 (7%)	Baicus et al., 2017
Additive information (e.g., single vs. multiple displays)	1 (7%)	Baicus et al., 2017
Other characteristics (varied denominators, area under curve, type of measure)	4 (27%)	Garcia-Retamero et al., 2020; Hickam, 1993; Mazur and Raina et al., 2005; Petit-Moneger et al., 2017
Not applicable	3 (20%)	Brundage et al., 2011; Moynihan et al., 2018; Windish et al., 2007
Study Design a
Studies with randomization
Between-subjects comparison	4 (27%)	Baicus et al., 2017; Friederichs et al., 2014; Garcia-Retamero et al., 2016; Garcia-Retamero et al., 2020
Within-subjects comparison	4 (27%)	Elting and Bodey, 1991; Mazur and Hickam, 1993; Petit-Moneger et al., 2017; Zikmund-Fisher et al., 2019
Both between- and within-subjects comparisons	3 (20%)	Elting et al., 1999; Raina et al., 2005; Snyder et al., 2017
Studies without randomization
Cross-sectional survey	2 (13%)	Kujipers et al., 2016; Windish et al., 2007
Qualitative interviews	2 (13%)	Brundage et al., 2011; Moynihan et al., 2018; Snyder et al., 2017
Medical condition context
Cancer	5 (33%)	Brundage et al., 2011; Elting and Bodey, 1991; Elting et al., 1999; Kujipers et al., 2016; Snyder et al., 2017;
Other or general	10 (67%)	Baicus et al., 2017; Friederichs et al., 2014; Garcia-Retamero et al., 2016; Garcia-Retamero et al., 2020; Mazur and Hickam, 1993; Moynihan et al., 2018; Petit-Moneger et al., 2017; Raina et al., 2005; Windish et al., 2007; Zikmund-Fisher et al., 2019
Setting
U.S.	7 (47%)	Elting and Bodey, 1991; Elting et al., 1999; Mazur and Hickam, 1993; Moynihan et al., 2018; Snyder et al., 2017; Zikmund-Fisher et al., 2019; Windish et al., 2007
Europe	6 (40%)	Baicus et al., 2017; Friederichs et al., 2014; Garcia-Retamero et al., 2016; Garcia-Retamero et al., 2020; Kujipers et al., 2016; Petit-Moneger et al., 2017
Canada	2 (13%)	Brundage et al., 2011; Raina et al., 2005

^aCategories are not mutually exclusive such that more than one classification can apply within a single study; DO=Doctor of Osteopathic Medicine; NP=nurse practitioners; PA=physician assistant; PROs=patient-reported outcomes. Refer to Appendix B for details on individual studies.

Table 2 provides a synthesis of measured outcomes across the 15 studies. The most commonly assessed outcomes in this literature (Table 2) were objective comprehension of information (i.e., accuracy of response), subjective comprehension of information (e.g., perceived understanding), preferences for alternate data display presentations, and hypothetical decision-making (e.g., drug prescribing) Other studied outcomes included confidence in applying or using the information in the displays, acceptability of data displays for use with patients, comprehension assessment response time, and other attitudes and perceptions related to the data displays. We describe key findings according to each outcome category.

Table 2.

Outcomes Measured in Included Studies (N=15)

	Objective Comprehension	Subjective Comprehension	Preference	Hypothetical Decision-making	Response Time	Other Attitudes/Perceptions	Confidence in Applying/Using
Baicus et al., 2017				X
Brundage et al., 2011			X			X
Elting and Bodey, 1991	X		X		X
Elting et al., 1999	X		X
Friederichs et al., 2014	X						X
Garcia-Retamero et al., 2016	X				X
Garcia-Retarmero et al., 2020	X	X
Kuijpers et al., 2016	X	X	X
Mazur and Hickam, 1993				X
Moynihan et al., 2018	X					X
Petit-Moneger et al., 2017		X		X
Raina et al., 2005	X			X			X
Snyder et al., 2017	X	X	X
Windish et al., 2007	X					X	X
Zikmund-Fisher et al., 2019			X			X

Objective and Subjective Comprehension

Ten studies assessed objective comprehension of information presented in data displays.^2,11–19 Three of these also studied subjective comprehension,^15,16,18 while one study captured subjective comprehension alone.²⁰ In four studies that tested icon arrays compared to other formats, the icon array consistently outperformed other formats.^11–15 In these four studies, icon arrays conveyed information about risk likelihood by using a matrix representation of individual units (people) within the larger at-risk population. One study tested and concluded that addition of a tree diagram (depicting how calculations on test sensitivity and specificity are derived) improved objective comprehension over numeric probabilities alone.¹³ In studies that assessed subjective comprehension, participants generally perceived information to be clear or easy to understand, regardless of display type.

In one study, specialty physicians performed better on objective comprehension tasks than nurses and paramedical professionals,¹⁶ while another found that nurses and lab technicians performed better than physicians.¹¹ In the latter, nurses and lab technicians had the highest scores with icon arrays, while physicians had the highest scores with tables.¹¹ In the former study, the discrepancy between subjective understanding of the data and accuracy in understanding for nonphysicians was large: 85% responded that the information was easy to understand, while accurate understanding was 35%.¹⁶ We identified one study that also assessed the influence of numeracy: the accuracy of comprehension scores for less numerate surgeons improved when they were shown an icon array rather than numerical information alone.¹⁴

In four studies that either assessed objective comprehension of clinical trial terms^2,17,19 or asked participants to calculate statistical quantities using data¹³, accuracy varied and overall was lower than what would be considered mastery of the information Aspects of data display design associated with greater objective comprehension of clinical trial information included presentation of equal denominators across displays,¹⁵ data tables that used negative as opposed to positive framing of outcomes,¹¹ axes where higher values indicated better outcomes rather than larger quantities, and threshold lines or red circles signaling concerning outcomes.¹⁸

Preferences

Six studies assessed preferences when comparing two or more types of displays, and four of these also assessed objective comprehension of information. Of these studies, two found that although icon arrays produced the greatest comprehension, they were not the preferred display.^11,12 Almost a quarter of participants in these two studies noted that they disliked the icon array format despite improved comprehension compared with pie or bar chart formats.¹² Tables were most preferred in both studies, despite placing second to icon arrays in comprehension. In another study, participants preferred heat maps even though they had higher comprehension scores with noncolored bar charts.¹⁶ In two studies that assessed preferences without comprehension, participants expressed a preference for line graphs over tables and bar charts,^21,22 particularly when displaying information on change over time.²¹

Decision-Making

Four studies assessed hypothetical decision-making about treatments and prescribing behavior.^17,20,23,24 In one study of prescription drug effectiveness data, healthcare providers’ motivation to change elements of their practice was associated with display format: pictographs (i.e., icon arrays) and bar charts increased motivation more than linear “sliders”, despite icon arrays being rated least clear on subjective comprehension measures.²⁰ In other studies, participants were more likely to say they would prescribe when graphs depicted a 5-year risk time frame as opposed to a 1-year risk time frame²³ and when meta-analysis data were more homogeneous or yielded a larger effect size.¹⁷ In another study, prescribers who viewed survival curves comparing two treatments reported being more influenced by the areas under the curve rather than the endpoints.²⁴

Other Outcomes (Response Time, Confidence, Perceptions/Attitudes)

In two studies that evaluated response time for comprehension questions, participants consistently had the fastest response times and greatest accuracy with icon arrays.^11,14 Across three studies that evaluated confidence with statistical concepts, participants reported low to moderate confidence with certain calculations such as number needed to treat and positive predictive value.^13,17,19 Participants felt more confident using more basic concepts, such as P values,¹⁹ and interpreting results with greater homogeneity or effect sizes.¹⁷ Other findings included a desire to learn more about statistics and acknowledgment that statistics encountered in research are difficult to comprehend¹⁹ and the perception that clinical trial data often lack enough detail to assess its validity.²¹ In one study, participants’ preferred formats were the same as those they perceived as acceptable for use with patients, such graphs featuring a goal range for target A1C levels (featured as a green shaded background area behind a grey target line) rather than the goal only display (grey target line).²²

DISCUSSION

We identified very few studies of healthcare providers that assessed understanding and processing of clinical trial data presented as visual displays. While numerous factors influence safe and appropriate prescribing decisions, accurate understanding of clinical trial information is a critical and understudied skill in the practice of evidence-based medicine. Numbers of studies identified in our review were small relative to the likely thousands of articles published on medical therapeutics during this same time frame. Our search for relevant research suggests that the topic is greatly understudied or is perhaps a nascent field of research, because about one-half of the identified 15 studies were published within the past 5 years (from 2016 to 2020). Conversely, a robust body of literature exists on patient understanding of and best practices related to data displays in medical contexts.^7,9,25–27 In a recent systematic review in which the authors concluded that well-designed visual aids can be very effective tools for improving informed decision-making, healthcare providers comprised only 6% of the study population across the 36 studies.²⁸ Thus, it is not clear if the same best practices designed with patients in mind are applicable to healthcare providers. For example, studies with patients suggest a “less is more” approach that favors simplifying data displays to increase comprehension.^5,7,8 However, this approach may not be beneficial to healthcare providers, because providing too little information can fail to convey the complexity or nuances of the data.⁴

We identified four key gaps in this body of evidence with healthcare providers. First, most studies included only physicians; fewer included other healthcare providers who prescribe medications, such as physician assistants and nurse practitioners, and none exclusively focused on nonphysician prescribers. Thus, it is not clear if areas of understanding (or misunderstanding) vary by type of medical training or profession among those who are licensed to prescribe medical interventions. Of interest were findings of one study where the gap between perceived understanding of information presented in a data display and accurate comprehension was large for non-physicians¹⁶, pointing to the need for education around clinical trial concepts for other professionals who prescribe. Second, studies that assessed decision-making relied on hypothetical or simulated scenarios; no studies assessed actual decision-making. Although the use of hypothetical or simulated scenarios (or vignettes) is a common methodology, the extent to which hypothetical prescribing translates to actual prescribing is not clear given the myriad of influences on this behavior. Third, our review identified only one study that assessed understanding of data displays presenting meta-analysis results (a statistical method that combines and analyzes results of all discoverable trials on the same topic). Given the growing importance of applying meta-analysis results to the practice of evidence-based medicine,^29,30 understanding how clinicians comprehend data displays often used in meta-analyses (e.g., forest plots) warrants further exploration. Similarly, evidence on how healthcare providers comprehend information in survival curves was also scant, and no studies assessed comprehension of different clinical endpoints in data displays such as overall survival versus progression-free survival.³¹ Findings from our review suggest that icon arrays may be ideal for communicating exact percentages while also conveying “gist” impressions of risk likelihood. However, icon arrays are not typically used to convey information on medication benefit or efficacy, and this format was often the least preferred by prescribers. When considering which formats to adopt, preference is secondary to comprehension of the information. Lastly, we did not identify any studies that examined how healthcare providers comprehend visual information conveying statistical uncertainty, such as interpretation of large confidence intervals. Evidence is scant about the best way to visually depict statistical ambiguity, signaling the need to explore this topic more deeply.³²

The small and varied body of evidence precluded us from drawing quantitative conclusions across studies. However, we offer several observations. Findings suggest that healthcare providers may not accurately interpret complex types of data displays; they tend to prefer display presentations that are less likely to be interpreted correctly, and they may lack confidence in their ability to understand statistical concepts and apply them to their practice. It is plausible that these limitations to comprehension can adversely affect decision-making, although this hypothesis has not been tested. Certain characteristics of the data displays themselves, such as framing outcomes as negative versus positive events or displaying longer versus shorter time frames, influenced study outcomes. These findings suggest that careful attention to data display details should be considered during their development to communicate findings effectively and clearly.

Limitations to our study were that despite use of a comprehensive database search strategy, we identified only 15 articles that met our criteria, several through hand searches, indicating that this literature is inconsistently indexed in databases of peer-reviewed literature. It is possible that we did not identify all relevant literature, but our thorough search methods should have mitigated this possibility. Our review was limited to peer-reviewed manuscripts published in English and highly-developed counties, although we do not suspect that these criteria would have eliminated large numbers of otherwise eligible studies. Second, we did not conduct a formal risk of bias assessment on our included studies, as would be done with a systematic review.¹⁰ However, our cursory review of the individual studies’ quality suggests that many are limited by use of small, convenience samples and idiosyncratic measures.

CONCLUSION

Our scoping review of the literature suggests that best practices are needed for developing data displays that visually communicate complex clinical trial information to prescribers. Further research is needed to guide these practices that is inclusive of nonphysician prescribers and rigorously evaluates comprehension of information and concepts, including data displays of meta-analysis results and visual presentations of statistical uncertainty. Such research should be informed by the large evidence base on patient understanding of data displays and best practices for their design more generally^{6,9,25,27,33,34} but adapted and tested for use with healthcare providers. Further, our findings could suggest the need for opportunities to better educate providers on interpreting complex data displays of clinical trial information within the context of prescription drug promotion.

LESSONS FOR PRACTICE.

• Healthcare providers’ understanding of clinical trial information in data displays was low to moderate, and preferred display format did not always correspond with comprehension.

• Best practices are needed for developing visual data displays that accurately and effectively communicate clinical trial information.

• Interventions are needed to enhance comprehension of complex data displays among healthcare providers.