Do you want to promote recall, perceptions, or behavior? The best data visualization depends on the communication goal

Abstract

Data visualizations can be effective and inclusive means for helping people understand health-related data. Yet numerous high-quality studies comparing data visualizations have yielded relatively little practical design guidance because of a lack of clarity about what communicators want their audience to accomplish. When conducting rigorous evaluations of communication (eg, applying the ISO 9186 method), describing the process simply as evaluating “comprehension” or “interpretation” of visualizations fails to do justice to the true range of outcomes being studied. We present newly developed taxonomies of outcome measures and tasks that are guiding a large-scale systematic review of the health numbers communication literature. Using these taxonomies allows a designer to determine whether a specific data presentation format or feature supports or inhibits the desired audience cognitions, feelings, or behaviors. We argue that taking a granular, outcomes-based approach to designing and evaluating information visualization research is essential to deriving practical, actionable knowledge from it.

INTRODUCTION

Visualizations are important for communicating many types of health data, including test results, risk calculator output, side effect information, and public health risks. Patient portals, mHealth applications, patient education, and online media all incorporate data visualizations. Using well-designed visualizations can mitigate disparities in interpretation by education, health literacy, or numeracy.^1–4

However, recognizing that data visualizations can provide communication benefits in general is not enough to determine which visualization to use in a particular context. In fact, one of the predominant questions in research on visualizations and other data presentation formats has been, “what data presentation format is best?” To answer this question, we need to take the findings from disparate research studies and distill them into practical guidance.

In this perspective, we argue that unreported decisions in designing and evaluating visualizations and highly ambiguous terminologies make it hard for designers, researchers, and communicators to select an information visualization for health purposes. A new set of taxonomies for defining terms and synthesizing the research literature provides a way forward.

PUBLISHED RESEARCH LACKS ESSENTIAL DETAILS

Some of the strongest informatics research on information visualization relies on the ISO 9186 method for assessing comprehensibility for visual symbols.^5,6 To use this method, researchers invite respondents to provide short interpretations that are rated as correct or incorrect by independent evaluators. This systematic data collection and evaluation produces more replicable results than unstructured qualitative approaches. It may also avoid some of the limitations associated with closed-ended multiple choice or true/false questions.⁶

However, it is more complicated to apply the ISO approach than this description might imply. Before a visualization can be evaluated, it must be designed. The researchers or communicators must figure out what type of data they want to communicate. They must decide whether to use a well-known graphic presentation format (such as a bar chart) or design a new visualization. Finally, they must develop a rubric, often grounded in clinical judgment, for rating the patient’s interpretation correct or incorrect. These decisions are rarely explained in detail.

For example, imagine 2 research groups communicating laboratory values to patients by developing a visualization and conducting an ISO 9186 evaluation of comprehension. Both teams show the lab values for the past year in a line graph visualization but make different decisions about the axes and color schemes. When applying the ISO 9186 process, Team 1 decides that the correct and most clinically relevant interpretation is to recognize that the value is increasing over time. Team 2 decides that the correct interpretation is for the patient to recognize that their value puts them in a high-risk category. To add to the confusion, Team 1 asks for the interpretation while the patient is looking at the visualization, but Team 2 has the patient close the webpage before answering questions. In the end, the teams have selected different data, designed visualizations that sound similar but differ on important dimensions, and have used the term “comprehension” to describe several different outcomes. As a result, they might publish studies coming to opposite conclusions about comprehensibility of line graphs.

How then would the informatics community synthesize these studies to provide guidance to future designers, communicators, and researchers?

BREAKING DOWN THE DETAILS IN THE INFORMATION VISUALIZATION LITERATURE

In making sense of this literature, we have argued that it is helpful to think of health numbers communication as a process in which a patient examines a stimulus consisting of some data structure presented in a data presentation format (Table 1).⁷ The patient conducts cognitive tasks to make sense of the stimulus, resulting in perceptual, affective, cognitive, or behavioral outcomes that can be measured with questionnaire instruments, ISO-style written responses or think-aloud protocols, or observations (Table 1). Our research team recently empirically developed taxonomies of each of the underlined terms, on the basis of our recently completed literature review, to help clarify terms. These 4 taxonomies (data structures, data presentation formats, tasks, and outcomes) were developed to cover a specific corpus of articles retrieved during a systematic review, which limits their scope; certain concepts may be missing, and some of our concepts could be further subdivided into smaller concepts if more data (ie, articles) were to be added.⁷ Despite these limitations, we have found this set of taxonomies to be valuable to ensure that our systematic review groups similar research together to synthesize evidence-based guidance. Throughout, we limit our scope to the communication of and responses to numbers and quantities.

Table 1.

Concepts used in describing research studies on the communication of numbers

Concept	Definition
Data structure	The numerical information to be communicated. Examples include absolute probabilities (eg, a risk of 5%), probability comparisons (eg, a relative risk of 2), and absolute probabilities measured multiple times over time (eg, chances of cancer recurrence as they change over time).
Data presentation format	Ways numerical information can be conveyed. Visual formats include data graphics, such as icon arrays and bar charts, together with design features such as axes and data labels. Numerical formats include percentages, frequencies, relative risks, and others.
Task	A cognitive activity conducted by the reader to extract meaning from the data presentation format. For example, a reader seeking to know whether the probability of cancer recurrence rises or falls over time might assess the slope of a survival curve (a “trend” task), whereas a reader wanting to know the survival probability at 10 years might identify the coordinates of a single point on the curve (a “point-level” task).
Outcomes	Measures of constructs that the communication could affect. For example, getting a mammogram in response to a communication is a health behavior outcome that could be measured with a questionnaire or review of medical records.

We focus here on our taxonomy of outcomes, which is perhaps most essential for understanding the information visualization literature. In our taxonomy, the terms “understanding” and “comprehension” (even the common distinction between “verbatim” and “gist” comprehension) are not granular enough to describe the unique, situation-specific questions used by different researchers.⁷Table 2 presents 14 distinct and mutually exclusive cognitive, perceptual, and behavior outcomes we have identified in the literature. We acknowledge that some of these 14 could also be further subdivided (eg, we have grouped concepts such as attractiveness, perceived helpfulness, and perceived understandability under the preference outcome) but found that grouping them for the systematic review helps to demonstrate trends in the literature and develop useful evidence-based guidance. Throughout, “perception” refers to a belief about or interpretation of information (such as whether a 1% probability of disease appears large or small), and “feeling” refers to an affective response to the information (such as whether a 1% probability of disease is frightening).

Table 2.

Outcome measures and example measurement questions

Outcome	Definition	Example Questionsa
Identification	Ability to answer questions about the stimulus, such as locating a particular value, with access to the stimulus.	“What is your asthma control score?”8
Recall	Ability to answer questions about the stimulus, such as remembering a particular value, without access to the stimulus.	“To the best of your ability, can you tell us the lifetime risk of kidney cancer from the website calculators earlier in this survey?”9
Categorization	When given information about classes (such as thresholds for low | medium | high or above average | below average), ability to determine which class a value falls into.	“[On the basis of the thresholds illustrated here,] Is your asthma mild, moderate, or severe?”8
Contrast	Ability to identify the option or attribute with highest (or lowest) value; for example, to determine the most effective treatment option.	“Which cancer treatment option is the most effective in improving health outcomes?”10
Computation	Ability to give a numerical answer to a question requiring calculations, for example, to calculate the arithmetic difference between 2 risks provided to the reader.	[The participant is given probabilities of stroke with and without medication and is asked for a numerical answer to the question] “How much lower is your risk of stroke if you take the recommended medication?”11
Probability or magnitude perceptions	Subjective interpretation of the likelihood of an event (sometimes called “perceived risk”) or size of a quantity, measured on an ordinal or qualitative scale.	“How likely do you think you would be to experience a side effect?”12
Probability or magnitude feelings	Affective response to the event likelihood, such as worry or concern (also sometimes called “perceived risk”); affective response to the quantity.	“How worried would you be about [side effect] if you took tamoxifen?”13
Effectiveness perceptions	Subjective interpretation of the difference between 2 numbers, such as the benefit or harm from an intervention or risk factor, measured on an ordinal or qualitative scale.	“How effective would Drug A be at reducing your child’s chance of experiencing pain after surgery?”2
Effectiveness feelings	Affective response to the difference between 2 numbers.	“Please indicate how significant you believe the reduction of the risk was.”14
Discrimination	Ability to differentiate between different values (particularly small differences) on the basis of differences in the stimulus.	Not directly measured. Instead, computed by the researchers by determining whether perceptions differ when the quantity portrayed by the visualization differs. Intended to determine whether a data presentation format tends to exaggerate or obscure differences in quantities. 15
Behavioral intention	Expressed plan to take an action; decision.	“How likely are you to quit smoking after viewing this information?”16
Health behavior	Completion of a particular health-related action.	May be measured through self-report, EHR data extraction, or observation. For example, observed time spent flossing after receiving a message about risks of gum disease. 17
Preference	Participant-reported affinity for, perceived ease of understanding of, or perceived usefulness of the data presentation format.	“Which of the following graph formats did you prefer?”18
Trust	Trust or confidence in the information.	“How much do you trust that this test result is accurate?”19

Note: Longer definitions of terms used here and throughout the paper are available in Ancker et al⁷ and in slightly updated form in Ancker et al.²⁰

^aEach example question would be part of a larger scenario where participants are provided explicit quantitative information (eg, the probability of disease with and without treatment).

Some aggregate measures observed in previous literature combine different outcomes from our taxonomy in a single item. For example, a “comprehension” score might include questions assessing recall, contrast, and computation outcomes within the same item. This approach is problematic because a visualization may be good at promoting one desirable outcome but poor at promoting another. As one example, a visualization that portrays the numerator of a probability without portraying the denominator improves recall but inflates probability perceptions.²¹ Inconsistencies such as these have led us to avoid the term “comprehension” as overly broad. More specific outcome-focused evaluation measures should bring clarity to the literature.

Another key component of our set of taxonomies is our concept of tasks, or cognitive activities conducted by the patient to extract meaning from a visualization. We differentiate between examining a single data point (point-level tasks), evaluating the difference between 2 points (difference-level tasks), making judgments involving multiple data points, such as assessing a tradeoff between a chance of harm and a chance of benefit (synthesis-level tasks), and interpreting time trend patterns (trend-level tasks). Although visualizations showing a single data point can support only point-level tasks, many complicated visualizations can enable patients to conduct different tasks.

By clarifying the outcome to be promoted, the communicator can determine what data presentation format will facilitate the task a patient will need to do to extract information relevant to that outcome. These decisions help communicators design information visualizations. For example, if a communicator’s primary goal is to help a patient recognize that their hemoglobin A1c level is elevated (a categorization outcome), this outcome can be facilitated with a visualization designed around a point-level task: comparing the data point to relevant contextual information to see if it is in the elevated category. However, if the primary communication goal is to help the patient recognize that the trend has been increasing, a more complex visualization is needed to enable a trend-level cognitive task. This complex visualization would also enable point-level tasks (such as the categorization outcome task), but the communicator will have to decide whether a visualization that supports many different tasks may also create confusion and be harder to use. More detailed mappings of outcomes to tasks are available in our longer paper.²⁰

Guided by these taxonomies, our team undertook an extensive review of research that conducted head-to-head comparisons of 2 or more data presentation formats for communicating health numbers to lay audiences, identifying 391 unique papers to be included. A critical step has been to classify each included study not only by each of the data presentation formats tested but also by each of the specific outcomes assessed and the tasks needed to extract information relevant to that outcome. In this approach, research generates findings that are uniquely defined by the combination of a task, an outcome, the stimuli (data and data presentation format), and a data presentation format comparison.

APPLYING THE TAXONOMIES TO MAKE SENSE OF THE RESEARCH LITERATURE

A few examples demonstrate the value of these taxonomies in unpacking the findings available from information visualization research. Specifically, these examples show that if 2 research teams do not clearly specify the outcomes they want to study, they may come to different conclusions about the same visualization.

In 2017, Zikmund-Fisher et al¹⁵ published a paper on visual displays of laboratory test results. This study compared tabular presentations to number line graphics that located “your result” on horizontal lines with different combinations of color cues, reference point anchors, and evaluative labels (eg, “borderline high”). However, this study did not measure anything labeled “comprehension,” instead focusing on discrimination: “the difference between participants’ subjective sense of urgency when given test results that were either slightly or extremely outside the standard range.” Discrimination is measured by comparing 2 independent observations of magnitude feelings (measured as “how alarming…” and “how urgent…”). A secondary outcome was behavioral intentions (how quickly the respondent would plan to seek medical care). The study also included an aggregate score described as “display format preferences.” However, this measure unfortunately aggregated responses to 3 preference questions (eg, “how helpful were these images…”) and 1 trust question (“how much would you trust what these images are telling you about your health?”), complicating interpretation.

A later paper by the same team focused on whether providing goal range information would improve “comprehension,” operationalized as “reduc[ing] unnecessary negative reactions to test results that are outside of the standard range, but near their goal range.”²² The “comprehension” measures assessed a contrast outcome by asking respondents whether their result was above, within, or below the goal range and whether their future results should ideally be higher or lower than the current results. The study did not ask participants to either identify their result or categorize that result into any framework. The study measured magnitude feelings but framed the question in terms of encouragement (“how discouraged or encouraged…”) rather than the “how alarming…” language of Zikmund-Fisher.¹⁵

By contrast, Arcia et al⁸ used very different outcome measures to assess similarly designed visual displays of various measures of asthma control. The goal was to identify the elements that affected “comprehension,” and the authors modified the ISO 9186 method to account for the complexity of the target information visualizations. We classify their questions on “both gist and verbatim comprehension” as identification tasks (“what percentage of your lung function do you have?”) and categorization tasks (“Is your asthma mild, moderate, or severe?”; “Is your asthma controlled or not?”). We classify this last question as a categorization task because the visual stimulus included 2 color-coded ranges labeled “controlled” and “not controlled.”

Examining these parallel lines of research through the lens of our outcome taxonomy provides clarity about what can or cannot be learned. In terms of “comprehension” outcomes, only the categorization questions in the Scherer et al²² and Arcia et al⁸ papers are measuring similar cognitive tasks, while the contrast question in Scherer et al²² cannot be compared to the identification tasks of Arcia et al.⁸ At the same time, the magnitude feelings findings in the Scherer et al¹⁵ and Zikmund-Fisher et al²² papers appear appropriate for comparison, even though they were measured using somewhat different questions.

Similar nuances appear when examining other types of visualization studies separately by each relevant outcome. For example, 2 studies comparing visualizations presenting multiple survival curves found that expanding the graphed time period increased treatment effectiveness perceptions.^23,24 Yet one of the studies found no effect of the time period on the ability to select the curve showing the highest survival (contrast outcome).²³ So, does time period matter? It depends on what outcome you care about.

COMMUNICATORS MUST DEFINE DESIRED OUTCOMES BEFORE DESIGNING INFORMATION VISUALIZATIONS

Perhaps the single most important takeaway from our systematic review is that the “best” data presentation format depends heavily on the outcome measured. Data presentation formats that support optimal data recall often differ from those that amplify probability or magnitude perceptions. Data presentation formats that appear to optimize ability to compare 2 numbers (contrast) do not always score highly on preference.

As a result, we emphasize the importance of selecting relevant, granular outcomes that match predetermined communication goals as a necessary precursor to deriving practical guidance from visualization-related studies. Our taxonomy also covers some nomenclature for the design features of visualizations (such as axes and data labels), but additional work is needed in this area to ensure that not only the outcomes but also the visualizations can be compared across studies. Communicators should ask themselves: “What do I want the recipient to think, feel, or do after receiving this information?” This question is how the communicator articulates their goal for the communication.

Our approach is therefore highly congruent with that of Adar and Lee,²⁵ who urge designers to develop visualizations around visual learning objectives. The similarities between our taxonomies (developed empirically during a systematic literature review of both numerical and graphical communication, launched in 2018) and that of Adar and Lee (drawn from interviews with visual information designers in the 2020s), despite some differences, reinforce the common lesson that the optimal design for visualizations depends upon the communicator’s goal.

Applying a standard set of taxonomies can ensure that studies on visualizations are grouped together for synthesis when they have similar tasks, outcomes, and data presentation formats. Conversely, using these taxonomies can help separate studies that might appear similar on their surface but are conceptually different. Only by being precise about the tasks that readers perform and the outcomes that are desired can we identify consistent research evidence and avoid mistakenly overgeneralizing. Doing so helps us avoid unhelpfully vague questions such as “what data presentation format is best for presenting probabilities?” Instead, it helps guide the field toward practical knowledge by inviting us to ask, “Given the information I have at hand, what data presentation format is best for promoting my communication goal?”

FUNDING

This work was funded by the National Library of Medicine (R01 LM012964, PI: JSA).

AUTHOR CONTRIBUTIONS

All authors jointly conceived the outline for this perspective. JSA and BJZ jointly drafted the manuscript with substantial input from NCB.

CONFLICT OF INTEREST STATEMENT

The authors have no competing interests associated with this work.

DATA AVAILABILITY

No new data were generated or analyzed in support of this research.