Men's interpretations of graphical information in a videotape decision aid

Abstract

Objective  To examine men's interpretations of graphical information types viewed in a high‐quality, previously tested videotape decision aid (DA).

Setting, participants, design  A community‐dwelling sample of men >50 years of age (N = 188) balanced by education (college/non‐college) and race (Black/White) were interviewed just following their viewing of a videotape DA. A descriptive study design was used to examine men's interpretations of a representative sample of the types of graphs that were shown in the benign prostatic hyperplasia videotape DA.

Main variables studied  Men provided their interpretation of graphs information presented in three formats that varied in complexity: pictograph, line and horizontal bar graph. Audiotape transcripts of men's responses were coded for meaning and content‐related interpretation statements.

Results  Men provided both meaning and content‐focused interpretations of the graphs. Accuracy of interpretation was lower than hypothesized on the basis of literature review (85.4% for pictograph, 65.7% for line graph, 47.8% for horizontal bar graph). Accuracy for pictograph and line graphs was associated with education level,  = 3.94, P = 0.047, and  = 7.55, P = 0.006, respectively. Accuracy was uncorrelated with men's reported liking of the graphs,  = 2.00, P = 0.441.

Conclusion  While men generally liked the DA, accuracy of graphs interpretation was associated with format complexity and education level. Graphs are often recommended to improve comprehension of information in DAs. However, additional evaluation is needed in experimental and naturalistic observational settings to develop best practice standards for data representation.

Introduction

Decision aids (DAs) are becoming more frequently advocated to help communicate key information about health treatment benefits and risks. DAs are a promising extension of standard patient education approaches that can better prepare patients to share decision‐making with health care providers. Although many DAs have been tested for efficacy, ¹ , ² the state of the science is still young regarding the specific mechanisms by which DAs impact knowledge, health behaviour and other outcomes. A variety of presentation formats are being used to make DAs easily understandable and feasible to use. It is particularly important to identify presentation formats that communicate well to people of varying education and literacy levels. ³ , ⁴

Recent ‘best practices’ guidelines for DA design have supported including graphs to enhance communication of data about health risks and treatment side‐effects. An international consensus panel recently compiled a set of recommendations representing the state‐of‐the‐art principles in DA design. The International Patient Decision Aid Standards Collaboration (IPDAS) ⁵ , ⁶ established 12 standards for best practice in DA design. Due to the relative paucity of research, the best practices for graph presentation are phrased in general terms, such as, ‘use of visual diagrams’, and ‘use multiple methods [including diagrams] to view probabilities’. Graphical information is often used to aid comprehension of complex benefit–risk information. However, there is currently limited empirical evidence regarding the specific features of graphs that aid understanding, especially for probability information. The usefulness of graphs depends upon their ability to enhance understanding and reinforce other modes of information presentation such as text and audio. Clarity is critical, especially if a graph is a principal mode of communicating information.

Study objective and hypotheses

We performed a descriptive investigation of the accuracy of interpretation of graphical formats (pictographs, line graphs and horizontal bar graphs) in the context of a previously developed and widely used video DA that used voice‐over, patient interview and narration to reinforce and interpret the data. The goal of this study was to analyse men's interpretations of graphical information that they had just seen presented in a videotape DA for benign prostatic hyperplasia (BPH) treatment. Interviews were conducted in a naturalistic context similar to actual clinical use, including asking the study participants to respond to questions about their decision‐making processes prior to, during, and following their videotape viewing.

The main research question for the analysis reported in this paper concerned men's interpretations of a selected representative subset of pictographs, line graphs and horizontal bar graphs that were presented in the videotape. Accuracy of interpretation was expected to be high (>75% correct) because of multiple exposures to the information during videotape viewing. Relatively higher accuracy rates were also expected for simpler graph formats (graphs with relatively fewer concepts and more visual ‘white space’). The number and types of errors were of special interest to identify the strengths and weakness of graphs to support informed decision‐making. How the men related the content of graphs to their personal situations was also of interest.

Data were obtained as part of a larger project, the Health Information for Patient Decision‐Making (HIPD) study. ⁷ The overall objective of the HIPD study was to examine how a racially and educationally diverse sample of community‐dwelling men in the Midwestern US interpret and use information to reach informed decisions about BPH treatment. African‐American men were over‐sampled, as were men with less than a college education, in order to investigate treatment preferences, and interpretations. We stratified by race^* because race has been shown in the screening literature to be associated with differences in use of health care. ⁸ , ⁹ , ¹⁰ Risk perception and psychological distress differences with screening have been attributed to African‐American values with regard to spirituality, interpersonal relationships and time orientation. ¹⁰ , ¹¹

Methods

Sample and setting

A convenience sample of 188 community‐dwelling men in Michigan of at least 50 years of age was recruited. The men were recruited by postings and personal contacts from community settings such as union halls, clinics, barber shops, fraternal and service organizations, churches, homeless shelters and retiree associations – community settings in which relatively large numbers of middle to older adult age men usually gather. The men were of an appropriate age to be experiencing at least mild symptoms of BPH, and on average reported mild to moderate BPH symptoms but low symptom bother. All subjects self‐identified as either black or white and were fluent in English. A 2 education level (college, non‐college) × 2 race (Black, White) stratified sampling design was used, with a target sample size of approximately 50 men in each design condition. Demographic, clinical and treatment decision‐making data were gathered prior to videotape viewing. Men then viewed the videotape, which was paused following key sections on the treatments to ask interview questions. College‐educated men had higher pre‐videotape and post‐videotape BPH knowledge scores compared with non‐college‐educated men, and had higher health literacy as measured by the S‐TOFHLA. ⁷ Both college and non‐college‐educated men showed a significant and equivalent increase in knowledge scores following viewing the videotape.

Intervention design and procedure

A qualitative descriptive study design was used for the analysis that is reported in this paper. Following the videotape, the men were asked evaluation questions about the videotape, and presented with paper versions of screenshots of three already‐seen graphs that were representative of the different graphs format (pictograph, line and bar) in the videotape. The analysis reported in this paper focuses on men's interpretations of the selected graphs as discussed with the interviewers. Men's overall impressions of the graphs and the accuracy/inaccuracy of men's interpretations of the content of graphs were of interest.

Interviewers and training

Ten male interviewers were matched with participants on race and approximate age. Each interviewer conducted 15–20 interviews. Prior to data collection, the interviewers were trained via role‐playing and pilot interview activities. ⁷

Human subjects protections

The University human subjects Institutional Review Board approved all study materials and procedures. Verbal and written consent for participation were obtained prior to data collection.

BPH videotape

A previously developed and tested educational videotape produced by Health Dialog Corporation (Health Dialog®) and the Foundation for Informed Decision Making (FIMDM) was used. The videotape features an African‐American narrator/educator and testimonials from men with BPH who appear to be of mainstream European or African‐American heritage who provide basic information regarding BPH symptoms and treatment experiences. Graphs in various formats illustrate information that is also provided verbally by the narrator. The 47‐min videotape was last updated in 1999. It has been extensively field‐tested and was developed to present balanced, non‐biased information about key BPH treatment options of watchful waiting, medications and surgery. The videotape includes detailed information about indications for treatments, benefits, risks and men's stories about symptom and treatment experiences. A variety of methods are used to communicate benefit/risk information, including audio narration, visual tables and graphs. The tables and graphs are presented in a variety of formats. This state‐of‐the‐art programme easily meets the IPDAS standards. ⁶ Data were obtained as part of a larger project, the HIPD study. ⁷ The overall objective of the HIPD study was to examine how a racially and educationally diverse sample of community‐dwelling men in the Midwestern US interpret and use information to reach informed decisions about BPH treatment. African‐American men were over‐sampled, as were men with less than a college education, in order to investigate treatment preferences, and interpretations. We stratified by race ^† because race has been shown in the screening literature to be associated with differences in use of health care. ⁸ , ⁹ , ¹⁰ Risk perception and psychological distress differences with screening have been attributed to African‐American values with regard to spirituality, interpersonal relationships and time orientation. ¹⁰ , ¹¹

The videotape was paused following each key section (Introduction, Watchful Waiting, Medications, Surgery, Summary) to interview the men about their treatment preferences. ¹²

Graphs evaluation task

Three graphs (horizontal bar, line and pictograph) were selected to represent the main types of graph formats included in the videotape. As the men viewed the videotape, the narrator provided an audio interpretation of the data included in a given graph while it was displayed on the screen. Colour paper copies of the graphs were printed from the videotape to present to study participants. Graphs were presented in a standard sequence (line, horizontal bar, pictograph). As each graph was presented, the men were asked to respond to the question, ‘Please tell me, in your own words, what this means to you, if anything?’ The purpose of using a relatively broad question (not specifically focused on content of the graph) was to enable a fuller range of responses by the men. This is consistent with the descriptive purpose of the analysis in which we sought to more closely mirror a naturalistic (non‐laboratory based) encounter with the graphical information, such as a man might encounter in a booklet in a clinic waiting area or see on a television news story. At the end of the interview, men were also asked what they liked most and least about the videotape, including the graphs. Responses were audiotaped and transcribed for analysis.

Description of graph features

The selected graphs were examined a priori to identify level of complexity based on: (i) number of concepts per graph; (ii) time displayed on screen in the videotape and (iii) visual complexity. These dimensions of complexity have been described previously in the visual perception literature; for a review, see Tufte. ¹³ , ¹⁴ As the purpose of this study was descriptive (non‐experimental design), no attempt was made to alter the graphs in any way, including standardization of the amount of information or key concepts.

Horizontal bar graph

The most complex of the three graphs was the horizontal bar graph. It is entitled BPH SYMPTOM CHANGES for 100 men with MODERATE symptoms after 4 years. Multiple concepts are presented in this graph. The data indicate that 17% will increase to severe symptoms, 13% will decrease to mild symptoms, 46% still have moderate symptoms and 24% will have elected surgery by this point. The videotape presentation of the horizontal bar graph is more complex in that the graph changes in appearance with the audio narration, identifying year‐by‐year changes in segments of the ‘moderate symptom’ demographic. One second of videotape playing time is devoted to each of the first 3 years presented; the final version, fourth year (as presented to the subjects in print) is on screen for 16 sec.

Gillan and Lewis ¹⁵ and Goolkasian ¹⁶ state that complex graphs with distracting features are associated with higher error rates than those presenting data in a simpler format. The horizontal bar graph may cause confusion due to several distracting features. The horizontal bar format may have facilitated labelling of the bars, but might sacrifice ease of interpretation. Western readers accustomed to reading left to right most often scan in this direction to interpret the variation between the categories on bar graphs, but this graph requires vertical scanning. The data are subsumed amidst colours, text and icons. Stylized human figures that cap each bar may effectively gain the attention of men who do not necessarily have an affinity for graphs, but the use of five densely saturated colours distracts attention from the data. ¹⁷ The graph also requires scanning the whole field, with no clear target point for the eye. Although the graph is labelled ‘BPH changes…’, it is actually communicating proportions of men in each of four categories. This type of data may be more effectively communicated via a pie chart or divided/split bar. ¹⁸ Lipkus and Hollands noted the importance of minimizing computational effort required to interpret graphs. ¹⁸

Line graph

The line graph is entitled BPH SYMPTOMS OVER TIME. It is intended to show that some men will spontaneously get better and some worse over time. Usually people interpret changes over time with high accuracy when data are presented in a line graph, especially when the slopes are clear and diverge from each other. ¹⁸ Multiple concepts are presented in this line graph. The graph contains five colours, three data lines (indicating those with severe symptoms improving and those with mild or moderate symptoms increasing in severity over time) and six pieces of data labelling the vertical axis. The slopes of the lines are distinct. The use of colour may be excessive, and in combination with the text labelling could contribute to confusion of interpretation. The placement of text and proportions of colour block are not evenly scaled. The same data could be communicated in a two‐colour graph with the numeric BPH symptom scores along the vertical axis.

Pictograph

The pictograph is entitled ‘CHANCE OF DYING within 3 months of surgery for every 100 men’. This is the least complex of the three graphs. It uses two colours and presents two pieces of data: percentage chance of dying within 3 months and the acronym TURP superimposed over the figures, in order to distinguish this surgery from others presented in the videotape. A single concept is presented in this graph.

Graph responses coding

Three independent coders (KKB, JL, JP) reviewed a 10% random sample of transcripts to determine initial coding categories based on graph relevant sections of interview transcripts. The initial categories were discussed and further refined on the basis of research team discussion. Transcripts were then coded to full consensus by two independent coders (JL, JP). Initial coding identified correct or incorrect responses to content. As the stimulus question was open‐ended, an additional category, ‘meaning’, was added to classify responses that were meaning rather than content‐focused. These included men's emotional responses to the information in the graphs. Some men gave only Meaning responses, some Content interpretation and some both. Responses could be coded into any of the three categories. Initial agreement for coding approached 100%. A few responses which could not be clearly coded were resolved by research team discussion and consensus.

Meaning

A ‘meaning’ statement was coded if the respondent provided a subjective interpretation (for self or others) in relation to a graph. Some men gave no content information at all and were coded only for their Meaning statements. Meaning statements were subclassified as global impressions or risk/outcome focused. Global impressions were defined as non‐specific, non‐content‐oriented statements, and could include statements of the meaninglessness of graphs, e.g. ‘…I don't think too many guys are into graphs unless they're office workers’. Risk/outcome focused statements were defined as statements of potential risks for treatment options, e.g. ‘anytime you have invasive surgery there's always a risk’.

Content

Content statements were those that provided a data‐based interpretation of a specific graph, as opposed to an evaluation of the meaning of the information for self or others. Content‐focused responses were coded as ‘correct’, ‘incorrect’ or ‘don't know’.

Correct response

This was defined as a response for a specific graph that provided an accurate interpretation of the objective meaning of the data presented in that graph, in the absence of any incorrect element of the response. For example, for the pictograph, a fully correct answer was ‘2% die within 3 months of having a TURP’. Responses that were less complete but having no incorrect statements were also counted as correct.

Incorrect response

This was defined as a response for a graph that provided an incorrect interpretation of the objective meaning of the data presented in the graph, even in context of some correct interpretation being provided. For example, for the horizontal bar graph, an incorrect response would be ‘if you were a hundred men with moderate symptoms after 4 years if you chose surgery, 24 men are going to be in better shape…’.

Don't know and missing data

Twenty men provided responses that indicated that they did not know how to respond, could not remember how to interpret the graph, or otherwise could not provide an explanation for the information provided by the graph. These types of responses were coded as ‘don't know’ if the individual provided no additional elaboration that could be coded into another category. Audiotapes for two men could not be transcribed due to taping problems.

Statistical analyses

Relationships between variables (education, correct/incorrect interpretation, liking of graphs, graph type) were analysed by chi‐squared tests of proportions.

Results

Overall sample characteristics

Detailed sample characteristics are presented elsewhere. ⁷ The sample included 51 college‐educated Black men, 56 college‐educated White men, 37 non‐college‐educated Black men and 44 non‐college‐educated White men. The average participant was 61.3 years of age (SD = 7.7 years), in good general health (M = 3.6/5, SD = 1.0), had moderate BPH symptoms (M = 7.8/35, SD = 5.8) and mild BPH symptom bother (M = 1.6/12, SD = 2.1). Only 15 men scored in the low literacy range on the S‐TOFHLA. In the larger quasi‐experimental study, statistically significant increases in knowledge about BPH occurred from prior to following the videotape viewing for each group of men. ⁷

Table 1 displays the overall numbers and percentages of correct and incorrect responses by graph types. As hypothesized, accuracy significantly declined with increasing complexity of the graph (P < 0.001).

Table 1.

 Number of individuals and their percentages for correct and incorrect interpretations by graph type (N = 186)

	N	%
Correct
Bar	65	46
Line	92	66
Pictograph	129	85
Incorrect
Bar	76	54
Line	48	34
Pictograph	22	15

‘Don't know’ responses are not included in the table.

Pictograph

This graph had the highest overall accuracy rate (85.4%). College‐educated men were more accurate compared with non‐college‐educated men (89.2% vs. 76.9%, P < 0.05).

Line graph

Overall, 65.7% of men provided an accurate interpretation of this graph. Accuracy by education level did differ significantly in favour of the college‐educated men (74.1% vs. 51%, P < 0.01).

Horizontal bar graph

This graph appeared to be the most challenging overall for the men to interpret accurately; only 47.8% of men provided a correct interpretation of this graph. Education was not significantly associated with accuracy, although a trend was shown (college educated = 53.3%; no college = 36.4%, P = 0.065).

Relationship between accuracy and other study variables

Men who stated a liking for graphical presentation of information were not more accurate in their interpretation of the graphs compared to those who stated that they did not like graphs,  = 2.00, P = 0.441. There was a statistically significant association between race and accuracy for the horizontal bar graph (White > Black, 59.6% vs. 39%, P < 0.05), but race and education were confounded in this sample. About 74% of White vs. 62% of Black men were college‐educated, preventing an unambiguous interpretation of this study finding.

Discussion

Our initial hypothesis of >75% correct answers for graphs content interpretation was demonstrated to be false. As the men had as much time as desired to view the graphs and had already seen them in context in the video, this result was surprising, especially given the high literacy rates. The relatively high rates of incorrect interpretations for the line and bar graphs highlight the current challenge of distilling visual information into a comprehensible information summary. Many men expressed confusion about the horizontal bar graph. The complexity of, and amount of distracting information may have contributed to its difficulty. In the videotaped animation, the bars changed lengths to illustrate change over time, though narrative voice‐over provided interpretation.

Our results are consistent with recent results of Fagerlin and colleagues, who found that people more easily comprehended pictographs, despite preferring bar graphs for their scientific credibility. ¹⁹ Our descriptive study was geared towards examining men's interpretations of several types of graphs formats that are commonly used to present health‐related information in non‐experimental, real world situations. Our findings support the observations of others that the effort to decipher a complex graph may adversely affect the communication of information in DAs. This highlights the need for attention to careful selection of data to present in graphs ¹⁸ , ²⁰ and additional empirical testing of the effects of graphical information presentation formats. It also emphasizes the need for additional research on best practices for visual aids for naturalistic situations in which DAs are used. Additional experimental research is needed to develop test graphs that present varying number of concepts, in varying formats to differentiate whether the source of difficulty is primarily complexity or whether specific formats contribute to misinterpretation. In experimental research, specific aspects of format can be manipulated, e.g. presenting bars vertically, use of no more than two colours and omission of extraneous content such as pictographs capping bars.

A second key finding of this research is that accuracy of interpreting complex graphic material was associated to some extent with educational level. College‐educated men were more accurate for both the pictograph and line graphs, and accuracy of interpretation of the horizontal bar graph was low across groups. Thus, increasing complexity of graphical information may inadvertently reinforce disparities produced by differences in education. As no content is lost by clarity and simplicity of information, it is very important to design DAs with the simplest possible representation of quantitative information. The degradation in accuracy rates with more complex formats was dramatic, from an average of 85% to an average of 48%. Over half of men misinterpreted at least one element of the data in the horizontal bar graph, rising to two‐thirds among non‐college‐educated men.

Literature on human perception experiments is potentially informative for best practices for DA design, but thus far has not been well‐integrated into DA research. Experimental research is typically well‐controlled for subjects’ length of exposure to stimuli and other confounding variables, ²¹ enabling strong tests of basic perceptual processes with regard to graphs design. Dependent variables in experimental studies to date include measures such as visual tracking, response accuracy and reaction time, as indicators of attention and ease of interpretation. ¹⁵ , ²² ‘Optimal’ graph formats, established in these experimental studies, are those that users evaluate with the highest accuracy in the least response time. Communication properties have been found to vary according to the type of format. ¹⁵ , ¹⁶ , ¹⁷ For example, pie charts are effective for communicating proportions, whereas line graphs can effectively express change over time. ²³ However, divided bar graphs have been shown to be interpreted more rapidly than pie graphs, especially when small proportion differences are being evaluated. ²⁴ Trevena et al., ²⁵ in a systematic review of approaches to communicate evidence to patients, concluded that probabilistic information presented in relevant natural frequencies was superior to more traditional approaches such as information presented in words or risk reduction effect.

The results of experimental human perception research may or may not be consistently applicable to DA design. While the line graphs used in the video were used to show change over time, they were also shown quickly. Men had more time to study the graphs when they were presented on paper. However, as ‘stand alones’, the printed versions may have been more difficult to understand. While the experimental literature suggests bar graphs should be easy to understand, clearly the somewhat unconventional horizontal bar graph format was frequently misinterpreted. We clearly found that it is critical to limit the concepts in any one graph to just one key concept. The experimental literature appropriately reminds DA designers to avoid the seductive appeal of more elaborate but potentially unclear graphs. In some experimental studies, subjects readily understand data presented in simple tables, ²⁶ but this ‘old‐fashioned’ format has been found to be unpopular and lacking in visual appeal in ‘real world’ standard patient education materials. ²⁷ , ²⁸ Some focus group research ²⁹ has documented that people prefer vertical bars over horizontal ones. We found that this convention appears to influence accuracy of interpretation. Graphs have been described as analytical and difficult to understand easily, while still having the benefits of simplicity and richness of information. ²⁹ We question the use of any format that adds difficulty.

There are some caveats to the interpretation of these results. First, in the videotape DA, the context of information interpretation was a multimedia presentation in which a ‘main points’ emphasis was taken in the narration of the videotape. In interpreting the results of this study it is important to distinguish between the ‘visual‐aiding’ vs. ‘stand alone’ use of a graph. In the visual‐aiding use of graphs, the graphics are intended to highlight key points that are made, e.g. in the overall text of an educational booklet or videotape. In the stand‐alone use of a graphic, the intent is to provide comprehensive detail about summary points that are made in the text or narration of educational materials. People may be better able to grasp the main points of information without necessarily recalling from narration, all the specific details about what a given graph shows. However, despite large average gains in knowledge, the main points were not universally gleaned from the videotape. ⁷ Even with a 47‐min videotape in a multimedia format, some basic points were missed by substantial number of viewers. It is important to bear in mind that in our study, only a small number of men in the sample had borderline or inadequate levels of health literacy. The sample sizes were too small to fully analyse the possible association of inadequate health literacy and accuracy of interpretation of graphs. A study of low literacy samples would likely find even more error in interpretation. In the future, studies should investigate whether there are particular characteristics of graphs that are specific to errors made by low literacy samples, or whether there is a linear relationship between literacy and ability to interpret increasingly complex graphs. However, as our sample included half non‐college‐educated men, it is likely that our results apply to most populations excluding those with inadequate health literacy.

Conclusion

The key findings of this analysis show that men's accuracy in interpretation of graphs depends upon both graph format and education level. As hypothesized, based on an analysis of the graph features, men were most accurate in their interpretations of information for simpler graph formats (pictograph) and least accurate for more complex graph formats (bar graph). Differences in accuracy were also systematically associated with level of education. Our results as reported in this paper highlight the role of educational level as a key variable in accuracy of interpretation of graphs. A key question for future research is how to package information in ways that can meet the needs of DA users who vary in educational level.

Contemporary research on shared decision‐making emphasizes the benefits of shaping decision support to the needs and preferences of health care users. ³⁰ Misunderstandings of treatment options can become a source of health care disparities. ³¹ Clinicians should do a careful assessment of which data patients actually need and prefer. Careful choices need to be made about the circumstances in which graphs are likely to facilitate vs. hinder understanding. It is important to provide complete information, but care should be exercised to avoid excessive complexity in the form of extraneous labelling, colour and data that may function as barriers to comprehension. In situations where DAs are used outside the encounter with a health care provider, clinicians cannot assume that full understanding has been achieved on the basis of the DA alone. Understanding should be formally checked as part of treatment planning.