A Usability Problem: Conveying Health Risks to Consumers On the Internet

Abstract

Effectively designing risk information for the public is challenging and selecting the appropriate medium to deliver disease risk information is crucial. In a usability evaluation of a colon cancer risk website, we evaluated the public’s understanding and interpretation of graphical displays of risk information (2 bar graphs). Results from this study suggest that many people do not understand risk and often misinterpret graphical displays of risk and associated terminology. This work shows the importance of not only including representational analysis within user-centered design of consumer health websites, but also evaluating the health and numerical literacy levels of these websites as an aspect of usability testing.

Introduction

With easy access to health information on the Internet, consumer retrieval of health information has never been more prevalent than it is today. According to The Pew Internet & American Life Project, 75–80% of Internet users search for health information¹ and are making decisions based on the information found on the Internet1. Yet, this repository of health information does not always help the consumer to critically evaluate the information and not enough consideration is given to the health and numerical literacy of these users. With the recent shift in the delivery of healthcare towards giving patients more responsibility for their healthcare choices, disease risk calculators are being promoted to help consumers understand their probabilities of various disease risks. Whereas in the past, physicians interpreted and filtered the information to consumers, these disease risk calculators are providing disease risk probabilities without significant explanation. Communicating disease risk to the public is challenging because of a number of issues, including the general public’s difficulty in understanding and using quantitative risk magnitude information to make decisions. Conveying information to the public about the magnitudes of risks is not only difficult, but poses significant challenges^2–4. “Source problems” or the use of technical language, which is uninterpretable to the public at large and “receiver problems” which includes the public’s perception of risk and their associated difficulty in understanding their individualized risk are two main issues with representing risk²; Additionally, risk is often communicated numerically, which imposes an additional burden on the public, since 50% of Americans lack minimal math skills⁵ and are unable to comprehend the notion of numbers and chance^6–8. Research has shown that people interpret statistical information in different ways⁹, have difficulty understanding odds greater than 50:1^{9, 10}, underestimate the cumulative nature of probabilistic events¹¹, place more weight on the testimonials of a few people than large studies¹², and rate probabilities differently based on how they are presented (e.g., rating 1286 out of 10,000 as greater than 24.14 out of 100)¹³, thus have difficulty understanding and making use of small numerical probabilities. Given the complexity of communicating probabilistic information and its potential effect on decision-making processes, selecting the appropriate medium and terminology to communicate this information to the public is crucial to the effectiveness of these processes.

Using visual metaphors such as graphs or schematic diagrams provides potent means to express quantitative information in a way that is more meaningful than numbers and words alone. Numerous types of graphical displays for communicating risk information have been proposed given the health and numerical literacy issues of the general public^{14, 15}. Designing websites that present risk information to the general public must take into account the graphical features of risk displays and numeracy to help consumers with their quantitative reasoning.

Websites that disseminate individualized disease risk information need to be designed with not only good utility and usability, but also must address the functional literacy of the general public. A colorectal cancer risk website was designed for healthcare consumers based on the preliminary work of a colorectal cancer risk model¹⁶. This risk model under development addresses an important health risk in the U.S., colorectal cancer (CRC), which is the third most common cancer and cause of cancer death in the United States¹⁷. Although screening strategies such as colonoscopy have been shown to reduce CRC incidence and mortality through the removal of precancerous polyps¹⁸, rates of screening remain suboptimal^19–20. Some of the possible reasons for these suboptimal screening rates are that the public lacks knowledge of CRC risk, CRC risk factors, and the screening tests and guidelines^21–22.

Thus, the reason for the development of this CRC risk model and the accompanying development of the website is to provide individualized CRC risk, risk factors contributing to an individual’s risk, as well as providing information on screening tests and guidelines. The website was developed using a user-centered framework for interface design²³ and additionally took into consideration the display aspects of risk representation¹⁵. The aim of this study was twofold: 1) To evaluate the usability of the website using scenarios and tasks, and 2) to evaluate the public’s understanding and interpretation of potential colorectal cancer risk representations to be potentially used on a colorectal cancer risk assessment website. We evaluated two different risk representations as well as risk terminology frequently found in representations of risk. We particularly focused on risk representation because in general people have difficulty understanding probabilities and especially low probability risks such as CRC^24–25. Thus, if we are going to build effective informative websites that help individuals to decide to get screened, we must begin to improve the public’s understanding of their risk.

Methods

A convenience sample of subjects was recruited through local advertisement (e.g. public library, hospital bulletin boards). Inclusion criteria were age 21 and older, fluent in English, and computer literate. Potential subjects were excluded if they had a previous diagnosis of cancer. The recruitment brochure conveyed that the investigators were attempting to improve the ease of use of a colorectal cancer risk assessment website and that participants could learn about the risk factors for colorectal cancer. Potential subjects who responded to the advertisement were evaluated for eligibility and enrolled in the study based on their eligibility. Subjects were recruited to ensure the best cost-benefit ratio which has been reported to be achieved with 5–12 users, as long as those users are representative of the target population^26–27. Since small numbers of users will identify 80%–90% of the major problems with a system, we recruited 30 subjects to ensure that over 90% of the usability problems would be uncovered²⁸. Approval for this study was obtained through the Duke University IRB. All participants signed an informed consent. Participants were compensated with a parking pass to cover the cost of parking, a $5.00 gift certificate and an American Cancer Society Colorectal Cancer Prevention brochure.

Once the participants signed the informed consent and all questions regarding the study were addressed, each participant was given training on the talk-aloud technique using a simple math problem (two digit addition problem) and scrambled letter technique (make as many words as possible from a string of letters out-loud). Although Ericsson²⁹ suggests using a multiplication problem, experience over the years has shown us that due to numeracy issues, participants from the general population learn the talk-aloud technique just as well with an addition problem with carry-over. Verbal protocols such as the talk-aloud technique allows collection of procedural information about mental processing, wherein the investigator can make deductions about the principal cognitive processes of the subject while performing a task and thus difficulties a user may experience with an interface or website²⁹. The participants were asked to say out-loud everything they would think or say to themselves while performing a set of tasks.

We measured the health literacy of the applicants to determine the basic reading and numerical skills among the participants and to determine if there was a large discrepancy among the participants. So we then tested the participants with a health literacy-screening instrument, the Newest Vital Sign³⁰. The Newest Vital Sign correlates well with the Test of Functional Health Literacy in Adults (TOFLHA). It consists of a nutrition label with 6 quantitative numerical questions measuring math skill, reading and comprehension, and abstract reasoning skills. Each question answered correctly is given a score of 1. A score of > 4 = adequate literacy. A score of < 2 = marginal or inadequate literacy skills. This instrument is suitable for use as a quick screening test for limited literacy in primary healthcare settings (alpha >0.76)³⁰.

Participants were then given a series of tasks to complete on the website while talking-aloud and were assessed for successful completion of each task within a defined time period. These tasks included finding information on the website, navigating through the website, the ease of understanding the questions for determining an individual’s colorectal cancer risk, and ease of using the functionality such as printing a report. We additionally administered the Computer System Usability Questionnaire (CSUQ) to assess the usability of the website³¹. Although this questionnaire measures three factors: system usefulness, informational quality, and interface quality, we only measured informational quality and interface quality. The CSUQ’s coefficient alpha is 0.91 for informational quality, and 0.89 for interface quality³². Administration and scoring of the questionnaire was followed according to the author’s instructions. In addition to determining usability issues with the website, we also asked both closed and open-ended questions on risk terminology used in the risk display formats and asked the participants to provide their interpretation of the presented risk representations (Figures 1 & 2). If the participants did not talk about certain aspects of the pictorial display format, we continued to probe the subject.

Figure 1.

Lifetime and ten-year bar graph

Figure 2.

Relative risk graph

The pictorial display format of Figure 1 consists of ten-year and lifetime chance of developing colon cancer of a hypothetical 55 year old white female and additionally shows non-modifiable (age, race, sex, family history of colon cancer, height and history of inflammatory bowel disease) and modifiable risk factors (risk factors with current risk exposure information showing if each factor increases or decreases the hypothetical person’s risk using red and green arrows). Participants were specifically asked the following questions: what does ten-year and lifetime risk of developing colon cancer mean, does the display show the risk of “getting” colon cancer in the future, does the display show the risk of dying of colon cancer in the future, are the graphs easy to understand (on a 5-point Likert scale), and in general what each aspect of the display means. We defined 10-year risk of developing colon cancer as the chance of developing colon cancer over the next 10 years. We defined lifetime risk as the chance over the course of a lifetime from birth to death of developing colon cancer³³.

The pictorial display format of Figure 2 (on a logarithmic scale) consists of the relative risks of a hypothetical individual presenting the risk factors that increase or decrease their risk of developing colon cancer. Participants were asked the meaning of relative risk, and specifically what factors were increasing or decreasing their risk and in general what the entire picture was displaying. Relative risk is the ratio of risk in a group exposed to a factor to the risk of those not exposed³⁴.

Using Weinstein’s dimensions of risk³⁵, we first used his four essential dimensions for understanding risk (understanding precursors of risk – CRC risk factors, likelihood, consequences or severity of disease, and methods to prevent risk such as modifying risk factors or screening) to develop a list of a priori codes. Coding is a way to reduce the data to allow for data clustering and additionally provides a foundation for further analysis³⁶. We developed a matrix using each open-ended question as the header for each column and the individual answers in each row. The answer to each open-ended question on risk was analyzed through a data reduction process to organize and code these data³⁶. We used the a priori codes and when the codes did not fit, we discussed the findings and added codes by consensus. For example, we added effects on emotions³⁷ such as worry or concern, which can be induced by risk communication. We iteratively reviewed these data until saturation occurred and themes emerged. This process allowed us to draw conclusions about our data.

Results

Thirty participants took part in this study. The sample was predominately female (83%), Caucasian (76%) with a Bachelor’s or graduate degree (67%). The average age of the participants was 38.5 years with a range of 28–64 years. Most participants had significant computer and Internet experience (see Table 1).

Table 1.

Subject demographics

N =	30
Gender	25 female
Age, mean years (range)	38.5 (23–64)
Race
African American	5
Hispanic	1
Caucasian	23
Other	1
Education
High school	3
Some college	4
Associates	3
Bachelors	14
Masters	5
PhD	1
Computer experience mean years (range)	17.5 (4–35)
Internet experience mean years (range)	12 (4–25)

The Newest Vital Sign literacy assessment revealed that out of the 30 participants, 28 (93%) demonstrated adequate literacy. This was achieved by a score of 4 or greater (scale range: 1–6). Two respondents scored a 2, which suggests limited literacy. These two respondents reported a high school education.

The small-scale usability study revealed very few general usability problems with the website. The majority of the problems were found with the risk representations. First there was a problem with navigation, where 80% (n =24) of the participants had difficulty finding the homepage after navigating to a different part of the website. Of the 30 participants, 37% (n =11) could not determine the purpose of the website from the description provided on the homepage. Finally, there was one major missing functionality issue, where 87% (n =26) could not determine how to print the provided report within the website. The users expected to see a print button as opposed to using the browser print option. 100% of the participants agreed to strongly agreed that the questions on the website to determine their risk for colon cancer were easy to understand. The users ranked all of the information quality and interface quality related questions via the CSUQ between agree and strongly agree.

Negative and positive feedback provided by the participants on the CSUQ correlated well with the usability problems participants experienced while performing the provided tasks. The three major themes provided by the participants on the negative aspects of the website were no printer links on the report page (n =12), navigation to the homepage problems (n=12) and problematic visuals (n = 11). The positive aspects of the website noted by the participants centered mainly around the purpose of the website or the ability to provide information on colon cancer risk and ease of understanding the information provided with the exception of the display formats (n = 23).

The main usability issues with the colon cancer risk website were with the display formats of colon cancer risk. Both display formats previously described in the methods section presented problems. When asked the definition of 10-year risk, 53% (n= 16) of the participants defined it as their chance or probability of developing colon cancer over the next ten years. The remaining 14 (47%) participants responded with one or a combination of the following: 23% (n=7) as an immediate or short-term risk, 7% (n=2) as a risk that is modifiable over 10 years, 7% (n=2) as a magnitude of risk, 7% (n=2) as a comparison to the average person, 3% (n=1) as modifiable and non-modifiable risk factors that lead to colon cancer, 3% (n=1) as a risk factor, and 3% (n=1) as something concerning. When asked the definition of lifetime risk, 40% (n=12) of the participants defined it as their chance of developing colon cancer over the course of a lifetime. 10% (n=3) of the participants did not know the answer and the remaining 50% (n=15) defined it as one or a combination of the following: 23% (n=7) as a risk from present to death, 20% (n=6) compared it to the 10-year risk and interpreted it as risk 10 years from present and beyond, 20% (n=6) said it was affected by modifiable risk factors, 13% (n=4) said it was affected by non-modifiable risk factors, 3% (n=1) said it was a magnitude of risk and 3% (n=1) said it was something concerning. Several participants identified lifetime risk as non-modifiable such as a genetic risk factor while 10-year risk was seen as more immediate and a modifiable risk factor. When asked the difference between 10-year and lifetime risk, the majority of participants simply either stated they did not know the difference or simply did not answer the question.

Participants were asked to comment on the meaning of Figure 1. The majority of the participants, 73% (n=22), stated they agreed or strongly agreed that the graph was easy to understand, 17% (n=5) were neutral, and 10% (n=3) disagreed. Participants were asked if Figure 1 showed them their risk of developing colon cancer in the future; 70% (n=21) said yes, 23% (n=7) said no, and 7% (n=2) were uncertain. When asked if Figure 1 showed them their risk of dying from colon cancer, 80% (n=24) said no, 3% (n=1) said yes, 13% (n=4) were uncertain, and 3% (n=1) did not know.

Several themes arose from people’s interpretation of Figure 1. 87% (n=26) identified the graph as showing modifiable risk factors associated with colon cancer and 77% (n=23) identified non-modifiable risk factors. However, only 43% (n=13) comprehended the meaning of modifiable risk factors and 37% (n=11) the meaning of non-modifiable risk factors. The main themes that emerged were probability, risk magnitude, consequences of risk factors, time frame, worry, comparison to other people their same age, race, and sex. 60% (n=18) said that the figure shows the probability or chance of developing colon cancer, 63% (n=19) said there was a magnitude of risk represented, 10% (n=3) said the consequences of not changing habits were represented, and 23% (n=7) said that it made them feel concerned about getting colon cancer. 43% (n=13) of participants also identified Figure 1 as showing a time frame of when they may develop colon cancer, 3% (n=1) said that it may occur in the future, and 43% (n=13) said that their risk was in comparison to others. In addition, 3 participants (10%) diminished the risk of developing colon cancer. For example, one participant stated, “…life time risk seems to be far greater than the next ten years, that would make me think that I have 10 years before I have to worry about it,” and another participant stated, “the tiny percentage doesn’t look so big…”.

The red and green arrows in Figure 1 (which represent the level of exposure leading to the risk) were understood by 77% (n=23) of the participants and the bar graph was completely understood by 20% (n=6) of the participants. Between the 10-year and lifetime bars, 23% (n=7) and 30% (n=9) interpreted each bar correctly, respectively. One participant did not understand that the bar graphs were on a scale of 100. The fact that the scale stopped at 30 was confusing. Another participant said the modifiable risk factor of taking aspirin must have meant the person “was in pain”. Additionally, many participants did not understand the meaning of the words modifiable and non-modifiable and did not know the meanings of the acronyms used in the risk lists such as FOBT (fecal occult blood test).

When shown Figure 2, 93% (n=28) identified the risk factors that increased or decreased their risk for colon cancer. However, none could identify how much each factor increased or decreased their risk. None of the participants understood the scale. When asked the definition of relative risk, 40% (n=12) said they did not know what it was, 30% (n=9) said it was in comparison to other people or the population, and 17% (n=5) said it was a risk or chance, 10% (n=3) stated it was risk factors, and 3% (n=1) said it was family history.

Discussion

Online disease risk calculators that provide risk estimates to the general public can be valuable tools, yet they also have the potential to confuse the general public and lead people to generate inaccurate risk perceptions, thus impacting their decision-making regarding their risk. Health information needs to be presented to consumers using easily understood formats. This is critically important if tools such as risk calculators are to be used for consumers to make healthcare decisions. Usability evaluation is one key to ease of use³⁸. Usability refers to “how useful, usable, and satisfying a system is for the intent users” ³⁸. When health information is presented online, we not only must develop these sites using user-centered frameworks, but we must also utilize suggested practices for designing risk representations^{14–15, 37, 39} and evaluate these sites for literacy and numeracy issues as well. The Website Developmental Model for the Healthcare Consumer is one model that can be used that includes not only the important facets of user-centered designed such as user analysis, task analysis, functional analysis, representational analysis, and various inspection methods such as heuristic evaluation, but also includes suggestions for testing domain specific visual display criteria, suitability assessment of materials and readability of all parts of a website⁴⁰. This model has been successfully used to develop user-centered websites⁴¹. This additional testing ensures that the website text and cognitive artifacts such as graphs are not only readable, but understandable at a level that enhances consumer decision-making.

Results indicated that we had few usability problems with the exception of the graphs. This is in part because we adhered to general principles of website design, a user-centered framework²³, the Health On the Net (HON) code⁴², and the American Medical Association (AMA) principles⁴³. However, consistent with the literature³⁵ results indicate that many people have difficulty understanding concepts of risk and there is difficulty in presenting quantitative information in an easily understandable form. Overall, participants correctly defined and understood 10-year risk (47%) more than those of lifetime risk (40%). Due to the relatively small amount of risk depicted in the 10-year risk bar graph (e.g. 2%), 28% of participants minimized their risk as “not that bad” even if it was twice the average. This is in line with Fisher et al. findings which suggested that people either dismiss a low-probability risk entirely or else focus primarily on the size of the expected loss⁴⁴.

There were two main issues with the risk graphs. As expected, one was a source problem or the use of risk terminology, which was unfamiliar to the many of the participants. Many of our participants had difficulty with the definitions of 10-year, lifetime and relative risk. It is therefore very important to provide to consumers simple yet complete explanations of the meaning of any risk terminology used and to explain the meaning of the graphs. This can be easily accomplished on the Internet with the use of embedded links. The second issue was a receiver and design problem. The scales on both Figures 1 and 2 posed problems. In Figure 2, a logarithmic scale was used to represent relative risk. Using logarithmic scales are generally poorly understood by the public³⁷, yet there have been some instances where the use of a logarithmic scale has been used successfully such as the Pauling Perspective Scale⁴⁵. Lipkus suggests that further study needs to be done in the area of presenting risk of on a logarithmic scale³⁷. Figure 1 used an end-point of 30% on the y-axis instead of 100%. Presenting an end point on a bar graph as 30 instead of 100 on the y-axis can be misleading because it emphasizes the numerator which may influence the interpretation of the graph and thus risk perceptions and ultimately health care decisions such as screening⁴⁶.

Although many of the participants perceived the graphics as easy to understand, results indicate fewer of them truly understand them. This suggests a discrepancy between what consumers like and what actually assists in their understanding. There were several other important results as well. Medical acronyms should be used sparingly and defined (e.g. FOBT is fecal occult blood test). A few participants interpreted aspirin not as a protective factor against developing colon cancer, but as an indication that someone is in pain.

The lowest literacy scores on the Newest Vital Sign test³⁰ were from those with the lowest education, emphasizing the importance of literacy levels and suitability of materials for the targeted audience. The participants who scored poorly on the Newest Vital Sign test were some of the poorer performers in answering the questions of risk and interpreting the graphs. However, even some of the well-educated participants misinterpreted the graphs. Numeracy is a problem, as demonstrated in numerous studies^47–48. Lipkus, Samsa and Rimer found that even highly educated people have problems with simple numeracy questions such as, “If the chance of winning a car is 1 in 1,000, what percent of tickets win a car?”⁴⁷. Numeracy problems have implications in how risk is presented to the general public. One proposed solution to this problem is to present risks as natural frequencies instead of conditional probabilities⁴⁹.

We had several limitations with this pilot study. One, since the scope of this study was mainly limited to usability, which generally only requires a small number of subjects; this study was not powered to determine statistical significance. Second, our participants were mainly female, highly educated and Caucasian. Conducting further research on the risk graphic aspect of the study with a more heterogeneous population is warranted to show the general utility of our findings. In particular, it would be important to focus on those who are not currently getting screened for colorectal cancer such as those with no health insurance, racial minorities, and those with a low socioeconomic status⁵⁰.

When designing online health information, performing a thorough representational analysis and adhering to guidelines for representing risk^{14–15, 37, 39} will determine the best way to present numerical information to health-care consumers. According to Zhang and Walji³⁸, “the essence of usability is the representational effect”. The wrong representation can pose problems with how information is interpreted and therefore can affect behavioral outcomes among other things³⁸. In short, this pilot study shows the importance of designing consumer health websites with a user-centered framework and with careful attention to a representational analysis and taking into consideration the health and numerical literacy of its users. Through continued research on optimal displays of health risk information, it is hoped that people will not only be able to adequately understand their health risks but initiate changes in modifiable risk behaviors; and in the case of colon cancer, get screened.