Usability Testing of a Web-Based Decision Aid for Breast Cancer Risk Assessment Among Multi-Ethnic Women

Abstract

Chemoprevention with antiestrogens could decrease the incidence of invasive breast cancer but uptake has been low among high-risk women in the United States. We have designed a web-based patient-facing decision aid, called RealRisks, to inform high-risk women about the risks and benefits of chemoprevention and facilitate shared decision-making with their primary care provider. We conducted two rounds of usability testing to determine how subjects engaged with and understood the information in RealRisks. A total of 7 English-speaking and 4 Spanish-speaking subjects completed testing. Using surveys, think-aloud protocols, and subject recordings, we identified several themes relating to the usability of RealRisks, specifically in the content, ease of use, and navigability of the application. By conducting studies in two languages with a diverse multi-ethnic population, we were able to implement interface changes to make RealRisks accessible to users with varying health literacy and acculturation.

Introduction

Breast cancer is the most commonly diagnosed and second deadliest cancer among women in the United States.^1,2 Based upon randomized controlled trials, chemoprevention with antiestrogens, specifically tamoxifen, raloxifene, anastrozole, and exemestane, can reduce the incidence of breast cancer among high-risk women by 30-70%.^3–6 It is estimated that at least 15% of women in the United States meet high-risk criteria for breast cancer, defined as a 5- year risk greater than 1.67% or a lifetime risk greater than 20%, according to the Gail model.⁷ However, of these women fewer than 5% who are offered chemoprevention therapy reported taking the medication.⁸ While this low uptake is partially attributed to concerns about side effects, lack of knowledge about chemoprevention among both patients and primary care providers (PCPs) is another a major contributor to low uptake.⁹

Recognizing and making decisions about health can be a difficult task for patients, particularly when there are multiple options for care and patient preferences are not well defined.¹⁰ The United States Preventive Services Task Force (USPSTF) recommends that physicians engage in shared decision-making with patients regarding chemoprevention and screening for breast cancer.¹¹ Studies have indicated that patient decision aids (PtDAs) are an important part of influencing a woman’s decision to seek certain types of treatment for breast cancer.¹² PtDAs for considering breast cancer screening options and prevention strategies have been developed to assist women with assessing their risk and making informed choices about their health.^13-17 While PtDAs for screening were successful in reducing over-screening rates, chemoprevention uptake among high-risk women exposed to a PtDA on breast cancer chemoprevention remained low up to 3 months after use.^15-17 PtDAs that provide information that is tailored to the risk status of the patient tend to be more efficacious than those that do not.¹⁰

We have developed an innovative web-based PtDA, RealRisks, to provide high-risk women with more information about breast cancer chemoprevention. RealRisks is designed to improve users’ accuracy of risk perception while presenting them with information about breast cancer risk and chemoprevention options that can be tailored to their preferred method of receiving health information. The PtDA offers users the ability to input personal information in order to engage with modules that reflect the prevention options that are available for women with the same risk level. Patients who complete RealRisks are provided with an action plan for discussing prevention options with their PCP. RealRisks is designed to be used in conjunction with the Breast cancer risk NAVigation (BNAV) tool, a clinical decision support tool for PCPs that is integrated into the electronic health record.¹⁸ Together RealRisks and BNAV provide comprehensive information to high-risk women and their PCPs in order to promote shared decision-making about chemoprevention.

We previously conducted focus groups and participatory workshops with an early prototype of RealRisks, which specifically targets women from diverse racial/ethnic and educational backgrounds. In particular, we demonstrated that after exposure to RealRisks, women with low or high numeracy levels had improved breast cancer risk perceptions and the majority found the tool useful and easy to use.¹⁹ The aim of this study was to assess the usability of a more advanced prototype of RealRisks, which is available in English and Spanish, by evaluating issues related to the design, content, and user engagement of the PtDA. We sought to make improvements to the application that would make RealRisks understandable to multi-ethnic women with a broad range of health literacy and acculturation levels.

Methods

Participants and Recruitment

Participants in the study were recruited from a database of women who had undergone routine screening mammography and previously consented to be contacted for future research. Participants were contacted by telephone and offered morning and afternoon sessions to partake in usability testing. Subjects who indicated that they were unable to use a computer or had serious issues relating to reading a computer screen were not eligible for the study.

We initially recruited 5 English-speaking subjects for the first round of usability testing. For the second round of testing, we recruited 2 English-speaking and 4 Spanish-speaking subjects after changes were made to the program. The literature has shown that conducting usability testing with at least 5 subjects will usually identify upwards of 80% of design issues, and that 10 subjects will identify a minimum of 82% of usability concerns before becoming repetitive.^20,21 In total, 11 subjects participated in the usability testing for RealRisks. All subjects signed written informed consent and the study was approved by the Institutional Review Board at the Columbia University Medical Center (CUMC).

The RealRisks Decision Aid

The RealRisks PtDA is tailored to provide personalized information about chemoprevention to high-risk women in both English and Spanish. The PtDA contains 6 modules that provide various information about risk assessment and chemoprevention (Figure 1 ). Each module includes a comic following a fictional character, Rose, who discusses breast cancer risk with family, friends, and health care providers (the light narrative), a series of informational slides (the dense narrative), and interactive games (Figure 2 ). For example, the first module introduces users to the concept of risk through a “clicking” game that demonstrates the 5-year and lifetime breast cancer risk for an average 50-year old woman using a pictograph with 100 clickable women (Figure 4 ). This experience-based risk interface asks players to continue to click (e.g., sample from a population of women) to better learn the meaning of a given pre-set probability (i.e., 12 out of 100 or 12% of women will develop breast cancer in their lifetime) and has been previously shown to improve risk perception in high and low numeracy women.^19,22 The second module outlines various risk factors for breast cancer and prompts the participant to calculate their Gail risk score and allows users to revisit the previous risk game using their personal risk for breast cancer. The third module asks users to input their family tree, with special attention to family members with a history of cancer. The fourth and fifth modules are tailored to the woman’s level of breast cancer risk and includes preference elicitation games (Figure 3 ). Module four gives information about BRCA genetic testing, whereas module five describes different breast cancer chemoprevention options. Preference elicitation is done through the use of scales that allow users to rate how important the risk of developing breast cancer, risk of side effects with chemoprevention, and various personal factors are to their decision about taking chemoprevention. The sixth module opens for all participants and provides information about lifestyle behaviors than can lower the risk for breast cancer. Completing all six modules generates an action plan that includes all of their inputted information that can be printed and shared with their PCP during their next clinic visit. A copy of the action plan is also provided to the PCP through the electronic health record via the BNAV tool. RealRisks is designed to be accessible on a computer, tablet, or smartphone. Development of the risk clicking game and other components of RealRisks have been addressed in previous studies.^18,19

Figure 1.

RealRisks dashboard view.

Figure 2.

RealRisks light and dense narratives for chemoprevention.

Figure 4.

RealRisks “clicking game” used to visually demonstrate breast cancer risk.

Figure 3.

Comparison of the original (left) and updated (right) versions of the preference elicitation scales for chemoprevention.

Usability Testing

Usability studies were conducted in English and Spanish by pairs of researchers who took on the role of moderator and observer and were bilingual for the Spanish-speaking participants. The moderator was tasked with receiving consent from the study subjects, orienting the subject to the PtDA, and providing guidance and assistance as the participant completed the study. The observer used Morae Observer software to view a livestream of the subject and code the usability study in real time on a separate laptop. This livestream captured the audio and visual image of each subject, as well as non-verbal reactions. The moderator was positioned across from the subject, while the observer was in a corner of the study room. All usability studies took place at CUMC.

Participants consented to be audio and video recorded. Prior to beginning the usability testing, subjects were asked to complete a short survey to indicate their familiarity with computer and web-based technology and validated measures of acculturation and health literacy.^23–26 Study subjects were then introduced to the RealRisks platform and instructed to create an account using predetermined patient profiles and scenarios. These preset profiles included simulated health information that would activate both the genetic testing and chemoprevention eligibility modules without requiring subjects to disclose their personal medical history to the research team. Participants were then asked to complete the six RealRisks modules in any order they desired and follow a think-aloud protocol as they moved through the PtDA.^27,28 In this protocol, subjects voiced their feelings and intentions about chemoprevention, the design of the application, and their understanding of the information as it was presented to them. When subjects ceased thinking-aloud as they used the PtDA, they were prompted by the Moderator with questions relating to the section of the application they were on. The most frequently used prompts included “how are you feeling about the information on this page”, “what are you currently thinking about”, and “why did you decide to move to that section”. Prompting was not necessary for the majority of study subjects. The moderator did not interact with the participant beyond prompting and answering direct questions regarding the application.

Each usability study lasted approximately two hours or until the subject completed the entire RealRisks PtDA. The PtDA was considered completed once a subject had gone through all six modules of RealRisks and viewed their action plan. At the end of the testing period, all subjects were asked to complete the System Usability Scale (SUS), a validated 10-item questionnaire used to determine the usability of a software.^29,30 The results of the SUS were used to determine the current usability status of RealRisks.

Data Collection and Analysis

Morae Recorder, Observer, and Manager software version 3.3.4 (Techsmith Corporation, Okemos, MI) was used for all usability recording and analysis. Morae software allows for the creation of usability study templates to be created with Tasks and Markers for ease of coding. Markers were used to label points in the video such as “error”, “participant prompted”, and “observation”. Tasks were used to identify when subjects had completed activities in RealRisks, such as “created an account”, “finished module 1”, and “viewed action plan”. All participant recordings were coded by two researchers – the first being the Observer and the second being an additional researcher involved in the study, usually the Moderator. The first round of coding was done in real time, while the second occurred immediately after the study.

Qualitative coding was used to identify themes regarding the design, content, and user engagement with RealRisks. Recurring issues brought up by participants were sorted into whether they related to the dashboard, light narrative, dense narrative, games, or action plan. SUS scores were calculated using the standard methodology described by Brooke.³⁰ The results of the usability studies were used by the research team to identify opportunities to improve RealRisks. Feedback given to the programming team was used to update RealRisks prior to implementation of the tool among high-risk women.

Results

Participants

For the first round of usability studies, a total of 5 English-speaking subjects were enrolled. Of these subjects, 1 had difficulty with reading some information on the screen because she forgot her glasses – this subject was given assistance by the moderator to read the screen. In the second round of usability studies, 2 were English-speaking subjects and 4 were Spanish-speaking subjects. A total of 11 subjects took part in the two rounds of usability testing for RealRisks. Our sample population was relatively diverse, in terms of education level, race and ethnicity, health literacy and acculturation (Table 1). The English-speaking group had a higher median health literacy than the Spanish-speaking group, however, in both cohorts health literacy ranged from low to high. The English-speaking group was more acculturated than the Spanish-speaking group. Two subjects spoke a language other than English or Spanish – the one subject who identified her additional language spoke Amharic.

Table 1.

Baseline characteristics of study subjects

Characteristic	English-Speaking (N = 7, 64%)	Spanish-Speaking (N = 4, 36%)
Median age, years (range)	57 (29-69)	56.5 (47-71)
Race and Ethnicity, N (%)
Non-Hispanic White	1 (14)	0
Non-Hispanic Black	5 (71)	0
Hispanic	0	4 (100)
Other	1 (14)	0
Education Level, N (%)
High School or Less	2 (29)	3 (75)
Some College or Vocational	2 (29)	1 (25)
Graduate School or Professional	3 (43)	0
Health Literacy
Median (range, 0 [low] – 4 [high])	3 (1.67-4)	2.33 (1.33-4)
Acculturation
Median (range, 1 [low] – 5 [high])	5 (3-5)	1.67 (1-3.67)

All subjects owned a mobile phone, and 9 owned a smart phone. All but 3 subjects owned a computer of some type, although at least 1 of these subjects indicated that she had a computer in the house but was not the owner. Of the 11 subjects, 10 had access to the internet in their home.

Theme Identification

Usability themes were grouped into Content, Ease of Use, and Navigation, which were used to identify potential usability problems that could be addressed by improving RealRisks. All themes were marked as being concerned with the dashboard, light narrative, dense narrative, games, or action plan. Major recurring themes and corresponding changes are described in Table 2.

Table 2.

Themes identified and addressed during the usability studies.

Theme	Location	Description	Changes Made
Content	Light Narrative	Comics are “talking down” and too simplistic; alternatively comics are highly relatable	Users can navigate through light and/or dense narratives based upon preferences for cognitive load
	Dense Narrative	Cancer development and DNA/genetic information is confusing and too technical	Language and pictures were simplified
	Games	Preference elicitation game does not have non-clinical factors, is confusing, not sensitive enough to patient preferences	Scales redesigned to include more preference factors, which were also summarized in the action plan
	Action Plan	Confusion over who the action plan is meant for and what the subject should do with it	Action plan redesigned and given additional instructions
Ease of Use	Dense Narrative	Too much text to read, would prefer audio	Audio feature made more prominent/visible
	Games	Risk assessment clicking game not giving enough feedback, took too long to complete	Clicking feedback improved, number of clicks needed were shortened
	Games	Preference elicitation game had many bugs, drag and drop functionality was difficult for many participants	Scales redesigned to use sliding scales for preset preference options
	Games	Family tree difficult to understand without a legend or key	Legend added to family tree
Navigation	Dashboard	Difficult for users to determine which module they had just completed and how far along they were in RealRisks	Checkbox functionality improved
	Dashboard	Users asked for some sort of aid to guide them through the application	Tutorial videos in development to assist users with low computer literacy

Content Issues

Most problems arose in the dense narrative and during the genetic testing and chemoprevention preference elicitation games. Two subjects were vocal about their distaste for the light narrative, indicating that they thought that the comics were “speaking down” or “not something I would read”. However, subjects who aligned more closely with the target population identified the cartoon figure of Rose as being “like me” and performing actions that were in line with what they would do in real life. Based on this feedback that some users prefer the light narrative whereas others dense, RealRisks was updated so that end users can complete either the light and/or dense narrative sections to trigger the creation of the action plan based upon their preferences for cognitive load.

Certain sections in the dense narrative were frequently confusing to subjects. Slides in depicting the growth of breast cancer cells was described as hard to understand and several subjects either asked the moderator for further information or described frustration with the use of technical words like “hyperplasia” or “atypia.” These concerns were addressed by improving the picture of breast cancer growth on this slide and using layman’s definitions in the rollovers on the slide. Similarly, use of the metric system on some slides was difficult to understand for at least two subjects. The dense narrative was edited to include measures in imperial units. A body mass index (BMI) chart that appears in the lifestyle factors module was difficult to read for subjects, with one staring at it for a period of time and expressing confusion at what she was looking at. This chart will be amended in future versions of the PtDA.

The most major content issue occurred with the preference elicitation games in modules 4 and 5 and overlapped significantly with ease of use. These games were designed to allow subjects to consider a variety of factors that may influence their decision-making about genetic testing or chemoprevention. In the original version of RealRisks, subjects were presented with a set of scales for genetic testing and each type of chemoprevention. Users would drag different factors onto the scale to indicate whether they considered it a positive or negative factor. The purpose of this activity was to elicit and visualize the user’s preferences. Content issues with this game included confusion over the purpose, lack of clarity regarding how the scale was weighted, and an inability of the scale to capture preferences outside of clinical factors such as “reduced risk of breast cancer” and “increased risk of endometrial cancer”. These concerns were addressed by completely redesigning the game to make the scale more intuitive. The changes included embedding the sliders directly into the page and allowing the scales to move with each slider. A Likert-type scale and color was also added to each slider. Examples of new nonclinical preference factors added to the game include “taking control of my health”, “don’t trust medications”, and “availability of social support”. A comparison of the before and after of this interactive game for chemoprevention can be seen in Figure 3 .

Users were also confused about the purpose and structure of the action plan at the end of RealRisks. Original versions of the action plan did not specify explicitly that it was to be shared with her health care provider, and the layout of the page made it difficult to read. These issues were addressed by including more information about the purpose of the action plan and restructuring the page to be easier to comprehend.

Ease of Use Issues

Ease of use concerns were the most frequently brought up usability issue with RealRisks, but were generally minor to fix. These concerns primarily related to how easy it was to use some of the features of the games. In regards to the preference elicitation scales, there were issues with the sensitivity of the scales and the drag and drop functionality. We amended the tool to have all options preloaded, so subjects would not have to drag individual boxes (Figure 3 ).

For the risk assessment games, users were asked to click on a pictograph of 100 figures to identify one with breast cancer, which is a novel experience-based format for conveying risk information to those with low numeracy. Participants stated that the game took too long to complete and that it wasn’t clear how they were supposed to move on from the game (Figure 4 ). These issues were amended by shortening the number of clicks necessary to identify a breast cancer patient, improving the feedback for when a person was clicked, and adding a pop-up window to automatically find the patient after a certain number of clicks. Another issue with the clicking game that was brought up by both English and Spanish-speaking subjects was that the different rounds of the game were difficult to tell apart. Explanations of the risk of breast cancer appear in a yellow box to the right of the game, however, some participants found it confusing that the 5-year risk game was not further differentiated from the lifetime risk game. This concern was addressed by expanding the instructions and clearly labeling the two games.

Some participants voiced difficulty with reading all of the information on the dense narrative screen – either it hurt their eyes or they would prefer to listen to the material. RealRisks was designed to include an audio option, and this feature was added in English and Spanish. Another ease of use issue occurred with the family tree in the family history module. In this game, subjects are asked to create a family tree by inputting information about their family. However, the final product did not include an index for what different shapes, colors, and lines meant, therefore, a legend was added to the family tree.

Navigation Issues

Navigation concerns were a minor recurring concern among study participants. Subjects generally chose to follow through the modules in order and found the process of moving through RealRisks to be intuitive. One recurring issue relating to navigation was confusion regarding where in the application the subject was supposed to go next. After completing a module, subjects would be brought back to the main screen with no indication of which module was meant to be next. This was addressed by improving the performance of a “checkbox” feature, that indicated to the subjects which items they had completed. When users start RealRisks, they are briefly guided through the features of the PtDA by a series of on-screen instruction boxes. One recommendation that was brought up by older English and Spanish-speaking subjects was the desire for some sort of tutorial to help guide participants through the rest of the PtDA. Another suggestion was to turn the introductory guide into a video to better explain the features of RealRisks.These recommendations will be implemented in a future version of the PtDA.

System Usability Scale Scoring

Every participant completed the SUS questionnaire at the end of the usability study. Scores varied appreciably between English and Spanish-speaking subjects. English-speaking users gave RealRisks a median score of 80.00 (range, 55.00-95.00), while Spanish-speaking users gave the application a median score of 66.30 (range, 55.00-75.00). The mean score between the two groups was 72.00. An empirical review of SUS scores has indicated that a SUS score between 65 and 70 is considered “ok” or “high marginal”, while a score above 70 is considered to be “good” or “acceptable” usability for a web-based application.^29,31 This would indicate the need for further analysis and improvement of the RealRisks application based on the feedback we received from Spanish-speaking users.

Discussion

The aim of this series of usability studies was to evaluate RealRisks in order to better tailor the patient decision aid (PtDA) to women at high-risk for breast cancer, particularly multi-ethnic women with a broad range of health and computer literacy. By conducting studies in both English and Spanish, we were able to identify several issues across three themes that reduced the usability of the application for our target population. This study illustrates the importance of usability testing in creating culturally relevant and targeted PtDAs, particularly for web-based tools designed to be used by diverse populations with varying health literacy and acculturation.

Uptake of antiestrogens for chemoprevention remains low, in large part due to patients and providers viewing the risk of side effects as disproportionately higher than the potential benefits of breast cancer risk reduction.^8,9 The literature supports that inaccurate perception of risk may lead women to overestimate their own breast cancer risk or the risk of side effects for chemoprevention agents.^32,33 Thus, one of the goals in designing RealRisks and the corresponding usability studies was to improve the accuracy of users’ perception of risk. The “clicking” games, designed to help users explore pre-set probabilities, were developed based on previous research on risk perception but were not adequately describing risk to study participants in the format they were presented.^19,22 By updating the feedback and explanations for these games, we were able to improve the application for all users.

Another key feature that differentiates RealRisks from other chemoprevention PtDAs is the use of preference elicitation games to engage users in making decisions based on their personal preferences. In the original version of RealRisks, users found these games difficult to use and reported confusion about their purpose. By redesigning the tool as a series of sliding scales, users are now able to adjust their preferences towards various aspects of genetic testing or chemoprevention in order to make more informed decisions..Furthermore, by adding more options to the scales, users are empowered to make decisions based on clinical and non-clinical factors, such as how chemoprevention will impact their family. By including the preference scales in the final action plan, users will be able to review their personal preferences towards chemoprevention with their family and primary care provider (PCP).

In our study, Spanish and English-speaking participants voiced similar concerns, and differences tended to arise based on health literacy and numeracy rather than acculturation. Spanish-speaking participants rated the PtDA as somewhat less useful based upon the SUS. Research on breast cancer decision-making among low-literacy Hispanic women, such as those targeted by RealRisks, has found that this population tends to report poorer decision outcomes when using a PtDA. PtDAs for this population tend to be more effective when they are culturally appropriate and better integrate clinicians and family members into the decision-making process.³⁴ RealRisks is designed to provide women with an action plan that they can discuss later with their PCP and family members to facilitate decision-making about chemoprevention. Furthermore, RealRisks will be integrated into clinic workflow with the BNAV tool for providers. In an ongoing pilot study with 50 patient-provider dyads and and upcoming randomized controlled trial of RealRisks for high-risk women and BNAV for PCPs, we will assess shared decision-making when both patients and providers are primed prior to the clinical encounter with these decision support tools.

Strengths and limitations

There were several strengths to our usability studies. We achieved a sufficient sample size to identify at least 80% of usability concerns with RealRisks. The recurrence of certain themes in later studies indicated that we reached the saturation point of usability testing.^20,21 Features that were problematic during the first iteration of tests, such as the preference elicitation scales, were viewed as more usable in the second round. Usability testing was conducted in both English and Spanish in order to address concerns relevant to the multi-ethnic population that the PtDA was designed for. Furthermore, the population included in the study was diverse and represented a broad range of ages, ethnicities, and acculturation levels. The majority of problems for English-speaking subjects were also found by Spanish-speaking users, indicating that there may be similarities in how both populations view PtDAs, despite differences in acculturation and health literacy.

Limitations in our study were primarily due to the structure of RealRisks and usability testing in general. The web- based format of the PtDA restricted the inclusion of women who were computer illiterate or unable to read a computer screen. Additionally, the think-aloud protocol was difficult for older participants to understand and utilize and necessitated frequent prompting by the study moderator. While think-aloud protocols are standard for usability testing, there is some evidence that using concurrent think-aloud methodology might decrease user engagement during studies.³⁵ While our sample was diverse, recruitment was limited to women who were able to come in for testing during work hours and thus may not have fully captured usability issues that would impact all populations that use RealRisks. Finally, the time limit placed on each usability study meant that some subjects were unable to complete the entire PtDA in a single sitting. Participants were guided towards relevant sections when time was low, and therefore certain problems may have been missed by users who skipped modules.

Conclusion

In conducting usability studies for the RealRisks decision aid, our team was able to identify several key issues that informed the design of the tool for our target population of low-literacy, multi-ethnic women. Our study is unique in that it incorporated the suggestions and feedback of a very diverse population of potential users and found that concerns were similar between English and Spanish-speaking participants but that differences arose based on health literacy and numeracy. Users reported moderate to high satisfaction with the overall usability of the tool, and future changes will be made to address concerns specific to Hispanic users. Future usability studies of web-based PtDAs should take into account the computer literacy, acculturation, and health literacy of participants when testing in order to develop tools which are accessible to diverse populations.