User Comprehension and EHR Integration of the RealRisks Decision Aid for Breast Cancer Risk Assessment: A Qualitative Study

Abstract

RealRisks is a decision aid that integrates patient-generated and electronic health record (EHR) data using Fast Healthcare Interoperability Resources (FHIR). It offers modules to enhance understanding of breast cancer risk and a way for individuals to review and modify their EHR data before it is used in their personal risk assessment. RealRisks intends to encourage high-risk patients to take risk-reducing measures. To better understand how patients understand risk and barriers to action, we conducted in-depth interviews as part of a usability study to assess the clarity and interpretability of RealRisks. Overall, participants demonstrated an improved understanding of breast cancer risk after using RealRisks. However, challenges were noted for certain concepts, in particular, lifetime risk, how benign breast disease affects your risk, and the differences between hereditary, sporadic, and familial cancer. The EHR download feature was well-received, but some raised concerns about insurance and privacy/security.

Introduction

Breast cancer is the most common cancer among women in the United States, with approximately 280,000 new cases and 40,000 deaths from breast cancer each year¹. While mortality has declined over the past three decades, this decline has begun to plateau, particularly among racial/ethnic minorities². Women identified as high risk for breast cancer may benefit from chemoprevention, testing for breast cancer susceptibility genes, screening for early detection, and other risk-reducing strategies. However, implementing these personalized approaches is hindered by several obstacles, primarily due to the time-intensive nature of conducting individualized risk assessments across populations³. Electronic health records (EHRs), a common source for populating risk assessment models, present their own challenges^{4, 5}, including mistakes⁶ and missing data^{7, 8}. Moreover, some patient-provided data tend to be more accurate than EHR data’. We previously extracted EHR data on age, race/ethnicity, family history of breast cancer, benign breast disease, and breast density to calculate breast cancer risk according to the Breast Cancer Surveillance Consortium (BCSC) model among 9,514 women’. When comparing self-reported data with EHR data, a significantly higher proportion of women, specifically those with a first-degree family history of breast cancer (14.6% vs. 4.4%) and benign breast biopsies (21.3% vs. 11.3%), were identified through patient-reported data. Conversely, EHR data identified more women with atypia or lobular carcinoma in situ (1.1% vs. 2.3%). Notably, the EHR had missing data on race/ethnicity for 26.8% of women and on first-degree family history of breast cancer for 87.2%. In another study, we calculated eligibility for genetic counseling using the six-point scale (SPS)¹⁰, a validated family history screener used to determine eligibility for BRCA1/2 genetic counseling¹¹. We then calculated SPS scores using structured and free-text EHR data and compared the results with SPS scores calculated from self-reported data. Our analysis revealed that more women met eligibility for genetic counseling referral based on self-reported data (12.1% vs. 1.4%)¹¹. Opportunely, Fast Healthcare Interoperability Resources (FHIR), application programming interfaces, and recent legislation, including the 21st Century Cures Act¹², offer an elegant solution for automated breast cancer risk assessment that integrates both patient-generated and EHR data to harness the strengths of each approach^12-14.

To that end, Kukafka et al. developed the RealRisks decision aid using an iterative design process to maximize acceptability, feasibility, appropriateness, and usability equitably^15-17. RealRisks promotes understanding of breast cancer risk and collects patient-entered data to calculate breast cancer risk according to the Gail model¹⁸, BCSC¹⁹, and BRCAPRO²⁰ to estimate a woman’s risk of developing breast cancer and carrying a high penetrance germline pathogenic variant (i.e., BRCA1/2). When FHIR became widely available, RealRisks was updated to automatically populate patient information for breast cancer risk calculation from the EHR. Moreover, Kukafka et al. designed a prototype user interface that displays this data to patients, allowing them to review and modify data before running the risk assessments. RealRisks was previously evaluated in two randomized controlled trials among 300 high-risk women eligible for chemoprevention and their providers^21-24 and 190 women for BRCA1/2 genetic testing and their providers^{25, 26}. Comparing the intervention and control arms, there were significant differences in some patient reported outcomes, such as informed choice and mean decision conflict, but no significant differences in chemoprevention or genetic testing uptake. The goal of this study is to improve the usability of RealRisks and better understand the nuances of how women understood the different modules of RealRisks. Ultimately, our goal is to increase the number of patients who take action as a result of their understanding of their personal risk.

RealRisks consists of seven core modules: 1. Understanding Risk; 2. Breast Cancer Risk Factors; 3. Family History; 4. Genetic Testing; 5. Breast Cancer Risk-Reducing Pill; 6. Lifestyle Behaviors; and 7. Action Plan. Each learning module presents information in two formats: comic-style “chapters” and detailed “learn more about” infographics. The comic-style “chapters” provide a topic summary with cartoon characters, while the “learn more about” infographics offer in-depth information on the topic. Module 1 (Understanding Risk) introduces statistical risk and presents the current breast cancer incidence, following which users can engage in a general risk simulation, allowing users to experience 5-year and 10-year risk scenarios. Module 2 (Breast Cancer Risk Factors) discusses major breast cancer risk factors, including genetics, family history, estrogen, dense breast tissue, benign breast disease, body mass index (BMI), and alcohol consumption. At the end of Module 2 (Breast Cancer Risk Factors), users are prompted to connect to their EHR, after which their data is downloaded for risk calculation. Users then have an opportunity to review and modify their data before proceeding with the risk assessments. Following this, users can engage with a personalized risk exercise to explore their 5-year and 10-year risk based on their individual risk factors. Module 3 (Family History) delves into family history, the difference between sporadic and inherited cancer, and genetic inheritance patterns. At the end of Module 3 (Family History), users are prompted to input their family history for genetic testing eligibility assessment. Family history was initially collected using a pedigree chart but was later replaced with a family history screener using a validated instrument, the SPS¹⁰. Upon navigating to Module 4 (Genetic Testing), users are informed of their eligibility for genetic testing based on the results of the family history screener. Regardless of their result, all users can view the content of Module 4 (Genetic Testing), which explores pros/cons, indications, and other matters relating to genetic testing. Module 5 (Breast Cancer Risk-Reducing Pill) informs users whether they qualify for chemoprevention based on their breast cancer risk. All users can view Module 5 (Breast Cancer Risk-Reducing Pill), which offers insights into different risk-reducing pills, including their potential risks/benefits and indications. Module 6 (Lifestyle Behaviors) presents several modifiable lifestyle factors that can influence risk, including weight, diet, alcohol use, and exercise. Finally, the user is invited to “build an action plan” in Module 7 (Action Plan) that summarizes their family history, indications (or lack thereof) for genetic testing and chemoprevention, and appropriate next steps.

This current study aimed to assess users’ ability to understand key concepts communicated in RealRisks, including general risk, personal risk, as well as risk factors like family history and genetics. Our focus is to uncover the nuance of participants’ grasp of the concepts communicated within the modules, as their understanding can potentially impact the effectiveness of the tool. Our analysis assesses the clarity and interpretability of the educational modules and what the women understood. Additionally, we were interested in the effectiveness of a new feature in RealRisks, which allows women to download, review, modify, and supplement their EHR data.

Methods

Participants. A total of ten participants were recruited for the study. We recruited from a cohort of women screened in a prior study while undergoing screening mammography²². We had information on their sociodemographics, breast cancer risk factors, and breast cancer risk, calculated based on the Gail model¹⁹. Our recruitment targeted this diverse cohort to ensure representation across various educational backgrounds and racial/ethnic groups. Eligibility criteria included: 1) Women aged 35-74 years, without a personal history of breast cancer, 2) High-risk defined as 5-year invasive breast cancer risk ≥1.7% or 10-year breast cancer risk ≥20% according to the BCSC²⁰ or Gail models¹⁹ 3) English- or Spanish-speaking, and 4) Able to sign informed consent. Initial contact was made via email, providing study details and inviting participation, followed by telephone screening to confirm eligibility. Eligible participants received informed consent forms (ICF) detailing the study objectives, procedures, potential risks, benefits, and their rights as participants. Upon the receipt of signed ICFs, participants were scheduled for in-depth interviews. Each participant was granted individual credentials to log in and engage with RealRisks on their own time before the interview. Finally, the participants were interviewed via Zoom for approximately 90 minutes. Participants received a compensation of $100 in the form of Amazon gift cards. The Columbia University Irving Medical Center (CUIMC) institutional review board approved the recruitment strategy and materials.

Interview Process. The interviews were facilitated via a semi-structured interview guide and comprised two distinct phases: concept elicitation²⁷ and cognitive debriefing²⁸. The initial 15 minutes were dedicated to concept elicitation, during which participants engaged in an open-ended conversation that delved into their understanding of breast cancer and breast cancer risk, allowing them to express their thoughts and perceptions freely. Following this exploratory phase, participants were cognitively debriefed as they were asked to go through RealRisks and provide feedback via the “think aloud” technique, wherein participants are encouraged to verbalize their thought processes²⁸. For this study, the scope of content and engagement was limited to the first three modules, i.e., Module 1 (Understanding Risks), Module 2 (Breast Cancer Risk Factors) and Module 3 (Family History). The purpose of this cognitive debriefing phase was to assess whether participants were able to interpret Module 1 (Understanding Risks), Module 2 (Breast Cancer Risk Factors), and Module 3 (Family History) accurately and found the content to be clear and comprehensible. We also assessed the fidelity of the understanding of the risk they came away with. Additionally, we explored participants’ feedback regarding the connection to their EHR, specifically assessing their comfort levels with the data download feature. For those who expressed hesitance, we inquired into their reluctance to identify potential barriers or concerns related to EHR connectivity. Interviews were audio- and video-recorded via Zoom and transcribed verbatim.

To describe RealRisks in more detail: Module 1 (Understanding Risk) consisted of three sections: 1) Chapter 1, a summary of breast cancer risk presented in a comic style format; 2) Learn More About Risks, featuring four infographic-style sections: What is Risk?, Breast Cancer is Common, Who Might Develop Breast Cancer?, and Knowledge; and 3) Risk Exercise, a general risk simulation allowing users to experience 5-year and 10-year risk scenarios by interacting with a visual representation of 100 women, some of whom may have breast cancer, without prior identification. Module 2 (Breast Cancer Risk Factors) consisted of three sections: 1) Chapter 2, a summary of breast cancer risk factors presented in a comic style format; 2) Learn More About Breast Cancer Risk Factors, featuring six infographic-style sections: Inheriting a Strong Breast Cancer Gene and/or Having a Family History of Breast Cancer are Strong Risk Factors, Estrogen Can Increase Breast Cancer Risk, Dense Breast Tissue is a Risk Factor, Benign Breast Disease is an Important Breast Cancer Risk Factor, Some Lifestyle Factors Can Lower Breast Cancer Risk (BMI), and Some Lifestyle Factors Can Lower Breast Cancer Risk (Alcohol Consumption); and 3) Find Out Your Breast Cancer, where users can download, review, and edit their EHR data prior to risk assessments, following which users are engaged in a Risk Exercise, a personalized risk simulation allowing users to experience their 5-year and 10-year risk scenarios by interacting with a visual representation of 100 women, some of whom may have breast cancer, without prior identification. Module 3 (Family History) consisted of three sections: 1) Chapter 3, a summary of family history presented in a comic style format; 2) Learn More About Family History, featuring five infographic style sections: Important Things to Know About Your Family History of Cancer, Gene Mutations can be Inherited or Acquired, Hereditary Cancer, Sporadic Cancer, and Patterns of Inheriting Genetic Mutations; and 3) Collect Your Family History, a section designed to collect user’s family history information.

Data analysis. For each of the three modules, participants were asked to interpret each section in their own words, and if their interpretation aligned with the intended meaning, it was considered interpretable. Participants who interpreted the section accurately were then asked to assess the clarity of that section; if they found it unclear, they were given the option to provide reword suggestions. Finally, participants provided suggestions for improving each section. Interpretation and clarity were quantified based on two coders’ reviews of the transcripts. The transcripts were also to understand users’ ability to understand key concepts communicated in RealRisks²⁹. The two coders met to develop a codebook of themes. All transcripts were uploaded into ATLAS.ti software to enable investigators to build the codebook and code into themes. The coders met regularly to compare coding as they emerged from interview transcripts and to modify the codebook. Discrepancies between the two coders were discussed, and the coding of each transcript was compared consecutively. All investigators participating in the study reviewed exemplar quotations and contributed to the iterative refinement of themes. Data was also used to suggest refinements to RealRisks to improve potential issues affecting interpretability, clarity, and ease of use.

Results

In this section, we present the demographics of the participants, their baseline understanding of breast cancer risk probed during the first phase of the interview, and review in detail the interpretability and clarity assessment of three modules within the RealRisks decision aid: Module 1 (Understanding Risk), Module 2 (Breast Cancer Risk Factors), and Module 3 (Family History). We use quotes from the subjects to illustrate some misunderstandings and, more importantly, to show how someone can understand a basic concept earlier in the decision aid but needs help to apply their understanding correctly as more concepts are introduced.

We present results from the concept elicitation phase of the interview to assess baseline understanding. During this phase, participants provided insights into their understanding of breast cancer, its prevalence, and their perceptions of individual risk factors. This phase offers crucial baseline information for assessing participants’ knowledge and attitudes towards breast cancer risk.

Subsequently, we present the assessment of three modules within the RealRisks decision-aid, focusing on their interpretability and clarity. Specifically, we examine participants’ responses to various visual aids and interactive simulations designed to convey information about breast cancer risk and risk factors. Through this exploration, we aim to uncover the reasons behind participants’ misinterpretations or perceptions of unclear information, thereby providing insights into the nuances of their comprehension processes.

Participant Demographics. The mean (standard deviation [SD]) age of participants was 59.7 (6.8) years (range 5067), and 100.0% of participants were female. Most participants were black or African American (50.0%) and not Hispanic or Latino (70.0%). The education levels among participants varied, with some having master’s degrees (n=3), followed by associate degrees (n=2), professional school degrees (n=2), and some college without obtaining a degree (n=2). The majority of participants (n=7) were employed.

Baseline Understanding. Participants were asked to describe breast cancer and estimate its prevalence. Almost all participants (n=9) described breast cancer accurately. Moreover, most participants (n=8) perceived breast cancer as “common”. Additionally, participants were asked to comment on their risk for breast cancer. Almost all participants (n=9) demonstrated a basic understanding of risk for breast cancer, with one participant remarking, “As a woman with breasts, you are at risk. There is a potential that you could get breast cancer.” When probed further on the nature of risk, 80% (n=8) emphasized that breast cancer risk entails the possibility rather than the certainty of developing the condition, and 70% (n=7) noted that risk could be modified. Two participants (n=2) expressed skepticism regarding the modifiability of breast cancer risk, with one stating, “I don’t think you can necessarily change the chance that you might get it.” Finally, participants varied in their frequency of considering breast cancer risk, with responses ranging from “never” (n=2) to “not often” (n=1), “a couple of times” (n=2) and “during mammograms” (n=1).

Module 1: Understanding Risk. The figure below (Figure 1) summarizes the interpretability and clarity of module 1 (Understanding Risks), designed to convey the concept of risk. The module starts with an introduction to risk, defining what risk entails, followed by a discussion on breast cancer as a prevalent condition and the individuals at risk of developing it. Users then engage in a general interactive 5-year and 10-year risk simulation.

Figure 1.

RealRisks Module 1 (Understanding Risks): Interpretability and Clarity*

*Due to missing data, not all categories sum up to N=10.

All participants (n=10) found Chapter 1, a comic-style summary of breast cancer risk, to be interpretable, and 90% (n=9) to be clear.

Infographic 1, What is Risk?, defines risk and illustrates different risk levels with two individuals standing on a dock next to a lake for low risk, seated in a boat equipped with life vests for medium risk, and seated in a boat without life vests for high risk. All participants (n=10) found it interpretable, and 70% (n=7) found it clear. Two participants (n=2) expressed clarity issues, with one participant noting that the infographic seemed unnecessary and the other participant noting that the definition of risk was confusing, stating the following: “I understand, but in the beginning, it was kind of confusing this scenario.”

Infographic 2, Breast Cancer is Common, emphasizes the prevalence of breast cancer among women in the United States and provides the risk of developing breast cancer for women who live to 90 years of age. While the majority of participants (n=7) found it interpretable, three participants (n=3) struggled with understanding the concept of lifetime risk and the specified age of 90 years. Two participants struggled to grasp the concept of probability, with one stating, “It says for women who live to 90 years of age, about 12 out of 100 of them will develop breast cancer. … I don’t agree with that statement because it’s like saying it’s a definite 12 out of 100 women who live to 90 years will develop breast cancer”, while one participant incorrectly stated, “1 out of 100 women will live 90 years old and develop breast cancer.” Of the participants who found it interpretable (n=7), one (n=1) found it unclear and was confused by why 90 years was specified.

Infographic 3, Who Might Develop Breast Cancer? distinguishes between sporadic, hereditary, and familial cancer. While the majority of participants (n=8) found it interpretable, two participants (n=2) did not interpret it as intended. One participant incorrectly equated hereditary with familial risk, while the other participant struggled to grasp the relative risk of sporadic, hereditary, and familial cancer: “It’s telling me that now it’s 26 out of 100 if you have a family history. Here it says only 5 out of 100. So only about 5 in 100 cancers are hereditary, but here you’re saying there’s what, 26?” Of the participants who interpreted it as intended (n=8), two participants (n=2) found it unclear; one participant found the word ‘familial’ unclear, and another participant thought the wording could be presented in a less confusing manner.

Infographic 4, Knowledge, emphasizes the intersection between lifestyle factors and genes in breast cancer risk. The majority of participants (n=8) found it interpretable, and 60% (n=6) found it clear. This infographic was later removed as it was considered to be redundant.

Regarding the comprehension of 5-year and 10-year risk, 80.0% (n=8) demonstrated interpretability, while two participants (n=2) did not grasp the concept accurately. One participant inaccurately stated that the 10-year risk is “Thirty out of 100 in 10 years”, while another participant equated 5-year risk to lifetime risk: “I would have thought 12%, that there would be 12 of them that would have it, so is this saying that 1%?” Of the participants who accurately interpreted the risk exercises (n=8), one participant (n=1) found the purpose of the exercise unclear, stating, “I just don’t know why you’re doing it.”

Module 2: Breast Cancer Risk Factors. The figure below (Figure 2) summarizes the interpretability and clarity of the RealRisks module 2 (Breast Cancer Risk Factors), designed to convey risk factors and engage users in interactive 5-year and 10-year risk simulations based on their patient-generated and EHR data.

Figure 2.

RealRisks Module 2 (Breast Cancer Risk Factors): Interpretability and Clarity*

*Due to missing data, not all categories sum up to N=10.

Sixty percent of participants (n=6) found Chapter 2, a comic-style summary of breast cancer risk factors, to be interpretable and clear. One participant (n=1) had trouble interpreting the contents of the comic, specifically struggling with benign breast disease: “… so this is saying that if you have a negative biopsy, you still have breast disease? … No matter what your breasts are doing, you have breast disease.”

Infographic 1, Inheriting a Strong Breast Cancer Gene and/or Having a Family History of Breast Cancer are Strong Risk Factors, highlights the significance of BRCA gene mutations and the varying impact of family history based on the degree of relatedness and age at diagnosis. Seventy percent of participants (n=7) found it interpretable, while one participant (n=1) incorrectly stated that individuals with first-degree relatives diagnosed before age 50 have the highest risk. Of those who found it interpretable (n=7), three participants (n=3) encountered clarity issues. Two participants experienced difficulty interpreting graphs related to this topic, while one participant found the terms ‘first-degree relative’ and ‘second-degree relative’ confusing.

Infographic 2, Estrogen Can Increase Breast Cancer Risk, explains the link between increased exposure to estrogen and heightened breast cancer risk. Most participants (n=8) interpreted the infographic as intended, and of the participants who found it interpretable (n=8), six participants (n=6) found it clear. One participant (n=1) found the concept to be slightly unclear, stating that, “If I was a hypochondriac, I would be at my doctor’s office going oh my God, I have 50% chance based on that …”

Infographic 3, Dense Breast Tissue is a Risk Factor, introduces the concept of breast density and its relation to breast cancer risk. Seventy percent of participants (n=7) found this infographic to be interpretable and clear, although one participant (n=1) was unable to make the connection between breast density and the relative risk of breast cancer.

Infographic 4, Benign Breast Disease is an Important Breast Cancer Risk Factor, introduces the different categories of benign breast disease and their relation to breast cancer risk. Only 40% of participants (n=4) found this infographic interpretable and clear. Three participants (n=3) were confused by the term ‘benign breast disease’, with one participant inaccurately equating lobular carcinoma in situ with a cancer diagnosis.

Infographic 5, Some Lifestyle Factors Can Lower Breast Cancer (BMI), highlights the importance of maintaining a healthy BMI to lower risk. The majority of participants (n=8) found this infographic interpretable, with most (n=7) also finding it clear. However, one participant (n=1) expressed reservations about the use of BMI, highlighting potential cultural and racial biases in its application: “I don’t talk in BMI … I read something about how the BMI readings and how they work it out is based on a white demographic and there’s some difference in terms of racial origin, I think.” Infographic 6, Some Lifestyle Factors Can Lower Breast Cancer (Alcohol Consumption), emphasizes the importance of monitoring alcohol intake to mitigate breast cancer risk. Seventy percent of participants (n=7) interpreted the infographic as intended, while one (n=1) struggled to grasp the concept of a serving size. Of the participants who found it interpretable (n=7), one participant (n=1) encountered some clarity issues, stating, “I was just wondering like one serving of alcohol means it is the same if I drink a cocktail or a glass of wine?”

Regarding personal risk assessment, most participants (n=8) could interpret their 5-year and 10-year risk based on individual risk factors. However, one participant (n=1) encountered challenges recognizing that the risk assessment was tailored to them: “Now, this one is in terms of family history, so that was the difference. … I know the first one was just random people who had nothing to do with family members or anything like that. It’s your risk as a normal person who doesn’t have any risk factors in the family. This one is in terms of the family.” Among participants who interpreted the exercise as intended (n=8), five participants (n=5) found it to be clear.

Module 3: Family History. The figure below (Figure 3) summarizes the interpretability and clarity of the RealRisks module 3 (Family History) designed to convey the importance of family history, genetic mutations (both inherited and sporadic), and patterns of inheritance in understanding risk for breast cancer.

Figure 3.

RealRisks Module 3 (Family History): Interpretability and Clarity*

*Due to missing data, not all categories sum up to N=10.

Sixty percent of participants (n=6) found Chapter 3, a comic-style summary of breast cancer risk factors, to be interpretable and clear.

Infographic 1, Important Things to Know About Your Family History of Cancer, outlines that breast cancer risk is dependent on the degree of relatedness to affected family members, the type of cancer diagnosis, and the age at which the diagnosis occurred. Most participants found this infographic interpretable (n=9) and clear (n=8).

Infographic 2, Gene Mutations can be Inherited or Acquired, introduces the concept of genetic mutations being either inherited or acquired. Most participants (n=9) found this infographic interpretable. Of those who interpreted it as intended (n=9), one participant (n=1) expressed some issues in clarity: “Okay, that’s not very clear, I don’t think. Can you only get a tumor if two genes are damaged?”

Infographic 3, Hereditary Cancer, explains inherited gene mutations. Six participants (n=6) interpreted it as intended, while three participants (n=3) did not. One participant expressed uncertainty, stating, “I mean I understand it but I don’t understand it. I understand the last sentence, but just looking at the cycle, trying to understand.” Another participant questioned the origins of DNA damage: “How does that happen? What can cause a gene to have additional is it from the alcohol, is it those things that I don’t understand?” Yet another participant incorrectly stated that the infographic was describing “stages of cancer being developed.” Among those who found it interpretable (n=6), one participant (n=1) noted that the infographic could be clearer.

Infographic 4, Sporadic Cancer, explains acquired gene mutations. Seventy percent (n=7) interpreted it as intended, while two participants (n=2) did not. One participant questioned the origins of DNA damage, while the other participant incorrectly stated the following: “Well this is basically the difference between a mutation that you inherit and a mutation that you acquire and that you need the two to basically develop the cancer.” Of the participants who found the infographic interpretable (n=7), six participants (n=6) also found it clear; one participant (n=1) noted that it could be clearer.

Infographic 5, Patterns of Inheriting Genetic Mutations, explains that BRCA mutations are autosomal dominant. Most participants (n=9) found this infographic to be interpretable, and of those who found it interpretable (n=8), five participants (n=5) also found it to be clear. Two participants (n=2) stated that the concepts could be clearer and that it took a while to unpack it.

Module 3 (Family History) ended with the collection of family history data. While we initially utilized a pedigree chart, this was later switched to the SPS¹⁰, to decrease participant fatigue and enhance the likelihood of completion. The pedigree chart was tested on five participants (n=5), two of whom (n=2) found it interpretable and clear. The SPS¹⁰, on the other hand, was tested on five participants (n=5), four of whom (n=4) found it interpretable and clear.

Comfort With EHR Download. Seven participants (n=7) chose to download their EHR data, while three (n=3) had trouble connecting to their patient portal due to trouble accessing their credentials. All participants, regardless of whether they were able to download their EHR data, were asked to express their levels of comfort with respect to connecting to their patient portal. While five participants (n=5) expressed feeling comfortable downloading their data, three participants (n=3) expressed some discomfort. All three participants expressed insurance concerns, with one stating, “It did flash through my mind that you don’t share with insurance companies, right?” and two participants had general privacy/security concerns, with one participant stating the following: “As I said, it’s just having that information out. I’m not on any insurance or anything like that, but just high risk and things of that nature being out or potentially somewhere I can be considered, as I’m older, in my 40s, like would that work against me in some way.”

Conclusions

Participants showed a basic understanding of breast cancer and perceived it as common. Most participants also showed a basic understanding of breast cancer risk, recognizing it as a possibility rather than a certainty, with many acknowledging the potential to modify risk factors. However, some participants who understood basic concepts earlier in the decision aid needed to be able to apply their understanding correctly or extend their understanding as more concepts were introduced. For example, they were understanding risk at a point in time but not understanding risk over a period of time (e.g., lifetime risk), understanding risk for a population but not understanding how risk can be different for different subsets of the population, and not understanding how risk can be conditional. These limitations in understanding risk are essential as they may limit the tool’s effectiveness in helping participants make informed decisions based on their personalized risk. They may also impact user motivation to take serious steps toward reducing breast cancer risk. We were also reminded that participants are balancing health risks with financial risks and might be hesitant to know more about their health or take other action if, by doing so, they perceive there is a risk of their insurance company reducing coverage.

These conclusions are based on close listening to ten participants thinking out loud as they navigated the RealRisks decision tool. What we learned from their in-the-moment reflections has for us added a new level of understanding of how they think about risk beyond the quantitative surveys that are mostly relied upon.

Overall, module 1 (Understanding Risk) was well-received, with most participants demonstrating a basic understanding of risk. However, some infographics, particularly regarding lifetime risk and distinguishing between sporadic, hereditary, and familial cancer, posed challenges for interpretability and clarity. Specifically, some participants needed help connecting risk and probability. Additionally, while most participants understood the 5-year and 10-year risk exercises, some needed clarification on the purpose.

Concepts within module 2 (Breast Cancer Risk Factors) varied in interpretability and clarity. Most participants understood the significance of BRCA gene mutations and family history as strong risk factors. However, some encountered clarity issues with terms like ‘first-degree relative’ and ‘second-degree relative.’ While risk factors such as estrogen and dense breast tissue were generally well understood, participants struggled with the term ‘benign breast disease. Lifestyle factors affecting breast cancer risk, such as BMI and alcohol consumption, were generally interpretable and clear. Most participants were able to interpret their personalized 5-year and 10-year risk assessments based on their individual risk factors. Still, some encountered challenges in recognizing that in this module, we were tailoring risk assessments to them as individuals.

Concepts within module 3 (Family History), such as family history, genetic mutations, and patterns of inheritance, were mainly well-received; the majority of participants found them interpretable and clear. However, some participants encountered challenges with some infographics, particularly those explaining hereditary and sporadic cancer. Specifically, participants seemed confused by the origins of genetic damage. Furthermore, the transition from using a pedigree chart to the SPS¹⁰ for collecting family history data improved interpretability and clarity.

Finally, the EHR download feature in RealRisks was well-received, with most participants comfortably connecting to their patient portal. The primary concerns raised were 1) insurance implications and 2) privacy/security risks.

Informed by this more detailed awareness of how our participants were taking in what we were attempting to teach, we have developed a deeper empathy for them and have revisited the design of RealRisks. Words, concepts, and rules that are obvious to researchers and clinicians might introduce uncertainty for patients. On each page, the specific words, images, colors, and layout affect interpretability and clarity. For instance, to alleviate concerns about insurance implications and privacy/security risks associated with the EHR download, we explicitly stated the following: “This information will only be known to approved research staff who have undergone HIPAA and clinical trial training at Columbia University Medical Center.” Refer to the screenshots below (Figure 4) for a comparison: one shows the webpage prior to the revision and the other afterward.

Figure 4.

Screenshots before and after the revision.

In another instance, to clarify terms like ‘first-degree relatives’ and ‘second-degree relatives’, we provided examples such as more everyday terms such as parents or siblings for first-degree and aunts, uncles, or grandparents for second degree. Similar nuanced changes were made to every page. Design refinement will be an ongoing process throughout the lifetime of the tool based on user feedback and new emerging evidence to continuously improve the tool’s effectiveness and user experience for different populations.

We also reconsidered what needs to be included in the RealRisk modules. For example, how much biology do participants need to know to make an informed decision about their bodies? We want to respect our participant’s need for knowledge and their time by providing the information most salient to meeting the tool’s purpose. The extent to which RealRisks can become most efficient when applying empirical and best practices to communicate only essential concepts using optimally designed interfaces designed for varying levels of literacy continues to be an area of investigation for our research team.

As for future directions, currently, when users modify their risk factors based on their EHR data download or enter new information such as family history, these updates are not written back to the EHR. To address this, we are exploring what is needed to write patient-generated and patient-corrected data back to the EHR at our institution. We are conducting stakeholder interviews with providers, patients, and IT and operational leaders to identify potential implementation barriers. We are also conducting a pilot study to evaluate the refined version of RealRisks with a larger and more diverse population. This pilot study aims to assess both the tool’s usability and its impact on patient-reported outcomes, including patient activation and risk perception. Future research also includes expanding RealRisks to use the data downloaded by patients to run risk assessment models for more than one disease, for example, breast cancer and cardiovascular disease, to understand risk perception and decision-making in the context of multiple diseases.

Limitations. One limitation of this study is the potential presence of social desirability bias due to the interviewer’s presence during user interactions with RealRisks. Participants may have altered their behavior or responses, including their decision to download EHR data because they knew they were being observed. Future research could mitigate this limitation by exploring user behavior in a more naturalistic setting or by employing methods that minimize the influence of the interviewer.