Usability Evaluation of an Adaptive Information Recommendation System for Breast Cancer Patients

Abstract

We report on the usability of a mobile application, MyPath, that connects patients with personalized information based on their diagnosis and care plan and adapts over time as they progress through the cancer trajectory. We conducted usability tests with cancer survivors and health professionals, measuring three usability factors which could be affected by adaptive content: learnability, errors, and effectiveness. Our results indicate that the adaptive information did not obstruct usability of the system. Participants identified several strengths of the application, including the integration of clinical and non-clinical information, the segmentation of a large information set to reduce mental burden, and the inclusion of multiple media types to accommodate different learning styles. Participants also identified potential barriers to use and offered ideas for future developments. We share how we integrated this feedback into the MyPath system design and reflect on lessons for future personal health information systems.

Introduction

Following a cancer diagnosis, access to informational resources can improve patients’ engagement in their care and overall quality of life^1,2. Information about the disease can increase patients’ confidence in their ability to cope with the effects of the disease at home^1,3. Most often, patients will rely on health professionals and printed materials for information, though use of online health information has increased over the years^4,5. Notably, each of these information sources have limitations that affect their usability and accessibility. For example, patients can be hesitant to raise questions to health professionals^6,7. Printed materials can be difficult to organize, particularly as the collection grows over time, therefore making the information difficult to recall or retrieve later in the cancer trajectory, when the information could be best put to use⁸. Finally, patients have varying degrees of trust in online health information as these resources can be inaccurate and unreliable⁹.

We are exploring ways to use ubiquitous, mobile technologies to connect breast cancer patients with trusted health information while overcoming these existing barriers to information access. Prior patient-facing cancer management interventions that focus on informational support have been associated with improved symptom management¹⁰, increased confidence and engagement in care^11,12, and reduced symptom severity and distress^10,13. These systems include various features to support patients, including symptom monitoring^10,11,14, decision support¹⁵, and health management advice¹², which can be tailored based on user input¹⁴.

While these systems have demonstrated a number of positive effects, they often support a subset of patients’ information needs. Many existing systems focus on a particular moment in the cancer trajectory, such as during chemotherapy or radiation therapy. However, patients’ information needs both persist and adapt over time, as they transition from diagnosis through survivorship¹⁶. In addition, current systems tend to focus on helping patients manage the physical effects of cancer. Research consistently finds that patients’ information needs extend beyond physical health, and that patients often feel they do not receive enough information to effectively cope with the disease^1,17. Finally, while a small set of systems tailor content to the individual, we have not found examples of automatically adapting content based on an individual’s care plan. Such adaptations can provide patients with timely information to help them prepare for upcoming changes in care. Transitional periods can be vulnerable times in the cancer trajectory which alter patients’ self-management abilities¹⁸.

We are working with a team of oncologists, cancer navigators¹⁹, cancer survivors, and computer scientists to design a novel patient-facing health management application. This application, MyPath, provides patients with comprehensive health information, spanning beyond physical health information. The information adapts over time based on an individual’s care plan and treatment dates, as well as user input of current information needs.

We are employing a user-centered design process to develop a usable application that provides patients with information that aligns with their particular support needs and preferences. User-centered design involves a systematic process for including the intended users (e.g., patients and healthcare professionals) in the design and testing of interactive health technologies, to ensure that new tools are important and usable to the intended users²⁰. A core component of user-centered design, and one of the most influential components of the design process, is usability testing²¹. In usability testing, a small number of potential users of a system interact with a prototype, with the goal of identifying usability problems before the application is built²². While usability tests have been used in the design of tailored informational support systems²³, we did not find any prior studies that specifically test the effect of adaptive content on usability. We assess three usability factors: learnability, errors, and effectiveness. Prior studies have shown that adaptive content can influence each of these usability factors^24,25. In addition to measuring usability factors, we used a think aloud protocol, a method often used within usability assessments, to understand users’ expectations and reactions towards the technology²¹. Thus, the objectives of this study were to measure the effectiveness, learnability, and error rates of adaptive information recommendations, identify usability issues through participant feedback, and inform the design of future systems that tailor content over time.

Method

MyPath Description

MyPath is a mobile application that connects a patient with a trusted and comprehensive set of information related to cancer management, which adapts over time as the patient progresses through the cancer trajectory. This design goal was supported by our empirical work with cancer navigators, understanding the different types of support and information needs across the patient population¹⁹, as well as our work with breast cancer survivors creating journey narratives to assess unmet support needs²⁶. This work highlighted the broad and changing needs of breast cancer patients as they progress through multiple phases of care, and the need for support systems that reach across multiple phases of care, helping patients to cope with current challenges and prepare for future changes to their care.

MyPath’s database includes over 300 information resources from American Cancer Society, breastcancer.org, and cancer.net. We included these three sources based on oncologists’ recommendations for trusted online health information. MyPath’s information recommendations adapt in two ways. First, there is a user survey that may be completed at any time to immediately receive information related to current questions or challenges. The survey asks the user to select which challenges they are currently dealing with, and includes 58 possible responses. Survey items are based on the NCCN distress thermometer²⁷ and amended with additional challenges captured in our research with cancer survivors²⁶. Metadata within the MyPath database associates specific survey responses with relevant informational resources. Thus, when a patient selects a particular response in the survey, such as ‘Nausea’, resources related to that subject will immediately appear in the application when the survey response is submitted. While the research team created the metadata tags, multiple oncologists and oncology nurses validated that resources were appropriately characterized and tagged by the application.

In addition to tailored content based on survey responses, MyPath automatically recommends information as patients prepare for and encounter new treatments. Each MyPath user has a profile that includes all treatments and treatment dates. These dates are used to signal a transitional event that generates recommendations based on the patients’ treatment path. Metadata in the database connects transitional events, (e.g. surgery, radiation therapy, chemotherapy, hormonal therapy) to relevant resources. Resources appear one week before a new treatment, or at the end of a treatment, to help a patient learn what to expect and prepare for this change. For example, if a patient is receiving chemotherapy, then the system will identify the start date minus seven days as a transitional event, and on this date the user will see new resources associated with chemotherapy, which help them learn about the treatment, what to expect, and how to prepare.

MyPath Prototype

Once we established MyPath’s features, we then worked on multiple design options using low-fidelity prototypes to generate a proof of concept and elicit initial feedback from members of the research team. This iterative process led to the creation of a high-fidelity prototype. The prototype was created using the Justinmind prototyping software and displayed on Verizon Ellipsis 8 tablets. The prototype includes four sections: the home page, the information recommendations, the resource view, and the user survey (Figure 1). Within the prototype, participants were able to interact with all buttons and hyperlinks, allowing them to freely explore the system. When selecting one of the PDF resources, the prototype would link to the original online source, though these resources will be available offline in the final application.

Figure 1:

Four screens in the MyPath application.

Participants

We enrolled breast cancer survivors who had recently completed treatment. Access to a high number of cancer survivors was difficult as the cancer clinic serves a large, rural geographic region, with many survivors living a far distance from the clinic. Therefore, we also included cancer navigators and oncology nurses, who are highly knowledgeable of common questions and informational needs of patients throughout the cancer trajectory.

Usability task development and procedures

Due to the limited work evaluating adaptive interfaces, particularly in a health context, the usability assessments for MyPath were critical for understanding any complications caused by adapting the content over time. We developed three tasks in the usability test to demonstrate changes to the content over time, summarized in table 1. In the first task, participants were presented with information that would be displayed at the time of diagnosis. Before interacting with the prototype, participants were asked to reflect on either their own information needs during this time, or a common information need of their patients. Participants were then instructed to open the application and find two specific resources relevant to the issue they identified that would be helpful to a newly diagnosed patient. To accomplish this task, participants needed to 1) open the application, 2) select a category, 3) select one of the information pages to read, and 4) repeat steps 2-3 to find an additional resource. In the second task, we presented participants with health information relevant to a patient who was beginning radiation in one week. Again, we asked participants to first reflect on their own information needs during this time, or the needs of their patients. Participants were then asked to find two resources that address those information needs, thus repeating the same actions described in the first task. Notably, between the first and second tasks the facilitator updated the prototype to a second homepage view which had new resources added, thus visually mimicking the adaptive feature of the MyPath system. Participants were not told that the information recommendations would be updated.

Table 1:

Description of tasks included usability tests.

In the final task, we asked participants to imagine they were in the middle of treatment and to complete the user survey. While the final MyPath application will suggest resources based on these selections, the prototype did not include this feature. Therefore, we asked participants to describe the types of information they would expect to see appear after completing the survey.

Data collection and analysis

All usability sessions took place at the Harbin cancer clinic, in a private meeting room. The study involved a one-hour, one-on-one session with a member of the research team, who has a background in human-computer interaction and over seven years of experience running usability studies. While the facilitator was a member of the team who designed the application, participants were not made aware of this, in order to reduce possible influences on participants’ responses. Each session included the usability test, a post-test questionnaire, and interview. To ensure consistency, the facilitator used a script throughout the study to introduce participants to the study goals and procedures. The usability study was approved by the Georgia Institute of Technology’s Research Ethics Board.

We captured participants’ task performance, usability scores, and post-study feedback in order to assess three usability heuristics: learnability, errors, and effectiveness. Learnability refers to the increase in users’ ability to efficiently accomplish tasks over time, as they become familiar with a new system²⁸. Prior research has shown that adaptive interfaces can be unpredictable, thus affecting a system’s learnability²⁴. We expected that if the adaptive recommendations in MyPath impaired the system’s learnability, we would not see a reduction in task performance time between the first and second tasks. We measured task performance time as the elapsed time between the participant opening the application and the participant identifying (audibly) two useful resources.

Error rates are another metric used to identify significant usability issues in a design²⁰. Errors, noted by the facilitator, included an event in which a participant navigated to a page that did not match their intention (for example, looking for treatment information in the Social Support page), or participants were unable to locate an intended screen (such as the user survey). We used a think aloud protocol to capture participants’ intentions while using the system. In a think aloud protocol, participants are encouraged to think out loud as they interact with the application, so that the researcher may note when participants feel confused or experience difficulties. This method is useful for capturing obstacles that participants face throughout the usability assessment²⁹.

Similar to prior research, we included measures of effectiveness in the usability test to ensure that the application features were considered useful by participants²⁰. We both objectively and subjectively measured the system’s effectiveness. We used task completion rates as a measure of effectiveness, as a task was only considered complete if participants located resources that they considered useful for coping with the information challenges described prior to the task start. Following the usability test, we also asked users to rate on a 5-point Likert scale if they felt the resources included in MyPath would be useful for addressing their information needs (1=not at all useful, 5=very useful).

Finally, we captured post-study feedback using semi-structured interviews. All interviews were audio recorded. Two researchers transcribed the recordings and conducted a thematic analysis of the qualitative data³⁰. This analysis included first open-coding the transcripts independently, and clustering segments to develop theme concepts. The two researchers then compared codes, discussed discrepancies, and revised the coding scheme. One researcher then did a final review of all transcripts, verifying themes across the full dataset. The research team then met to discuss the final themes and identify necessary design changes.

Results

Participant characteristics

We enrolled 11 participants in the usability evaluation, including four breast cancer survivors and seven healthcare professionals. The healthcare professionals had an average of 16 years of experience working with cancer patients, ranging from 1 to 45 years. All participants were female, with an average age of 52 years. Participants’ average technology comfort score was 3.8 on a 5-point Likert scale (1 being very uncomfortable, 5 being very comfortable).

Learnability

We compared the completion times of the first and second usability tasks to assess learnability. We use the completion times of ten participants, as one participant was unable to complete the second task (see Effectiveness discussion below). The average completion time of the second task (mean=33.1, SD=10.5) was significantly lower compared to the first task (mean=58.9, SD=9.8) (t=5.0, df=9, p<.001).

Errors

Participants performed a total of 33 tasks in the usability tests, summarized in table 1. Across these tasks, we identified 13 errors. The majority of errors (70%) occurred in the third task. The most common error was incorrectly navigating to the user survey, with five participants unable to locate the survey button during the assessment. Three participants also had difficulty selecting the checkboxes within the survey, due to the widgets being too small.

Effectiveness

The task completion rate was 97%, with one participant not completing the second task due to not finding information that addressed her described information need. In this case, the participant was looking for transportation support offered within the community. The lack of local resources also came up frequently in the participants’ feedback, as we discuss below. All other participants were able to find multiple resources that they felt would be useful for addressing their described information needs.

During the first two tasks, participants were each asked to identify four informational resources to address patient questions or challenges. Participants were free to view any categories that they believed would include information relevant to their described information need. During these two tasks, participants identified 43 resources which spanned all of the MyPath categories. The most commonly viewed category was Treatments (n=18), followed by Day to Day Matters (n=8), Emotional Support (n=7), Health and Wellbeing (n=6), Overview (n=2), and Social Support (n=2). The range of information needs described by participants, and categories viewed, validates the importance of comprehensive informational support.

The average usefulness score from the post-test questionnaire was 4.2 out of 5. No participant rated the system’s usefulness below a 4, indicating an agreement in the expected utility of the application. However, in the post-test interviews, described below, participants highlight two issues that could impede on the system’s overall effectiveness.

Participant Feedback

We identified six themes during the analysis of the interview data. These themes fell into three categories: system strengths, anticipated barriers, and design recommendations. Table 2 shows a summary of these themes and the number of participants who mentioned each theme at least once during the interview.

Table 2:

Participant feedback themes, and the number of participants who mentioned each theme at least once during the post-usability study interview.

All participants believed MyPath would be helpful for future patients. Most often, participants said that they liked that MyPath included a breadth of information, addressing needs spanning beyond physical symptoms and side effects. One participant shared that her priorities did not always align with the information that doctors shared, and she was able to find useful information in the prototype:

I think sometimes physicians, because they do it every day, they don’t realize that there’s a lot of things that are not discussed with patients. Day to day matters, social support. [Pointing to MyPath] I would have looked through here for how do I support my family. I’m a mom. I’m a working mom. What’s there for my family? - Breast cancer survivor

Seven participants commented that MyPath recommended short lists of information. Participants liked that the application would populate with small, digestible sets of information during treatment transitions. Both cancer survivors and health professionals noted that large information sets often become overwhelming and ignored:

I like that the lists are short. Sometimes if we give them too many resources they don’t even want to read them. - Oncology nurse

I was bombarded by brochures [when I was diagnosed]. This seems like these things are sent directly, specifically to me by the creators. - Breast cancer survivor

Finally, six participants said that they liked that MyPath included multiple media types. While most of the information was in html or pdf format, the database also included videos and discussion forums. Both the health professionals and cancer survivors thought this would be useful, due to patients’ different learning styles:

I see there are several videos, and because people learn in different ways, that’s something that I think might be pretty good. - Breast cancer survivor

[Pointing to a video] I would rather listen than read. - Breast cancer survivor

Participants also noted barriers that they believed hurt the usability of the system. Five participants commented on navigation challenges they encountered, or would expect to encounter, which further substantiated the errors we identified in the usability test. Participants were particularly concerned about how to navigate to new resource recommendations:

Could it highlight the new resources, and a message that said please tap the highlighted field? Could it direct me in some way to go there? - Breast cancer survivor

While participants were overall positive about the potential usefulness of the MyPath application, four participants also noted that patients respond differently to health information. Participants cautioned that this type of application may not be useful for all cancer patients. As one cancer survivor shared, some people choose not to seek out information:

There is so much information out there. Some of it is not good. Some people want to know everything. I am not that kind of person. - Breast cancer survivor

Finally, participants shared ideas for future information systems. The most common recommendation was to incorporate information about patients’ care team and local resources within the patient’s community. Both health professionals and cancer survivors saw this application as an opportunity to help patients connect with in-person support systems:

You could do a little sentence at the end. If your distress level is 4 or higher, please share this information with your healthcare team. - Cancer navigator

I would like more about what is there actually for ME in this community? Is there transportation help? Is there financial help? - Breast cancer survivor

Discussion

We are developing MyPath, a mobile health application that connects cancer patients to tailored health information based on user input and their care plans, to assess opportunities and barriers of using adaptive systems to increase cancer patients’ access to health information. Thus far, little is known about how dynamic content within an adaptive intervention may affect the system’s overall usability. In this study, we assessed three usability factors that we hypothesized could be influenced by dynamic content: learnability, errors, and effectiveness. Our results indicate that adaptive health information presents a useful approach for making large information sets more manageable for patients, while also raising important usability issues requiring design alterations.

Our first goal within this usability study was to determine if the automatic addition of content over time would interfere participants’ ability to learn how to use the application and find relevant content. Usability studies within other contexts have suggested that adaptive interfaces can interfere with one’s ability to learn a system’s features²⁵. To test the system’s learnability, we compared the task completion times of the first and second tasks in the usability tests, which require participants to complete the same set of interactions, but with new information recommendations added to application prior to the second task. We found that task completion times were significantly lower in the second task, suggesting that dynamic content did not interfere with participants’ ability to find relevant information.

We found that the majority of errors were related to navigating to the user survey. Further, participants were concerned about navigating from the user survey to new resources that were added as a result of the survey submission. These errors highlight the importance of adding navigation support to help participants move between system features. Prior work exploring the design space for adaptive systems have discussed the importance of helping users to navigate these complex interfaces, especially if individuals interact with the system infrequently³¹. Our study shows that when recommending health information to patients, adaptive systems need to visualize when, why, and where new information recommendations have been added.

Participants’ high task completion rates and usefulness scores suggest a consensus that participants believe MyPath will be helpful for breast cancer patients. However, during the interviews, participants did suggest that integrating information about one’s care team and other local resources would be important for patients and increase the system’s utility. In-person resources, such as support groups, self-management programs, and peer navigation services, can improve a patient’s quality of life throughout the cancer trajectory^32,33. However, a lack of information about available resources can limit patients’ ability or willingness to access local services⁷. Participants in this study frequently commented that adaptive information systems could be help bridge these gaps, connecting patients with in-person support services.

Based on this feedback, we made a number of changes to MyPath. First, we added a local resources category that includes links to the cancer clinic and other support services for cancer patients. Second, we moved the user survey to a more prominent location on the homepage and added a text label. Third, we added a number of navigational support features. We changed the labels within each category to more accurately represent why new resource recommendations were added (labels originally only included the date the resources were recommended). We also included animation to show which categories include newly recommended resources, as well as messages with a description of new resource recommendations that are added after a survey submission or at the start of a treatment transition. Fourth, we added a question to the user survey that asks participants how much distress they are experiencing. When a patient selects a distress level above 4, a message appears suggesting they get in touch with a healthcare professional. These changes are depicted in Figure 2.

Figure 2:

Updated MyPath prototype.

The results from this study point to design guidelines for adaptive, patient-facing tools. As noted above, adaptive tools should include clear navigational support for users and may be useful in connecting patients with in-person support when necessary. Participants in this study also raised the issue of cancer patients having different attitudes towards information seeking. We see this issue as an important opportunity for future research. Many cancer patients use information avoidance as a coping strategy³⁴. Open questions remain about how health management tools may adapt over time to accommodate this behavior. Future research addressing this question could support the development of informational support that is beneficial for a broader patient population.

Limitations

Our usability assessment included a number of limitations. We worked with a small number of participants, and a larger participant group may identify more usability issues, though research suggests that this participant size typically captures a majority of the usability issues²². We chose not to include patients who were recently diagnosed or in the middle of treatment. Often these patients are traveling far distances to come to the cancer clinic for treatment, and we wanted to be respectful of their time and mental well-being. We opted to recruit healthcare professionals and cancer survivors who had progressed through diagnosis and treatment, who would be able to reflect on how patients’ information needs change over time. However, it is possible that the emotional burden of the diagnosis and treatments will influence the effectiveness of the tool. We will use a deployment study to assess real-world barriers to using adaptive health information recommendations throughout the cancer trajectory.

Conclusion

Adaptive content can have unforeseen consequences on the usability of an application. The findings from this study emphasize the importance of incorporating navigation support within adaptive systems. Participant feedback underscored the importance of visible system recommendations, with clear explanations of when and why new recommendations are being presented. Despite navigation challenges, participants consistently stressed the potential benefits of adaptive health information for breast cancer patients, such as helping to make large information sets more manageable and less overwhelming, while also revealing an important opportunity to use such tools to help patients connect with healthcare professionals and local support services. These findings may useful in designing technologies for other illness trajectories in which patients’ information needs change over time, including other types of cancer and illnesses with intermittent serious episodes, in which patients’ and caregivers’ information needs may significantly change over time. We also expect that as adaptive patient-facing tools become more common, this approach to usability testing may be useful for assessing how dynamic content influences usability in other contexts.