Qualitative user experience evaluation of the MS trust's online treatment decision aid tool's accommodation of planning pregnancy

Abstract

Background

Decision-making about treatment when planning a pregnancy (family planning) is complex for women living with multiple sclerosis (MS). Decision tools can help this process, in 2016 MS Trust launched their online digital treatment decision tool to support people with MS.

Objectives

To evaluate user-experience of this tool by exploring women's opinions about its content, interface, and usefulness in the context of family planning; and to synthesize recommendations to improve the tool.

Methods

Thirty participants qualitatively evaluated the tool using Think Aloud methodology. Sessions were conducted online using Microsoft Teams and were video recorded. Transcription was automated and data were thematically analyzed.

Results

Women's first impression was that the tool presented a lot of information at once, which was difficult to take in, and they found it difficult to navigate. Although the tool was helpful in allowing them to compare treatment options, the filters were confusing, and the information related to pregnancy sometimes contradicted advice from their healthcare practitioners. They suggested rewording the pregnancy recommendations and filters, updating some content, and making some changes to the interface to meet users’ cognitive needs.

Conclusion

The MS Trust treatment decision tool is excellent in helping women with treatment choices at initial diagnosis. However, it is not currently as useful when considering family plans. Recommendations were conveyed to MS Trust where some are now applied to the new live version and the rest are to be considered for future updating projects.

Introduction

Multiple Sclerosis (MS) is a chronic neurological condition that is more common in women than men (2–3:1), ¹ and onset is typically around age of 30 years ² which is co-incident with the typical age at which women who have children first become pregnant in the UK. ³ Over the past two decades over 20 Disease Modifying Drugs (DMDs) have been introduced for MS,^1,4 with different profiles, making decisions about treatment difficult. Historically family planning advice has been cautious in the context of MS treatments, but with growing maternal and fetal safety data for older medications there has been a clear change in practice towards planning MS treatments around pregnancy rather than pregnancy around treatments. This has added additional level of complexity to treatment decision making.

MS is considered a preference sensitive condition which means there is no single best treatment option, instead, few eligible treatments are offered for the patient to choose from. Taking patients’ preferences into account increases the likelihood of adherence to treatment, improves health outcomes, decreases all health costs, and enhances their quality of life^5–9 which fulfills a shared decision-making approach SDM that is increasingly encouraged. SDM is the process of making treatment decisions that takes into consideration both healthcare practicioner (HCP) advice and patient's preferences. ¹⁰ Decision aids (DAs) support patients to make a treatment choice. The majority of DA prototypes for MS were developed to help with DMD choices alone,^11–15 not DMD selection in the context of pregnancy planning. To the best of our knowledge, only one study has evaluated DA in deciding starting or enlarging a family but not to decide which DMD to choose accordingly. ¹⁶ All these DAs were found to increase patient knowledge and reduce decision conflict which is defined as uncertainty about what course of action to take and none of them comprehensively evaluated the user experience. ^{11 ,12–15,17}

In 2014, MS Trust started to develop an online DMD DA. People with MS and HCPs were involved in this work. This online DA has now been live for seven years. The tool uses a decision tree algorithm where it lays out all DMD options in a shopping interface format, where “customers” (MS patients) can use filters (different characteristics) to narrow down their treatment options according to their preferences. Up to three drugs can be compared side-by-side with a shortlist of the most important side effects. The tool also enables the user to review more details about each drug individually. Pregnancy recommendations come as a sub-heading when reading more about the drug and as one of the drug characteristics in the comparison function. However, the DA was developed prior to the recent substantial change in clinical practice away from a very cautious approach to family planning towards one where this is now much more routine.

In our study, we aimed to qualitatively evaluate the user experience of this publicly available digital DA, exploring its usability for a sample of women with MS specifically in the context of family planning which was not undertaken from this angle before. This work was undertaken in partnership with the MS Trust, with a view to learning from user experience as a revised DA tool was being built in 2020 but was held back during COVID and resumed in early 2023.

Objectives

• To explore women's opinions about the usability of MS Trust's DA (content, interface, navigation) and how useful it is specifically in the context of family planning.

• To synthesise patients’ recommendations to improve the DA and its user experience in a way that supports women's treatment choices and the management of treatment when planning for pregnancy.

Methods

This is a qualitative study that was conducted in the UK, from December 2019 and was completed near the end of 2022. All the screenshots of the DA's different sections used in this study were taken on 19 September 2022.

Participants

Women with relapsing-remitting Multiple Sclerosis (RRMS), UK residents, recruited by social media adverts and with the help of a research recruitment agency. Purposive sampling was used because we wanted to include women who have different motherhood choices to explore a wide range of views.^18–20

Design

A 15-min online Think Aloud session was used to qualitatively evaluate each user's experience of the tool. Think Aloud is a method of testing usability by asking the participant to speak their thoughts while being given a task to perform. ²¹

All sessions were video recorded using Microsoft Teams and auto transcribed using Microsoft Stream, then manually checked for punctuation, and to add context from observed body language. Transcripts were thematically analysed. Codes and themes were inductively identified and arranged using Microsoft Excel. Any disagreement was solved with team discussions.

Procedure

At the beginning of the session, the link to the tool was sent to participants, and they were asked to share their screens with the interviewer. The task they were set was to use the tool as if they were going to choose a DMD to start on soon, or to switch to, and to voice any thoughts they had while browsing the tool.

Ethics

This research was approved by UCL Research Ethics Committee (UCL Ethics Project ID Number: 18923/001). Participants were emailed all study papers (privacy policy, information sheet and consent form) two days before the session. Screenshots from the MS decision tool have been used with permission from the MS Trust.

Results

Participants were women with RRMS, living across the UK. All participants spoke good English even the minority whose English was not their first language (4/30). All participants were able to use Microsoft Teams efficiently and use the share screen feature independently or with minimal help. None of the participants were familiar with the tool.

(Table 1. Demographics)

Table 1.

Participant demographics.

Age at the time of interview	Range Mean	24–49 37
Age at diagnosis	Range Mean	16–42 30
Social status	Single With partner	14/31 17/31
Number of children	0 none 1 child 2 children 3 children	15/31 10/31 5/31 1/31
DMD use	Yes No	25/31 6/31
Ethnicity	White British Other	26/31 5/31

Four main themes were inductively generated from the sessions which were:

1. First impression

2. Women's opinions of each section

   • a. Instructions
   • b. Filters
   • c. Side effects and comparison
   • d. Pregnancy and conception

3. Interface

4. Recommendations (See supplemental material)

(Figure 1. Themes mapping)

Figure 1.

Theme mapping of women think aloud session.

1. First impression

During their initial viewing of the tool, women found the tool confusing by verbally expressing so “this is confusing”, and explaining this was due to the amount of information being presented in one go. This was also clear in their body language (looking annoyed, showing furrowed eyebrows, and random scrolling up and down). One woman mentioned that the term “decision aid” was not explained in any part of the tool sections and that she found it unclear.

2. Sections

a. Instructions

Nine women noticed the instructions and read them carefully before starting to use the tool. They appeared to be able to execute the instructions well afterwards. Twelve women noticed the instructions, started to read a couple of words, but then skipped them. Four did not appear to notice the instructions at all, and directly scrolled down to the body of the tool. Lastly, two women noticed the instructions but expressed they found it challenging to retain all six instructions in one go due to brain fog and affected memory.

b. Filters

The women's comments on the filters were categorised into the following sub-themes:

• Idealising what cannot be ideal

• Wording and clarity

Idealising what cannot be ideal

When answering the filter questions, women mostly chose the combined options most convenient for them, such as a drug that can be taken less frequently, with fewer hospital visits. All women interpreted the filters as questions that all needed to be answered in order to generate a recommended option, and because there is no ideal drug, many of these women were left with no DMD option that matched their preferences because they had used too many filters.

Wording and clarity

For many participants, it was not clear what questions under filters were for. Questions were not defined as “filters” on the PC version, opposed to the smartphone version where these questions are labelled as “filters”. The instructions, however, did not indicate that the user could choose more than one option for each filter, could skip filters that were not important to them, and could skip them entirely if they wish. As a result, many participants did not appear to understand the purpose of these filtering questions and could not utilize them effectively.

The wording of the filter function was also commented on for being confusing because it used the present tense, as if the patient is already using a DMD “how do I take the drug?”. This phrasing was confusing for those who are already on treatment and want to switch to a new drug.

The question about their MS type choosing from the following categories: “clinically isolated syndrome” (CIS), “active RRMS” or “very active RRMS” seemed to be tricky to answer by participants. One participant chose the active category but couldn’t find the treatment she was currently taking, then finally found out she was in the very active category.

This confusion might be a result of many factors. To begin with, MS phenotypes are relapsing remitting (RR), primary progressive MS (PP) or secondary progressive MS (SP) and these are the types that patients usually are familiar with, and DMDs are approved in relation to these phenotypes. Activity, in the context of DMD prescription criteria, is based on relapses and MRI activity, and people with MS may not be familiar with their MRI findings, or differentiate relapses from pseudo-relapses or fluctuating symptoms. However, choosing a treatment does not only depend on whether MS is active or very active, in people with clinically and radiologically inactive MS, a switch in treatment may still be considered for other reasons, e.g., pregnancy planning. Finally, listing CIS as a type of MS is controversial too, because CIS is not considered an MS phenotype but a possible precursor to MS.^22,23

The “more info” tab defining the MS types, was not always noticed by participants, and even after reading the definitions, some still found the information difficult to understand.

c. Side effects and comparison

Unlike the instructions and the filters, the side effects drop-down list, and the comparison function were clearer for women. They didn’t were able to independently use these functions. Participants described these functions as “very useful” and “helpful”. One participant was critical of the use of medical jargon such as “lipoatrophy” without an explanation or hyperlinking.

d. Pregnancy and contraception recommendations

Women's comments on pregnancy and contraception recommendations were categorized into three subthemes:

• Wording and strength of advice

• Contradictory recommendations

• Missing information

Wording and strength of advice

Women generally thought that the conception and pregnancy recommendations in the comparison table were not clear enough. They used to read:

- You must not become pregnant during treatment.

OR

- Pregnancy is not recommended during treatment.

Women found the first statement clear and firm in a preventative manner, but provided minimal hope and more to worry about due to the lack of time frame. In contrast, they felt that the second statement was less firm but not clear enough about the consequences. Both phrases lacked time frame such as (You must not become pregnant during treatment and for X time after stopping it), which was important to the participants in order to give them reassurance and help those approaching the end of fertility window with their decision.

Contradictory recommendations

Women who had used medications during their pregnancies found it concerning to read that the medication they had been told to use during pregnancy is not recommended for use during pregnancy.

Missing information

Women also expressed that because the DA's recommendations were brief and to the point, they would not help them to decide, especially if family planning was their priority. The following information was not covered by the tool: medication management while planning pregnancy, the time needed for a treatment to work, the time needed for a treatment to be ‘flushed out of the system’, guidance about what to do if a pregnancy accidently occurred while using a treatment and what implications this would have for their health and the health of their developing foetus, as well as breastfeeding recommendations.

3. The interface

The women's main comments about the tool's interface were categorised into the following sub-themes:

1. Infographics and colours

2. Automatic filtering response

3. Panes

4. Top bar

5. PC vs smartphone version differences

• a. Infographics and colours

Women generally liked the tool layout, colours, graphics, especially the ones representing the route of administration. While some found it quite tricky to differentiate between the syringe and the drip using only the graphics.

One woman's toddler, who was on her lap commented “Too blue!!” while pointing at the screen. This matched another participant's opinion that the monochrome scheme did not help to instantly visually distinguish the different drug efficacies.

• b. Automatic filtering response

When selecting an option in the filter tool, the tool immediately refreshed the page to show the remaining options, before users had finished making all of their selections. This proved to be confusing for some women, who would have preferred it if they could have selected all filters they wanted before the recommended options then appeared at the end.

• c. Panes

In the comparison table, the first column and the first row were not frozen. For users who might suffer from brain fog and memory problems such as MS patients, it was therefore challenging for them to remember which drug the displayed information related to. This meant they had to keep scrolling up and down to relate drug to its information. Although many women looked annoyed from this (facial expression of frustration when scrolling up and down). One woman only recommended freezing these panes for easier use.

• d. Top bar

The top bar navigated women who clicked on it away from the tool.

• e. PC vs smartphone interface differences

The interface of the filter tool in the PC version was slightly different from the way it looked on the smartphone version. The main difference that affected the ease of use was the way the filters were presented. Six women used their phones to navigate through the tool, and five of them found it hard to find the filters in the first place, even though labelled as “search and filter” on the drop-down list. In comparison, the filter tool was easier to find on the PC version. However, the fact that this was not labelled as “filters” confused some participants.

(Table 2. Quotations)

Table 2.

Quotations representing themes identified from think aloud sessions.

Theme	subtheme	quotation
First impression	Confusion Much of information in one go	P1: “I was a little bit confused as to…. where I start”. P17: “it's quite a lot to go through!”
Filters	Idealizing what cannot be ideal. Wording and clarity (using present tense)	P1: “How often you want to take the drugs? Like, why would you choose to take a drug several times a week if you had an ideal world?” P25: “I’m not quite sure in terms of this [pointing with the cursor at FILTERS]. What this is for?!” P9, “It's easy for me because I take no drugs currently, so it's all to do with the future. I think if people take drugs, they might get a little bit confused”.
Pregnancy and conception recommendation	Wording and strength of advice Contradictory information Missing information	P30: “I think for… ‘you must not become pregnant’, yes…. but that [pointing to ‘pregnancy is not recommended during treatment’] is not pretty clear.” P7: “they’ve always said the same thing to me, that copaxone you can stay on it during like trying to get pregnant on, you can stay on it while you're pregnant, so this is this something completely different… So yeah, that's a bit concerning…” P11: “Basically what I want to know is if I accidentally got pregnant on when taking either these drugs, what are you meant to do?” P17: “would probably want a timeline in there to say when I could have a baby because it would probably panic me at the moment.”

Discussion

Our study found that this DA, in the format available at the time of the study, showed limited utility for women with MS making medication choices when considering family planning. This was due to it containing some outdated information, restrictive family planning recommendations that doesn’t go in line with practice and missing some important management details.

Using restrictive Special Product Characteristics SPC recommendations, to avoid any legal implications of giving off-label advice, could contradict the advice given by MS consultants based on real world practice, and this was found to be concerning and detrimental for patients. This was made evident by two participants who were assured by their consultants of the safety of their chosen DMD use during pregnancy but were concerned when they read on the tool that its use was not recommended during pregnancy. Inconsistent information may impact the patient-consultant trust and could also result in unwanted health outcomes (e.g., stopping a treatment and relapsing during pregnancy). The increasing difference in practice between that expected on the basis of SPCs and real-world experience can be acknowledged in a DA, but to avoid undue confusion and distress, the specifics of particular DMD use in pregnancy will still need to be addressed by HCPs rather than a DA.

For making decision about medication use around different family plans, users wanted to know beforehand what to do if unplanned pregnancy had occurred, which is a very common event in the general population (45% in Britain) ²⁴ and in women with MS, as shown in a 590 Danish women survey as 10% of its participants reported unplanned pregnancies while using DMDs. ²⁵ They also asked for information about the amount of time needed for wash-out of medication before they can start trying for a baby. This was very important for women who had not yet started their families and those approaching the end of their fertility windows, of whom there may be many as the age of women having their first child in the UK has risen to an average of 30.9 years in 2021. ²⁶

Strengths

Studies of DAs are usually assessed based on how they increase users’ knowledge about a particular subject or decrease decisional conflict.^13,14,16 In our study however, we focused on exploring the process and the quality of the users’ experience with a goal of improving both the information presented by the tool and the experience of using it. Our use of this comprehensive methodology (Think Aloud) rather than conventional surveying methods, provided in depth experience evaluation and recommendations from users, stemming from their needs, so instead of asking whether or not participants found this tool useful for choosing treatment in the context of family planning, we explored why they found it useful (or not). We were also the first to look at a DA for treatment choices to test its usability in the context of family planning.

Limitations

We had to conduct the sessions virtually due to the research taking place during the COVID-19 pandemic and thus we might have missed people who were less familiar with using technology and digital DAs. There may have been sampling bias from recruitment via social media, although some participants were recruited directly by a research recruitment company. Given that some participants found navigation of the tool difficult, it seems likely that they might not be very familiar with technology. In addition, it would have been more informative to conduct similar sessions with HCPs as they are part of the decision-making process too, however, due to the limited time the HCPs had during COVID this was not feasible.

Impact and future direction

The results of this study were presented to the MS Trust who applied some of the recommendations we’ve suggested (rewording the pregnancy and conception recommendations and some interface changes) and took the rest in consideration for future tool updating projects. A re-evaluation of the tool after the completion of its update project, based on our findings is recommended to test its usability through user feedback.

Conclusion

The MS Trust DA is an excellent tool that helps women with DMD choices, however, at the time the research was conducted, it was less useful when considering user's family plans. Family planning statements needed to be reworded to provide clearer recommendations which do not contradict HCP's advice. Participants recommended updating the tool's interface to meet users’ cognitive needs. Our findings and recommendations have been presented to MS Trust and some have been already applied and the rest will be considered for future tool updating projects.