Skip to main content
NCBI home page
Health Science Reports logoLink to Health Science Reports
. 2018 Mar 6;1(3):e24. doi: 10.1002/hsr2.24

Producing an evidence‐based treatment information website in partnership with people affected by multiple sclerosis

Anneliese J Synnot 1,2,✉, Melanie Hawkins 3, Bronwen A Merner 1, Michael P Summers 1, Graziella Filippini 4, Richard H Osborne 3, Sue DP Shapland 5, Catherine L Cherry 6,7, Rwth Stuckey 8, Catherine A Milne 9, Paola Mosconi 10, Cinzia Colombo 10, Sophie J Hill 1
PMCID: PMC6266475  PMID: 30623063

Abstract

Background and Aims

In earlier work, we identified that people affected by multiple sclerosis (MS) can have difficulty finding online treatment information that is up to date, trustworthy, understandable, and applicable to personal circumstances, but does not provoke confusion or negative emotional consequences. The objective was to develop online consumer summaries of MS treatment evidence (derived from Cochrane Reviews) that respond to identified treatment information needs of people affected by MS.

Methods

A 2‐phase mixed‐methods project, conducted in partnership with consumers and an MS organisation. Phase 1 included review panels with consumers (Australians affected by MS) and health professionals to test paper‐based treatment summaries before development, and pilot testing of the website. Phase 2 involved an online survey after website launch.

Results

Eighty‐three participants (85% affected by MS) took part. Phase 1 participants strongly endorsed key review summary components, including layering information, and additional sections to aid personal applicability. Participants additionally suggested questions for health professionals. Participants across both phases were receptive to the idea of being provided with Cochrane Review summaries online but were seeking other types of evidence and information, such as personal experiences and the latest experimental treatments, which could not be provided. While the small survey sample size (n = 58) limits application of the results to a broader population, the website was viewed favourably, as a useful, understandable, and trustworthy information source.

Conclusion

We describe a partnership approach to developing online evidence‐based treatment information, underpinned by an in‐depth understanding of consumers' information needs.

Keywords: Cochrane, consumer participation, evidence‐based treatment information, health education, multiple sclerosis, patient and public involvement, systematic review

Short abstract

Background and Aims

In earlier work, we identified that people affected by multiple sclerosis (MS) can have difficulty finding online treatment information that is up to date, trustworthy, understandable, and applicable to personal circumstances, but does not provoke confusion or negative emotional consequences. The objective was to develop online consumer summaries of MS treatment evidence (derived from Cochrane Reviews) that respond to identified treatment information needs of people affected by MS.

Methods

A 2‐phase mixed‐methods project, conducted in partnership with consumers and an MS organisation. Phase 1 included review panels with consumers (Australians affected by MS) and health professionals to test paper‐based treatment summaries before development, and pilot testing of the website. Phase 2 involved an online survey after website launch.

Results

Eighty‐three participants (85% affected by MS) took part. Phase 1 participants strongly endorsed key review summary components, including layering information, and additional sections to aid personal applicability. Participants additionally suggested questions for health professionals. Participants across both phases were receptive to the idea of being provided with Cochrane Review summaries online but were seeking other types of evidence and information, such as personal experiences and the latest experimental treatments, which could not be provided. While the small survey sample size (n = 58) limits application of the results to a broader population, the website was viewed favourably, as a useful, understandable, and trustworthy information source.

Conclusion

We describe a partnership approach to developing online evidence‐based treatment information, underpinned by an in‐depth understanding of consumers' information needs.

1. BACKGROUND

Multiple sclerosis (MS) is a chronic neuro‐inflammatory condition with much uncertainty around individual prognosis. Treatments include a diverse array of drugs with some confronting risk‐benefit profiles1 and many novel and experimental treatments.2 Like many health care consumers, people affected by MS, that is, people with MS and their families and carers, increasingly seek information about such treatments online.3, 4 While most trust the advice of their health professionals,3 many people affected by MS gain confidence over time to find and interpret online treatment information.5

Within this context, it is important to provide people affected by MS with up‐to‐date, reliable, and independent evidence‐based treatment information. Systematic reviews, as summaries of multiple studies on a topic, should be the basis for much of the information about benefits and harms.6 However, previous research has found that consumers, that is, patients, families, carers, and their advocates, can find systematic reviews, and plain language summaries, either difficult to understand or insufficiently detailed to use in decision making.7, 8 They also want to know how research evidence relates to them individually, and determining this can be difficult.7, 9 While the evidence is building on how to best share systematic review evidence with consumers,7, 9, 10 including people affected by MS,11, 12 until recently, there has been scant literature or examples about how this translates into an online environment within the context of MS13 and how to integrate systematic review data with other information needs.9 Recent developments in consumer‐friendly online presentations of benefits and harms include interactive Summary of Findings tables and plain language summaries14 and the MAGICApp rapid recommendations and decision aids.15

This paper describes the development of Making Sense of MS Research,16 an Australian website launched in 2012, that provides summaries of MS treatment evidence from Cochrane Reviews for people affected by MS. We created the website as part of “Integrating and Deriving Evidence, Experiences and Preferences” (IN‐DEEP), a mixed‐methods study conducted in parallel in Australia5, 17 and Italy.18, 19 Here, we describe the Australian arm of the project.

For the first stage of IN‐DEEP, we undertook a qualitative exploration of the online treatment information‐seeking experiences, preferences, and needs of 60 Australians with MS and their family members.5 We identified that people affected by MS can have difficulty finding online treatment information that is up to date, trustworthy, understandable, and applicable to personal circumstances, but does not provoke confusion or negative emotional consequences.

In stage 2, described here, we sought to respond to the unmet treatment information needs identified in stage 1, by developing Making Sense of MS Research.16 As such, the aim of this project stage was to develop online consumer summaries of MS treatment that were easy to understand and applicable to personal circumstances, met additional information needs, relied on up‐to‐date and trustworthy sources, and mitigated confusion and negative emotional consequences.

2. METHODS

The methods for this project were informed by the principles of knowledge translation6 and consumer participation in research.20 In knowledge translation, the focus is not just on the content and message being shared but is also founded on an understanding of the target audience and the broader context.21 Involving consumers in research has an intrinsic value of encouraging democracy, accountability, and transparency and an extrinsic value of improving the quality, impact, and relevance of the research produced.22, 23 In practice, both involve interactive processes that share research between its creators and users, with varied activities and multiple iterations.6

We describe this study in 2 phases. Phase 1—developing the website—involved the development of paper‐based treatment summaries that were tested in review panels involving people affected by MS and others, followed by website creation and pilot testing. The purpose of these review panels and website pilot testing was to help inform the website content and design the information presentation format. Phase 2—website feedback survey—occurred after the website was launched and involved an online survey of website visitors. The purpose of the evaluation of the consumer summaries and the website was to gather user feedback about the presentation and usefulness of the consumer summaries and about website navigation. Reflecting commonly used terminology in Australia, in this paper, we use the term “consumer,” which means patients, families, carers, their advocates, and representatives, interchangeably with “people affected by MS.”

2.1. Project team, and consumer and operational advisory groups

The project team included Australian (SH, AS, MH, and RO) and Italian researchers (GF, CC, and PM), and staff from Australian and local (Victorian) MS societies (MPS and SS). We also sought input from 2 advisory groups convened for the project: a consumer advisory group and an operational advisory group. The consumer advisory group (CLC, RS, and CAM) included 2 people with MS and an MS health information specialist. They prioritised the Cochrane Reviews to be included on the website and provided advice on data collection and analysis, and input into website layout, content, and wording.24 The operational advisory group included clinical managers (n = 2), IT staff (n = 2), and communications personnel (n = 2) from the local Australian state‐based MS organisation. They played a major role in the design and technical aspects of the website and reviewed content from a clinical perspective.

2.2. Phase 1: developing the website

The development of the treatment summaries from the Cochrane Reviews, along with all additional website content, and overarching structure and format, was informed by existing literature and best practice documents. There were also shaped by the findings of the previous IN‐DEEP study (stage 1),5 together with input from people with MS, and others, described in here (IN‐DEEP stage 2).

2.2.1. Developing paper‐based summaries

We selected 2 Cochrane Reviews of MS treatments (prioritised by the consumer advisory group) and developed them into paper‐based treatment summaries for feedback by the review panels. We chose these reviews (one on a medication used by many people affected by MS25 and the other about exercise26) because of the relevance of this information to many people and because of the different types of data to be summarised. We followed research‐based principles for presenting treatment information to consumers, such as layering,27 and used a combination of words, numbers and pictures, and absolute, rather than relative, frequencies.28, 29 The information we presented differed from the usual format of a Cochrane Review in the following ways: It was significantly shorter and focused on key information only (ie, inclusion criteria and results); plain language was used throughout; graphical illustrations of numerical information were used; and novel sections, explaining how results applied to individuals and questions that consumers can ask their health professionals, were included.

2.2.2. Review panels

The aim of the review panels was to seek formal feedback on the paper‐based treatment summaries before they were transferred to an online format. Recruitment involved purposive sampling from participants of the first stage of the IN‐DEEP project5 and from the networks of the project team and the MS organisation. We sought to include people affected by MS, clinicians, and staff of the MS organisation.

One week prior to the review panels, we mailed participants a pack containing a document explaining the nature of the project, the 2 treatment summary templates, and information and consent forms. We included a one‐page feedback form, whereby participants were encouraged to record their immediate impressions of the templates and bring this to the panel discussion.

We held 2 review panels (a form of group interview)30, 31 at a local MS organisation office, with each panel session lasting 2 hours. Two researchers were involved as facilitator (SH) and note‐taker (AS). The following 5 questions, based on those used in the DECIDE project (work package one),32 were used to guide the discussion: “do you understand the information presented?”; “is it helpful to you?”; “is there anything missing?”; “is there anything superfluous?”; and “do you have any suggestions for improvement?”

One researcher (AS) collated participants' feedback sheets, the researchers' detailed written notes, and audio recordings of the review panel sessions, which were reviewed by a second researcher (SH). We used the audio recordings to supplement the written notes. One researcher (AS) grouped all participant feedback under 1 of 5 categories: one overarching category on content, formatting, or general feedback, and 4 specific categories corresponding to the 4 main sections of the treatment summary template (ie, “The short answer,” “The detailed answer,” “What does this mean for me?” and “Using this information”). Within each of these categories, individual feedback items were grouped according to “likes,” “dislikes,” and “suggestions.” We considered the implications of each individual feedback item, culminating in a list of recommendations for changes and/or additions to the individual review templates or, more broadly, for the website. We subsequently discussed and agreed on the recommendations with the project team and consumer advisory group.

2.2.3. Developing the pilot website

We created a pilot website after integrating feedback from the review panels. To guide the website layout and format, we used the HONcode principles33 and the Harvard School of Public Health Guidelines.34 Web design and functionality were informed by the Web Accessibility Initiative Guidelines,35 as well as by input from the consumer advisory group. To cater for people who did not have access to the Internet and to enable website visitors to discuss the treatment summaries with their health professional, we created downloadable PDF versions of the treatment summaries. MS organisation clinical managers and an MS organisation librarian reviewed the content for relevance to the Australian context and for readability.

2.2.4. Website pilot testing

After the development of the pilot website, we invited people affected by MS to take part in pilot testing. Participants were recruited from people who took part in the review panels and through the networks of the project team.

We asked pilot testing participants to visit the website and complete an online survey (see Supporting Information). Participants subsequently took part in a 30‐minute telephone interview to discuss their survey responses with a researcher who had not been involved in the development of the website (MH). In response to the survey data and interview responses, we made minor changes to the website, such as incorporating additional links to further information and reordering of the site page tabs.

2.3. Phase 2: website feedback survey

Following incorporation of the changes after pilot testing, we launched the website in November 2012, with promotion to relevant organisations and individuals across Australia and internationally. The primary promotion strategy was through the networks of the local MS organisation and MS Australia, via their websites and social media pages, e‐mailing lists, and newsletters.

2.3.1. Online survey

The purpose of the evaluation of the consumer summaries and the website was to gather user feedback about the presentation and usefulness of the consumer summaries and about website navigation. Development of the survey was based on the unmet treatment information needs identified in stage 1 of IN‐DEEP.5 On the basis of these unmet needs, we developed survey sections with brief descriptive rationales for survey questions. Two project team members (RO and MH) developed the survey questions using a rapid literature review and their extensive survey‐writing experiences and resources. The questions chosen for survey sections were informed by previously validated patient‐reported outcomes measures.36, 37 Collaboration with the research team about survey content was ongoing. The final survey included qualitative and quantitative questions under sections about users' initial expectations of the site and what else they would like to see on it; how the information was presented on the website and if users might use the information to consult health professionals; the site's usability; and demographic questions. The survey was pilot tested with people affected by MS as part of the broader website pilot testing, and some minor revisions were made to improve user understanding of the questions. This survey was designed for the purpose of gathering user feedback about presentation of the online consumer summaries and website usability and was not intended for use in other contexts. We included the survey on the website for 2 months after the site was launched (see Supporting Information) but did not actively recruit people to complete the survey. It displayed as a box of highlighted text that website users could accept or decline.

2.4. Ethical approval and informed consent

Participants in all phases provided informed consent to participate, via a written or online form. We received approval from the Human Ethics Committee, Faculty of Health Sciences, La Trobe University (numbers 11‐169 and 12‐128). We offered review panel participants a $50 voucher for their participation. We did not offer reimbursement to participants in other project phases.

3. RESULTS

3.1. Participants

A total of 83 participants took part of the project, participating in review panels (n = 16), website pilot testing (n = 9), or by completing the website feedback survey (n = 58) (see Table 1). Of the 61 participants who provided demographic information, the majority were people with MS (75%) or family members of people with MS (12%). Over two‐thirds of participants were female (71%) and aged between 41 and 65 years (73%). All but one participant resided in Australia. Approximately half of the participants affected by MS had a university degree (48%), although those in the final project stage (website feedback survey) included a broader range of educational backgrounds, with 34% of respondents having an educational level of high school or below. The median length of time since MS diagnosis was 8 years (range < 1 to 30 years), and most participants (69%) had the relapsing‐remitting form of the disease.

Table 1.

Participant demographic characteristics

Characteristics Developing the Website Website Feedback Survey Total (N = 83)
Review Panels (N = 16) Pilot Test (N = 9) Online Survey (N = 58)a
Participants (n, %)
People with MS 10 (63) 6 (67) 30 (83) 46 (75)
Family members 1 (6) 3 (33) 3 (8) 7 (12)
Health professionals/otherb 5 (31) 0 (0) 3 (8) 8 (13)
Gender (n, % female) 10 (63) 7 (78) 22 (73) 39 (71)
Age (n, %)
21 to 40 y 3 (19) 3 (33) 6 (20) 12 (22)
41 to 65 y 11 (69) 6 (67) 23 (77) 40 (73)
66 y and over 2 (13) 0 (0) 1 (3) 3 (6)
Highest education levelc (n, %)
High school (not completed) 0 (0) 0 (0) 5 (17) 5 (10)
High school (completed) 1 (9) 1 (11) 5 (17) 7 (14)
Occupational certificate 6 (55) 4 (44) 4 (13) 14 (28)
University degree 4 (36) 4 (44) 16 (53) 24 (48)
Time with MSe (y; median, range) 19 (2 to 24) 4.5 (1 to 21) 4 (<1 to 30) 8 (<1 to 30)
Type of MS (n, %)d , e
RRMS NA 5 (83) 20 (67) 25 (69)
PPMS NA 1 (17) 5 (17) 6 (17)
SPMS NA 0 (0) 3 (10) 3 (8)
Other/unsure NA 0 (0) 2 (7) 2 (6)
Open in a new tab

Abbreviations: MS = multiple sclerosis; N, total number of participants; NA, not available; PPMS = primary progressive multiple sclerosis; RRMA; relapsing remitting multiple sclerosis; SPMS, secondary progressive multiple sclerosis.

a

Twenty‐two participants did not provide full demographic details, so none of the total number of participants for each demographic characteristics adds up to 58.

b

Included neurologists, MS nurses, and MS organisation staff.

c

People with MS and family members only.

d

People with MS only.

e

Not asked of people with MS in the review panels.

3.2. Phase 1: developing the website

We present the results of this phase grouped under the 6 aims of the website.

3.2.1. Easy to understand

To present the treatment information in a format that was straightforward and clear, we created 2 sections (the “short answer” and the “detailed answer”) to summarise the benefits and harms of the treatment, so that the information was effectively layered.27 In “the short answer” section, we provided a one‐paragraph summary of the Cochrane Review results. In “the detailed answer” section, a combination of words, numbers, and graphical images was used to describe the effect of the intervention compared with placebo, on the main outcomes collected in each review, thereby presenting the same review findings in 3 different formats28, 29 (see Figure 1). We expressed absolute numbers as natural frequencies (ie, 5 of 100).

Figure 1.

Figure 1

Open in a new tab

Screenshot showing presentation of the evidence in “the detailed answer”

Review panel participants endorsed the concept of layering the information and providing detailed scientific information in a combination of formats. Some of the nonconsumer participants were concerned about consumers' abilities to make sense of the information, but this was not reflected by consumer participants. There were mixed responses to the 100 smiley faces graphic that we used in the review panel sessions (see Kasper's multifigure pictographs12 for an example). After consultation with colleagues, a new horizontal comparison graphic was used on the website instead (see Figure 1). Several participants suggested that the addition of a hyperlinked glossary would be preferable to trying to explain scientific terms in the text. The glossary was subsequently created for the website.

3.2.2. Applicable to personal circumstances

To provide treatment information that was more personally applicable, we created an additional section, specific to each treatment summary, called “Does this apply to me?” There, we described the characteristics of the people included in the studies and the parameters of the interventions studied therein.9 We also added a section called “Questions for my health professional” that was specific to each review.33 We developed a set of questions, in consultation with the consumer advisory group, that consumers could use as a prompt for discussion with their health professionals. Example questions included “How soon after diagnosis is it recommended to take interferons?” and “What kind of rehabilitation would be right for me?” Both the “Does this apply to me?” and “Questions for my health professional” sections were strongly endorsed in the review panels as ways to help readers apply the information to their specific situation.

3.2.3. Meets additional information needs

To acknowledge the additional information needs of consumers, we created a section called “Find out more,” with the aim of connecting readers to their local MS organisation and providing links to other online information and evidence sources (selected in consultation with MS organisation staff).9, 33 This was one of the most strongly endorsed concepts in the review panels and the pilot testing, but many consumers wanted more information and links than those provided. Some review panel participants wanted links to patient websites where they could read about “real people's” responses to treatments. Others wanted more detailed information about interventions, such as what specific types of exercise (eg, walking or gym classes) one might look at starting or practical tips on how the medications were administered.

While the scope of the project was limited to treatment information only, we provided links to MS organisation resources and created a new section called “New to MS?” with some brief information about MS and a range of web links for further information. In pilot testing, we found that participants were expecting more treatment topics on the website. To address this, we created a poll on the website with 5 additional review topics and invited website visitors to vote for the topic to be summarised next. Visitors could leave their email address if they wished to be notified when new treatment summaries were added to the website.

3.2.4. Trustworthy

At the review panels, there were questions about Cochrane and how Cochrane Reviews were created. For example, some participants asked if Cochrane was trustworthy and if the information was up to date. In response to this, we created a Frequently Asked Questions page, drawn from some of the questions posed at the review panels (for example, “What is the Cochrane Collaboration?” and “Why don't the treatment summaries always give a clear answer?”), and a page called “About the Research,” which explained the website development process. In addition, logos from the project partners of La Trobe University, the MS organisation, and Cochrane were included in the footer of every page to make clear the independence of the website creators from industry influence.33

3.2.5. Up to date

To ensure website visitors would know the information was up to date, we included the date of the review publication and the search dates for each Cochrane Review that was summarised.33 Review panel participants endorsed the inclusion of these dates but queried the accuracy of the treatment information because the reviews were published more than 2 years earlier. As such, in addition to the information about the date of publication, we included a statement about whether or not newer studies had been published since that time. We only included Cochrane Reviews on the site when we were confident that no further trials had been published that met the inclusion criteria for the review (as advised by the Cochrane Multiple Sclerosis and Rare Diseases of the Central Nervous System Review Group).

To align available Cochrane Review topics with consumer preferences, we asked the consumer advisory group to prioritise the reviews to be included on the website. The prioritisation criteria included the expected interest to people affected by MS and how current the review was. Unfortunately, the Cochrane Reviews on topics of broad interest such as exercise,26 diet,38 and vitamin D supplementation39 were not, at that time, sufficiently up to date to be included. The final reviews selected for the website were on immunotherapies for different types of MS,25, 40, 41, 42 hyperbaric oxygen therapy,43 and rehabilitation.44

3.2.6. Mitigates confusion and negative emotional consequences

To mitigate confusion or unnecessarily negative emotional consequences, consumer advisory group members reviewed the language used in the summaries, to ensure that it was appropriate to a broad range of consumers. We also added a question to the Frequently Asked Questions section to offer reasons as to why the review summaries could not always provide clear answers. To mitigate the risk of confusion or concern for website readers, we included questions such as “Why do these numbers look different from what I have seen elsewhere?” The website encouraged readers to contact their local MS organisation if they were concerned about conflicting information.

3.3. Phase 2: website feedback survey

Once launched, the website received 3873 visits (75% of which were unique visits) in the 2‐month period during which the feedback survey was live. During this evaluation period, 61 people elected to respond to the survey, with 58 useable responses included in the analysis (responses from 3 people were too incomplete to include). Given we did not actively recruit participants to the survey, but instead recruited them to view the website, the small sample size is not unexpected but does limit interpretation of the usefulness of the site to a broader population. However, of those who completed the survey, nearly 80% (n = 46) responded positively to the research summaries and the website as a whole. As well as quantitative data, respondents gave free‐text responses that indicated they found the information and website layout to be clear, aesthetically pleasing, and easy to navigate and understand. Overall, the presentation of the benefits and harms of each intervention was well received, with more than 75% of respondents agreeing or strongly agreeing with questions about understanding benefits and risks (see Supporting Information, S2 Table).

Other questions were designed to probe reflections about personal applicability (“The website has helped me to understand how the treatments might be useful for me”), trustworthiness (“The Cochrane Collaboration is a trustworthy source of information about treatments for health conditions”), meeting additional information needs (“I feel the About the Research section helped me to understand the summaries about MS treatments”), and emotional consequences (“I find it frightening to read about the risks of these treatments”). These were rated as “agree” or “strongly agree” by approximately two‐thirds (61%‐71%) of participants.

The main reasons for dissatisfaction with the information in the summaries were that they did not cover the latest approved and experimental MS medications and treatments, or nonmedical treatments. Questions with the lowest endorsement were about whether or not participants now felt able to make an educated decision about the treatments (21%), if the website had helped them understand how the treatments might be useful for them (19%), if the website met their information needs (17%), and if they found the “Does this apply to me?” section useful (17%).

4. DISCUSSION

In this study, we sought to address the unmet treatment information needs of people affected by MS, as established by phase 1 of this project, by developing a website based on MS Cochrane Review summaries. We undertook all project stages in partnership with people affected by MS and with the local MS society.24 As a consequence of this stakeholder consultation, the website was underpinned by an in‐depth understanding of treatment information–seeking experiences of people affected by MS. Responses across all project phases indicated there was a general receptivity by people affected by MS to the concept of having online, independent, evidence‐based treatment information. The website feedback survey responses, while modest in number, suggest the information was understandable, viewed as trustworthy, and addressed how people might apply the information to themselves. While the treatment summaries included information that was up to date, in that is was recent evidence, the feedback across all project stages and phases was that respondents wanted us to incorporate evidence about the latest, and sometimes experimental, treatments, which were not available as Cochrane Reviews at the time of this research.

While Cochrane Reviews remain the gold standard in systematic reviews of treatment research,45 and systematic reviews should form the basis of evidence‐based treatment information shared with consumers,6 they are insufficient for a complete treatment information website. Consumers also need general information about the disease and practical information, such as how treatments are administered.46 Additional challenges of using Cochrane Reviews are that they can only provide an answer if there is sufficient existing trial evidence and, like all systematic reviews, can date quickly in fast‐moving fields.47 This makes it particularly challenging to provide up‐to‐date information that includes coverage of new or experimental treatments on such a website. Originally, our treatment summaries were intended to be incorporated on our MS partner's website (where much of this additional information is provided). However, due to unforeseen circumstances during the project, we created a stand‐alone website to house the summaries. So, while we provided links to local MS organisations for further information, it was beyond the project scope and resources to provide all this additional information.

Similarly, the inclusion of personal narratives, or stories of the illness experiences of others in evidence‐based websites, is an emerging issue for health information providers. Echoing our results, other researchers have found that consumers want to know about the personal experiences of people with health conditions,48, 49 but there is uncertainty among the research community about how these experiences can be incorporated into evidence‐based resources. Personal narratives do affect people's judgment and choices, but how to harness this to facilitate, as opposed to hinder, patients' decision making is unclear.50 The use of scientifically rigorous methods to source and summarise personal narratives, such as those underpinning the Italian IN‐DEEP website51 and the HealthTalk website,52 offers, at least, a partial solution to this problem.

Our findings affirm the importance of partnerships between evidence producers and trusted information providers, such as peak bodies and member organisations. Not only do such organisations provide some of the additional information that consumers seek, but also groups like MS societies are considered trustworthy sources of health information.3 This is important within the context of Cochrane reviews, given they were unfamiliar to consumers in our study. This unfamiliarity has been found by others to result in scepticism about the trustworthiness of information based on Cochrane Reviews.9, 49 Additionally, presenting the evidence from systematic reviews is a science in itself. It requires a solid understanding of evidence‐based health care, the evolving body of health communication research, and information design principles that may be beyond those without specialist training.53 Better links between researchers and information providers could facilitate the integration of evidence from systematic reviews with other information that consumers need, and aid its dissemination.54

Subsequent to the IN‐DEEP study, researchers have experimented with new and interactive ways to present treatment benefits and risks, online.32 Recent examples include interactive Summary of Findings tables and plain language summaries,14 and the BMJ's consumer‐friendly rapid recommendations.55 While their graphical presentation would likely improve the presentation of benefits and harms used on our website, our results speak to the need for embedding such resources within the broader information and support needs of consumers. Notably, a recent randomised controlled trial of web‐based pictorial formats to present MS treatment information revealed that the animated graphical presentations of benefits were less well understood, compared with static presentations.12

Another relevant development is a broadening of systematic review approaches, including network meta‐analysis (where multiple interventions can be compared head‐to‐head),56 mixed‐method reviews (where quantitative data are synthesized with qualitative experience data),57 and reviews of observational data about side effects.58 These types of reviews open the door for evidence summaries to better meet consumer needs, by presenting empirically derived comparative effectiveness, longer‐term data on potential harms, and the lived experience of others.

We acknowledge a number of limitations. First is the small number of participants who completed the online survey, limiting application of the evaluation results to a broader population. The low response rate to the survey may be due, in part, to dissemination or recruitment shortcomings: We were reliant on our project partners for website promotion, and we elected to promote the website, rather than explicitly recruit people to complete the survey. Alternatively, those who did not find the website helpful may not have chosen to complete the survey, potentially biasing results towards more favourable responses. Second, it is unclear how well the website caters to those with low literacy levels. The educational background of consumers and family members involved in the development was high (mainly university educated), and we did not use a standardised tool (eg, Flesch‐Kincaid Grade) to assess the readability of the website.59 However, the fact that we received largely positive feedback from survey responses, including from those who did have limited levels of education, suggests we may have met, at least, some of the information needs of people with limited literacy. A final limitation was that the assessment of important concepts, such as understanding, were self‐reported, rather than objectively measured. A more objective measure of understanding is likely to have generated more modest results,12 although this was not within the scope of this project.

5. CONCLUSION

This paper describes an approach to developing evidence‐based treatment information that is underpinned by in‐depth understanding of the information needs of consumers. This understanding was facilitated by the research team's ongoing partnership with consumers, throughout the development of the Cochrane Review information summaries, and the website that hosts them. The evaluation indicates that some of the unmet treatment information needs of people with MS were met through this novel partnership approach. Equally, we have demonstrated that consumers seek more than just the information contained in Cochrane Reviews to help them understand and apply treatment information. Future projects that include partnership with consumers, as well as with peak health organisations and other treatment information providers, may offer opportunities to fulfil these additional consumer treatment information needs that extend beyond treatment risks and benefits.

FUNDING

Financial support for this study was provided in part by a grant from Multiple Sclerosis Research Australia (Grant number 00025) and MS Australia ACT/NSW/VIC. Richard Osborne was funded, in part, by a National Health and Medical Research Council Senior Research Fellowship (APP1059122). Graziella Filippini, Paola Mosconi, and Cinzia Colombo were funded in part by the Italian Multiple Sclerosis Foundation (Grant number 2010/R/19). The funding agreement ensured the authors' independence in designing the study, interpreting the data, writing, and publishing the report.

CONFLICT OF INTERESTS

The authors declared no conflict of interest with respect to the research, authorship, and/or publication of this article.

AUTHOR CONTRIBUTIONS

Conceptualization: Sophie Hill, Michael Summers, Graziella Filippini, Richard Osborne, Sue Shapland

Formal analysis: Melanie Hawkins, Anneliese Synnot, Richard Osborne, Sophie Hill, Bronwen Merner

Funding acquisition: Sophie Hill, Michael Summers, Graziella Filippini, Richard Osborne, Sue Shapland

Investigation: Anneliese Synnot, Melanie Hawkins, Sophie Hill, Richard Osborne

Methodology: Anneliese Synnot, Melanie Hawkins, Michael Summers, Graziella Filippini, Richard H Osborne, Sue Shapland, Catherine Cherry, Rwth Stuckey, Catherine Milne, Paola Mosconi, Cinzia Colombo, and Sophie Hill

Project administration: Anneliese Synnot

Supervision: Sophie Hill

Writing – original draft: Anneliese Synnot, Melanie Hawkins, Sophie Hill, Bronwen Merner

Writing – review and editing: Anneliese Synnot, Melanie Hawkins, Bronwen Merner, Michael Summers, Graziella Filippini, Richard H Osborne, Sue Shapland, Catherine Cherry, Rwth Stuckey, Catherine Milne, Paola Mosconi, Cinzia Colombo, and Sophie Hill

Supporting information

Supporting info item

Click here for additional data file. (304.7KB, docx)

ACKNOWLEDGEMENTS

We thank all the participants who contributed their time and invaluable feedback at each project stage. We gratefully acknowledge the time and expertise contributed by current and previous MS Australia ACT/NSW/VIC staff, in particular Elizabeth McDonald, Joël Lignier, Stefan Jayawardena, Robyn Hunter, and Emma Jeffries and our colleagues, Trung Do, Dawn Stacey, Sarah Rosenbaum, Claire Glenton, and Andy Oxman. Additionally, we gratefully acknowledge the contribution to this work of the Victorian Operational Infrastructure Support Program received by the Burnet Institute.

Synnot AJ, Hawkins M, Merner BA, et al. Producing an evidence‐based treatment information website in partnership with people affected by multiple sclerosis. Health Sci Rep. 2018;1:e24 10.1002/hsr2.24

REFERENCES

  • 1. Filippini G, Del Giovane C, Vacchi L, et al. Immunomodulators and immunosuppressants for multiple sclerosis: a network meta‐analysis. Cochrane Database Syst Rev. 2013;6:Art. No.: CD008933. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Cross AH, Naismith RT. Established and novel disease‐modifying treatments in multiple sclerosis. J Intern Med. 2014;275(4):350‐363. [DOI] [PubMed] [Google Scholar]
  • 3. Marrie R, Salter A, Tyry T, Fox R, Cutter G. Preferred sources of health information in persons with multiple sclerosis: degree of trust and information sought. J Med Internet Res. 2013;15:e67 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Hay MC, Strathmann C, Lieber E, Wick K, Giesser B. Why patients go online: multiple sclerosis, the internet, and physician‐patient communication. Neurologist. 2008;14:374‐381. [DOI] [PubMed] [Google Scholar]
  • 5. Synnot AJ, Hill SJ, Garner KA, et al. Online health information seeking: how people with multiple sclerosis find, assess and integrate treatment information to manage their health. Health Expect. 2014;19(3):727‐737. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Grimshaw J, Eccles M, Lavis J, Hill S, Squires J. Knowledge translation of research findings. Implement Sci. 2012;7:50 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Glenton C, Santesso N, Rosenbaum S, et al. Presenting the results of Cochrane Systematic Reviews to a consumer audience: a qualitative study. Med Decis Making. 2010;30(5):566‐577. [DOI] [PubMed] [Google Scholar]
  • 8. Glenton C, Underland V, Kho M, Pennick V, Oxman AD. Summaries of findings, descriptions of interventions, and information about adverse effects would make reviews more informative. J Clin Epidemiol. 2006;59(8):770‐778. [DOI] [PubMed] [Google Scholar]
  • 9. Brehaut JC, Santesso N, O'Connor AM, et al. Presenting evidence to patients online: what do web users think of consumer summaries of Cochrane Musculoskeletal reviews? J Med Internet Res. 2011;13(1):e5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Santesso N, Maxwell L, Tugwell P, et al. Knowledge transfer to clinicians and consumers by the Cochrane Musculoskeletal Group. J Rheumatol. 2006;33(11):2312‐2318. [PubMed] [Google Scholar]
  • 11. Heesen C, Schäffler N, Kasper J, Mühlhauser I, Köpke S. Suspected multiple sclerosis—what to do? Evaluation of a patient information leaflet. Mult Scler. 2009;15(9):1103‐1112. [DOI] [PubMed] [Google Scholar]
  • 12. Kasper J, van de Roemer A, Pöttgen J, et al. A new graphical format to communicate treatment effects to patients—a web‐based randomized controlled trial. Health Expect. 2016. 10.1111/hex.12522 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Köpke S, Solari A, Khan F, Heesen C, Giordano A. Information provision for people with multiple sclerosis. Cochrane Database Syst Rev. 2014;4:Art. No.: CD008757. [DOI] [PubMed] [Google Scholar]
  • 14. Langendam MW, Akl EA, Dahm P, Glasziou P, Guyatt G, Schünemann HJ. Assessing and presenting summaries of evidence in Cochrane Reviews. Syst Rev. 2013;2(1):81 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Siemieniuk RA, Agoritsas T, Macdonald H, Guyatt GH, Brandt L, Vandvik PO. Introduction to BMJ rapid recommendations. BMJ. 2016;354:i5191. [DOI] [PubMed] [Google Scholar]
  • 16. IN‐DEEP project team . Making sense of MS research. 2012; http://www.makingsenseofmsresearch.org.au.
  • 17. Hill S, Filippini G, Synnot A, et al. Presenting evidence‐based health information for people with multiple sclerosis: the IN‐DEEP project protocol. BMC Med Inform Decis Mak. 2012;12:20 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Colombo C, Mosconi P, Confalonieri P, et al. Web search behavior and information needs of people with multiple sclerosis: focus group study and analysis of online postings. Interact J Med Res. 2014;3(3):e12 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19. Colombo C, Filippini G, Synnot A, et al. Development and assessment of a website presenting evidence‐based information for people with multiple sclerosis: the IN‐DEEP project. BMC Neurol. 2016;16:30 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Boote J, Telford R, Cooper C. Consumer involvement in health research: a review and research agenda. Health Policy. 2002;61:213‐236. [DOI] [PubMed] [Google Scholar]
  • 21. Davison C. Knowledge translation: implications for evaluation. New Directions for Evaluation. 2009;124:75‐87. [Google Scholar]
  • 22. Nilsen E, Myrhaug H, Johansen M, Oliver S, Oxman A. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database Syst Rev. 2006;3:Art. No.: CD004563. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. Kreis J, Puhan M, Schunemann H, Dickersin K. Consumer involvement in systematic reviews of comparative effectiveness research. Health Expect. 2013;16(4):323‐337. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Synnot A, Cherry CM, Summers MP, Stuckey RL, Milne CA, Lowe DB, Hill SJ. Consumer engagement critical to success in an Australian research project: Reflections from those involved. Australian Journal of Primary Health In press. [DOI] [PubMed] [Google Scholar]
  • 25. Rice G, Incorvaia B, Munari LM, et al. Interferon in relapsing‐remitting multiple sclerosis. Cochrane Database Syst Rev. 2001;4:Art. No.: CD002002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26. Rietberg M, Brooks D, Uitdehaag BMJ, Kwakkel G. Exercise therapy for multiple sclerosis. Cochrane Database Syst Rev. 2005;1:Art. No.: CD003980. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Lavis JN, Wilson MG, Grimshaw JM, et al. Supporting the use of health technology assessments in policy making about health systems. Int J Technol Assess Health Care. 2010;26(04):405‐414. [DOI] [PubMed] [Google Scholar]
  • 28. Trevena LJ, Barratt A, Butow P, Caldwell P. A systematic review on communicating with patients about evidence. J Eval Clin Pract. 2006;12(1):13‐23. [DOI] [PubMed] [Google Scholar]
  • 29. Bunge M, Mühlhauser I, Steckelberg A. What constitutes evidence‐based patient information? Overview of discussed criteria. Patient Educ Couns. 2010;78(3):316‐328. [DOI] [PubMed] [Google Scholar]
  • 30. Green J. The use of focus groups in research into health In: Saks M, Allsop J, eds. Researching Health: Qualitative, Quantitative and Mixed Methods. London: SAGE publications; 2013. [Google Scholar]
  • 31. Coreil J. Group interview methods in community health research. Medical Anthropology: Cross‐Cultural Studies in Health and Illness. 1994;16:193‐210. [DOI] [PubMed] [Google Scholar]
  • 32. Treweek S, Oxman AD, Alderson P, et al. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33. Health on the Net Foundation . The HON Code of Conduct for medical and health Web sites (HONcode). 2010; http://www.hon.ch/HONcode/Webmasters/Conduct.html.
  • 34. Rudd R. Guidelines for creating materials: resources for developing and assessing materials. n.d.; http://www.hsph.harvard.edu/healthliteracy/practice/innovative-actions/.
  • 35. Web Accessibility Initiative . Web Content Accessibility Guidelines (WCAG) Overview. 2012; http://http://www.w3.org/WAI/intro/wcag.
  • 36. Osborne R, Elsworth G, Whitfield K. The Health Education Impact Questionnaire (heiQ): an outcomes and evaluation measure for patient education and self‐management interventions for people with chronic conditions. Patient Educ Couns. 2007;66:192‐201. [DOI] [PubMed] [Google Scholar]
  • 37. Ciciriello S, Buchbinder R, Osborne R, Wicks I. Improving treatment with methotrexate in rheumatoid arthritis—development of a multimedia patient education program and the MiRAK, a new instrument to evaluate methotrexate‐related knowledge. Semin Arthritis Rheum. 2013;43:437‐446. [DOI] [PubMed] [Google Scholar]
  • 38. Farinotti M, Vacchi L, Simi S, Di Pietrantonj C, Brait L, Filippini G. Dietary interventions for multiple sclerosis. Cochrane Database Syst Rev. 2012;12: [DOI] [PubMed] [Google Scholar]
  • 39. Jagannath VA, Fedorowicz Z, Asokan GV, Robak EW, Whamond L. Vitamin D for the management of multiple sclerosis. Cochrane Database Syst Rev. 2010;12:Art. No.: CD008422. [DOI] [PubMed] [Google Scholar]
  • 40. La Mantia L, Vacchi L, Di Pietrantonj C, et al. Interferon beta for secondary progressive multiple sclerosis. Cochrane Database Syst Rev. 2012;1:Art. No.: CD005181. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41. Rojas JI, Romano M, Ciapponi A, Patrucco L, Cristiano E. Interferon beta for primary progressive multiple sclerosis. Cochrane Database Syst Rev. 2010;1:Art. No.: CD006643. [DOI] [PubMed] [Google Scholar]
  • 42. La Mantia L, Munari LM, Lovati R. Glatiramer acetate for multiple sclerosis. Cochrane Database Syst Rev. 2010;5:Art. No.: CD004678. [DOI] [PubMed] [Google Scholar]
  • 43. Bennett MH, Heard R. Hyperbaric oxygen therapy for multiple sclerosis. Cochrane Database Syst Rev. 2004;1:Art. No.: CD003057. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44. Khan F, Turner‐Stokes L, Ng L, Kilpatrick T, Amatya B. Multidisciplinary rehabilitation for adults with multiple sclerosis. Cochrane Database Syst Rev. 2007;2:Art. No.: CD006036. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45. Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross‐sectional study. PLoS Med. 2016;13(5):e1002028 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Elwyn G, O'Connor A, Stacey D, et al. Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. BMJ. 2006;333(7565):417 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147(4):224‐233. [DOI] [PubMed] [Google Scholar]
  • 48. Fox S, Purcell K. Chronic disease and the internet. Washington DC: Pew Internet and American Life Project; 2010: http://www.pewinternet.org/files/old-media//Files/Reports/2010/PIP_Chronic_Disease_with_topline.pdf.
  • 49. Glenton C, Nilsen E, Carlsen B. Lay perceptions of evidence‐based information—a qualitative evaluation of a website for back pain sufferers. BMC Health Serv Res. 2006;6(1):34 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50. Bekker H, Winterbottom A, Butow P, et al. Chapter E: Using personal stories In: Volk R, Llewellyn‐Thomas H, eds. Update of the International Patient Decision Aids Standards (IDPAS) Collaboration's Background Document. International Patient Decision Aid Standards Collaboration; 2012. [Google Scholar]
  • 51. IN‐DEEP project team. IN‐DEEP. 2012; http://indeep.istituto-besta.it/.
  • 52.DIPEx. http://healthtalk.org. 2017; http://www.healthtalk.org/.
  • 53. Rosenbaum S. Improving the user experience of evidence: a design approach to evidence‐informed health care [PhD]. Oslo, The Oslo School of Architecture and Design (AHO); 2010.
  • 54. Mitton C, Adair C, McKenzie E, Patten S, Perry B. Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q. 2007;85:729‐768. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55. Vandvik PO, Otto CM, Siemieniuk RA, et al. Transcatheter or surgical aortic valve replacement for patients with severe, symptomatic, aortic stenosis at low to intermediate surgical risk: a clinical practice guideline. BMJ. 2016;354:i5085 [DOI] [PubMed] [Google Scholar]
  • 56. Mills EJ, Thorlund K, Ioannidis JPA. Demystifying trial networks and network meta‐analysis. BMJ: British Medical Journal. 2013;346: [DOI] [PubMed] [Google Scholar]
  • 57. Gough D. Qualitative and mixed methods in systematic reviews. Syst Rev. 2015;4(1):181 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58. Chou R, Aronson N, Atkins D, et al. AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health‐care program. J Clin Epidemiol. 2010;63(5):502‐512. [DOI] [PubMed] [Google Scholar]
  • 59. Badarudeen S, Sabharwal S. Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res. 2010;468(10):2572‐2580. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supporting info item

Click here for additional data file. (304.7KB, docx)

Articles from Health Science Reports are provided here courtesy of Wiley

RESOURCES