Development of a checklist for people communicating evidence-based information about the effects of healthcare interventions: a mixed methods study

Abstract

Objectives

To make informed decisions about healthcare, patients and the public, health professionals and policymakers need information about the effects of interventions. People need information that is based on the best available evidence; that is presented in a complete and unbiased way; and that is relevant, trustworthy and easy to use and to understand. The aim of this paper is to provide guidance and a checklist to those producing and communicating evidence-based information about the effects of interventions intended to inform decisions about healthcare.

Design

To inform the development of this checklist, we identified research relevant to communicating evidence-based information about the effects of interventions. We used an iterative, informal consensus process to synthesise our recommendations. We began by discussing and agreeing on some initial recommendations, based on our own experience and research over the past 20–30 years. Subsequent revisions were informed by the literature we examined and feedback. We also compared our recommendations to those made by others. We sought structured feedback from people with relevant expertise, including people who prepare and use information about the effects of interventions for the public, health professionals or policymakers.

Results

We produced a checklist with 10 recommendations. Three recommendations focus on making it easy to quickly determine the relevance of the information and find the key messages. Five recommendations are about helping the reader understand the size of effects and how sure we are about those estimates. Two recommendations are about helping the reader put information about intervention effects in context and understand if and why the information is trustworthy.

Conclusions

These 10 recommendations summarise lessons we have learnt developing and evaluating ways of helping people to make well-informed decisions by making research evidence more understandable and useful for them. We welcome feedback for how to improve our advice.

Strengths and limitations of this study.

• Our approach to preparing this checklist has been pragmatic in terms of the methods we have used.

• We have provided explanations of the basis for each recommendation and references to supporting research.

• We did not conduct a systematic review to inform our guidance.

• We did not review non-English language literature.

• We did not systematically grade the certainty of the evidence or strength of our recommendations.

Introduction

Access to healthcare information is necessary if people are to be involved in decisions regarding their own health.¹ Recognising this, governments in several countries have included the right to healthcare information in patients’ charters. These charters commonly establish people’s right to access information about treatments,² including the benefits and harms of these treatments.³ Patients’ charters also underline the need to provide this information in a way that people can understand and that is adapted to each individual’s needs.^{2 4}

Having the right to information does not necessarily mean that this information is available, and many patients and members of the public struggle to find information that is relevant to their circumstances. At the same time, most people are bombarded with claims in the media and other aspects of day-to-day life about what they should and should not do to maintain or improve their health.

Many health claims are unreliable and conflicting.^5–14 When they are purported to be based on research, this might also contribute to a lack of trust in research. For example, surveys in the UK have shown that only about one-third of the public trust evidence from medical research, while about two-thirds trust the experiences of friends and family.¹⁵

It cannot therefore be assumed that people will trust advice simply because it is based on research evidence and given by authorities. Nor should they, as the opinions of experts or authorities do not alone provide a reliable basis for judging the benefits and harms of interventions.^{16 17} Doctors, researchers and public health authorities—like anyone else—often disagree about the effects of interventions. This may be because their opinions are not always based on systematic reviews of fair comparisons of interventions.¹⁸ Government authorities and professional organisations host many websites that provide health advice to the public. However, these websites often provide information that is unclear, incomplete and misleading.¹¹ We were able to find only three websites that provide information about the effects of healthcare interventions that were explicitly based on systematic reviews.¹⁹ Even where information is based on systematic reviews, it may still be unclear, incomplete and misleading.

People who summarise lengthy research reports to make them more accessible are faced with many choices. This includes decisions about which evidence to present, how this evidence should be interpreted, and the format in which it should be presented. Our own experiences creating summaries based on Cochrane reviews have shown us that there are many pitfalls.^20–25 A fundamental challenge is to find an appropriate balance between accuracy and simplicity. On the one hand, summaries should give a reasonably complete, nuanced and unbiased representation of the evidence. On the other hand, they should be succinct and understandable to people without research expertise.

Another challenge to making research evidence easier to use is that people with expertise in a field have been found to pay attention to read and interpret information differently from people without expertise.²⁶ A common publishing strategy is to accommodate these differences by creating different versions of information for experts and non-experts; for example, for health professionals and for patients. However, both health professionals and patients frequently lack research expertise.^{22 26–29} In terms of understanding evidence-based information about the effects of treatments, ‘experts’ are the people who have acquired the skills needed to understand and interpret results from quantitative studies and systematic reviews. Everybody else could be considered ‘non-experts’ in this area.

This does not mean that this large group of non-experts are universally similar regarding their information needs. They may have different levels of language literacy, health literacy and numeracy, or they may need to use evidence for different kinds of decision-making tasks. However, when it comes to the specific task of understanding research evidence and using this information to weigh the trade-offs between possible benefits and harms, most users are non-experts. Consequently, most people would benefit from information about the effects of interventions that are presented in a way that recognises the needs of non-experts. This includes patients, health professionals and policymakers.

In summary, to make informed choices or decisions, people need information that is accessible, easy to find, relevant, based on the best available evidence, accurate, complete, not misleading, nuanced, unbiased, easy to understand and trustworthy.

The aim of this paper is to provide guidance and a checklist to anyone who is preparing and communicating evidence-based information on the effects of interventions (ie, information based on systematic reviews of fair comparisons) that is intended to inform decisions by patients and the public, health professionals or policymakers.

Methods

Ethical considerations

Development of this checklist was guided by ethical considerations underlying informed consent and patients’ rights. Informed consent in medical research has received a huge amount of attention.³⁰ Informed consent in clinical and public health practice has received far less attention,³¹ and a double standard has existed for at least 50 years.³² Consent in clinical and public health practice is reviewed, if at all, only in retrospect. Health professionals are exhorted to obtain informed consent, but in daily practice, as opposed to in clinical trials, they often minimise uncertainties about interventions and they may feel duty bound to provide unequivocal recommendations.³²

Our starting point in preparing this checklist was the belief that patients and the public have the right to be informed when making health choices—such as a personal choice about whether to adhere to advice, a decision about whether to participate in research or in taking a position regarding a health policy. Specifically, they should have access to the best available research evidence, including information about uncertainty, summarised in plain language. We do not assume that everyone wants this information.

Many people are not interested or prefer for someone else to make healthcare decisions on their behalf. For example, a systematic review of patient preferences for decision roles found that a substantial portion of patients prefer to delegate decision-making to their physician, although in most studies most patients reported a preference for shared decision-making.³³ Some patient’s rights charters take this into account—for instance, the right to waive one’s ‘right to be informed’ is specifically mentioned in the Norwegian Patient Rights legislation.⁴ We would argue that under most circumstances it is good clinical practice to respect patient preferences.³¹ Those people who do not want information on the effects of treatments do not need to read or listen to information, but it should be there for those who want it.

Literature review

To inform the development of this checklist, we compiled research evidence that is relevant to giving guidance on how to communicate evidence-based information about the effects of interventions. We started with our own research and then identified related research through a snowballing and citation reference method. We supplemented this with broad searches for evidence on communicating research evidence and intervention effects and specific searches for each item in the checklist. We did not conduct a systematic review. We have, however, referenced systematic reviews to support each item in the checklist when one was available. When we were not able to find a relevant systematic review, we have referenced the best available evidence that we have found. In addition, we have reviewed relevant guidance and reference lists. This included guidance for plain language summaries of research evidence,³⁴ for reporting and using systematic reviews,^{35 36} for making judgements about the certainty of evidence and for going from evidence to recommendations^37–39 and for risk communication.⁴⁰

Synthesis

We used an iterative, informal consensus process to synthesise our recommendations. This was informed by our own experience and research spanning over three decades, our review of the literature, comparing our recommendations to other relevant guidance, and feedback from colleagues. We met initially to discuss our recommendations, divided up tasks, prepared drafts and then discussed these until we reached agreement on a final set of recommendations. In addition to the checklist summarising our main recommendations, we prepared a flowchart, providing guidance for implementing our recommendations. After agreeing on a set of recommendations, we compared these to recommendations made by others and sent a draft report to 40 people and received feedback from 30 (see Acknowledgements section) requesting structured feedback (online supplementary additional file 1).

Patient and public involvement

We did not directly involve patients in planning or executing this study.

Results

Our recommendations are summarised in a checklist with 10 items (box 1). The basis for each recommendation is provided in online supplementary additional file 2 and explanations for each of the recommendations are provided in online supplementary additional file 3. All of our recommendations could be considered ‘good practice statements’. Good practice statements are recommendations that do not warrant formal ratings of the certainty of the evidence.⁴¹ One way of recognising such recommendations is to ask whether the unstated alternative is absurd.⁴¹ Arguably, that is the case for all the recommendations in box 1.

Box 1. Checklist for communicating effects.

Make it easy for your target audience to quickly determine the relevance of the information, and to find the key messages.

• Clearly state the problem and the options (interventions) that you address, using language that is familiar to your target audience—so that people can determine whether the information is relevant to them.

• Present key messages up front, using language that is appropriate for your audience and make it easy for those who are interested to dig deeper and find information that is more detailed.

• Report the most important benefits and harms, including outcomes for which no evidence was found—so that there is no ambiguity about what was found for each outcome that was considered.

For each outcome, help your target audience to understand the size of the effect and how sure we can be about that; and avoid presentations that are misleading.

• Explicitly assess and report the certainty of the evidence.

• Use language and numerical formats that are consistent and easy to understand.

• Present both numbers and words and consider using tables to summarise benefits and harms, for instance, using Grading of Recommendations Assessment, Development and Evaluation (GRADE) summary of finding tables or similar tables.

• Report absolute effects.

• Avoid misleading presentations and interpretations of effects.

• Help your audience to avoid misinterpreting continuous outcome measures.

• Explicitly assess and report the credibility of subgroup effects.

• Avoid confusing ‘statistically significant’ with ‘important’ or a ‘lack of evidence’ with a ‘lack of effect’.

Help your target audience to put information about the effects of interventions in context and to understand why the information is trustworthy.

• Provide relevant background information, help people weigh the advantages against the disadvantages of interventions and provide a sufficient description of the interventions.

• Tell your audience how the information was prepared, what it is based on, the last search date, who prepared it and whether the people who prepared the information had conflicts of interest.

Flowchart

The flowchart (figure 1) outlines a process for producing evidence-based information about the effects of interventions. It provides examples that illustrate each step of the process. ^42–46 The process begins with making sure that you know your target audience. It is important to consider how members of your target audience will be involved in the process. The next steps in the process are designing and user testing a template for the information that you will prepare, organising an editorial process and training and considering ways of making it easy for your target audience to find your information. Although the flowchart suggests a linear process, development should be approached as an iterative, cyclical process. The last step in figure 1 is to collect feedback on each individual piece of information from people in your target audience; to make changes if needed (to your template as well as to individual pieces of information); and to evaluate again, if needed. It also includes establishing routines for updating the information that you prepare, if this is planned.

Figure 1.

Flow chart outlining a process for producing evidence-based information about the effects of interventions

Discussion

How our checklist compares with related checklists and guidance

Although our guidance overlaps with other guidance,^{38 47–54} for the most part other guidance does not specifically addressing preparation of evidence-based information for decision-makers about the effects of interventions. The one exception or which we are aware is the ‘guideline for evidence-based health information’ prepared by the German Network for Evidence-based Medicine (DNEbM),⁵⁵ which is only partially translated to English as of April 2020. The DNEbM recommendations are consistent with or recommendations to present both numbers and words and report absolute effects. They do not explicitly address our other recommendations. Comparison of our guidance with other guidance is summarised in table 1.

Table 1.

Comparison of our checklist with other guidance

Guidance	Purpose	Comparison with our checklist
The Conference On Guideline Standardisation checklist for reporting clinical practice guidelines47	The checklist is intended to minimise the quality defects that arise from failure to include essential information and to promote development of recommendation statements that are more easily implemented.	Focus is on content of a full guideline report rather than on presentation of information. It does not include guidance for how to present information about benefits and harms. It is consistent with our checklist for the items that overlap. Some of the 18 items are outside of the scope of our checklist.
DISCERN instrument for judging the quality of written consumer health information on treatment choices48	To enable patients and information providers to judge the quality of written information about treatment choices and to facilitate the production of new, high-quality, evidence-based consumer health information.	There is some overlap, but the focus is on content of information for patients and the public rather than on presentation of that information; and the checklist is presented as an instrument for assessing the quality of information rather than as a guide for preparing it.
Ensuring Quality Information for Patients tool49	To provide a practical measure of the presentation quality for all types of written healthcare information.	There is some overlap, but it does not address how to present evidence-based information about the effects of interventions. It includes some relevant suggestions that we have not included:
		Use short sentences
		Personally address the reader
		Be respectful
		Include easy-to-understand illustrations.
Evidence-based risk communication50	Key findings to inform best practice from a systematic review of the comparative effectiveness of methods of communicating probabilistic information to patients who maximise their cognitive and behavioural outcomes.	The findings from this systematic review are largely consistent with our recommendations for how to help people understand the size of effects. It includes some suggestions that we have not:
		Add bar graphs or icon arrays to natural frequencies or event rates.
		Consider the use of icon arrays with smaller numerators and bar graphs with larger numerators.
		Place a patient’s risk in context by using comparative risks of other events.
		Realise that positive framing (stating benefits rather than harms) increases acceptance of therapies.
GRADE guidelines38	To provide guidance for use of the GRADE* system of rating the certainty of evidence and grading the strength of recommendations in systematic reviews, health technology assessments and clinical practice guidelines.	This is a series of articles that provide detailed guidance for people preparing systematic reviews, health technology assessments, or guidelines. We have helped to develop this guidance and have drawn on it. Our checklist is consistent with GRADE guidance for summary of finding tables and communicating information about uncertainty.
	International Patient Decision Aid Standards (IPDAS) Patient Decision Aid User Checklist51 52		To provide a set of quality criteria for patient decision aids.	Many of the items in the IPDAS checklist overlap with our checklist. It also includes items that are outside of the scope of our checklist (eg, decision aids for tests, helping users to clarify their values and evaluation of decision aids) as well as some items that are within our scope, which we have not included. They are reformulated here as guidance:
Use visual diagrams to show the probabilities (eg, faces, stick figures or bar charts).
Allow patients to select a way of viewing the probabilities (eg, words, numbers, diagrams).
Present probabilities using both positive and negative frames (eg, showing both survival and death rates).
Describe the features of options to help patients imagine what it is like to experience their physical, emotional and social effects.
Provide stories of other patients’ experiences.
Identify the reading level at which it is written and the formula (method) used to determine the level.
Provide ways to help patients understand information other than reading (eg, audio, video or in-person discussion).
Risk and uncertainty communication53	Explores the major issues in communicating risk assessments arising from statistical analysis and concludes with a set of recommendations.	Largely consistent with our checklist. Includes a set of recommendations about visualisations, such as:
		Illuminate graphics with words and numbers.
		Design graphics to allow part-to-whole comparisons on an appropriate scale.
		Helpful narrative labels are important.
		Be cautious about interactivity and animations.
		Avoid chart junk.
		Most importantly, assess the needs of the audience, experiment, test and iterate towards a final design.
US National Standards for the Certification of Patient Decision Aids54	To provide criteria for a potential decision aid certification process in the USA	Although there is some overlap with our checklist, the criteria do not address how to present information about the effects of interventions other than ‘adopting risk communication principles’.

*Grading of Recommendations Assessment, Development and Evaluation

The Ensuring Quality Information for Patients (EQIP) tool⁴⁹ and the International Patient Decision Aid Standards (IPDAS) checklist^{51 52} include specific recommendations related to using plain language (short sentences and a reading level not exceeding a reading age of 12). We have included key principles for plain language in our detailed guidance (online supplementary additional file 3).

The EQIP tool,⁴⁹ the IPDAS checklist^{51 52} as well as a systematic review on evidence-based risk communication by Zipkin and colleagues⁵⁰ recommend using visual aids. The last two recommend using graphs to show probabilities. We agree that information for people making decisions about interventions should be visually appealing and that well-designed visualisations can help some people to understand information about the effects of interventions. The DNEbM guidelines⁵⁵ recommend that ‘graphics may be used to supplement numerical presentations in texts or tables’ based on ‘low-quality’ evidence. They also recommend that ‘if graphics are used as a supplement, then either pictograms or bar charts should be used’ based on ‘moderate-quality’ evidence. Spiegelhalter⁵³ recommends visualisations in communication about risk and uncertainty, which seems sensible. However, we do not think there currently is enough evidence to support recommendations about when to use visualisations or what type of visualisation to use.50 53 56 57

The systematic review on evidence-based risk communication⁵⁰ suggests being aware that positive framing (stating benefits rather than harms) increases acceptance of therapies. The IPDAS checklist^{52 53} recommends presenting probabilities using both positive and negative frames (eg, showing both survival and death rates). We do not think there currently is enough evidence for either of these recommendations.⁵⁸

Zipkin and colleagues⁵⁰ suggest placing a patient’s risk in context by using comparative risks of other events. We do not think there is currently enough evidence to support this recommendation and question its relevance for many decisions about interventions.

The DNEbM guidelines⁵⁵ suggest ‘interactive elements may be used in health information’ based on ‘moderate-quality’ evidence. Similarly, the IPDAS checklist^{51 52} recommends allowing patients to select a way of viewing the probabilities (eg, words, numbers, diagrams). We agree this is sensible and, in previous work, we have designed an interactive summary of findings with this in mind.⁴⁵ However, there is limited evidence to support this recommendation. We attempted to test this hypothesis in a randomised trial.⁵⁹ Because of technical problems (the interactive summary of findings and data collection did not work for some participants), we were not able to complete the trial. The qualitative data that we collected suggested that participants (people in Scotland with an interest in participating in randomised trials of interventions⁶⁰) had mixed views about their preferences for an interactive versus a static presentation. They also had mixed views regarding which initial presentation they preferred in the interactive presentation.

Finally, the DNEbM guidelines conclude that ‘narratives cannot be recommended’ based on ‘low-quality’ evidence. In contrast, the IPDAS checklist^{51 52} recommends including stories of other patients’ experiences and using audio and video to help users understand the information. We agree that this may be helpful. However, it is also possible that stories that specifically describe patients’ experiences of treatment effects and side effects can have unintended consequences. For example, people’s perceptions of their own risks of experiencing a benefit or harm could be influenced by whether they identify with the person telling the story or not. We are not aware of evidence from randomised trials comparing information with and without patients’ experiences, audio or video; or comparing different types of presentations. A recent systematic review on the use of narratives to impact health policymaking did not find any trials.⁶¹

Strengths and weaknesses of our checklist

We did not conduct a systematic review to inform our guidance, review non-English language literature, assess the certainty of the evidence supporting each recommendation, grade the strength of our recommendations or use a formal consensus process. However, we have provided explanations of the basis for each recommendation and references to supporting research. Our approach to preparing this checklist has been pragmatic in terms of the methods we have used. We hope that others will find the checklist practical and helpful. To facilitate use of the checklist, we have prepared a flowchart with examples (figure 1).

Implementation of the guidance can be facilitated by developing a template, specific guidance for those charged with using the template to prepare the information and training for those people. Links to examples of these are found in the flowchart. User testing can help to ensure that people in your target audience experience the information positively and as intended. We have provided links to examples of user tests of information about the effects of interventions and to resources for user testing in the flowchart.

Implications for research

There remain many important uncertainties about how best to present evidence-based information about the effects of interventions to people making decisions about those interventions There is a need for more primary research and more systematic reviews in this field. We have summarised key uncertainties that we identified while preparing this checklist in table 2. In addition, there is a need for a methodological review and a consensus on appropriate outcomes for studies evaluating different ways of communicating evidence-based information about the effects of interventions.⁶²

Table 2.

Important uncertainties about how to present evidence-based information about the effects of interventions to people making decisions

Question	What is known	Research that is needed
What are the effects of alternative visual displays of intervention effects on understanding and users’ experience of the information?	Not all visual displays are more intuitive than text or numbers, some visual displays can be misleading, some may require explanation in order for people to understand them and people tend to prefer simplicity and familiarity, which may not be associated with accurate quantitative judgements.50 53 56 57 63 64	Design and user testing of ways of visualising effects of multiple outcomes; randomised trials comparing different graphs or visualisations to each other and to information (tables and text) without visualisations and a systematic review of those trials.
What are the effects of positive versus negative framing for different types of decisions on people’s understanding and decisions?	Low to moderate certainty evidence suggests that both attribute and goal framing may have little if any consistent effect on patients’ behaviour.58 Unexplained heterogeneity between studies suggests the possibility of a framing effect under specific conditions.	Randomised trials comparing positive to negative framing for different types of decisions and a systematic review of those trials.
When should CIs be reported and how should they be presented and explained?	Although CIs are more informative than p values, CIs can also be misinterpreted.43 65 66 There are pros and cons to reporting CIs and little evidence to support a recommendation either to include them or exclude them, or how to present and explain them, if they are included. Deciding whether and how to report CIs may depend on the target audience.	User testing of ways of presenting and explaining CIs; randomised trials comparing different ways of presenting and explaining CIs to other ways and to not presenting CIs and a systematic review of those trials.
What are the effects of interactive presentations of information about the effects of interventions compared with static presentations, on comprehension, ease of use and usefulness in decision-making for people across a broad range of target audiences?	Different people prefer different types of presentation formats, and access information for different reasons that require different amount of detail. Instead of offering multiple tailored static formats to different audiences, an alternative solution is making multiple types of presentations available to all viewers through an interactive solution. Unpublished qualitative data from a failed trial with patients and the public59 suggest that there may be mixed preferences for an interactive versus a static presentation. There is also uncertainty about which initial presentation to use for interactive presentations.	Design and user testing of interactive presentations; randomised trials comparing interactive to static presentations in a heterogeneous group, comparing alternative initial presentations across different subgroups and a systematic review of this evidence.
What are the effects of including stories of patients’ experiences in patient information?	People want this information and value it.20	Design and user testing of ways of incorporating patients’ experiences, including the use of patients’ stories to describe treatment benefits and harms or to describe the treatment or condition; randomised trials comparing information with and without patients’ experiences and a systematic review of this evidence.
What are the effects of audio and video presentations of information about the effects of interventions on peoples’ understanding, decisions and experience of the information?	Audio and video presentations are likely to be helpful for people with poor reading skills and some people may prefer these presentations either as an alternative or as a supplement to reading.	Design and user testing of audio and video presentations; randomised trials comparing information with and without audio and video presentations and a systematic review of this evidence.

Conclusions

The checklist that we have developed, which includes 10 items, is the top layer of our recommendations for how to prepare evidence-based information on the effects of interventions that is intended to inform decisions by patients and the public, health professionals or policymakers. These 10 recommendations summarise the lessons that we have learnt from our review of relevant research. The recommendations draw on our own experience over the past 20–30 years in developing and evaluating ways of helping people to make well-informed health choices by making research evidence more understandable and useful to them. We welcome feedback and suggestions for how to improve our advice.