Guidance on Communication of Uncertainty in Scientific Assessments

Abstract

This document provides guidance for communicators on how to communicate the various expressions of uncertainty described in EFSA's document: ‘Guidance on uncertainty analysis in scientific assessments’. It also contains specific guidance for assessors on how best to report the various expressions of uncertainty. The document provides a template for identifying expressions of uncertainty in scientific assessments and locating the specific guidance for each expression. The guidance is structured according to EFSA's three broadly defined categories of target audience: ‘entry’, ‘informed’ and ‘technical’ levels. Communicators should use the guidance for entry and informed audiences, while assessors should use the guidance for the technical level. The guidance was formulated using evidence from the scientific literature, grey literature and two EFSA research studies, or based on judgement and reasoning where evidence was incomplete or missing. The limitations of the evidence sources inform the recommendations for further research on uncertainty communication.

Short abstract

This publication is linked to the following EFSA Supporting Publications article: http://onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2019.EN-1540/full

Summary

The European Food Safety Authority (EFSA) requested its Scientific Committee and Emerging Risk Unit (SCER) and its Communication Unit (COM) to establish a working group to develop practical guidance for EFSA communicators on how to communicate the various expressions of uncertainty resulting from the uncertainty analyses (e.g. qualitative, quantitative) described in EFSA's document: ‘Guidance on uncertainty analysis in scientific assessments’ (EFSA Scientific Committee, 2018a; henceforth, ‘Uncertainty Analysis GD’).

The European Union (EU) Food Law identifies the target audiences of risk communication on EU food and feed safety: ‘risk assessors, risk managers, consumers, feed and food businesses, the academic community and other interested parties’. EFSA tailors its communication messages according to the expected scientific literacy of these audiences and their varying interests, splitting them into three broad categories – ‘entry’, ‘informed’ and ‘technical’ levels. EFSA delivers layered content to these audiences through a mixture of channels (meetings, media, website, social media, journal) and formats (scientific opinions, news stories, multimedia, posts) depending on their information needs and behaviours.

The application of EFSA's Uncertainty Analysis GD can result in eight main types of expressions of uncertainty: (1) unqualified conclusions with no expression of uncertainty; (2) description of a source of uncertainty; (3) qualitative description of the direction and/or magnitude of uncertainty; (4) inconclusive assessment; (5) a precise probability; (6) an approximate probability; (7) a probability distribution; and (8) a two‐dimensional probability distribution. In this communication guidance document (henceforth, ‘Uncertainty Communication GD’), these eight expressions provide the framework for the discussion of evidence sources and the guidance on communication of uncertainty.

This Uncertainty Communication GD is primarily aimed at communicators at EFSA, but could be applied by risk communicators in other institutions that provide scientific advice. It should be used as a supporting document alongside the EFSA handbook: When food is cooking up a storm – proven recipes for risk communications (EFSA, 2017; henceforth, EFSA Risk Communication Handbook), which is a practical guide for food safety risk communicators.

This document contains general and specific guidance for communication to audiences within the three levels: guidance for the entry and informed levels is addressed to communicators producing supporting communications (e.g. news stories), while the guidance for the technical level is addressed to assessors producing scientific outputs (e.g. opinions, reports). The guidance consists of clear instructions, precautions, examples and choices to consider. The general guidance should be used for all uncertainty communication. Use of the specific guidance is case‐by‐case; an easy‐to‐use form is the main tool provided for identifying the specific guidance applicable to the eight different uncertainty expression types.

The general and specific guidances were developed from an analysis of key evidence sources including selected academic literature, extracts from similar frameworks or guidance documents from other national or international advisory bodies, and the results of EFSA's own target audience research studies.

Although the available evidence on the best ways to communicate the uncertainty expressions was limited overall, the expert analysis of selected academic literature provided a useful starting point. All but two of the eight expressions of uncertainty resulting from uncertainty analyses, i.e. ‘unqualified conclusion with no expression of uncertainty’ and ‘inconclusive assessment’, were extracted from the selected papers, either as formulated by the authors themselves or interpreted and drafted by the working group. The recommendations from this literature were applied and tested on real examples of EFSA scientific assessments, to draft messages and select visual aids for communicating the related uncertainties. Since the available evidence does not, however, address every aspect of communicating uncertainty, some guidance in the current document is based on judgement and reasoning.

EFSA commissioned a focus group study in 2016 (Etienne et al., 2018) and carried out its own follow‐up multilingual online survey in 2017 (EFSA, 2018) that represented the early development phase of the Communication GD. The studies provided indications of target audience perspectives of the usefulness of uncertainty information, the impact of language, culture and professional background, and on audiences’ understanding of and/or preferences for messages describing four types of expressions of uncertainty. Both studies have limitations in their design and execution but, considered cautiously, they are useful sources of insights directly from EFSA's key target audiences.

How uncertainties should be conveyed to enable non‐scientists to make informed decisions is still an under‐researched field. Experience gained during the implementation of this Uncertainty Communication GD itself will provide new insights into the best way to communicate different expressions of uncertainty in scientific assessments. EFSA therefore intends to review and, if needed, update the Uncertainty Communication GD over the next five years.

Future research should address how various subjective probabilities could be communicated to laypeople so that they understand the information. Additional research should examine how well test subjects understand uncertainty communication and whether various communication formats result in different decisions. All such research should also involve decision‐makers in the risk management domain, because these stakeholders may reach different conclusions depending on how uncertainty is communicated.

1. Introduction

1.1. Background

The European Food Safety Authority's (EFSA) Guidance document on uncertainty analysis in scientific assessment (EFSA Scientific Committee, 2018a; henceforth, ‘Uncertainty Analysis GD’) defines ‘uncertainty’ as a general term referring to all types of limitations in available knowledge that affect the range and probability of possible answers to an assessment question.

Available knowledge refers here to the knowledge (evidence, data, etc.) available to assessors at the time the assessment is conducted and within the time and resources agreed for the assessment. Sometimes ‘uncertainty’ is used to refer to a source of uncertainty (see separate definition), and sometimes to its impact on the conclusion of an assessment. In science and statistics, we are familiar with concepts such as measurement uncertainty and sampling uncertainty, and that weaknesses in methodological quality of studies used in assessments can be important sources of uncertainty. Uncertainties in how evidence is used and combined in assessment – e.g. model uncertainty, or uncertainty in weighing different and sometimes conflicting lines of evidence in a reasoned argument – are also important sources of uncertainty. General types of uncertainty that are common in EFSA assessments are outlined in Section 8 of its ‘Scientific opinion on the principles and methods behind EFSA's guidance on uncertainty analysis in scientific assessment’ (EFSA Scientific Committee, 2018b). There the Scientific Committee recommended closer interaction between assessors and decision‐makers both during the assessment process and when communicating the conclusions. Understanding of the type and degree of uncertainties identified in the assessment helps to characterise the level of risk to the recipients and is therefore essential for informed decision‐making (EFSA Scientific Committee, 2018b). Communication helps them to understand the range and likelihood of possible consequences.

During the development of the Uncertainty Analysis GD, the Scientific Committee reviewed literature on the communication of uncertainty information as a basis for developing a common approach for EFSA's communications on uncertainty to different target audiences, including decision‐makers and the general public. That initial review indicated that the literature is equivocal about the most effective strategies to communicate scientific uncertainties and that, on the whole, there is a lack of empirical data in the literature on which to base a working model. In EFSA's own publication on best practice in risk communication ‘When food is cooking up a storm – proven recipes for risk communications’ (EFSA, 2017), which was developed with national partners, the advice to communicators is only of a general nature.

Therefore, the Scientific Committee recommended that EFSA should initiate research activities to explore best practices and develop further guidance in areas where this would benefit implementation of the Uncertainty Analysis GD, and the communication of uncertainties in scientific assessments targeted at different audiences. This would allow EFSA to identify how changes could be made to its current communications practices in relation to uncertainties and to tailor key messages to specific target audience needs.

As EFSA completed its research activities on communication of uncertainties to different target audiences (EFSA, 2018; Etienne et al., 2018), the Scientific Committee proposed to develop a separate Guidance document on communication of uncertainty in scientific assessments (henceforth ‘Uncertainty Communication GD’). The Scientific Committee considered that the significance of the research results and the different purpose, scope and target of the communication methodology warranted a stand‐alone document for communication practitioners. The Uncertainty Communication GD is a companion document to the Uncertainty Analysis GD.

1.2. Terms of Reference as provided by the requestor

The European Food Safety Authority (EFSA) asked its Scientific Committee and Emerging Risk Unit (SCER) and its Communication Unit (COM) to establish a working group to develop guidance on how to communicate uncertainty on the basis of EFSA's Uncertainty Analysis GD.

The Scientific Committee advised that to carry out this work expertise in social sciences (e.g. psychology, risk communication, uncertainty communication and public perceptions) was needed to join its working group on uncertainty. The working group had the following three objectives:

1. Develop practical guidance for EFSA communicators on how to communicate the various expressions of the uncertainty analyses (e.g. qualitative, quantitative) described in EFSA's Uncertainty Analysis GD, and in the ‘Scientific opinion on the principles and methods behind EFSA's guidance on uncertainty analysis in scientific assessment’ (EFSA Scientific Committee, 2018b) with the aim of bridging risk assessors and the different EFSA target audiences.

2. Advise assessors on the ways in which uncertainties are reported in EFSA assessments in relation to the need to communicate.

3. Advise EFSA on its current communication approach for dealing with uncertainty as described in the EFSA Risk Communication Handbook (EFSA, 2017).

1.3. Interpretation of the Terms of Reference

The working group developed the following work plan to reach the three objectives of the Terms of Reference:

1. Develop practical guidance for EFSA communicators on how to communicate the various expressions of the uncertainty analyses (e.g. qualitative, quantitative) described in EFSA's Uncertainty Analysis GD and in the ‘Scientific opinion on the principles and methods behind EFSA's guidance on uncertainty analysis in scientific assessment’ (EFSA Scientific Committee, 2018b). The following tasks were planned to reach objective 1:

   • – Identify review papers for risk communication and relate them to uncertainty communication.
   • – Perform a literature search on uncertainty communication and review the resulting literature.
   • – Complement the literature with the insights gained from the EFSA research projects on communication of uncertainty from 2016 and 2017.
   • – Perform an online search on approaches to communicating uncertainty by relevant national and international organisations.
   • – Identify a representative set of outputs with case studies upon which EFSA might communicate, map the different sensitivities of topics dealt with by EFSA, target audiences, different types of assessment and expressions of uncertainty (e.g. probabilities, quantitative, qualitative), linking them to the methods described in the Uncertainty Analysis GD.
   • – Consider whether different communications on uncertainty are needed for EFSA's defined target audiences and/or whether new categories are required.
   • – Develop a practical communications approach and supporting tools for communications practitioners who are required to communicate scientific uncertainties to different target audiences. Use the literature review and the results of EFSA's target audience research activities conducted in 2016 and 2017 to inform the approach.
   • – Draft this Guidance document on communication of uncertainty for consultation and follow‐up on feedback from the consultation activities with a report and input for finalising the Guidance document.
   • – Consult EFSA's risk communication partners in the European Union (EU) Member States, the EU institutions and other interested parties (e.g. other EU agencies, international organisations, non‐EU countries) before finalising the Communication GD.

2. Advise assessors on the ways in which uncertainties are reported in EFSA assessments in relation to the need to communicate. The following tasks were identified to reach objective 2:

   • – For the results of objective 1, consider whether there are additional requirements and/or recommendations (e.g. terminology, data format, graphics) that Panels should be aware of when drafting their opinions and especially the conclusions of their assessments.

3. Advise EFSA on its current communication approach for dealing with uncertainty as described in the EFSA Risk Communication Handbook (EFSA, 2017). The following tasks were identified to reach objective 3:

   • – Review the relevant section of the EFSA Risk Communication Handbook.
   • – Identify key examples of past communication challenges in which uncertainty was a decisive issue to determine the impact of a new approach to uncertainty communication.

1.4. Scope, audience and use

The Terms of Reference require providing guidance for EFSA on how to communicate uncertainty on the basis of its Uncertainty Analysis GD. This Uncertainty Communication GD provides a practical framework for communicating uncertainties in scientific assessments. It does not provide a template for EFSA's risk communication activities as this takes place within a well‐defined legal framework. However, a short description of EFSA's risk communication role, strategies and target audiences follows below (Section 1.5) as background and context.

1.4.1. Audience for this guidance

This Uncertainty Communication GD is primarily aimed at risk communicators at EFSA, but it may also be helpful to risk communicators at other institutions that provide scientific advice. It does not replace the current EFSA Risk Communication Handbook (EFSA, 2017), which describes the overall framework and practical approaches for risk communication at EFSA, but supports and complements that publication. In addition to guidance for communicators, this document contains some guidance for assessors on how best to report the various expressions of uncertainty resulting from their uncertainty analyses so that they can support effective uncertainty communication.

1.4.2. Structure of the guidance document

This Uncertainty Communication GD first explains EFSA's three broad communication target audiences (Section 1.5.2) and the eight different possible expressions of uncertainty as described in the Uncertainty Analysis GD (Section 2). As shown in Figure 1, Risk communicators looking for straightforward instructions on how best to communicate the various expression of uncertainty should go to Section 3. The guidance for the entry‐ and informed‐level audiences is addressed to communicators, while the guidance for the technical level is addressed to assessors and advises them on the information to provide in their assessments that is needed for communication. Section 4 describes the evidence sources that contributed to development of the guidance. Additional reasoning is summarised in Appendix B. Section 5 provides recommendations for further research and Section 6 outlines EFSA's plans for implementing the approach in its working practices and subsequent evaluation.

Figure 1.

Visual map of the structure and how to use this Uncertainty Communication GD

1.5. Risk communication at EFSA

EFSA's science communications role within the EU food safety system is discussed in this section. For terminology, the Uncertainty Analysis GD refers to ‘scientific assessment’ rather than ‘risk assessment’ to recognise that the Uncertainty Analysis GD is applicable to all of EFSA's scientific advisory work, not solely its risk assessments. Notwithstanding this, most of the following section refers to EFSA's ‘risk communication’ role because this is the terminology used in the relevant legal texts establishing EFSA. However, this Uncertainty Communication GD applies to all of EFSA's ‘science communication’ activities and so is titled in full: ‘Guidance on communication of uncertainty in scientific assessments’.

1.5.1. EFSA's risk communication role and strategies

Under the EU Food Law, Regulation (EC) No 178/20021, by which EFSA was founded, EFSA is ‘an independent scientific source of advice, information and risk communication to improve consumer confidence’. It defines risk communication as ‘the interactive exchange of information and opinions throughout the risk analysis process as regards hazards and risks, risk‐related factors and risk perceptions’. As stated by Codex (2018), risk communication should include a ‘transparent explanation of the risk assessment policy and of the assessment of risk, including the uncertainty’.

EFSA is charged with communicating the results of its work in the fields within its mission (food and feed safety, animal and plant health, nutrition) and with explaining its risk assessment findings. The European Commission is responsible for communicating its risk management decisions and the basis for them (i.e. scientific and/or other considerations). EU Member States are also charged with public communication on food and feed safety and risk. Given these overlapping roles, the Regulation also requires that EFSA collaborate closely with the Commission and the Member States to ‘promote the necessary consistency in the risk communication processes’.2 Risk management measures, if needed, usually take further time to agree and implement after the completion of an assessment. Public perceptions of the related risks may be affected by such a timescale, requiring linked communication over the entire public information campaign. Critically, this also allows opportunities for feedback and dialogue with the representatives of consumers, producers and other interested parties, which is also required under the Food Law.

With a clear mandate to communicate its scientific assessment results ‘on its own initiative’, EFSA follows a communications strategy based on guidelines in its EFSA Risk Communication Handbook (EFSA, 2017) that was developed together with the European Commission and members of EFSA's Communications Experts Network, which comprises communications representatives of EU national competent authorities on food and feed safety. The guidelines provide a framework to assist decision‐making about appropriate communication approaches in a wide variety of situations that can occur when assessing and communicating on risks related to food safety in Europe.

EFSA publishes over 500 scientific assessments and reports annually, but only about one‐tenth of these are accompanied by supporting communications (e.g. press releases/interviews, FAQs, videos, briefings). EFSA's communicators weigh up several factors in selecting assessments for supporting communication and the mix of communication approaches and formats to employ for each. Broadly, these include:

• significance of the scientific results (e.g. routine vs new findings);

• nature of the risk (e.g. emerging, possible, identified or confirmed);

• potential public/animal health/environmental impact (e.g. there is a safety concern);

• public perception and anticipated reactions/sensitivity of subject area;

• legislative and market contexts (e.g. a request for authorisation);

• urgency of request (e.g. in an outbreak situation, or a long‐term review);

• institutional risk (national, European, international contexts).

The existence of one or more of the above factors in relation to an upcoming assessment or other activity is a potential trigger for such supporting communications. The communicator analyses the key results of the assessment or report, discusses key messages and supporting points with the assessors (scientific officers, EFSA Panel or working group members), agreeing a communications plan in the process. The plan identifies the rationale for communicating, the key messages and supporting points, and also defines the target audiences for the communication.

1.5.2. EFSA's target audiences

The EU Food Law identifies the target audiences of risk communication on EU food and feed safety as ‘risk assessors, risk managers, consumers, feed and food businesses, the academic community and other interested parties’. For EFSA's risk communication, the potential audience is therefore 500 million people residing in the European Union. Their interest, knowledge and concerns about food and food safety vary widely. Language, tradition, culture, age and education (e.g. scientific literacy) are among the variables that affect their understanding of messages about food safety‐related assessments.

Communicating directly to everyone is unrealistic given this diversity and complexity. Therefore, in the EU food safety system, many players share responsibility for communicating about food risks (and benefits) to consumers, food chain operators and other interested parties. EFSA cooperates with these other players: its partners in the EU Member States, EU institutions and ‘stakeholder’ groups (e.g. consumer organisations and public health professionals) to further disseminate the outcomes of its scientific assessments. To enable this, EFSA tailors its communication messages in layers to the expected scientific literacy and interests of these audiences and targets them through a mixture of channels (meetings, the media, website, social media, scientific journal) and formats (scientific opinions, news stories, multimedia, tweets).

In devising its external communications, EFSA follows an approach for mapping and targeting these audiences (see Table 1) that was codified internally in 2014–2015 for the redesign of EFSA's corporate website. It was subsequently adapted to other communication channels and formats; more detailed segmentation of the audiences is possible for specific types of communication (e.g. to attract participants to events on specialist topics). The approach was based on the analysis of extensive user‐centred research involving interviews, online surveys, analytics data (web metrics, media pick up) and external expertise, as well as the frameworks that guide EFSA's work: EU Food Law, EFSA's strategic documents and plans (see Annex A for further details).

Table 1.

Mapping EFSA's target audiences for external communications (2015)

The table shows key target audience groups for EFSA that were identified through this research: decision‐makers, assessors, industry, non‐governmental organisations (NGOs)/specialised media, general media, and informed/concerned citizens. It also clusters them according to their scientific literacy and temporal relationship with EFSA's communications into three broad categories – ‘entry’, ‘informed’ and ‘technical’ levels. EFSA generally seeks to layer its communications content to improve accessibility for users in these broad audience categories. Certain audiences may generally prefer different communications products and channels, and the content can be tailored accordingly. The layering of information can also be within a single communication output. For example, a news story headline and introductory paragraphs are worded to be understandable to ‘entry’ level users while subsequent paragraphs may provide more details for ‘informed’ and ‘technical’ audiences.

The mapping of target audiences and strategy for content development comes with important caveats. There is much diversity within the target audiences and considerable overlap between them in terms of the assumptions made about them (e.g. their scientific literacy) and the communication products they may use when informing themselves about EFSA. For example, industry representatives have widely varying degrees of scientific literacy and users in all categories have personal preferences that may guide them in selecting one format over another. Nevertheless this approach proved practical and was already in use at EFSA, making it a functioning framework for the purposes of the Communication GD. Therefore, these groups of target audiences will be used as a parameter in structuring the guidance in Section 3. For communicators not involved in developing or further disseminating EFSA's communication activities but who may wish to use this guidance, this approach could probably be easily adapted to the characteristics of their target audiences.

1.6. Uncertainty communication

Communicating uncertainty aims to increase the transparency of the scientific assessment process and to provide the risk managers with a more informed evidence base by reporting on the strengths and weaknesses of the evidence (EFSA Scientific Committee, 2018b). However, several factors require consideration in developing strategies for uncertainty communications. Aversion to ambiguity has been observed in many studies (Ellsberg, 1961). People tend to prefer a risky option (e.g. 30% chance of a gain) over an ambiguous option (e.g. 20–40% chance of a gain). This aversion does not imply, however, that uncertainty should not be communicated. Transparent and open communication requires that uncertainty is communicated. Furthermore, studies suggest that people prefer to be openly informed about uncertainty associated with scientific findings (Frewer et al., 2002; Miles and Frewer, 2011).

The willingness to receive information about uncertainty also depends on education and age, with young and highly educated people wanting more details about uncertainty than less educated groups (Lofstedt et al., 2017). Additionally, beliefs about science interact with the level of scientific uncertainty, with high uncertainty being more influential on the willingness to act for people considering science as debate than for people understanding science as a search for absolute truth (Rabinovich and Morton, 2012).

To understand how and when uncertainty could be best formulated in qualitative terms, Frewer et al. (2002) tested the reception of 10 uncertainty statements, finding that people wanted to be provided with information on uncertainty as soon as it was identified and in full. Miles and Frewer (2011) tested the communication of eight types of uncertainty: uncertainty about who is affected, about past and future states, measurement uncertainty, due to scientific disagreement, from extrapolation from animals to humans, about the size of the risk and about how to reduce the risk. Their results were different across five different food hazards and confirmed higher risk perceptions in the presence of uncertainty for technological risks (genetically modified organisms, pesticides) than for natural risks (BSE, fat diets, Salmonella).

Frewer et al. (2002) found that uncertainty associated with the scientific process was more readily accepted than uncertainty due to lack of action by the government. This suggests that communication of uncertainty is less likely to cause public alarm if it is accompanied by information on what actions are being taken by the relevant authorities to address that uncertainty. However, such actions are risk management measures, which are outside the remit of EFSA (see Section 1.5.1). Therefore, when having assessed a risk, and when EFSA communications include information on uncertainty, consideration should be given to coordinating with the Commission on what can be said about any measures aimed at addressing the uncertainty.

1.6.1. EFSA's context

Transparency is one of EFSA's basic principles (EFSA, 2003) and has important implications for risk communication. A transparent approach to explaining how an organisation works, its governance and how it makes its decisions, is intended to build trust. Communications should clearly convey the most important areas of uncertainty in the scientific assessment, whether and how these can be addressed by the assessors and decision‐makers, and the implications of these remaining uncertainties for public health (EFSA, 2017).

It is important to distinguish between the understanding (comprehension) of EFSA's message by the audience, and the audience's perception of risk and uncertainty after receiving the message. Article 40 of EFSA's founding Regulation requires that EFSA ‘shall ensure that the public and any interested parties are rapidly given objective, reliable and easily accessible information, in particular for the results of its work’. This implies that EFSA's communications should be designed to ensure the results of its work (including its assessment of risk and uncertainty) are correctly understood by its audience, which is therefore the objective of this guidance. How people perceive risk and uncertainty themselves, after receiving EFSA's communications, will be influenced by many factors including their own prior beliefs, their stake in the issues involved, and their values. While authorities may consider it appropriate to communicate options for managing risk and uncertainty, this is outside EFSA's remit. This has implications for the review of the literature and the guidance, as most studies have focused on the perceptions of subjects after receiving information on risk and uncertainty, rather than their understanding of the information as it was communicated (see Section 4).

1.6.2. International context

Many national and international risk communication Guidance documents include only general rather than specific considerations about communication of uncertainty (Bloom et al., 1993; NRC, 2003; Fischoff et al., 2012; EEA, 2013; NASEM, 2017). A few national and international guidance documents on uncertainty analysis include a section with suggestions on how to communicate the findings of uncertainty analysis most effectively (Wardekker et al., 2013; Mastrandrea et al., 2010; BfR, 2015). Some of these suggestions reiterate some of the basic principles of good risk communication in the context of uncertainty, such as the need for transparent reporting of, for example, lack of sufficient knowledge, shared assumptions, or criteria by which evidence is included or dismissed.

Specific guidance on how to communicate on uncertainty in general has not yet been developed, with rare exceptions such as that included in the ‘Guidance for uncertainty assessment and communication’ by the Dutch environmental assessment agency (Wardekker et al., 2013) and in ‘Environmental decisions in the face of uncertainty’ by the Institute of Medicine in the USA (IOM, 2013). In both, general recommendations for communicating uncertainty are provided along with more specific recommendations for addressing the different target audiences and on how to present the uncertainty. These documents are more general in nature and do not provide detailed guidance on the different expressions of uncertainty that may result from the application of EFSA's Uncertainty Analysis GD (see next section).

2. Expressions of uncertainty

Messages about uncertainty need to be based on information that is provided in EFSA's scientific assessments. Therefore, this section describes the main types of expressions of uncertainty that are expected to result from uncertainty analyses following EFSA's Uncertainty Analysis GD. Section 3 then provides guidance on communicating each type of uncertainty expression, based on the different sources of evidence that contributed to the guidance and additional reasoning that is summarised in Appendix B.

The Uncertainty Analysis GD contains several options for carrying out an uncertainty analysis. Table 2 lists the types of expressions of uncertainty that are normally produced when uncertainty analysis is conducted by following those options. It does not include expressions that are discouraged by the Uncertainty Analysis GD, e.g. a range or bound for a quantity of interest that is presented without a specified probability and therefore lacks a defined meaning.. For conciseness, Table 2 and Sections 3 and 4 also do not include other expressions that may be generated by some more specialised methods included in the Uncertainty Analysis GD, such as probability boxes (Ferson et al., 2004) and the outputs of some methods for sensitivity analysis (Annex B17 of EFSA Scientific Committee, 2018b).

Table 2.

Types of expressions of uncertainty produced by uncertainty analysis when following the Uncertainty Analysis GD (EFSA Scientific Committee, 2018a) and included in the conclusion, summary or abstract of a scientific assessment. The same assessment may produce one or more of these expressions

Type of uncertainty expression	Description
Unqualified conclusion, with no expression of uncertainty	This occurs in two situations: When a standardised assessment procedure only takes into account standard uncertainties, its outcome may be reported as an unqualified conclusion, without any expression of uncertainty (see EFSA Scientific Committee, 2018a,b for more explanation) When uncertainty is present in an assessment, but decision‐makers or legislation requires an unqualified conclusion (e.g. safe, not safe or ‘cannot conclude’), without any expression of uncertainty. In some of these cases, uncertainty expressions may be included elsewhere in the assessment report, e.g. in the detailed results or in an annex
Description of a source of uncertainty	Verbal description of a source or cause of uncertainty. In some areas of EFSA's work, there are standard terminologies for describing some types of uncertainties, but often descriptions are specific to the assessment in hand (EFSA Scientific Committee, 2018b)
Qualitative description of the direction and/or magnitude of uncertainty using words or symbols	Words or an ordinal scale describing how much a source of uncertainty affects the assessment or its conclusion (e.g. low, medium or high uncertainty; conservative, very conservative or non‐conservative; unlikely, likely or very likely; or symbols indicating the direction and magnitude of uncertainty: —, –, ‐, +, ++, +++) Because the meaning of such expressions is ambiguous, EFSA's Uncertainty Analysis GD recommends that they should not be used unless they are accompanied by a quantitative definition (EFSA Scientific Committee, 2018a)
Inconclusive assessment	This occurs in two situations: When decision‐makers or legislation require an unqualified conclusion but assessors judge there is too much uncertainty to give one and report that they cannot conclude. The basis for this uncertainty expression should be documented in the body of the assessment report or an annex, and may include one or more uncertainty expressions When it is not required that conclusions must be unqualified, but the assessors are unable to give any quantitative expression of uncertainty or, where they judge that their probability for a conclusion could be anywhere between 0% and 100%. This should be accompanied by a qualitative description of the uncertainties (see description of ‘A precise probability’)
A precise probability	A single number (in EFSA outputs: a percentage between 0% and 100%) quantifying the likelihood of either: A specified answer to a question (e.g. a ‘yes’ answer to a ‘yes/no’ question) A specified quantity lying in a specified range of values, or above or below a specified value (e.g. 90% probability that between 10 and 100 infected organisms will enter the EU in 2019; 5% probability that more than 100 infected organisms will enter) Note that the term ‘precise’ is used here to refer to how the probability is expressed, as a single number, and does not imply that it is actually known with absolute precision, which is not possible
An approximate probability	Any range of probabilities (e.g. 10–20% probability) providing an approximate quantification of likelihood for either: A specified answer to a question (e.g. a ‘yes’ answer to a ‘yes/no’ question) A specified quantity lying in a specified range of values, or above or below a specified value (e.g. 1–10% probability that more than 100 infected organisms will enter the EU in 2019) The probability ranges used in EFSA's approximate probability scale (see Table 4) are examples of approximate probability expressions. Assessors are not restricted to the ranges in the approximate probability scale and should use whatever ranges best reflect their judgement of the uncertainty (EFSA Scientific Committee, 2018a)
A probability distribution	A graph showing probabilities for different values of an uncertain quantity that has a single true value (e.g. the average exposure for a population). The graph can be plotted in various formats, most commonly a probability density function (PDF), cumulative distribution function (CDF) or complementary cumulative distribution function (CCDF) (see Section 4.1.4.2)
A two‐dimensional probability distribution	In this guidance, the term ‘two‐dimensional (or 2D) probability distribution’ refers to a distribution that quantifies the uncertainty of a quantity that is variable, i.e. takes multiple true values (e.g. the exposure of different individuals in a population). This is most often plotted as a CDF or CCDF representing the median estimate of the variability, with confidence or probability intervals quantifying the uncertainty around the CDF or CCDF

It is currently expected that, in many assessments, the conclusion, summary and abstract of the assessment will not contain expressions of uncertainty. This may arise in two types of situation, as indicated in the description for ‘unqualified conclusions’ in Table 2.

The first type of situation arises in standard assessment procedures, which are most commonly used in the assessment of regulated products. A standard procedure is an assessment methodology prepared for routine use in a specified type of assessment (e.g. acute or chronic risk for a specified class of chemicals). Standard assessment procedures include standard provisions (e.g. uncertainty factors) for addressing ‘standard uncertainties’, i.e. uncertainties that routinely occur in that type of assessment, and their conclusions are expressed in a standard form (e.g. ‘no health concern’, ‘health concern’, etc.) (EFSA Scientific Committee, 2018a,b). Provided only the standard uncertainties are present, the standard conclusion can be reported and communicated without qualification.

The second situation relates to other contexts in which legislation or decision‐makers require an unqualified conclusion (EFSA Scientific Committee, 2018a,b). This may apply to standard procedures in which non‐standard uncertainties are present, and to some types of non‐standard (case‐specific) assessments. In both cases, the assessment conclusion will have been based on an analysis of the uncertainties that are present but will be reported in an unqualified manner, which should then also be the primary message in communication. If the uncertainties are reported in the body of the assessment or an annex, then they may also be referred to in supporting parts of the communication but should not be included in the primary message.

When non‐standard uncertainties are present, the Uncertainty Analysis GD recommends that assessors quantify the combined impact of as many of the uncertainties they identify as possible, for reasons explained in detail by EFSA Scientific Committee (2018b). However, in some assessments, the assessors will be unable to include all the identified uncertainties in their quantitative expression of overall uncertainty, which will then be accompanied by a qualitative description of the unquantified uncertainties.

The Uncertainty Analysis GD contains the form of uncertainty expression ‘probability bound’. Note that a probability bound includes either a ‘precise probability’ or an ‘approximate probability’, and should be communicated accordingly. Precise and approximate probabilities are addressed separately in the present guidance because they have different implications for communication.

Note also that the Uncertainty Analysis GD quantifies uncertainty using probability expressed as a percentage (0–100%), which leads to potential for confusion when the uncertainty refers to a quantity that is itself a percentage, e.g. a 10% probability that 10% of people have exposures above a reference dose. This and other potential sources of misunderstanding are addressed by the general guidance in Section 3.1.

3. Guidance on communicating uncertainty

This section contains practical guidance for communicators and some guidance for assessors on providing information in their assessments that is needed for communication. It gives straightforward instructions, practical tips and examples, as well as further choices to consider. It is presented as general guidance and specific guidance: the general guidance applies universally to all communication of uncertainty in EFSA scientific assessments; the specific guidance is structured to address and communicate the different expressions of uncertainty described in Section 2.

The guidance is based, as far as possible, on the scientific literature and author recommendations described in Section 4. The available evidence does not address every aspect of communicating uncertainty, however, so some general and specific guidance is based on judgement and reasoning. This guidance was formulated following the testing and interpretation of concrete EFSA examples, which are listed in Section 4.4. For transparency, the literature source and/or detailed reasoning underpinning each point of guidance is summarised in Appendix B.

This guidance complements the EFSA Risk Communication Handbook (EFSA, 2017), which details general best practice in science communication: e.g. use plain language as far as possible, explain technical terms when they are unavoidable, and provide links to more detailed information (e.g. factsheets, videos, FAQs or EFSA's scientific outputs) for interested readers.

3.1. General guidance for communicators

Follow all the general guidance below for communicating uncertainty to the ‘entry’ and ‘informed’ level audiences. Apply it together with the specific guidance related to the different uncertainty expressions (described in Section 2). Complete the template in Table 3 to find the specific guidance, contained in separate boxes (Section 3.3), that is relevant for your communication.

Table 3.

Template for identifying messages with associated uncertainty expressions, and specific guidance for their communication

3.1.1. Alignment with the assessment

1. Be consistent with the degree of certainty or uncertainty the assessors give to their scientific conclusions in all parts of your communication, including entry‐ and informed‐level material and any accompanying titles or headlines. Avoid using any forms of expression that would imply more or less certainty than expressed in the assessment.

2. State clearly what the message and uncertainty information refer to, e.g. a specific event, outcome or quantity, and the population, geographic region and time period for which it has been assessed. Indicate if the outcome has any particular importance (e.g. exposure exceeding a reference dose such as a tolerable intake).

3. State clearly if the conclusion described in your message or the assessment of its uncertainty are subject to any conditions or assumptions (e.g. ‘this assessment assumes consumers comply fully with the proposed dietary advice, and would change if compliance was partial’).

3.1.2. Describing uncertainty with words

1. If numbers are provided in the scientific output to express uncertainty, do not replace them with words – follow the guidance in the following sections.

2. If the scientific output includes words that describe a degree of uncertainty or probability (e.g. low uncertainty, unlikely, high probability), use exactly the same wording in your communications. Only alter the wording (e.g. if you consider it too technical or unclear) after checking the rewording with the assessors to ensure it still conveys the intended message.

3.1.3. Describing uncertainty with numbers

1. When assessors quantify uncertainty using probability, always communicate it numerically and do not convert it into words (e.g. likely, unlikely).

2. Refer to uncertainty as percentage certainty (e.g. ‘the experts considered it 90% certain that…’) rather than percentage probability, as this helps to make clear that it is an expert judgement and not a measure of frequency. The word ‘certainty’ is preferred to confidence because the latter has different connotations, including a special technical meaning when used in the term ‘confidence interval’.

3. At the entry level, frame the message positively as % certainty for the outcome, conclusion or range of values that EFSA considers more likely. This is important because expressing, e.g., 5% probability as 5% certainty is misleading, since there is then a 95% probability that the outcome will not occur. This is better expressed as 95% certainty of non‐occurrence. At the informed level, repeat the certainty statement from the entry level and explain clearly the main reasons why the outcome might occur. Then, give the main reasons why the outcome might not occur.

4. If the assessors provide a verbal expression as well as probability (e.g. when using the ‘approximate probability scale’ (APS), see Table 4), always communicate the probability first (expressed as % certainty) and the words second. For example, ‘the experts considered it 66–90% certain (likely) that…’ should be preferred to ‘the experts considered it likely (66–90% certain) that…’. This is because, in English, people interpret numeric and verbal probabilities more consistently in this order. This may be different in other languages where word order is more flexible or follows other conventions.

5. When communicating a range for an uncertain quantity, always accompany it with an expression of probability that the quantity lies in this range, e.g. ‘exposure was estimated to be between 5 and 20 mg/kg bw per day, with 95% certainty’. Without this, the meaning of the range is ambiguous. Also consider indicating which values within the range are more likely, e.g. by providing a central estimate (see Point 6), to counter the tendency for people to focus excessively on the upper or lower bound.

6. Do not use the expression ‘best estimate’, rather use ‘central estimate’ and make clear at the informed level which type of central estimate it refers to, e.g. a mean or median estimate or the most likely value (mode). The meaning of ‘best’ is ambiguous and might lead people to focus excessively on that value.

7. Communicate the quantities that are part of the message – do not require the audience to infer them. For example, if you have to communicate a range, communicate the upper and lower bounds explicitly and do not require the audience to derive them by looking at a graph.

8. Link, when relevant, to general FAQs for explanation of all commonly used ways of communicating uncertainty. They should explain probabilities, approximate probabilities and the terms in EFSA's APS (e.g. ‘likely’) and also any visualisations that might be used in communication (e.g. box plots, see Box 8 below).

Table 4.

Approximate probability scale recommended for harmonised use in EFSA to express uncertainty about questions or quantities of interest

Probability term	Subjective probability range	Additional options
Almost certain	99–100%	More likely than not: > 50%	Unable to give any probability: range is 0–100% Report as ‘inconclusive’, ‘cannot conclude’, or ‘unknown’
Extremely likely	95–99%
Very likely	90–95%
Likely	66–90%
About as likely as not	33–66%
Unlikely	10–33%
Very unlikely	5–10%
Extremely unlikely	1–5%
Almost impossible	0–1%

This table was adapted from a similar scale used by the Intergovernmental Panel on Climate Change. Additional details and guidance on use can be found in EFSA Scientific Committee (2018b).

3.1.4. Precautions when using numbers

1. Avoid using ‘hedging words such as ‘about’, ‘approximately’, ‘up to’, etc. to qualify numerical expressions. People interpret the meaning of these words differently.
Warning: if you choose to use hedging words anyway – e.g. saying ‘about 80% certain’ in spoken communication if ‘66–90% certain’ seems less natural – be aware that people interpret the implied range around 80% differently. This means that at least some of your audience will understand incorrectly what the assessors intended.

2. Try to word the communication so that it makes clear that expressions of % certainty are the consensus judgement of the experts involved for the probability of a conclusion being correct, and do not refer to the percentage of experts who support the conclusion.

3. Avoid using percentages for the outcome of interest (e.g. a proportion of something, or the incidence or risk of an outcome or effect), to avoid confusion with percentages quantifying uncertainty. Where possible, use frequencies (e.g. 1 in 20) to express incidence. Example: ‘The Panel was 90% certain that at most 3 in 10,000 sheep would be affected by the virus.’

4. People are generally more familiar with the use of probability for expressing variability, frequency, incidence or risk. Therefore, if you use probability to express uncertainty, use wordings that avoid people misinterpreting it. For example, if communicating EFSA's percentage probability that a hazard exists, either use % certainty as recommended in Section 3.1.3, or explain clearly that the probability refers to uncertainty about whether the hazard exists, and not to the percentage of people who will be affected.

5. Ranges are often used to represent variability, e.g. the range of intakes in a population. Therefore, state clearly when ranges represent uncertainty. Also, people are more familiar with the use of graphs (e.g. box plots or graphs) to represent variability, so explain clearly when they represent uncertainty, or both variability and uncertainty (see Boxes 8 and 9 respectively).

3.1.5. Describing sources of uncertainty

1. Provide information on sources of uncertainty at the informed level, when this is included in the assessment report, as it helps recipients to understand why uncertainty is present (see Box 2 for guidance on this).

2. Make clear that all uncertainties referred to in the communication have been taken into account in the overall conclusion of the assessment, to avoid any impression that the conclusion is undermined by the uncertainties. For example, if the conclusion is ‘no safety concern’, then it must be made clear that this is the final conclusion of the assessors after considering the uncertainties.

3.1.6. Addressing the uncertainties

1. Communicate information about options for addressing uncertainty, especially if the uncertainty is substantial or might cause concern.

2. If the assessment evaluates any risk management options for dealing with uncertainty (e.g. precautionary action), present these as options. Do not imply any preference between options or recommend particular options as in most cases this would involve risk management considerations (e.g. affordability, feasibility, proportionality), which are outside EFSA's remit.

3. If the assessment specifies any options or requirements for further data or analysis aimed at reducing uncertainty, communicate these as follows:

   • – At the entry level, mention that the assessors identified aspects in which further data are needed; listing the most important of these is optional.
   • – At the informed level, briefly list the options or requirements and differentiate formal requirements for applicants (i.e. linked to market authorisations) from other non‐statutory options outside EFSA's explicit remit, e.g. proposals for future research.

3.2. General guidance for assessors

Follow the guidance below for expressing uncertainties in scientific assessments. Apply it together with the specific guidance related to each of the eight uncertainty expressions as listed in Section 3.3 Specific guidance.

1. Assessors should make clear whether scientific conclusions relate to real world conditions and outcomes or to specific conditions and/or assumptions. When the conclusion is based on the result of a model or statistical analysis, remember to consider uncertainties not quantified within the model or analysis, including uncertainties about the assumptions of the model or analysis and any extrapolation from it to the real quantity or question of interest. This should be carried out as part of characterising overall uncertainty, which is one of the steps in the Uncertainty Analysis GD.

2. Do not express more precision than is justified by the scientific assessment. In particular, do not express quantities or probabilities with more significant figures than is justified – usually one or two.

3. When using probability to express uncertainty, always express it numerically. If you also provide a verbal expression (e.g. when using the APS), then report the numeric expression first and the verbal expression second, as this has been shown to improve the consistency of interpretation by recipients. If you cannot provide a numeric expression of probability, state explicitly that the probability cannot be assessed.

4. When estimating the probability, frequency or incidence of outcomes, express them as frequencies (e.g. 10 per 1,000), as this makes it easier for people to understand and use them. This is especially true if probabilities are conditional and the audience needs to infer unconditional probabilities for outcomes of interest to them (e.g. infer the probability they have a disease from three different pieces of information: the conditional probability for having the disease given a positive test, the accuracy of the test, and the base rate for the disease in the population), as it has been shown that people can do this more reliably using frequencies. Using frequencies to quantify variability also reduces the risk of confusion that can arise if probability is used to quantify both variability and uncertainty

5. When reporting a range for a quantitative estimate, always accompany it with a precise or approximate probability for that range. Also indicate which values within the range are more likely if this might be important for understanding and decision‐making. Presenting a central estimate as well as the range will indicate whether the distribution is skewed to one side. If more detail on the shape of the distribution is needed, consider including quartiles or a box plot (see Box 8).

6. When giving an approximate probability, always provide a lower and upper bound (e.g. 66–90%). When the probability refers to a range of values of a quantity of interest, give both a lower and upper bound for the range when both are available from the assessment (e.g. a confidence interval from a statistical analysis). When using assessment methods that produce one‐sided bounds, e.g. when using probability bounds analysis, consider providing also a probability for a second value of the quantity, to reduce the tendency for people to anchor if only one value is mentioned. For example, if presenting a probability for exposure exceeding a safe dose, consider also assessing and presenting the probability of exceeding a specified multiple of the safe dose (e.g. 2× or 5×). Similarly, if presenting a probability for any (> 1) infected animals entering the EU, consider also assessing the probability of the true value being greater than a higher number of interest.

7. A clear and concise summary of the outcome of the uncertainty analysis should be included in the abstract of the scientific assessment where it is easily accessible to communicators. This should include the overall impact of uncertainty on the conclusions and the major sources of uncertainty. More detailed information on the uncertainty analysis should be provided in the summary and main text of the assessment report.

3.3. Specific guidance

This section contains specific guidance for communicating different uncertainty expressions as listed in Table 2 to different audiences. Communicators should use the specific guidance for the entry and informed levels together with the general guidance in Section 3.1. The specific guidance at the technical level provided in Boxes 1–9 refers to tasks for assessors to support the communication process, and should be applied in conjunction with the general guidance in Section 3.2 of this document and in the Uncertainty Analysis GD. Assessors should use this guidance to inform how information is reported in their assessments.

Communicators should take the following steps for each communication:

1. Examine the scientific output to identify proposed messages and supporting points for communication according to the EFSA Risk Communication Handbook.

2. Consult the appropriate assessor (scientific officer, head of unit, working group chair, etc.) to confirm the selection and validate the accuracy of messages in line with the standard operating procedure (EFSA, 2015).

3. For messages that refer to scientific conclusions, either ask the assessor to fill in the template below (Table 3) or complete it yourself with their input. Answer all the questions in the template for each of these messages.

4. The completed template tells you which types of uncertainty expression are associated with each message. It also directs you to the specific guidance for communicating each type of expression (Boxes 1–9). One or more types of uncertainty expression might be used in a single message (e.g. description of some sources of uncertainty and an approximate probability for the conclusion). Always consider them together.

5. Craft entry‐level and informed‐level communications for each message, applying both the general guidance in Section 3.1 and the relevant specific guidance from Boxes 1–9.

6. Integrate the crafted communications into a coherent narrative in the chosen format (e.g. news story). Material for the entry and informed levels may be used in a single communication, e.g. top‐line messages followed by more details lower down. If published separately, link the entry level to more detailed material (e.g. FAQs, the scientific output).

Box 1: Guidance for communicating assessments using standardised procedures

Box 2: Guidance for communicating a description of sources of uncertainty

Box 3: Guidance for communicating qualitative descriptions of the direction and/or degree of uncertainty using words or symbols^(a)

Box 4: Guidance for communicating inconclusive assessments

Box 5: Guidance for communicating unqualified conclusions with no expression of uncertainty

Box 6: Guidance for communicating a precise probability

Box 7: Guidance for communicating an approximate probability

Box 8: Guidance for communicating a probability distribution

Box 9: Guidance for communicating a two‐dimensional probability distribution

4. Sources of evidence for the guidance

This section describes the key sources of evidence on which the guidance in Section 3 is based. These evidence sources include an expert analysis of selected academic literature including direct author recommendations, quotes and interpretation (Section 4.1), the results of research commissioned or carried out by EFSA to inform the development of this Uncertainty Communication GD (Section 4.2) and extracts from frameworks or guidance documents similar in scope and purpose to this Uncertainty Communication GD from other national and international advisory bodies (Section 4.3). It also includes a description of the concrete EFSA examples, which were used to formulate additional guidance with expert judgement and reasoning (Section 4.).

For transparency, Appendix B provides a comprehensive overview of each literature source and/or the reasoning underpinning each point of guidance described in Section 3.

This section concludes that the available evidence does not address every aspect of communicating uncertainty and therefore feeds directly into the ‘Further research needs’ described in Section 5.

4.1. Scientific literature

An expert analysis of key papers on communicating uncertainty follows below. The criteria used to select the key papers are described first.

4.1.1. Scope of literature search

An initial list of 37 references was drafted based on:

• working group members’ knowledge of the literature;

• references included in a major recent review paper on risk and uncertainty communication, (Spiegelhalter, 2017);

• references already available at EFSA from different sources.

This list was used for a literature search to identify current publications that reported primary research on communicating uncertainties. However, owing to the ubiquitous appearance of the search terms ‘communication’ and ‘uncertainty’ in scientific literature, the usual search strategies were not specific enough to narrow the search to a feasible number.

A list of criteria for including papers in the literature review was drafted collectively by the working group. Studies were included that simultaneously:

• provide guidance on how uncertainty should be communicated (best practice),

• include experimental data, but also literature reviews on uncertainty communication,

• provide recommendations for uncertainty communication,

• cover at least one of the possible expressions of uncertainty from the Uncertainty Analysis GD.

The relevance of the subject area was also considered, with studies on food safety/public health/environment being preferred.

Studies essentially arguing about the importance of communicating uncertainty were excluded, as they were considered to be of low relevance for the practical needs of the Communication GD.

Application of these criteria by the working group and resulted in 25 papers being considered in the first round of the literature study, selected from the 37 papers proposed initially. These papers were read in full and further relevant references were identified from them. This list of 25 papers was then crosschecked with a review paper by van der Bles (2018), which is currently in preparation. This work has also provided inspiration for structuring our literature review and recommendations. Finally, additional references have been suggested during the public consultation. Finally, 57 papers have been included in the literature review.

4.1.2. Relevance of the scientific literature included

This review does not include a large set of risk communication studies because they examined how best to communicate risks (e.g. the risk of having cancer given a positive test result (Gigerenzer and Hoffrage, 1995)) and are less relevant to the communication of uncertainty. Although both risk and uncertainty are often expressed in terms of probability, probabilities for risk either quantify the frequency of an adverse outcome in a population, or are frequentist estimates of the probability of the outcome for an individual member of the population. Probabilities for uncertainty quantify the likelihood of single events, for example the likelihood that the true frequency of the adverse outcome is within a specified interval. EFSA's Uncertainty Analysis GD uses subjective probabilities for this purpose. There is considerable variation among the studies included in this review as to how they dealt with uncertainty and it is often not explicitly stated whether the authors focused on how to communicate frequentist or Bayesian expressions of uncertainty. There seemed to be a lack of research that focused on the communication of subjective probabilities, which is the type of uncertainty expression used in assessments following EFSA's Uncertainty Analysis GD.

However, the distinction between relative and absolute risks is important for both risk and uncertainty communication (Gigerenzer et al., 2007). Although relative risk information is widely used for communicating risks, it is not an effective strategy when the target audience consists of laypeople (Siegrist and Hartmann, 2017). The presentation of relative risk information results in larger and more persuasive numbers than when the same information is expressed in absolute terms. This can be easily demonstrated using a simple example: a given treatment reduces the risk of an adverse outcome from 8% to 4%, consequently decreasing the risk in relative terms by 50% but by only 4% when expressed in absolute terms. Research suggests that relative risk information increases the likelihood of people misinterpreting it and displaying a biased reaction (Malenka et al., 1993). Providing both, relative and absolute risk information is not advisable, because experiments suggest that the relative risk reduction information influences people's preferences even when absolute risk reduction information is also presented (Gyrd‐Hansen et al., 2003). These results imply that for the communication of risks and uncertainties, absolute rather than relative numbers should always be used.

There is a body of literature which recommends communicating with frequencies rather than probabilities (e.g. Gigerenzer and Hoffrage, 1995; Gigerenzer et al., 2007). However, many of the studies supporting this recommendation involved tasks which required subjects to make inferences involving combination of conditional probabilities and base rates. EFSA assessments and communications generally provide direct estimates of the outcomes of interest to the audience, and hence do not require them to deduce these from conditional probabilities. Furthermore, applying EFSA's guidance on uncertainty analysis results not in probabilities representing frequency but in subjective probabilities expressing uncertainty about single outcomes or events, and it is unclear whether these would be better understood if they were expressed using frequencies. A study by Joslyn and Nichols (2009) in which the tasks did not require conditional inferences found that subjects understood weather forecast information better when expressed in a probability format rather than a frequency format. Furthermore, expressing single‐event probabilities in terms of frequency would require framing each assessment in terms of a hypothetical population or reference class of similar cases (e.g. other chemical risks with similar evidence and risk to the one under assessment), which increases the complexity of the communication and therefore poses different challenges to understanding.

Most of the reviewed studies presented uncertain information to subjects and then asked them to express their response to the information. The form of response that was requested varied between studies, but in nearly every study it was explicit or implicit that the requested response referred to the subject's own interpretation or perception after receiving the information, and/or what actions they might take or wish to be taken in response to it. Only one study (Ibrekk and Morgan, 1987) was identified in which the participants were explicitly asked ‘what the forecast says’. In some further studies (e.g. Edwards et al., 2012), it was ambiguous whether the subjects were being asked for their understanding (comprehension) of the communication or for their own opinion, having seen the communication. The former is more relevant for this guidance as its purpose is to ensure that EFSA's communications are understood (see Section 1.6.1). This varying relevance has been taken into account when using the findings from the literature (reviewed below) to inform this Guidance (Section 3).

Whereas many of the selected studies investigated the communication of uncertainty expressed as a range for a quantity of interest, in only few of them were the ranges accompanied by confidence levels or probabilities.

The subjects considered in the reviewed studies were mostly selected to represent the common ‘layperson’, and in many cases reflected convenience choices for the experimental settings. Most studies used convenience samples: students form the biggest pool of subjects, followed by people recruited in shopping malls, parks, at cinema exits, via advertisements on university networks or campuses, via the Decision Research web panel subject pool, and/or via Amazon Mechanical Turk.

Only four studies (Patt and Dessai, 2005; Wardekker et al., 2008; Pappenberger et al., 2013; Beck et al., 2016) experimentally dealt with the clearly identifiable target audiences with whom EFSA interacts on a regular basis, namely decision‐makers, policy advisers and stakeholder representative groups such as NGOs, consumer groups or industry associations.

4.1.3. General findings

In a study of climate change, Visschers (2018) found that respondents differentiated between different types of uncertainty: ambiguity in research, measurement uncertainty and uncertainty about future impacts. Similarly, Wardekker et al. (2008) reported a preference among experts and users of uncertainty information at the science–policy interface for communication of different types and sources of uncertainty. Together, these results indicate that communicating different sources of uncertainty together with communication of aggregated uncertainty might be welcomed.

4.1.4. Scientific literature contributing to the specific guidance

The scientific literature that contributes to the specific guidance in Section 3.3 are sorted by the expressions of uncertainty that may be produced when following the EFSA Uncertainty Analysis GD.

4.1.4.1. Ranges

Ranges can refer to a quantity of interest or to a probability used to express uncertainty (Section 2). Most of the literature reviewed studied ranges for a quantity of interest (e.g. the 95th percentile of a population), sometimes accompanied by a probability for the range. Some papers studied communication of ranges for probabilities describing uncertainty on the occurrence of an event. The parameters examined in the studies were diverse and are reviewed below in six categories:

• influence on risk perception,3

• effectiveness of communication leading to accurate understanding of the uncertainty information,

• impact on decision‐making (of forms of uncertainty communication, use of uncertainty information in taking decisions, ambiguity aversion,4 ability to take a decision),

• effects on emotions (e.g. worry),

• role of source credibility on risk perception,

• focus on the end values of the range.

Although not all of these effects relate to the specific needs of this Uncertainty Communication GD they may be valuable for deriving further recommendations for uncertainty communication.

The influence of uncertainty communication on risk perceptions is by far the most frequent parameter investigated, followed by effectiveness.

Influence on risk perception

Kuhn (2000), Dieckmann et al. (2010) and Han et al. (2011) found no significant difference in the magnitude of risk perception associated with communication of a probability range (i.e. a probability with associated uncertainty expressed by giving a range: x% to y%) as compared with communicating a precise probability (i.e. a point estimate for the probability without expressing associated uncertainty: x%). Han et al. (2011) provided both uncertainty information and comparative risk information. They found that uncertainty information moderated the increase in risk perception that was observed when comparative risk information was provided in addition to individual risk information (i.e. the individual risk compared to the average risk in a population). For the hypothetical situation of a terrorist attack, tested by Dieckmann et al. (2010), there was no difference in perceived harm with or without uncertainty information.

In the experiments of Dieckmann et al. (2010), uncertainty information was presented as scenarios that included probabilities and narrative information, the latter adding context and an explanatory story in addition to describing the logic and evidence used to generate the assessment. Risk perception increased when the probability increased, and also when the narrative was provided. The effect of increased probability was the same, whether it was presented as a point probability or a range, and whether or not it was accompanied by narrative information or not.

Johnson and Slovic (1995) reported an increase in risk perception when uncertainty was communicated as probability ranges rather than as compared with point estimates. More generally, low risk estimates were deemed more ‘preliminary’, whether or not uncertainty was communicated. In Kuhn (2000), when no mention was made of uncertainty, or when the ends of the probability range were explained as the conclusions of two different sources with opposing biases, peoples’ environmental attitudes predicted risk perception. However, environmental attitudes predicted risk perception less well when the probability range was centred on the best estimate. The average level of risk perception did not differ between point estimates and uncertainty communication. Viscusi et al. (1991) emphasised that the number of people who are influenced by the most adverse or the most favourable uncertainty estimates is quite constant.

Overall, although influence on risk perception is relevant to risk management, the main consideration for this Communication GD is the impact of uncertainty information on accurate understanding.

Effectiveness in terms of accurate understanding of uncertainty information

Dieckmann et al. (2015) presented subjects with ranges for various quantities, but no information on the underlying distribution. Subjects were then asked to choose between three options for the underlying distribution: uniform, central values more likely or extremes more likely. Distributional perceptions were similar whether the range referred to a quantity (e.g. temperature) or to the probability of an event occurring. Without clues about the form of the distribution, people tended, in relatively balanced proportions, either to consider all the values in a range as being equally probable (uniform distribution) or to perceive the central values as more likely (particularly the more numerate). A small percentage perceived a U‐shaped distribution. Providing a best estimate and a confidence level with the range reduced the perceptions of both uniform and U‐shaped distributions. Further studies by the same team (Dieckmann et al., 2017) confirmed that perceptions of the distribution underlying numerical ranges were affected by the motivations and worldviews of the recipients. This ‘motivated reasoning effect’ remained after controlling for numeracy and fluid intelligence, but was attenuated (in one instance described as ‘eliminated’) when the correct interpretation of the distribution was made clear through the use of a graph. In other words, people interpret the uncertainty information in a motivated way only if they are given the opportunity, and they do not do so when the correct information about the distribution is made clear.

Impact on decision‐making

Participants in the experiments of Dieckmann et al. (2010) thought that at higher levels of probability the probability range was more useful for decision‐making than the point probability, whereas at a lower probability the point estimate was more useful. However, the magnitude of the difference was not provided and the authors stated that the reasons for it were unclear.

Kuhn (1997) showed that for the low to high range condition (e.g. 10–30%), vague options are more likely to be preferred when decision problems are framed negatively (i.e. losses) than when they are framed positively (i.e. gains). Morton et al. (2011) showed that communication of a probability range combined with a positive framing (possibility of losses not materialising) increased individual intentions to behave environmentally‐friendly for limiting climate change effects.

The impacts of probability bounds for which either the lower or the upper bound were specified have been studied by Hohle and Teigen (2018). They compared how research participants perceived climate‐related forecasts communicated with lower‐bound (‘more than x% chance’) versus upper‐bound (‘less than x% chance’) probability ranges. They found that ‘more than’ statements guide listeners’ attention towards the possible occurrence of the event, while the contrary was found for the ‘less than’ statements, focusing participants’ attention towards the non‐occurrence of the event, accompanied by negative explanations of why that event might not appear. Furthermore, most people thought of estimates for the probability of the event rather close to the boundary, even if the range allowed by that boundary was very large (e.g. 30–100% for ‘over 30%’). A second finding was that boundaries were perceived as indicating the forecaster's belief about either an increasing (‘more than’) or a decreasing (‘less than’) trend over time (Teigen, 2008; Hohle and Teigen, 2018). Similarly, a ‘more–less’ asymmetry has been found by Hoorens and Bruckmüller (2015), who showed that ‘more than’ statements seem to be easier to process cognitively than ‘less than’ statements. For this reason, people use ‘more than’ statements more frequently, agree with them more, more readily believe them and like them better.

Consequently, probability bounds could be used strategically to suggest specific attitudes to take regarding the message communicated (e.g. purchasing a good) (Teigen et al., 2007; Teigen, 2008). The effects of communicating probability bounds also differ if they are inclusive (‘minimum’, ‘at least’/‘maximum’, ‘at most’) or exclusive (‘more than’, ‘above’/‘less than’, ‘below’): while the inclusive ones suggest possible values, the exclusive ones suggest implausible outcomes and can allow a greater range of values (e.g. ‘more than 50 victims’ can describe a higher number of victims than ‘at least 50’). Lower bounds indicate that the target value is higher than could have been expected, and higher bounds indicate the opposite (Teigen et al., 2007). However, no study compared the effect of replacing the use of one‐sided intervals (e.g. more than 30%) with full ranges (30–100%).

Løhre and Teigen (2017) comparatively assessed people's understanding of either a range of the outcome of interest, or a probability reflecting experts’ confidence in that range. They found two distinct ways of reflecting about confidence intervals, namely distributional (wide intervals are seen as more probable than narrow interval) and associative (wide intervals are seen as more uncertain than narrow intervals). However, their experiments did not include communication of both the range and the associated confidence. No study could be identified as having tested the understanding of the simultaneous communication of both ranges of outcomes expressed using probabilities and an associated probability expressing expert confidence (as recommended in this guidance, see Box 9).

Using examples on weather forecasts for their experiments, Savelli and Joslyn (2013) showed that people provided with uncertainty information, expressed as a range of temperatures with a specified probability, were more decisive than people provided with deterministic forecasts, i.e., they were more likely to issue a weather warning). The authors suggested that this was because providing a range and probability enabled people to distinguish better between situations in which a target event is likely and unlikely.

Worry

Cancer‐related worry was found to increase (Han et al., 2011) when uncertainty on individualised cancer risk was communicated in the form of probability ranges, compared with point estimates. However, this effect was moderated by the format and people's personality: textual communication (vs. visual) and low optimism (vs. high) were associated with higher worry. In the study by Johnson and Slovic (1998), giving zero as the lower bound created problems, as on average people agreed that zero (instead of one in 10 million) as a lower bound meant that the government could be wrong and that the true risk was higher. Respondents were therefore less likely to worry if the lower bound was one in 10 million than if it was zero. But the effect is relatively small (see Table 4 of Johnson and Slovic, 1998) and of limited practical relevance for recommendations on describing the bounds.

Uncertainty can also be the consequence of experts revising values previously communicated, either upwards or downwards, to take new evidence into account. The literature studying the impact of such changes shows that people tend to consider revised probabilistic forecasts as indicating a trend for the future, with changes upward indicating the same increasing pattern in the future, and changes downwards indicating a decrease (Hohle and Teigen, 2015). This effect can be found despite probabilities being the same after revision (e.g., 30%), for example a change from 20% to 30% for the probability of an event will produce a different effect in people, who will consider the event to be less remote, than if the chance were from 40% to 30% (Maglio and Polman, 2016).

Role of source credibility on risk perception

Han et al. (2011) found no relationship between the credibility of the probability ranges provided in the study and worry or risk perceptions. The results of Johnson and Slovic (1995, 1998), who studied communication by a governmental agency in charge of risk assessments, indicated that provision of uncertainty ranges is deemed by most people to be honest and competent. Most people in the studies of Johnson and Slovic (1998) said they did not expect risk assessment to be certain, or that a single number could describe an environmental health risk.

Dieckmann et al. (2010) found no substantial difference in perceived source credibility between point and range communication related to a forecast situation except when the forecast was evaluated in hindsight. In this latter case, source credibility was higher for the range. Also, the credibility of the narrative evidence about uncertainty was found to be directly correlated with the credibility of the source.

Focus on end values of the range

In all of the studies, having discussed the choice of extreme values among those in a range, a substantial minority of the respondents strongly focused on the extreme values of the ranges. In Dieckmann et al. (2015), 11%, 8.3% and 12.1% (for the three different risks considered) perceived the values towards the ends of the range (i.e. further from the best estimate) as being more likely (U‐shaped distribution). One of their experiments included the additional option ‘None of the above: please specify’ and participants who selected this indicated that it was due to a lack of trust in the data or forecaster. In a subsequent study, Dieckmann et al. (2017) showed that subjects were less likely to say extreme values were more likely when provided with a graphic and narrative indicating a normal‐shaped distribution, although some still did so.

In Johnson and Slovic (1998), when a range of risk values (frequencies) was presented, 22% of the subjects focused on the highest value of the range. In the study by Viscusi et al. (1991), 20% of the sample chose an extreme value from the range presented. The authors concluded that there is quite some prevalence of people who are influenced by the most adverse or the most favourable piece of information presented.

4.1.4.2. Probability distributions

Only two studies (Ibrekk and Morgan, 1987; Edwards et al., 2012) looked experimentally at how people understand probability distributions, a third (Pappenberger et al., 2013) was of low relevance for the current guidance, because of its high specificity to the field of meteorology.

Ibrekk and Morgan (1987) studied nine pictorial displays: a traditional point estimate with an error bar that spans a 95% confidence interval; a conventional PDF; a discretised version of a PDF; a discretised PDF using a pie chart; a PDF and its mirror image plotted above and below a horizontal line to help users focus on the area enclosed between the curves; horizontal bars of constant width shaded to display a PDF using dots and vertical lines; a box plot modified to indicate the mean as a solid point; and a CDF; and the integral of a PDF. Their results indicated that the performance of a display depends on the information that a subject is trying to extract. So a CDF, used alone (without indicating the mean and without explanation), severely misled some subjects in estimating the mean. However, a CDF allowed very good estimations of the probability that x > a, and the probability that b > x > a. The results obtained with engineers were similar to those obtained with a non‐technical audience.

As regards PDFs, when asked to indicate the mean, people rather selected the mode, unless the mean was explicitly marked. The various PDFs tested gave similar results.

This was the sole study to explicitly ask the participants what the assessor was trying to communicate (e.g. about the best estimate) rather than their own understanding of the true value.

Edwards et al. (2012) looked at 10 commonly used graphical display methods: PDF, CDF, CCDF, ‘multiple probability density function’, ‘multiple cumulative probability distribution function’, ‘multiple complementary cumulative probability distribution function’, box plots (box‐and‐whisker plots), standard error bars, ‘multiple complementary cumulative probability distribution function’ and scatterplots. Participants were asked to estimate the mean for each variable and the probability that an observed measurement of each variable was more or less than a given amount. The influence of these methods on decision‐making was studied.

Complementary cumulative probability functions led to the highest probability estimation accuracy, and CDFs and 3‐D PDFs led to poor accuracy, which for CDFs is contrary to the results of Ibrekk and Morgan (1987). However, similar to Ibrekk and Morgan (1987), Edwards et al. (2012) found that background knowledge in statistics or familiarity with a display method did not influence accuracy.

Both studies focused on the participants’ abilities to correctly extract specific information from the graph (e.g. best estimate), but not on which type of graph would be most helpful for people to make informed decisions. Ibrekk and Morgan (1987) did not investigate the use of their graphical formats for decision‐making. Edwards et al. (2012) found that the box plot and the error bar performed best in a simple decision task, which only required the estimation of probabilities and did not require any weighing of probabilities against outcomes. They also asked participants to rate the different formats for usefulness for decision‐making. The box plot, error bar and scatterplot were perceived as more useful for deciding what to undertake than the CDF, ‘multiple cumulative probability distribution function’, ‘multiple complementary cumulative probability distribution function’, ‘multiple probability density function’ or ‘three‐dimensional probability density function’.

4.1.4.3. Expression of expert judgement using ordinal scales

There is a rich and long‐established literature on the use of verbal uncertainty expressions (e.g. ‘likely’) and their associated numerical formulations. The most recent papers looked at the likelihood scale used by the Intergovernmental Panel on Climate Change (IPCC). Table 4 in this guidance is an adaptation of the IPCC scale, recommended for use in EFSA (see EFSA Scientific Committee, 2018b). All of this literature consensually criticises the use of verbal uncertainty expressions alone, which can be responsible for significant misunderstanding of uncertainty messages. Consistency between people's numerical interpretation of IPCC's verbal expressions, and IPCC's conversion table, was found to be low. Making the conversion table available to subjects slightly reduced this inconsistency (Budescu et al., 2009). The variability in the numerical meanings assigned to the probability terms was found to be very high, with significant overlap between the terms (Druzdzel, 1989; Budescu et al., 2009, 2012) and reversal of the order established between the terms (Budescu et al., 2009). Interpretations were regressive, i.e. underestimating high probabilities and overestimating low probabilities (Budescu et al., 2009, 2012, 2014). The range of values associated with each term and the level of agreement with the IPCC guidelines increased when dual verbal and numerical expressions were communicated (Budescu et al., 2009, 2012, 2014). Negatively worded terms (e.g. ‘unlikely’) resulted in greater variability than positively worded terms (e.g. ‘likely’) (Smithson et al., 2012). Furthermore, when the target outcome is positive, positive probabilistic terms (‘probable’, ‘possible’) are rated to be more optimistic and more correct than negative statements (‘not certain’, ‘doubtful), even for identical perceived probabilities of occurrence (Teigen and Brun, 2003). The directionality (positive or negative) of verbal probabilities informs listeners about the speaker's reference point (Honda and Yamagishi, 2017). Positive framings draw listeners’ attention to what might happen, and negative formats invite them to consider that it might not happen after all (Teigen and Brun, 1999; Holleman and Pander Maat, 2009).

Authors repeatedly concluded on the context‐dependency of individual interpretations of these verbal terms (Wallsten et al., 1986; Druzdzel, 1989; Mosteller and Youtz, 1990; Weber and Hilton, 1990; Patt and Schrag, 2003; Patt and Dessai, 2005; Harris and Corner, 2011). Verbal uncertainty expressions embody not only their probabilistic meaning, but also other things such as an expression of the magnitude of the event: language referring to a severe (in terms of consequences) event might be interpreted as indicating either a lower probability event (Patt and Schrag, 2003; Patt and Dessai, 2005) or a higher probability event (Harris and Corner, 2011) than those referring to a more neutral event when both events have the same objective probability. Contextual influences can also be related to the expected frequency/base rate of the event being described (Wallsten et al., 1986) and the emotional valence of the terms being used (unpleasantness, desirability) (Weber and Hilton, 1990). As a matter of principle, qualitative probability terms express both amounts of uncertainty and degrees of confidence in that uncertainty, expectations that uncertainty may change with information, and probably other factors (Wallsten et al., 1986).

However, variability in the understanding of verbal expressions is severely underestimated by those using them, who think these expressions are more precise and more extreme (farther from 0.5) than they really are (for the receivers) (Druzdzel, 1989).

Some of the authors cited above (Patt and Schrag, 2003; Patt and Dessai, 2005; Budescu et al., 2009, 2012; Harris and Corner, 2011; Smithson et al., 2012) recommend at least accompanying verbal uncertainty expressions with their corresponding probabilities at each use (and not solely somewhere in the annexes where a correspondence table is usually included). Budescu et al. (2009) additionally recommend adjusting the width of the numerical range within the generic range (i.e. as specified in IPCC's correspondence table), to match the uncertainty of the target events.

Furthermore, Ho et al. (2015) developed evidence‐based methods (i.e., based on listeners’ understanding and characterisation of verbal probabilities) to go beyond the ad hoc scales developed by different expert committees and to standardise verbal expressions of uncertainty so that their meaning is shared between the parties in a given communication.

Whereas the body of research presented above concerns essentially comparisons between uses of verbal versus mixed (verbal and numerical) probability expressions, recent research has compared the use of mixed formats with numerical communication of probabilities.

Specific research on the words ‘unlikely’ and ‘improbable’ (Teigen et al., 2013) showed that they are most often associated with outcomes that have a 0% probability of occurrence (events that are not going to happen) rather than simply to low frequency events (e.g. an ‘improbable’ exam grade is one that has not yet been observed rather than one that has been obtained by a small percentage of students). Starting from this finding, Jenkins et al. (2018) further looked at four formats: verbal (‘unlikely’), numerical (‘20% chance’), verbal–numerical (‘unlikely [20% chance]‘) and numerical–verbal (‘20% chance [unlikely]‘). They found that including the word ‘unlikely’ (for verbal and verbal–numerical formats) increased the tendency to describe outcomes at the very end of the probability range or beyond it; this tendency was not apparent in the numerical format, but responses were preponderant below the correct answer. The numerical‐verbal format led to responses rather evenly distributed above and below the correct answer. The authors hence labelled an ‘order effect’ in which the first part of the expression is given the most weight in verbal–numerical formats. The ‘extremity effect’5 found for the verbal and verbal–numerical formats of ‘unlikely’ but not for the numerical and numerical–verbal formats indicates that the effect is related to how probabilities are expressed and not to how people understand probabilities in general. These results were found to be particularly relevant for less numerate audiences.

These results are in line with previous research showing that numerical probabilities generate much less variability in people's decisions based on numerical probabilities compared with verbal probabilities, and significantly outperform verbal probabilities regarding the criteria of reliability, internal consistency and construct validity of judgements (Budescu et al., 1988). This conflicts with older research indicating that people process better verbal expressions for predicting future events, compared with numerical variables (Zimmer, 1983).

Research on listeners’ preferences showed that people prefer to receive numerical information but more often use verbal formats for communicating about the probability of an event (Wallsten and Budescu, 1995), a phenomenon labelled ‘communication mode preference (CMP) paradox’ (Erev and Cohen, 1990). Preferences for communication mode are partly explained by the ‘congruity principle’, referring to a desire to achieve congruence between the precision of the communication and the precision of the target event and its underlying uncertainty (Du et al., 2011).

4.1.4.4. Qualitative expressions of uncertainty

Studies testing qualitative expressions of uncertainty are diverse in their objectives and formulations of uncertainty. Several authors looked into accompanying numerical expression of uncertainty with narrative evidence. Dieckmann et al. (2010) found that narrative description of the available evidence had a large effect on people's judgements (in particular, risk perception) when they were associated with probabilistic point estimates of a risk, but this effect was smaller when the narrative was combined with a probability range. Also experimenting on the communication of a verbal qualification of a single point probabilistic estimate (‘estimated to be around x%’), Kuhn (1997) highlighted framing effects: verbal qualification led to greater vagueness aversion with a positive frame, whereas a negative frame reduced vagueness aversion. A best estimate with verbal qualification of associated uncertainty (i.e. due to incomplete knowledge) did not amplify perceived risks for environmentally oriented people (Kuhn, 2000). More generally, the study concluded that aggregate levels of uncertainty in hazard risk evaluation might not be sufficient, and using several approaches to present uncertainty could increase the potential for reaching different audiences.

Similarly, discourse‐based uncertainty expression (entire sentences describing limitations of studies) did not have a significant effect on the parameters studied by Jensen et al. (2017), namely fatalism, backlash and informational overload.

Han et al. (2011) found greater ambiguity aversion for textual explanations related to cancer treatment (e.g. breast screening) than visual representations (bar graphs) of a probability range, when people were generally prone to low dispositional optimism. However, textual description had similar effects on people's worry when visual representations were changed (depicting a confidence interval using a bar graph, and adding blurred edges to reinforce the presence of ambiguity and the idea that probability distributions lack firm boundaries).

Dieckmann et al. (2012) found that evaluative labels (‘high’, ‘low’) associated with numerical probability ranges facilitated the understanding of uncertainty; they were considered easy to use and appeared to be employed even when numerical uncertainty ranges were provided. In that study, participants were more likely to choose the option with the highest confidence (e.g. ‘high’ confidence, corresponding to the lowest uncertainty) when presented with the range and evaluative labels, as opposed to the range alone. They were also more likely to choose the lowest confidence, lowest cost option with the range only. While such evaluative labels may be useful for reflecting the amount of uncertainty, they might be less relevant for expressing the confidence levels. Indeed, it might be difficult to separate likelihood and confidence, as likelihood implicitly contains confidence levels. (A statement about an extremely likely event can hardly be characterised as being of low confidence – if the latter were true, why would the likelihood be high?) (Han, 2013).

Hedging – the use of natural language words (‘about’, ‘around’, ‘almost’, etc.) to express numerical values – is understood in a wide variety of ways by different users, with important inter‐individual variation and overlap for many approximators. Furthermore, its use is modulated by context, including the magnitude of the number, its roundness and even its units (Ferson et al., 2015). However, Jensen (2008) found that hedging was positively associated with perceptions that study limitations were reported clearly.

Communication of uncertainty with plain language necessarily brings in the rhetorical properties of the words used. Even if numbers are used, the phrasing that accompanies these naturally also inserts perspectives (Moxey and Sanford, 2000).

4.1.4.5. Visualisation

Ibrekk and Morgan (1987) found that a box plot or simple error bar performed well if the mean was the primary attribute to be communicated, but less well if one was trying to communicate probability intervals for which CDFs were most effective. The results of Edwards et al. (2012) indicated that error bars and box plots were the most accurate for estimating the mean. Participants rendered more accurate means when using a graphical display method that explicitly provided the necessary information. Error bars with the mean explicitly labelled led to the highest accuracy in mean estimation of all graph types, followed by the box plot, while CCDFs led to the highest accuracy for probability estimation. The box plot, error bar, and scatterplot were perceived by the subjects as more useful for deciding what to undertake than the CDF, ‘multiple complementary cumulative probability distribution function’, ‘multiple probability density function’ and ‘three‐dimensional probability density function’.

While efficient for conveying information about the mean, box plots performed less well for conveying information about the distribution underlying a range. For this kind of distributional information, a simple density graph was better, confirming the idea that visualisation tools have different levels of performance depending on the kind of information to be conveyed (Dieckmann et al., 2015).

Similarly, bar charts with error bars can be misleading, as values inside the bar are seen as likelier than values outside the bar (Correll and Gleicher, 2014).

Only one experimental study produced results about the pedigree chart (NUSAP), which was considered useful by the interviewed experts (Wardekker et al., 2008). NUSAP (Numeral, Unit, Spread, Assessment, Pedigree) is a notational system for displaying different sorts of uncertainty as quantitative information. The pedigree chart allows an evaluative description of the mode of production of that information, and can be represented as a spider/radar chart using traffic lights symbols (e.g. green for good quality, red for low quality) (EFSA Scientific Committee, 2018b).

Visualisation might have an effect on understanding and emotions, but the evidence in the literature is weak. For example, Han et al. (2011) found that people with a personal predisposition to optimism were more tolerant of ambiguity when uncertainty of cancer risk was communicated using bar graphs rather than a textual description. However, enhanced visual representations (depicting a confidence interval using a bar graph, and adding blurred edges to reinforce the presence of ambiguity and the idea that probability distributions lack firm boundaries) had a similar impact to the textual description. In Johnson and Slovic (1995), accompanying numerical information on uncertainty with simple graphics (different values of risk indicated on a vertical line) helped people recognise uncertainty but made the information seem less trustworthy.

Based on a review of various visualisation examples, Spiegelhalter et al. (2011) and Spiegelhalter (2017) concluded that the choice of visualisation depends closely on the objectives of the presenter, the context of the communication and the audience. They formulated a number of general recommendations for visualising uncertainty. Even in areas that are not controversial, the wide variability of audiences, in terms of numeracy and ability to understand graphs, needs to be considered.

According to practitioners (Saltelli et al., 2000, 2008), the most intuitive way to display the results from a sensitivity analysis are scatterplots of the output variable vs each input variable.

In general, there are few studies on the benefits of visualisation specifically for understanding uncertainty information. One such study, by Savelli and Joslyn (2013), provided subjects with ranges of temperatures accompanied by specified probabilities, presented in different formats. When the ranges were depicted graphically, with a separate key indicating the probability, there was a tendency for them to focus on one end of the range. The authors suggest that the ranges were interpreted as representing diurnal variation rather than uncertainty. The same tendency occurred in other formats where the key indicating probability was presented separately. However, this tendency was significantly reduced when the ranges and probabilities were presented together in text form. The authors concluded that their results showed no advantages for visualisations over the text‐only format.

4.1.5. Summary of recommendations from the scientific literature

In this section, we have listed recommendations formulated by the cited authors themselves with additional recommendations that we have drawn from further interpretation of their results. Our general guidance for uncertainty communication and specific guidance for each expression of uncertainty (described in Section 3) are based, as far as possible, on these sources, which are mapped and referenced in Appendix B. Recommendations formulated by the authors are referenced below with their names; and instructions developed by the authors of this GD based on interpretation of the results found in the literature, are referenced using the wording ‘based on…’.

General aspects for uncertainty communication are summarised as follows:

As no single representation suits all audiences, a pluralistic approach to uncertainty communication (multiple formats) can allow different target audiences to use different clues in your message that are most adapted to their ways of understanding (Patt and Dessai, 2005; Spiegelhalter et al., 2011; Spiegelhalter, 2017; also based on Kuhn, 2000).

• Repetition of the same information under different forms is not a problem as long as it is consistent, while misunderstanding of your message can become a problem. You may consider simultaneously using numbers, verbal descriptions and graphs, more or less detailed information in the same document or in different (e.g. background) documents or communication platforms (based on Spiegelhalter et al., 2011; Edwards et al., 2012; Kuhn, 2000; Dieckmann et al., 2017).

• Provide information on uncertainty in full as soon as it is identified. Information about what is being carried out to reduce uncertainty should be included (Frewer et al., 2002).

The main aspects related to specific expressions of uncertainty are summarised in the following subsections.

4.1.5.1. Ranges

• Each probability range should be accompanied by a verbal, numerical or graphical description of the distribution of values behind it (e.g. uniform, normal, etc.). Providing a central estimate (e.g. mean, mode) and a confidence interval is recommended, as well as a description of how to interpret the range (Dieckmann et al., 2015, 2017).

• Whatever range is communicated, a (substantial) minority of people will focus on the values close to the ends of the range for a variety of reasons: lack of clarity in the communication, general lack of trust in the scientific advisory process, or by strategic choice for influencing political decisions.

• In communicating ranges, two options are possible; which option is chosen cannot be influenced by the communicator and will depend on each particular case considered. One option is to communicate the ‘largest’ range of scientific relevance (e.g. P1–P99 or P5–P95), which has the advantage of ensuring transparency about all the available values. The disadvantage of this option is that the range can potentially be large, with the difference between the minimum and maximum values extremely wide. A second option is to communicate a range with end values narrower than the scientifically possible minimum and maximum (e.g. P25–P75). In this case, a minority of people will still focus on values at the ends of the range, but the advantage is that the ends of this range are less extreme. The disadvantage of this option is that it may lead to criticism of EFSA for not communicating the higher values available in the full assessment (based on Viscusi et al., 1991; Johnson and Slovic, 1998).

4.1.5.2. Probability distributions

• For highlighting the mean, using a CDF alone (without indicating the mean and without explanation) is not a good way to communicate (Ibrekk and Morgan, 1987).

• CDFs and multivariate density function graphs should be used with caution (Edwards et al., 2012).

• A CDF plotted directly above a PDF with the same horizontal scale, and with the location of the mean clearly marked on both curves, should be used (Ibrekk and Morgan, 1987).

• CCDFs should be preferred for communicating precise probabilities (Edwards et al., 2012).

• Pie charts to represent uncertainty (discretised PDFs) show potential for confusion and should be avoided (even if many people declared that they prefer this representation) (Ibrekk and Morgan, 1987).

4.1.5.3. Expression of expert judgement using ordinal scales6

• Verbal expressions of the different levels of the ordinal scale (APS) should be accompanied with their numerical translation each time they are used, whether in a short or longer text. Providing only a correspondence table (between verbal and numerical) is not sufficient, especially if such a table is difficult to locate in an annex (Patt and Schrag, 2003; Patt and Dessai, 2005; Budescu et al., 2009; Harris and Corner, 2011; Budescu et al., 2012, 2014; also based on Wallsten et al., 1986; Mosteller and Youtz, 1990).

• A verbal–numerical probabilistic ordinal scale should be used primarily for unambiguous events (e.g. ‘large and abrupt event’ could be interpreted in different ways). The target events should be very precisely defined to avoid confusion between the ambiguity in the description of the event itself and the uncertainty associated with it (Budescu et al., 2009).

• However, Jenkins et al. (2018) suggest caution in using verbal–numerical formats and rather recommend the use of numerical–verbal formats.

• Negatively worded terms (e.g. ‘unlikely’) produce higher levels of misunderstanding than positively worded terms (e.g. ‘likely’). It should be borne in mind that these verbal terms are not interpreted as the inverse of each other when presented without accompanying quantitative definitions (Budescu et al., 2012; Smithson et al., 2012).

• People tend to believe that ‘unlikely’ or ‘improbable’ events are those that are not going to happen (rather than events that happen at low frequency). Therefore, the use of these words should therefore be avoided, and infrequent but not negligible events should rather be described using terms having a positive directionality (e.g. ‘low probability’) (Teigen et al., 2013).

• Online versions of communication documents could provide links between probability terms and optional graphic displays related to the sources of uncertainty and to uncertainty scales (Spiegelhalter et al., 2011; Spiegelhalter, 2017).

4.1.5.4. Qualitative expressions of uncertainty7

• The various sources of uncertainty that underlie key events should be specified (e.g. incomplete scientific understanding of a process, lacking data for informing the parameters of a model, unreliability of measurements, etc.), as well as their nature and magnitude (Budescu et al., 2009; also based on Kuhn, 2000; Maxim et al., 2012; Maxim and Mansier, 2014; Visschers, 2018; Wardekker et al., 2008).

• When appropriate, the extent to which some of these uncertainties could be reduced should be communicated (Frewer et al., 2002).

• Hedging terms should be avoided (‘about’, ‘almost’, etc.) (based on Ferson et al., 2015).

• There may be advantages in using evaluative labels (‘high’, ‘moderate’, ‘low’) together with numeric uncertainty ranges, as they can highlight aspects of uncertainty information that may otherwise be overlooked in more complex numerical displays. However, the cut‐offs for high, medium and low have to be precisely defined. Also, the decision about the aspects of uncertainty to which evaluative labels are applied has to be explained. Be aware, however, that evaluative labels are a double‐edged sword, as they may signal to a layperson that this element is paramount to the decision at hand, even though other factors could also be important for decision‐making. Furthermore, people might focus on the evaluative labels and ignore other ways of looking at uncertainty like examining upper and lower values (Dieckmann et al., 2012).

4.1.5.5. Visualisation (other than probability distributions)

• Visual communication should include the aspects that are important. For example, simple visual cues (e.g. highlighting where the mean is on a PDF) could increase the interpretability of display methods (Edwards et al., 2012; also based on Spiegelhalter et al., 2011).

• Error bars and box plots are preferred for allowing mean estimation and deriving a follow‐up action (Edwards et al., 2012).

• A simple density graphic accompanying an uncertainty range is useful to decrease the inter‐individual variance in distributional perceptions (Dieckmann et al., 2015).

• A box plot should not be used to represent an uncertainty range without additional information describing the underlying distribution (verbal or graphic) (Dieckmann et al., 2015).

• Graphics should be accompanied by verbal and numerical descriptions of what they represent (Spiegelhalter et al., 2011; Spiegelhalter, 2017).

• For the online versions of communication documents, interactive links and animations can help into adapting the information to the user's needs and ways of understanding (Spiegelhalter et al., 2011; Budescu et al., 2012).

4.2. EFSA research studies

EFSA commissioned a target audience research study in 2016 and carried out its own follow‐up study in 2017 when preparing for this Communication GD. In the first study (Etienne et al., 2018), the authors used qualitative methods to design and communicate an opinion summary and related statements about the uncertainty. Evidence was collected from selected representatives of EFSA's key partners in the EU institutions and Member States and stakeholders (n = 39), who answered a questionnaire individually (22 of 39 completed the questionnaire in full) and then discussed their responses in five focus groups of 6–10 participants (39 in total). The second study (EFSA, 2018) was undertaken to further test the initial findings with an increased sample size. The authors adapted materials from the first study and conducted an online survey in six EU languages. EFSA and eight members of its Communication Experts Network, composed of communication representatives from the EU Member State food safety authorities, promoted the survey, generating over 1,900 responses. Appendix C contains an overview of the statements that were tested and the questions that the participants were asked.

Both studies have limitations in their design and conduct that restrict their use. Nevertheless, considered cautiously they provide some useful insights and indications retrieved directly from the key target audiences for EFSA's communications: officials in the EU Institutions and EU Member States, civil society organisations, members of the public, the scientific community, the media and food industry operators.

4.2.1. Usefulness of uncertainty information

In both EFSA studies, the response to receiving uncertainty information was positive. In the first study, a majority of the focus group participants indicated that most of the statements on uncertainty were informative. Opinions were more evenly divided on the usefulness of the information. The discussions revealed differences among individuals within the groups, in particular some participants in the general public group found the information overwhelming. In the second study, 96% of respondents replied positively about the need to communicate uncertainty information, considering it at least helpful, important or essential. The trend was similar for all languages and professions. Favourable comments indicated that uncertainty communication aids transparency, but should be clear, concise and tailored to different audiences whose perceptions, experiences and expectations vary. A few less favourable comments were made, mainly by industry or political decision‐makers. These commenters considered uncertainty information useful for decision‐makers and scientists but not for the general public, whom they suggested may misinterpret the information, which they thought might decrease public trust in EFSA and regulatory bodies.

4.2.2. Positive vs negative framing

The framing effect is a type of cognitive bias, in which people react to the same choice in different ways (i.e. positive or negative) depending on how it is presented. The EFSA studies included several examples of framing including the following, with the framings in italics: ‘EFSA's scientists said that, based on what we know, there is about an 80% chance that exposure of the most exposed group of consumers is within the level’ (positive); and ‘EFSA's scientists said that, given the uncertainties, there is about a 20% chance that exposure of the most exposed group of consumers exceeds the safe level (negative)’.

The focus group study provided some indication that the framing of uncertainty statements had an impact on risk perception. However, only one statement was tested for alternative framings and the sample was small. With these caveats, the findings suggested that emphasising the high probability of a low risk to the most exposed group contributed to keeping risk perceptions stable. Emphasising the low probability of a higher risk stimulated participants to reconsider their assessment of the risk, with some revising it up and some down.

In none of the focus groups did a majority of participants prefer statements combining both framings, as in ‘POSITIVE FRAMING. In other words, NEGATIVE FRAMING’. This contradicts common advice that uncertainty statements should be in a format that combines positive and negative framings. For example, one commenter in the online survey suggested that negative framing overestimates uncertainty and a way to overcome this is to present negative and positive framings.

In the online survey, positive framings received many more first rankings for understanding and for clarity than negative framings, although the large size of the difference may have been partly due to the order in which they were presented in the survey (positive first).

4.2.3. Confidence in EFSA

In the focus group, most participants’ confidence in EFSA was unchanged after reading uncertainty statements, although there was some impact on trust from statement to statement and from group to group. The small sample size meant it was not possible to build any firm conclusions on the basis of the observed variations. However, the focus group findings suggested that the perceived lack of clarity of some statements negatively affected trust in EFSA. Statements considered to be more simply worded were received more positively and generally did not lead participants to revise their confidence in EFSA. For some stakeholders, their predispositions towards EFSA, positive or negative, coloured and possibly amplified the manner in which they responded to the uncertainty statements.

4.2.4. Professional background

Most of the participants of the focus group were carefully selected and therefore this study provides insight into responses by partners and stakeholders who have frequent contact with EFSA, its scientific advice and its communication outputs:

• Technical policy‐makers disagreed on the best ways to communicate scientific uncertainties. Some preferred detailed information on the sources, extent and direction of uncertainties. Others preferred a qualitative assessment of the general direction (over‐/underestimate the risk). Some preferred qualitative statements because these improved their own understanding, while others expected that these statements would be easier to communicate to others (i.e. their own audiences). Some expressed concerns that detailed uncertainty information could confuse laypeople and create unnecessary concerns.

• Political decision‐makers’ views on uncertainty statements were influenced by how such information might be perceived and understood by their own audiences, such as citizens. This group stressed the need to ensure consistency between the content of scientific opinions and communication materials.

• NGOs/civil society groups believed that EFSA should communicate on uncertainties, but had negative views on the uncertainty statements presented during the focus group. The way statements were formulated (e.g. double negatives) was deemed confusing. Some participants believed that the presence of uncertainties contradicted EFSA's risk assessment conclusions on the absence of health concerns (in the example used).

• Industry representatives had varying degrees of scientific literacy skills. Some preferred statements providing information on the probabilities associated with uncertainties. However, most preferred a qualitative description of uncertainty, with an assessment of whether safe exposure levels are exceeded. Industry was the only group in which a strong majority preferred a qualitative statement when asked to rank their preference for three similar statements (the other two were quantitative).

• The general public differed in their views on the best formats for uncertainty communication: some participants preferred qualitative statements, while others were more interested in statements that provided the percentages and ranges associated with uncertainties. Those individuals less familiar with scientific research found uncertainty information to be confusing, and preferred to be informed only about risk assessment conclusions rather than about uncertainties.

In the online study, responses were grouped differently. More respondents in the political/NGO, scientist/academic and industry groups considered the qualitative statement to be the most helpful, while in the general public/media group, more individuals considered the quantitative probability statement to be the most helpful. The differences between groups, however, were not large, i.e. a large proportion in all groups found both quantitative and qualitative statements helpful in understanding the risk assessment. The real causes of the differences were unclear, because the differences between professional backgrounds were confounded by differences between languages, and also perhaps other factors.

4.2.5. Culture and language

The online survey produced some results that might indicate some differences in understanding, and preference for receiving, uncertainty information depending on language and/or cultural background. For example, differences between English and French were larger than any differences between professions. The quantitative statement received more first rankings than the qualitative option in French (54%) and to a lesser extent Greek (35%). The higher proportion of general public respondents in these languages (30% in Greek, 16% in French) may have influenced this outcome. The English responses included the highest proportion (45%) of scientists/academics (who preferred qualitative) and the lowest proportion of the general public (who preferred quantitative), but the real causes of the preferences are not completely clear.

When participants in the focus groups were asked to rate how understandable the different uncertainty statements were, preferences for qualitative or quantitative wordings varied depending on the clarity and simplicity of the expressions. This factor also influenced their preferences when choosing between different wording of similar statements. Generally speaking, therefore, the results indicated that the more concise the statements, the greater likelihood that the respondents would find them favourable.

One statement tested in the online survey included a hedging word (about an 80% chance) and another included an approximate probability (likely (66–90% probability)). However, due to the sequential structure of the survey, preferences between these were not assessed.

4.2.6. Messages for expressing uncertainty

In the focus group study, participants rated how understandable, informative and useful they found different uncertainty statements. More concise statements were generally rated more understandable than complexly worded statements. The online survey provided indications on which expressions participants found most helpful for understanding the risk. Five types of expressions of uncertainty were included in the two studies, although not all of these were directly compared:

• qualitative description of the direction of uncertainty (conservative or non‐conservative), including the ‘uncertainty table’ containing brief qualitative descriptions of sources of uncertainty accompanied by plus and minus symbols indicating the direction of their impact on the assessment;

• qualitative expressions of probability (e.g. ‘likely’);

• approximate probabilities (‘66–90% chance’ and ‘more than 50% chance’);

• a precise probability qualified by a hedging word (e.g. ‘about 80% chance’).

Table 5 summarises the key indications on each.

Table 5.

Indications of the audiences’ perceived understanding of different expressions of uncertainty and their preferences among different formats of similar statements

	Entry	Informed	Technical
Qualitative descriptions of sources and/or direction of uncertainty
Focus group	The whole group found the uncertainty table confusing and misinterpreted the +/− signs as cancelling each other out	–	Two out of nine of the technical group participants described the +/− table as ‘incomprehensible’
Qualitative expressions of probability
Focus group	Two out of seven in the public group preferred the qualitative statement	Three out of seven in the policy group and zero out of five in the NGO group preferred the qualitative statement	Six out of seven in the industry and four out of nine in the technical groups preferred the qualitative statement
Online survey	35% of the public/media group ranked the qualitative statement most helpful for understanding the risk	47% of the policy/NGO group ranked the qualitative statement most helpful for understanding the risk	46% of the industry, 39% of the risk assessors and 41% of scientist/academic groups ranked the qualitative statement most helpful for understanding the risk
A precise probability with hedging word (‘about’)
Focus group	Five out of seven of the public group preferred the quantitative probability	Four out of five in the NGO and three out of seven in the policy groups preferred the quantitative probability	Four out of nine in the technical group but only one out of eight in industry groups
Online survey	45% of the public/media group ranked the quantitative probability most helpful for understanding the risk	37% of the policy/NGO group ranked the quantitative probability most helpful for understanding the risk	35% of the industry, 38% of the risk assessor, and 35% of the scientist/academic groups ranked the quantitative probability most helpful for understanding the risk
An approximate probability (range)
Focus group	None of the public group preferred the approximate probability using ranges	One out of five in the NGO and one out of seven in the political groups preferred the approximate probability using ranges	One out of eight in the industry and one out of 10 in the technical groups preferred the approximate probability using ranges

Some technical and political participants found the uncertainty table useful when accompanied by a qualitative expression of the overall uncertainty. However, the table is easily misunderstood by individuals from all audience levels.

In both studies, a large share of participants was favourable to qualitative statements about the uncertainties, with differences among professions. Comments from the focus group discussions are consistent with a preference for qualitative information. A majority of industry participants ranked them higher than the quantitative options for understanding, as did large minorities of risk assessors and other scientists, but in many cases they were answering according to what they thought was the best format for the general public. Fewer political and public respondents favoured the qualitative information and in the focus group study the NGO participants rejected them outright when given the choice. By language, the English and Spanish language groups ranked these statements most useful in the online survey. As noted above, the differences between languages and professional backgrounds were confounded, so the real cause of the differences is unclear.

In both studies, a large share of participants were favourable to probability statements expressed quantitatively, with differences among professions. The public/media and NGO groups ranked them slightly more helpful for understanding risk than qualitative expressions, while political participants were more balanced between the two. In the focus group studies, the industry group opposed use of such information but one‐third ranked them highest in the online survey (also about one‐third of risk assessors in both studies). By language, the French, Spanish and Greek groups ranked these statements most useful in the online survey.

The findings may support the use of quantitative probability statements about the uncertainties in communication aimed at the entry‐level audiences. Support for such an approach may be higher in languages/cultures more open to the use of numerical expressions (e.g. French).

In the focus groups, the precise probability statement with a hedging word (‘about’) was clearly preferred to the approximate probability, with strong first preferences among the public, NGO and political decision‐maker groups in particular. In the online survey, the approximate probability statements were not directly compared with precise probabilities.

4.3. Implications of the grey literature for the guidance

Recommendations on how to communicate uncertainty have been published by various institutions (IOM, 2013; Wardekker et al., 2013). The approaches taken in these documents differ considerably, however. Some of them provide a comprehensive literature review without concrete examples of how best to communicate uncertainty (e.g. IOM, 2013), while others describe in a very practical way what needs to be taken into account when communicating uncertainty to the public without providing much evidence for the recommendations (Wardekker et al., 2013). Because this grey literature does not report original research, these publications have not been included in the literature review. In the examples described in the present document, insights from the grey literature have been taken into account when necessary and, in Section 4.1.5, the implications of the scientific literature for guidance, differences and commonalities between the present document and guidelines from other institutions are discussed.

4.4. EFSA examples used to develop the guidance

The insights from the scientific literature and the findings of EFSA's research provided a starting point for the proposed guidance. These were applied and tested on real examples from EFSA scientific assessments to draft messages and select visual aids for communicating the related uncertainties. The effective application of the proposed guidance strengthened the case for including them among EFSA's guidance in Section 3. The available evidence does not, however, address every aspect of communicating uncertainty, so some of the guidance presented in this document is based partly or wholly on expert judgement and reasoning. Preliminary instructions for the guidance were also tested, refined, and further elaborated by applying them to concrete EFSA examples, which are listed in Table 6. For transparency, the basis for each point of guidance is summarised in Appendix B.

Table 6.

Selected EFSA scientific assessments used as examples

Expressions of uncertainty	EFSA scientific assessments used as examples
Unqualified conclusion with no expression of uncertainty	Zearalenone in feed (EFSA CONTAM Panel, 2017)
Description of a source of uncertainty	Zearalenone in feed (EFSA CONTAM Panel, 2017)
Qualitative descriptions of the direction and/or magnitude of uncertainty	Zearalenone in feed (EFSA CONTAM Panel, 2017)
Inconclusive assessment	Zearalenone in feed (EFSA CONTAM Panel, 2017)
A precise probability	Nematodes 2017 (EFSA PLH Panel, 2017)
An approximate probability	Animal welfare aspects in respect of the slaughter or killing of pregnant livestock animals (EFSA AHAW Panel, 2017) Listeria monocytogenes in ready‐to‐eat (RTE) foods (EFSA BIOHAZ Panel, 2018)
A probability distribution	Animal welfare aspects in respect of the slaughter or killing of pregnant livestock animals (prevalence) (EFSA AHAW Panel, 2017) Nematodes 2017 (EFSA PLH Panel, 2017)
A two‐dimensional probability distribution	IESTI Opinion (EFSA, 2007)

The selection of examples aimed at providing a representative view of the different types of EFSA assessments (e.g. food safety, animal and plant health, chemical, biological) and also at providing examples of all the eight types of uncertainty expressions explained in Section 2. Table 6 lists the examples. It is important to note that the choice of EFSA scientific assessments for this procedure was limited as, at the time of writing, only a few of these had been developed according to approaches and methods required by the Uncertainty Analysis GD. A smaller number of older assessments was also available, but in many cases the uncertainty information they contained and the way that information was presented is not fully in line with the requirements of the Uncertainty Analysis GD. Once applying the Uncertainty Analysis GD becomes the norm for EFSA's scientific assessments, further appraisal of this Uncertainty Communication GD is planned.

The authors provided expertise in scientific assessment, social sciences and communications. This expertise was used to analyse the above examples and develop communications for each one. The process involved: extraction of the salient information from the assessments (EFSA's scientific opinions) including conclusions on risk, types of effects, affected populations and foods/feed, and type of uncertainty analysis. Different questions were applied, e.g. ‘What is the central estimate?’ and ‘Do the assessors use words or numbers to describe probabilities about the uncertainties?’ Messages for different target audience levels were then developed and requirements for further information (e.g. visual aids, FAQs) investigated. These results then provided the basis for feedback to assessors to make them aware of how best to support effective communication by including the required information in their assessments.

The most critical outputs from this procedure are the examples of messages, descriptions and visual aids and the resulting guidance that in some cases – particularly when the academic literature is weak or missing – are derived solely from this process. For transparency, the table in Appendix B indicates which evidence source formed the basis of each point of guidance.

5. Further research needs

Communication of uncertainties is a challenge; even researchers often incorrectly interpret confidence intervals or other measures of uncertainties (Greenland et al., 2016). Even more difficult is the communication of scientific uncertainties to laypeople or non‐scientists. The present literature review shows that the way in which uncertainties should be conveyed to non‐scientists to enable informed decisions is still an under‐researched field.

The guidance for communicators provided in this document is based on the best available evidence. Nevertheless, the guidance needs to be carefully evaluated (see Section 6). The research community can not only benefit from studying how EFSA's approach evolves, but also contribute to this process. Studies are needed that examine whether the target audience understands the communicated uncertainty information, and whether they perceive this information to be useful for decision‐making. Furthermore, it should also be examined whether communication of uncertainties impacts on trust in EFSA; will trust increase because EFSA is open and transparent about its conclusions or would such openness have the opposite effect, because people may wrongly infer that the conclusions of EFSA are not reliable, i.e. they do not grasp that uncertainty is inherent in any decision. Implementing uncertainty communication is a process. The present document initiates this process, but the guidance provided needs to be adapted by practitioners based on the outcomes of the evaluations and future research that examine how to best communicate uncertainty information.

Many of the studies reviewed in this Uncertainty Communication GD examined whether the different communication formats in which uncertainty was expressed evoked different levels of concern or risk perception. Such research is of interest, but it provides only limited information about the more relevant question for EFSA, namely which format is best understood by participants. Ultimately, EFSA is most interested in which communication format helps people best to make informed decisions. The question ‘what gold standard should be used to evaluate the quality of uncertainty communication’ has received little attention, however. ‘Understanding’ could be measured in various ways, of course. One approach would be to examine whether objectively different levels of uncertainty evoke different levels of subjectively perceived uncertainty. Another approach would require that participants have to extract meaningful information from the communicated uncertainties and consequently to make decisions that take account of uncertainty information. Other approaches are also possible. Therefore, future research should address how to measure people's understanding of uncertainty communication in a comprehensive way.

The question of whether there are different kinds of probability judgements can be viewed from a mathematical/statistical perspective or from a psychological perspective. The latter perspective is especially important for the communication of uncertainties, because there is a strong intuition that probabilities based on frequency information are different from those based on other types of evidence and in particular expert subjective estimations (Frisch and Baron, 1988).

It is not the aim of the present guidance to propose a comprehensive research program, but it is the hope of the authors of this Uncertainty Communication GD that future research will take the following points into account:

• Future research should examine how well participants understand the uncertainty information and whether various communication formats result in different decisions. More specifically, it is important to test which formats are best understood and how they influence the audience's perception of uncertainty and the decision‐making process.

• There is little research that focused on the communication of subjective beliefs expressed by experts. Future research should address the question of how such probabilities should be communicated to laypeople so that they understand the information correctly (e.g. as the group of experts’ consensus opinion about the probability of their conclusion being correct), and that they do not make erroneous inferences, e.g. that the probability refers to the proportion of experts who support the conclusion. As part of this, it would be useful to investigate understanding of messages where probability is expressed as % certainty (section 3.1.3 point (2)) and the effect of giving two‐one‐sided bounds rather than one (section 3.2 point (6)), as evidence on these aspects of the guidance is currently lacking.

• Another important question that needs to be deepened in future research is whether prior attitudes towards hazards or trust in the communicator interact with the understanding of uncertainty information and its use in decision‐making, and whether provision of information on uncertainty influences trust.

• Whereas most of the current studies used participants from the general public, future research should also include decision‐makers in the risk management domain, because these stakeholders may come to different conclusions depending on how uncertainty is communicated.

• Finally, much of the research has examined convenience samples therefore future research efforts should follow more structured approaches, e.g. random sampling.

6. Implementation and evaluation

Implementation of this Uncertainty Communication GD requires changes to current communication working practices at EFSA. These changes require internal testing, periodic evaluation and a review within a suitable period (e.g. three years). This ongoing process will provide opportunities to adapt and improve the approach and to check that it is helping EFSA to meet its overall objectives in dealing with uncertainties.

6.1. Implementation

Implementation of this Uncertainty Communication GD will shadow the gradual application of the Uncertainty Analysis GD by EFSA's scientific panels and scientific staff in their assessments. In agreement with EU decision‐makers, EFSA scientific panels carrying out assessments in areas that are not directly related to market authorisation procedures began to apply the Uncertainty Analysis GD from autumn 2018 onwards. Communication on assessments in those areas and resulting from work begun from that date onwards will be developed in line with this Uncertainty Communication GD. Communication on assessments in areas related to regulated products will follow the Uncertainty Analysis GD implementation plan, thus benefiting from experience gained in applying this Uncertainty Communication GD in the non‐regulated product areas. It should be noted that in some areas, for example plant health, where uncertainty analysis is part of the existing scientific process, communicators can already use the Communication GD to support their work.

At a practical level, EFSA's staff and experts need training and support to ensure they recognise and understand the types of uncertainty expressions used in scientific assessments and can apply the step‐by‐step process presented in Section 3. Support will include training on ensuring accuracy in framing messages on uncertainties, and internal testing of messages at EFSA. Where feasible, training for EFSA's communications staff will be opened to communications counterparts in the EU Institutions and in EU Member States, for example, press office officials at the European Commission and at national authorities.

Further support will be provided by establishing an online resource of ‘frequently asked questions’ (‘FAQs’) including entries on key concepts and expressions for communicating uncertainty, including the interpretation of probability expressions (see Section 3.1). This will be useful to assessors and communicators at EFSA as well as providing supporting information for interested members of the target audiences. These FAQs will be made available publicly on EFSA's website.

The authors recognise the limited type and number of example assessments used in Section 3.2. A proposal in the implementation of the Uncertainty Analysis GD is the creation of a database of examples, which will grow over time and assist assessors when dealing with similar types of assessments. EFSA proposes to repeat this activity with the Uncertainty Communication GD, expanding the body of ‘case studies’ that can inform a future review of the GD and further assist communicators.

6.1.1. Engagement

While researching and developing the Uncertainty Communication GD, EFSA regularly consulted and engaged with its institutional partners and other stakeholders, including presentations to academic audiences at international conferences. Further attention is needed to ensure the mutual understanding of uncertainty information in the regular dialogue and coordination of risk communication activities between EFSA and decision‐makers in the EU Institutions and Member States. Efforts to inform and where necessary support understanding of EFSA's approach will also extend to other stakeholders: consumer organisations, NGOs, industry representatives and – arguably most crucially for communication – the media.

Communication of uncertainty is one element among many in EFSA's systematic interaction with its partners and stakeholders. For example, EFSA's press office proactively engages with and assists journalists reporting on its scientific assessments. The Authority has several channels for interaction with national, European and global partners (Advisory Forum, Communication Experts Network, International Risk Communication Liaison Group) and stakeholders (EFSA Stakeholder Forum, Communicators Lab). The European Commission also participates to varying degrees in these groups.

6.2. Evaluation

Following consultation with its partners and stakeholders, EFSA recommends a series of test and evaluation steps throughout the implementation of this Uncertainty Communication GD:

• Feedback from ad hoc testing of understanding of uncertainty messages and formats on receivers of EFSA's communications including decision‐makers, consumers, NGOs, the media and other stakeholders as appropriate and resource permitting.

• Following a suitable period of implementation (e.g. three years), carry out a review, focusing primarily on how the approach affects understanding of uncertainty information, and eventually consider updating the Uncertainty Communication GD.

• Maintain a database of example communication activities to support use of the Uncertainty Communication GD by communicators and contribute to the review (see above).

• Within the context of EFSA's social research activities, explore generally and where feasible how the roles of different audiences (e.g. decision‐makers, journalist) affects their information needs related to EFSA's scientific advice including information in relation to uncertainties.

• In the context of EFSA's overall approach on uncertainty analysis and communication, further consider how to evaluate the understanding and use of uncertainty information by decision‐makers and other stakeholders to assess the impact and effectiveness of this approach.

Glossary

Ambiguity

The quality of being open to more than one interpretation. A type or cause of uncertainty that may apply for example to questions for assessment, evidence, models or concepts, and assessment conclusions

Approximate probability

Any range or bound for a probability. This includes, but is not limited to, the ranges that are used in the approximate probability scale (q.v.)

Approximate probability scale

A set of approximate probabilities with accompanying verbal probability terms, shown in Section 12.3 of the EFSA Uncertainty Guidance Document (EFSA Scientific Committee, 2018a) and recommended for harmonised use in EFSA scientific assessments

Assessor

A person conducting a scientific assessment and/or uncertainty analysis

APS

Acronym for approximate probability scale (q.v.)

Bayesian

A school of statistics that uses probability to quantify judgement on the likelihood of a particular range of a quantity, or the likelihood of a specified category (e.g. low/medium/high). See also the glossary entries for ‘probability’ and ‘frequentist’

Best estimate

Often used as a synonym for ‘central estimate’ in the literature (q.v.), but not recommended in this Guidance because the term ‘best’ has a normative content, as it implies a preference for a value, whereas ‘central’ just communicates the place of that value in a set of relevant values

Box plot

A simple way of representing statistical data on a plot in which a rectangle is drawn to represent the second and third quartiles, usually with a line inside to indicate the median value. Lower and upper probability bounds are shown as lines (‘the whiskers’) outside the rectangle

Characterising uncertainty

The process of making and expressing an evaluation of uncertainty either for an assessment as a whole or for a specified part of an assessment. Can be performed and expressed either qualitatively or quantitatively

Central estimate

Descriptive term for a central value in a distribution, most commonly the arithmetic mean, the geometric mean, the median or the mode. Which one is used depends on the shape of the distribution, and should be specified

Certainty

Used in this document as a synonym for probability (when the probability is over 50%), to facilitate communication when appropriate. For example, 90% probability of X would be communicated as 90% certainty of X

Confidence

Assessors sometimes use qualitative expressions of confidence (e.g. high, low) to express their evaluation of the strength of evidence or the likelihood that a conclusion is correct. Various scales have been proposed for this purpose, but the Scientific Committee is not aware that any of these have a formal operational definition of the type that exists for probability itself (see section 5.10 of EFSA Scientific Committee, 2018b). Therefore they are not recommended in EFSA Scientific Committee (2018a) or the present guidance for expressing the overall uncertainty of assessment conclusions. However, they can be useful for expressing individual sources of uncertainty and in weight of evidence assessment (EFSA Scientific Committee, 2017), provided the overall uncertainty of the conclusion is expressed quantitatively when possible

Confidence interval

In frequentist statistics, a confidence interval is a range that would include the true value of the parameter to be estimated in a specified proportion of occasions (the confidence level) if the experiment and/or statistical analysis that produced the range was repeated an infinite number of times

Conditional probability

A probability for a specified event given that another specified event has occurred (e.g. probability of exceeding a safe dose of a chemical, given that food containing the chemical has been consumed)

Decision‐maker

A person with responsibility for making decisions; in the context of this document, a person making decisions informed by EFSA's scientific advice. Includes risk managers but also people making decisions on other issues, e.g. health benefits, efficacy, etc

Description of sources of uncertainty

Verbal description of the nature or causes of sources of uncertainty

Distribution

A probability distribution is a mathematical function that relates probabilities with specified intervals of a continuous quantity or values of a discrete quantity. Applicable both to random variables and uncertain parameters

Expert judgement

The judgement of a person with relevant knowledge or skills for making that judgement

Expert knowledge elicitation (EKE)

A systematic, documented and reviewable process to retrieve expert judgements from a group of experts, often in the form of a probability distribution

Expression of uncertainty

A verbal or numerical statement that characterises uncertainty associated with the outcome of scientific assessments. In this document is used to refer to any characterisation of the source, direction or magnitude of uncertainty, which may include words, numbers or graphics or any combination of these

Frequency

The number of occurrences of something, expressed either as the absolute number or as a proportion or percentage of a larger population (which should be specified)

Frequentist

A school of statistics that uses probability to quantify the frequency with which sampled values arise within a specified range of a quantity or for a specified category (e.g. low/medium/high). See also the glossary entries for ‘probability’ and ‘Bayesian’

Hedging

Words used to limit or qualify (something) by conditions or exceptions, e.g. ‘about’, ‘approximately’, ‘nearly’, etc

Inconclusive assessment

Used in this document to refer to two types of situation: (1) an assessment where an unqualified conclusion is required by legislators or risk managers, but the assessment does not reach the level of certainty regarded as necessary for that (see Section 3.5 of EFSA Scientific Committee, 2018b); and (2) an assessment in which assessors are unable to give a probability or approximate probability for any possible answers to the assessment question, and therefore no conclusion can be drawn

Influence analysis

Quantitative or qualitative analysis of the extent to which plausible changes in the overall structure, parameters and assumptions used in an assessment produce a change in the results

Laypeople/layperson

In this document, people who are not professionally involved in uncertainty assessment, scientific assessment or science communication

Likelihood

In this document, likelihood is used with its everyday meaning, referring to the chance or probability of something: generally replaced with ‘probability’ in this document

Lower bound

Lower value of a range, e.g. a range of probabilities

Measurement uncertainty

Non‐negative parameter characterising the dispersion of the quantity values being attributed to a measurand (a quantity intended to be measured), based on the information used (JCGM, 2012)

Non‐standard uncertainties

Any deviations from a standardised assessment procedure or standardised assessment element that lead to uncertainty regarding the result of the assessment. For example, studies that deviate from the standard guidelines or are poorly reported, cases where there is doubt about the applicability of default values, or the use of non‐standard or ‘higher tier’ studies that are not part of the standard procedure (see EFSA Scientific Committee, 2018a,b for detailed guidance and explanation)

Numeracy

A person's ability to understand and process numerical concepts

Ordinal scale

A scale of measurement comprised of ordered categories (e.g. low/medium/high), where the magnitude of the difference between categories is not quantified

Overall uncertainty

The assessors’ uncertainty about the question or quantity of interest at the time of reporting, taking account of the combined effect of all sources of uncertainty identified by the assessors as being relevant to the assessment

Point estimate

Used in this document to refer to a single value that is used by assessors as an estimate for an uncertain quantity

Plain language

Clear, straightforward expression, using only as many words as are necessary and avoiding obscure or inflated vocabulary, technical or scientific terms and convoluted construction

Precise probability

Probability expressed as a single number, as opposed to a range of probabilities.

1. Probabilistic Representation of uncertainty and/or variability using probability distributions.
2. Calculations in which one or more inputs are probability distributions and repeated calculations give different answers. Related term: deterministic.

Probability

Probability can be used for quantifying risk, but in this GD it is used only to describe uncertainty. Defined depending on philosophical perspective:

1. From a frequentist perspective, the frequency with which sampled values arise within a specified range of a quantity or for a specified category (e.g. low/medium/high).
2. From a Bayesian perspective, quantification of judgement on the likelihood of a particular range or category.

Probability interval

Used in this document for an interval of a quantity of interest for which a subjective probability is specified, i.e. an interval within which it is judged that the true value of the quantity lies with the specified probability. Referred to in Bayesian statistics as a credible interval

Probability range

The upper and lower bound for an approximate probability

Probability judgement

A probability, approximate probability or probability bound obtained by expert judgement

Qualitative assessment

Sometimes refers to the form in which the conclusion of an assessment is expressed (e.g. a verbal response to a question of interest), or to the methods used to reach the conclusion (not involving calculations), or both

Qualitative conclusion

Used in this document to refer to an assessment conclusion that refers to something (e.g. an outcome, condition or mechanism) that is described in words, and not quantified

Qualitative expression of uncertainty

Expression of uncertainty using words or ordinal scales

Qualitative description of the direction and/or magnitude of uncertainty

Verbal description of how large the overall uncertainty or the single uncertainties are and how this impacts the outcome of the scientific assessment

Quantitative estimate

The result of assessing an uncertain quantity, expressed numerically (e.g. a bound, range or distribution)

Quantitative expression of uncertainty

Expression of uncertainty using numerical measures of the range and relative likelihood of alternative answers or values for a question or quantity of interest

Quantity

A property or characteristic having a numerical scale

Range

Two numbers (an upper and lower bound) used to specify an inclusive set of values for an uncertain or variable quantity, or for an approximate probability

Risk communication

The interactive exchange of information and opinions throughout the risk analysis process as regards hazards and risks, risk‐related factors and risk perceptions, among assessors, risk managers, consumers, feed and food businesses, the academic community and other interested parties, including the explanation of scientific assessment findings and the basis of risk management decisions

Risk perception

Laypeople's subjective or intuitive risk judgement to evaluate hazards

Risk manager

A type of decision‐maker, responsible for making risk management judgements

Scientific assessment

The process of using scientific evidence and reasoning to answer a question or estimate a quantity

Sensitivity analysis

A study of how the variation in the outputs of a model can be attributed to, qualitatively or quantitatively, different sources of uncertainty or variability. Implemented by observing how model output changes when model inputs are changed in a structured way

Source of uncertainty

Defined by EFSA Scientific Committee, 2018a as an individual contribution to uncertainty, defined by its location (e.g. a component of the assessment) and its type (e.g. measurement uncertainty, sampling uncertainty). Used in this document more informally, to refer to any distinct cause of uncertainty

Standard outcome

The conclusion of a standardised assessment procedure (q.v.), expressed in a standardised way (e.g. no concern, safe, etc.)

Standard uncertainties

Sources of uncertainty that are considered (implicitly or explicitly) to be addressed by the provisions of a standardised procedure or standardised assessment element. For example, uncertainties due to within and between species differences in toxicity are often addressed by a default factor of 100 in chemical risk assessment

Standardised assessment

An assessment that follows a standardised procedure (q.v.).

Standardised procedure

A procedure that specifies every step of assessment for a specified class or products or problems, and is accepted by assessors and decision‐makers as providing an appropriate basis for decision‐making. Often (but not only) used in scientific assessments for regulated products

Subjective probability

Quantification of judgement on the likelihood of a specified category or a specified range of a quantity. This is the Bayesian concept of probability (q.v.), which is used in EFSA's guidance on uncertainty analysis (see Section 5.10 in EFSA Scientific Committee, 2018a)

True value

The actual value that would be obtained with perfect measuring instruments and without committing any error of any type, both in collecting the primary data and in carrying out mathematical operations. (OECD Glossary of Statistical Terms, https://stats.oecd.org/glossary/detail.asp?ID=4557)

Uncertainty

In this document, uncertainty is used as a general term referring to all types of limitations in available knowledge that affect the range and probability of possible answers to an assessment question. Available knowledge refers here to the knowledge (evidence, data, etc.) available to assessors at the time the assessment is conducted and within the time and resources agreed for the assessment. Sometimes ‘uncertainty’ is used to refer to a source of uncertainty (see separate definition), and sometimes to its impact on the conclusion of an assessment

Uncertainty analysis

The process of identifying and characterising uncertainty about questions of interest and/or quantities of interest in a scientific assessment

Unqualified conclusion with no expression of uncertainty

A conclusion that is presented as a statement without being qualified by any expression of uncertainty, because the respective regulation or risk managers require this (see Section 3.5 of EFSA Scientific Committee, 2018b)

Unquantified uncertainty

An identified source of uncertainty in a scientific assessment that the assessors are unable to include, or choose not to include, in a quantitative expression of overall uncertainty for that assessment

Variability

Heterogeneity of values over time, space or different members of a population, including stochastic variability and controllable variability (e.g. body weight measured in different individuals in a population, or in the same individual at different points in time)

Weight of evidence assessment

A process in which evidence is integrated to determine the relative support for possible answers to a scientific question

Upper bound

Upper value of a range, e.g. a range of probabilities

Abbreviations

APS

approximate probability scale

bw

body weight

CCDF

complementary cumulative distribution function

CDF

cumulative distribution function

COM

EFSA Communication Unit

CMP

communication mode preference

EEA

European Environment Agency

EKE

expert knowledge elicitation

EPA

Environmental Protection Agency

GD

Guidance

IESTI

International Estimated Short‐term Intake

IPCC

Intergovernmental Panel on Climate Change

NGO

non‐governmental organisation

NUSAP

Numeral, Unit, Spread, Assessment and Pedigree

PDF

probability density function

RTE

ready‐to‐eat

SCER

EFSA Scientific Committee and Emerging Risk Unit

Appendix A – Target audience research by EFSA (2014–2015)

1.

EFSA revamped its corporate website in 2015, modernising the graphical interface, revising and simplifying most of the copy, restructuring the information architecture and moving the entire site to a new content management system. The project benefitted from a user‐experience design approach involving extensive target‐audience research. This formed the basis of the mapping of EFSA's users in ‘entry’, ‘informed’ and ‘technical’ levels of information needs and scientific/technical literacy as used in the Communication GD (see Section 1.5.2).

A.1. User mapping

The EFSA website project team mapped existing and potential users of EFSA's website through a mixture of desk research and interviews. Previous annual web user surveys (n = 4,382, 2013) and a dedicated survey (n = 1,762, 2014) provided substantial information on user profiles, their information needs and opinions about the website as well as the EFSA Journal and other communication products (e.g. multimedia tools). Subscriptions to information services such as alerts and newsletters (about 30,000 subscribers) provided additional data. Web analytics were available for several years giving a detailed picture of web usage and trends. Requests for information sent to the ‘Ask EFSA’ service for the general public also provided insights on users’ concerns/behaviours. There were also 12 interviews with representatives of partners in the European institutions, Member States, and stakeholders, including consumer groups, NGOs, industry and media.

A.2. Information types

Alongside the user mapping, internal business owners (e.g. scientific units, human resources, procurement) were interviewed to understand the kinds of information/services they need to provide, as well as the direction and development of their interaction with users. Primary scientific communications (i.e. scientific outputs) accounted for most traffic to the EFSA website. However, there are several types of secondary information related both to core activities (i.e. expert biographies, meetings, networks, consultations, calls for data, press releases) and management and reporting (i.e. work plans, annual report, audit report, quality report). A third level related to services mainly includes calls for tender and job vacancies. Different categories of users expressed different needs in terms of the type of pages/products they wanted to access.

The project team mapped the users, their information needs and EFSA's communication objectives in a matrix, producing a three‐tiered content structure for the website. Personas were developed to assist in converting the data into realistic scenarios that were then further tested with actual users of the website.

Figure A.1.

Mapping of EFSA content by target audience (2015)

A.2.1. Entry

An entry level is pitched at a broad range of partially informed/uninformed users. The tone of voice is clear, simplified and to‐the‐point, with key messages kept to a minimum and with a greater emphasis on context. Current and potential users include political decision‐makers (MEPs, EC cabinet members, MS officials), consumers/some consumer organisations in MS, general media, SMEs, some NGOs. They are mainly interested in high level content, e.g. the EFSA homepage, news stories, general FAQs, topics, videos, multimedia, social media, brochures, the ‘about’ section. They navigate via search, the homepage and the newsletter.

A.2.2. Informed

Informed users already have some knowledge of EFSA/the EU, food safety, scientific risk assessment and/or their specialist area. The tone of voice is clear, the level of complexity more challenging with additional details and supporting messages. Current/potential users include risk managers (policy officers), international organisations, some regulated products applicants (SMEs), non‐specialist stakeholders including specialist media, issue‐specific NGOs, health professionals, European auditors. In addition to content like news, issue‐specific FAQs, data summaries, summaries of scientific opinions, reports, minutes, agendas, EFSA's Register of Questions, declarations of interest, and more specific about pages. They are likely to navigate via search, bookmarks, newsletter.

A.2.3. Technical/expert

Most scientific users who are engaged in EFSA's work possess a medium to high level of technical literacy. The tone of voice is scientific journal/article level. Users include EFSA and Member State risk assessors, academics, public/private research institutes, third country risk assessors, international organisations, regulated product applicants (e.g. multinationals), reference laboratories. Their preferred content types are scientific opinions, technical reports, guidance documents, data files (both summary and raw), methodologies, models. They find their content via search, bookmarks, and closed networks.

Figure A.2.

Mapping of EFSA's target audiences for external communications (2015)

The chart shows six key target audience groupings for EFSA identified through this research: decision‐makers, risk assessors, industry, NGOs/specialised media, general media, and informed/concerned citizens. It also clusters them according to their scientific literacy and temporal relationship with EFSA's communications into three broad categories – ‘technical’, ‘informed’ and ‘entry level’.

Appendix B – Summary of evidence and reasoning for the guidance

1.

As far as possible, this Guidance is based on the evidence sources described in Section 4. The available evidence does not address every aspect of communicating uncertainty, however, so some guidance is based partly or wholly on expert judgement and reasoning. Preliminary recommendations for the guidance were tested, refined and further elaborated by applying them to concrete EFSA examples, which are listed in Table 6, Section 4. For transparency, the basis for each recommendation is summarised in the table below.

Section	Guidance point no.	Evidence/reasoning
3.1.1 Alignment with the assessment	1	Reasoning: It is essential for transparency and consistency that all parts of the communication should convey the same message
	2	Reasoning: Minimises ambiguity about what has been assessed and what conditions and outcome the scientific conclusion and uncertainty relate to, which is essential for the information to be understood, and to allow unambiguous expression of its uncertainty in terms of probability
	3	Reasoning: This is critical for correct understanding of the conclusion and its associated uncertainty
3.1.2 Describing uncertainty with words	1 and 2	Evidence: Many studies demonstrate that verbal expressions of probability (e.g. likely) are ambiguous and interpreted by different people in widely different ways (see Sections 4.1.4.3 and 4.1.4.4) Reasoning: To be understood, a meaningful and unambiguous metric is needed for expressing the magnitude of uncertainty. This requirement is met by probability but not by other quantitative or qualitative means of expressing uncertainty. See EFSA (2018b) for further discussion and evidence on these points
3.1.3 Describing uncertainty with numbers	1	Evidence: Multiple studies show that interpretation is more variable for verbal probabilities than numeric probabilities (see Sections 4.1.4.3 and 4.1.4.4)
	2	Reasoning: Because both probabilities and percentages can be used to express other things as well as uncertainty, there is potential for misunderstanding. To avoid this, it was proposed in the public consultation draft of this guidance to refer to ‘% certainty’. This also helps to indicate the subjective nature of the probability and provides a framing in terms of certainty rather than uncertainty, which assessors and communicators may prefer. This approach could be misleading for low probabilities (5% certain sounds low but is actually 95% certainty of non‐occurrence), so it is recommended to communicate % certainty for the outcome that EFSA considers more likely (e.g. 10% probability of X would be communicated as 90% certainty of X not occurring). Some comments received in the public consultation suggested that varied framing (% certainty for some outcomes occurring, and for others not occurring) might lead to misunderstanding, and it is unclear whether recipients would understand % certainty as equivalent to the corresponding probability. On balance, it is judged preferable to retain the use of % certainty due to the advantages stated above, and to address the issues raised by the commenters by a) ensuring the outcome each % certainty refers to is clearly stated, and b) ensuring that it is clearly explained in FAQs linked to the communication that % certainty is to be interpreted as % probability. The use of % certainty in communication has not been tested in any existing study, so it is recommended to evaluate it by monitoring its effects in practice and through appropriate research studies, and to revise the guidance if this appears necessary There is a body of literature that recommends communicating with frequencies rather than probabilities (e.g. Gigerenzer, 1994). However, many of the studies supporting this involved tasks which required subjects to make inferences involving combination of conditional probabilities and base rates. EFSA assessments and communications generally provide direct estimates of the outcomes of interest to the audience, and hence do not require them to deduce these from conditional probabilities. Also, applying EFSA's guidance on uncertainty analysis results in subjective probabilities expressing uncertainty about single outcomes or events and it is unclear whether these would be better expressed using frequencies. Gigerenzer and Hoffrage (1995) have argued that they would, but a study by Joslyn and Nichols (2009) in which the tasks did not require conditional inferences found that subjects understood weather forecast information better when expressed using probability than frequencies. Furthermore, expressing single‐event probabilities in terms of frequency requires framing each assessment in terms of a hypothetical population of similar cases (e.g. other chemical risks with similar evidence and risk to the one under assessment), which increases the complexity of the communication and therefore poses different challenges to understanding. The Working Group therefore decided to retain the recommendation to communicate subjective probabilities as % certainty, but recommends that this be reviewed when further research on the issue is available
	3	One idea emerging from the literature was that it might be desirable to present probabilities both for occurrence and for non‐occurrence of an event, to avoid audience understanding or perception being biased by the framing An obvious strategy to reduce bias would be to communicate both positive and negative framings. However, in the EFSA focus group study (summarised in the draft guidance) where this was done in a simple way (there is a P% probability of X occurring, and a 100 − P% probability of X not occurring), subjects said they considered this form of communication to be unnecessarily complex We therefore propose a layered strategy, where only a positive framing (the outcome EFSA considers more likely) is used but, at the informed level, this is accompanied by concise information on reasons why both the positive and negative outcomes have some probability. Taking the more likely outcome as positive framing is necessary to avoid the problem of expressing low probabilities as % certainty, and also has some support from the finding of Teigen and Brun (2003) that positive and negative frames were induced by complementary ways of presenting numeric chances. We are unaware of any evidence on whether presenting reasons for both positive and negative outcomes at informed level would reduce framing bias or affect communication in other ways, so this should be subject to future review of its performance in practice, based on appropriate studies
	4	Evidence: There is evidence that when both verbal and numeric probabilities are presented together, interpretation is less variable when the numeric probability is presented first (Jenkins et al., 2018)
	5	Reasoning for indicating a probability: When a range is given on its own, the probability for it is ambiguous and could in principle be anything, although it is usually implicit that it is at least possible that the value lies in the range (probability greater than zero). To avoid ambiguity, a probability must be indicated Evidence and reasoning for indicating which values within the range are more likely: Several studies provide evidence that, when uncertainty is represented by a range, people tend to focus on one or other end of the range (Viscusi et al., 1991; Johnson and Slovic, 1998; Dieckmann et al., 2015, 2017). It seems likely that this results not from misunderstanding of the range (which is a primary concern in this guidance) but from subsequent interpretation and value judgements by the recipient (e.g. choosing to be precautionary), which are outside EFSA's remit. Nevertheless, it seems prudent to communicate both a central estimate and range, and to distinguish clearly between these, because this indicates which part of the range is most likely (except in rare cases, when the underlying distribution is multimodal) and provides more understanding for the recipients
	6	Reasoning: The term ‘best’ has a normative content, as it embeds already a preference for a value, whereas ‘central’ just communicates the place of that value in a set of relevant values
	7	Evidence: Several studies have shown that people perform poorly at inferring central estimates from graphical representations of distributions (e.g. Ibrekk and Morgan, 1987)
	8	Reasoning: It is essential to provide explanation and background on the concept of uncertainty and the methods used for expressing it, and of any unavoidable technical terms. Providing this in the form of general FAQs on EFSA's website avoids repeating the same information in every communication, which would be impractical and result in significant risk of inconsistencies between communications
3.1.4 Precautions when using numbers	1	Evidence: Hedging is understood in different ways by different users, with important inter‐individual variation and overlap. Furthermore, their use is modulated by context, including the magnitude of the number, its roundness and even its units (Ferson et al., 2015)
	2	Reasoning: The potential for this type of misunderstanding was raised in a comment received from the public consultation. Careful wording is the only practical option for countering it
	3	Reasoning: Referring to % certainty about an outcome that is expressed in % (e.g. % of people affected) may cause confusion for some readers
	4	Reasoning and evidence: The risk of this type of misinterpretation in the example of hazard has been raised by EFSA experts and others on several occasions and instances of it were observed during the trial period for EFSA's Uncertainty Analysis GD
	5	Reasoning: most people are more familiar with the use of ranges and graphs to communicate variability, so may not recognise when they are used for uncertainty unless this is communicated clearly
3.1.5 Describing sources of uncertainty	1	Evidence: Many publications emphasise the importance of specifying the sources of uncertainty and/or describing their nature (Budescu et al., 2009; also based on Kuhn, 2000; Maxim et al., 2012; Maxim and Mansier, 2014; Visschers, 2018; Wardekker et al., 2008). Also, people's response to uncertainty depends in part on what caused the uncertainty (Frewer et al., 2002; Miles and Frewer, 2011)
	2	Reasoning: assessors and risk managers have in the past raised concerns that communication of conclusions might be undermined if accompanied by description of sources of uncertainty, so it needs to be communicated clearly that conclusions take the uncertainties into account
3.1.6 Addressing the uncertainties	1 and 2	Evidence and reasoning: Frewer et al. (2002) found that uncertainty associated with the scientific process was more readily accepted than uncertainty due to lack of action of the government. This suggests that communication of uncertainty is less likely to cause public alarm if it is accompanied by information on what actions are being taken by the relevant authorities to address that uncertainty. While such actions are risk management measures and are therefore outside the remit of EFSA, as is current practice, such communication may take place if carried out in coordination with decision‐makers (e.g. the European Commission)
	3	Evidence and reasoning: Frewer et al. (2002) found that uncertainty associated with the scientific process was more readily accepted than uncertainty due to lack of action of the government. Investigations aimed at reducing uncertainty are one type of action that can be taken. However, it is important to make distinguish actions that have already been decided (e.g. in areas in which EFSA has the authority to set data requirements) and those that are options for risk managers to consider
3.1.7 Guidance for assessors	1	Reasoning: This is necessary to ensure communicators can interpret correctly what the scientific conclusion refers to and convey this correctly in their communications
	2	Reasoning: this is a generally accepted principle in science, to avoid implying more precision than is justified
	3	Evidence: Multiple studies show that interpretation is more variable for verbal probabilities than numeric probabilities (see Sections 4.1.4.3 and 4.1.4.4). Some of these show that, when both verbal and numeric probabilities are presented together, interpretation is less variable when the numeric probability is presented first (e.g. Jenkins et al., 2018)
	4	Evidence and reasoning: the findings of Gigerenzer and Hoffrage (1995), Gigerenzer et al. (2007), and other publications) show that people (including doctors and people with statistical training) perform very poorly at deducing probabilities of interest from conditional probability information and perform much better if provided with information in the form of frequencies. It is better to avoid the problem altogether by providing frequencies for the outcome of interest directly and not requiring readers to make deductions from conditional information
	5	See Evidence and reasoning for 3.1.3 point 5 (above)
	6	Evidence: when receiving a message such as ‘more than X’, people tend to infer extreme values (Teigen (2008), Hohle and Teigen (2018), Hoorens and Bruckmüller (2015))
	7	Reasoning: This is desirable because it requires assessors to draft an expression of the uncertainty that is concise enough to serve as a starting point for crafting communications, without the communicators having to interpret it (and potentially misinterpret it) from more detailed technical material. It also makes the information more accessible to the communicators
Box  1	Reasoning: Standardised procedures usually have standard language for expressing their conclusions (e.g. no concern), which should be used also in communications. If there is a requirement for conclusions to be unqualified, then this should be respected by both assessors and communicators Reasoning: If there is not a requirement for the conclusion to be unqualified then any information about uncertainty that is provided by the assessors may optionally be included in the communication, following the recommendations applicable to the forms of expression involved
Box  2	Evidence: Many publications emphasise the importance of specifying the sources of uncertainty and/or describing their nature (Budescu et al., 2009; also based on Kuhn, 2000; Maxim et al., 2012; Maxim and Mansier, 2014; Visschers, 2018; Wardekker et al., 2008) Also, people's response to uncertainty depends in part on what caused the uncertainty (Frewer et al., 2002; Miles and Frewer, 2011) Reasoning: The relative importance of uncertainties is important information for targeting further analysis or research, when needed. Conflicting evidence is an important source of uncertainty (as stated in a comment to the public consultation) but is not explicitly listed in EFSA Scientific Committee (2018a,b). The opportunity is therefore taken to highlight it here and refer assessors to EFSA's guidance on weight of evidence assessment, which addresses it directly
Box  3	Reasoning: This form of expression is discouraged by the Uncertainty Analysis GD due to its ambiguity. If the assessors have nevertheless used such expressions, then it is important that the communicators avoid rewording them in ways that the assessors would consider incompatible with their judgement of the uncertainty It seems likely that the informed‐level audience would like to receive some description of the uncertainties that contribute to the qualitatively described magnitude, and how they were assessed. Various qualitative scales have been proposed for expressing uncertainty, confidence or the strength and quality of evidence, but the Scientific Committee is not aware that any of these have a formal operational definition of the type that exists for probability itself (see section 5.10 of EFSA Scientific Committee, 2018b). Therefore, qualitative scales and expressions are not recommended in EFSA Scientific Committee (2018a) or the present guidance for expressing the overall uncertainty of assessment conclusions. However, they can be useful for expressing individual sources of uncertainty and in weight of evidence assessment (EFSA Scientific Committee, 2017), provided the overall uncertainty of the conclusion is expressed quantitatively when possible Evidence: Findings from EFSA studies support the use of general qualitative statements on the direction of the uncertainties in communication aimed at the informed‐level and technical‐level audiences. The +/− table currently used in some EFSA assessments is considered helpful by some users, but limited or misleading by others including some users with high scientific literacy. Providing quantitative definitions will reduce ambiguity when such scales are used
Box  4	Reasoning: If the assessors were unable to make any conclusion then it is essential that the communication should reflect this clearly and not use any language that implies more is known When there is this much uncertainty, it seems prudent to give some indication of the reasons for it at entry level as well as informed level. At the informed level it seems desirable to mention also any options that have been identified for reducing uncertainty
Box  5	Most unqualified conclusions follow a standard wording that has special meaning and should therefore be retained in all communications. For transparency, the special meaning should be made explicit at the informed level and/or in a linked FAQ. To provide this, actions are needed by assessors and risk managers at the technical level It seems likely that the informed‐level audience may want to have access to a concise plain language explanation of how the conclusion was reached (evidence and methods), and of the types of uncertainty that are present, and to know that the latter were taken into account in reaching the conclusion
Box  6	Reasoning: For entry level, see reasoning for general guidance It seems likely that the informed‐level audience may want to have access to a concise plain language explanation of how the conclusion was reached (evidence and methods), and of the types of uncertainty that are present; to know that the latter were taken into account in reaching the conclusion; and to have a plain language outline of how that was carried out
Box  7	Evidence: Many studies have shown that using verbal expressions to communicate approximate probabilities results in widely varying interpretations, and that this is reduced when the numeric range is presented alongside the verbal expression (Patt and Schrag, 2003; Patt and Dessai, 2005; Budescu et al., 2009; Harris and Corner, 2011; Budescu et al., 2012, 2014; also based on Wallsten et al., 1986; Mosteller and Youtz, 1990) Reasoning: It is recommended to communicate the quantitative range of probabilities, because this expresses the assessors’ conclusion without ambiguity. Although other options may be simpler (e.g. replacing 66‐90% with ‘likely’ or ‘about 80%’), and there was some indication from the EFSA studies that more subjects preferred them, they will be interpreted by different people in different ways. If a simpler option is used at the entry level, it is essential that the assessors’ range of probabilities is provided at the informed level, so that the meaning is clear to those who read both Reasoning: It seems likely that the informed‐level audience may also want to have access to a concise plain language explanation of the basis for the approximate probability (evidence and methods), and of the types of uncertainty that were taken into account
Box  8	Evidence and reasoning: Some studies in the literature review compare alternative means for visual representations of probability distributions. In summary, box plots appear to be best understood, although they only provide partial information about distribution shape. CDFs and PDFs provide more information but will be badly misinterpreted by some people. Therefore it is proposed to use only box plots in communications except at the technical level, in which the box plot may be accompanied by a PDF and/or CDF if this is useful to communicate more information about distribution shape (PDF) or enable readers to read off additional quantiles (CDF); in all cases all graphics should be well labelled and the key results should be marked on them Experience at EFSA has shown that it is important to make clear when box plots and PDFs are used to communicate uncertainty rather than variability, as people are more familiar with the latter use There is evidence that the choice of quantiles to report will influence how people respond (their perception of the uncertainty) but it is unclear whether this implies a failure to understand what is communicated or subsequent interpretation by the recipient. There is a trade‐off between giving excessive weight to extreme values and understating the uncertainty. Also, it is unlikely to be prudent for EFSA to limit reporting to ranges that are expected not to contain the outcome in a substantial proportion of cases. Therefore, it is proposed in general to communicate the median together with both the 50% and 95% range (the latter because this is conventionally used in science). In addition to this, results for specific values or quantiles of interest when these are known
Box  9	Reasoning: None of the selected studies examined the communication of 2D probability distributions, but it is expected that they will be less well understood than 1D distributions. Therefore it is proposed to focus communication for entry and informed levels on selected results extracted from the 2D distribution, and communicate quantiles for these numerically at entry level with the addition of box plots at informed level 2D distributions should be used only at the technical level, and then with great care and explanation

Appendix C – EFSA studies design overview

1.

EFSA commissioned a target audience research study in 2016 and carried out its own follow‐up study in 2017 when preparing for this Uncertainty Communication GD. Both studies have limitations in their design and conduct which restrict their use. Nevertheless, used cautiously, they are a rich source of insights and indications retrieved directly from the key target audiences of EFSA's communications: officials in the EU Institutions and EU Member States, civil society organisations, members of the public, the scientific community, media and food industry operators.

C.1. Focus group study (Etienne et al., 2018)

Evidence was collected from selected representatives of EFSA's key partners in the EU institutions and Member States and stakeholders, who completed a questionnaire individually then discussed their responses in five focus groups of 6‐10 participants (39 in total). After seeing a one‐page summary of a scientific assessment (based on an EFSA assessment of the natural toxins T‐2 and HT‐2), the participants were provided different information about the resulting uncertainties and asked to answer a series of questions. This took place during the written part of the test. They discussed the results in the subsequent focus group stage with a moderator.

The uncertainty information statements were as follows (A–G):

• ASources of uncertainty taken into account by the EFSA scientists in their assessment of health risks posed by the sum of T‐2 and HT‐2 toxins in food.

  Sources of uncertainty	Directiona
  Uncertainties in the analytical measurement of T‐2 and HT‐2 in foods	+/−
  Effect of food processing	+/−
  Use of highest and lowest toxin level estimates in the exposure estimations	+/−
  Limited data on exposure for infants	+/−
  Limited data on exposure for vegetarians	+/−
  No data on absorption, metabolism and elimination of T‐2 and HT‐2 toxins in humans	+/−
  Lack of information on the contribution of the toxicity of HT‐2 toxin and other metabolites to overall toxicity	+/−
  Limited data on combined effects with other mycotoxins or other toxic substances in food	+/−

  ^a

  The ‘+’ and ‘−’ signs indicate only in which direction uncertainty affects the risk assessment.

  + = uncertainty with potential to cause overestimation of exposure/risk.

  − = uncertainty with potential to cause underestimation of exposure/risk.

• BTaking account of the uncertainties, EFSA's experts concluded that the risk assessment of human exposure to the sum of T‐2 and HT‐2 toxins is more likely to over‐ than underestimate the risk.
• CTaking account of the uncertainties, EFSA's experts concluded that there is a more than 50% chance that the risk assessment of human exposure to the sum of T‐2 and HT‐2 toxins overestimates the risk, and a less than 50% chance the risk is underestimated.
• DTaking account of the uncertainties, EFSA's experts concluded that it is unlikely that exposure for the highest consuming toddlers exceeds the safe level. In other words, it is likely that exposure for the highest consuming toddlers is below the safe level.
• ETaking account of the uncertainties, EFSA's experts concluded that it is unlikely (10–33% probability) that exposure for the highest consuming toddler exceeds the safe level. In other words, it is likely (66–90% probability) that exposure for the highest consuming toddler is below the safe level.
• FTaking account of the uncertainties, EFSA's experts concluded that there is about 20% probability (1 in 5 chance) that exposure for the highest consuming toddler exceeds the safe level. In other words, there is about 80% probability (4 in 5 chance) that exposure for the highest consuming toddler is below the safe level.
• GTaking account of the uncertainties, EFSA's experts concluded that there is about 75% probability (3 in 4 chances) that the exposure for the highest consuming toddler is between 23 and 91 ng/kg bw per day. In other words, there is about 25% probability (1 in 4 chances) that the exposure for the highest consuming toddler is either below 23 or above 91 ng/kg bw per day.

The participants answered the following questions multiple times for each uncertainty statement, qualifying their answers (e.g. strongly agree, agree, disagree, strongly disagree). The full list of questions is available in the report (Etienne et al., 2018). An overview of the key questions follows below:

1. To what extent do you agree or not with the following statement: ‘It is normal that scientific advice is subject to a degree of uncertainty’?

2. Having read the summary of EFSA's opinion, how much confidence do you have in EFSA to give you accurate information about the risk to human health posed by T‐2 and HT‐2?

3. Considering that additional piece of information, how likely do you think it is that any toddler will experience ill health caused by T‐2 and HT‐2?

4. How does this additional piece of information influence how much confidence you have in EFSA to give you accurate information about the risk to human health posed by T‐2 and HT‐2?

5. On a scale from 1 to 4 (1 = strongly agree; 4 = strongly disagree), please indicate whether you find this statement: Understandable; Informative; or Useful.

The participants were also asked to compare statements B and C (expressing effects of combined uncertainties on risk assessment), and also statements D, E and F (describing the uncertainties qualitatively and quantitatively) and to answer questions about their preferences among these statements.

C.2. Online survey study (EFSA, 2018)

The second study (EFSA, 2018) was undertaken to further test the initial findings with an increased sample size. The authors adapted materials from the first study and conducted an online survey in six EU languages. The survey was designed by EFSA staff/experts and run on the EU Survey platform hosted by the European Commission's DG DIGIT. EFSA and eight members of its Communication Experts Network, composed of communication representatives from the EU Member State food safety authorities, helped to translate and promote the survey, generating over 1,900 responses.

As with the focus group study, the respondents were given a short text describing the risk assessment of a natural toxin found in cereals. They were then asked the following questions in relation to the information on the uncertainties:

1. The outcome of EFSA's uncertainty assessment for the natural toxin can be described in various ways. Here are three examples. Please rank the three additional sentences by how helpful they are for understanding the risk.

   1. EFSA's scientists said that, based on what we know, the assessment is more likely to over‐ than underestimate the risk for the most exposed group of consumers and therefore the actual risk is more likely to be lower than estimated.

   2. EFSA's scientists said that, based on what we know, it is likely that exposure of the most exposed group of consumers is within the safe level.

   3. EFSA's scientists said that, based on what we know, there is about an 80% chance that exposure of the most exposed group of consumers is within the safe level.

   4. Depending which of the answers A, B or C they ranked highest they were then asked to consider the clarity of some alternative wordings of their first ranked statements.

2. Please rank these three expressions in order of the clarity of information.

   1. EFSA's scientists said that, based on what we know, the assessment is more likely to over‐ than underestimate the risk for the most exposed group of consumers and therefore the actual risk is more likely to be lower than estimated.

   2. EFSA's scientists said that, given the uncertainties, the assessment is more likely to over‐ than underestimate the risk for the most exposed group of consumers and therefore the actual risk is more likely to be lower than estimated.

   3. EFSA's scientists said that, based on what we know, there is a more than 50% chance that the assessment overestimates the risk for the most exposed group of consumers and therefore there is a more than 50% chance that the actual risk is lower than estimated.

3. Please rank these three expressions in order of the clarity of information.

   1. EFSA's scientists said that, based on what we know, it is likely that exposure of the most exposed group of consumers is within the safe level.

   2. EFSA's scientists said that, given the uncertainties, it is unlikely that exposure of the most exposed group of consumers exceeds the safe level.

   3. EFSA's scientists said that, based on what we know, it is likely (66–90% probability) that exposure of the most exposed group of consumers is within the safe level.

4. Please rank these three expressions in order of the clarity of information.

   1. EFSA's scientists said that, based on what we know, there is about an 80% chance that exposure of the most exposed group of consumers is within the safe level.

   2. EFSA's scientists said that, given the uncertainties, there is about a 20% chance that exposure of the most exposed group of consumers exceeds the safe level.

   3. EFSA's scientists said that, based on what we know, there is about a 4 in 5 chance that exposure of the most exposed group of consumers is within the safe level.

They were also asked how useful they thought their preferred form of uncertainty information was.

Regarding your selected top ranked uncertainty statement, please rate the usefulness of this information on uncertainties: [Please select one answer only]

• Essential (must always be mentioned)

• Important (should be mentioned)

• Helpful (nice to have)

• Unnecessary (should only be mentioned on request)

• Unhelpful (should never be mentioned)

Other questions asked at the beginning and end of the survey provided information about professional/geographical background, the wish for further information (e.g. about the hazard, about the risk assessment) to provide a more detailed indication of the profiles of respondents.

Suggested citation: EFSA (European Food Safety Authority) , Hart A, Maxim L, Siegrist M, Von Goetz N, da Cruz C, Merten C, Mosbach‐Schulz O, Lahaniatis M, Smith A and Hardy A, 2019. Guidance on Communication of Uncertainty in Scientific Assessments. EFSA Journal 2019;17(1):5520, 73 pp. 10.2903/j.efsa.2019.5520