Evaluating science communication

Abstract

Effective science communication requires assembling scientists with knowledge relevant to decision makers, translating that knowledge into useful terms, establishing trusted two-way communication channels, evaluating the process, and refining it as needed. Communicating Science Effectively: A Research Agenda [National Research Council (2017)] surveys the scientific foundations for accomplishing these tasks, the research agenda for improving them, and the essential collaborative relations with decision makers and communication professionals. Recognizing the complexity of the science, the decisions, and the communication processes, the report calls for a systems approach. This perspective offers an approach to creating such systems by adapting scientific methods to the practical constraints of science communication. It considers staffing (are the right people involved?), internal collaboration (are they talking to one another?), and external collaboration (are they talking to other stakeholders?). It focuses on contexts where the goal of science communication is helping people to make autonomous choices rather than promoting specific behaviors (e.g., voter turnout, vaccination rates, energy consumption). The approach is illustrated with research in two domains: decisions about preventing sexual assault and responding to pandemic disease.

Communicating science effectively can require an unnatural act: collaboration among experts from professional communities with different norms and practices. Those experts include scientists who know the subject matter and scientists who know how people communicate. They include practitioners who know how to create trusted two-way communication channels and practitioners who know how to send and receive content through them. They also include professionals who straddle these worlds, such as public health officials managing pandemics and climate scientists defending their work. Communicating Science Effectively: A Research Agenda (1) calls for a systems approach to recruiting and coordinating individuals with these skills and connecting them with those whom they might serve (ref. 1, pp. 8 and 84–86).

This perspective offers an approach to answering that call. It is grounded in the research presented in the report and at the National Academies’ Colloquia on the Science of Science Communication (2–4). It is designed to accommodate the limited resources of many organizations engaged in science communication. To that end, it proposes simplified versions of scientific methods that should degrade gracefully in practical applications.

The approach’s conceptual framework is grounded in Herbert Simon’s two general strategies (5, 6) for addressing complex problems. One is “bounded rationality,” looking for the best possible solution to a manageable subset of a problem, while deliberately ignoring some aspects. The second is “satisficing,” looking for an adequate solution while considering all aspects. Both strategies rely on heuristics to identify potentially useful strategies. With bounded rationality, heuristics indicate which aspects to ignore and how to optimize within those bounds. With satisficing, heuristics indicate how to generate and evaluate integrative solutions. To a first approximation, scientists pursue bounded rationality, whereas practitioners satisfice. However, this characterization, like Simon’s distinction (5, 6), is a heuristic one. It captures some general features while breaking down in ways that reveal the communities’ need for one another.

Scientists’ bounded rationality entails ignoring issues that they cannot treat systematically, hoping to reach strong conclusions within their discipline’s self-imposed constraints. Scientists from different disciplines struggle to collaborate, because they bound problems differently. Experimental researchers may be uncomfortable with unruly field observations. Field researchers may question the artificial conditions of experiments. Both may puzzle over computational models, while modelers may have little patience for the simplification of experiments or the qualitative evidence of field research. Scientists who study individuals may not know what to do with the context provided by those who study groups or cultures, who may shake their heads at being ignored. Each discipline owes its success to its tacit knowledge of how to work within its bounds. Those bounds can be so incommensurable that scientists from different disciplines struggle even to agree about how to disagree (7, 8). Nonetheless, as argued by Communicating Science Effectively: A Research Agenda (1), the success of science communication depends on collaboration across disciplines.

Practitioners’ satisficing entails paying attention to anything that might be relevant and accepting imperfect solutions. Practitioners of different persuasions struggle to collaborate, because they have different skills and norms. Those skills might include designing visual materials, crafting text, attracting media attention, convening stakeholders, and branding programs. Those norms might include how relevant they find social science evidence, whether they subscribe to a design philosophy, and what their professional code of ethics is. Their organizing design constructs maybe so different that they effectively speak different languages. Nonetheless, as argued in Communicating Science Effectively: A Research Agenda (1), serving diverse audiences and decisions requires practitioners with diverse expertise.

When these two worlds fail to connect, each is the worse for it. Scientists can overestimate how far their results generalize and offer practitioners unsupported advice or summaries. Practitioners can absorb a fragment of science and exaggerate its value. Scientists can unwittingly or naively let their values color their research or expositions. Practitioners can selectively pursue or accept convenient truths. Conversely, the two worlds support one another when they do connect, with practitioners helping scientists to identify the results that matter to their audiences and scientists helping practitioners to structure those interactions (9–11).

Ashby (12) introduced the term “requisite variety” to describe solutions that were as complex as the problems that they address. It is a heuristic construct for systems without easily characterized dimensions. However, it evokes the challenge facing a systems approach to science communication. The effort may fail unless the requisite sciences and practices are recruited and coordinated. As a result, the present approach focuses on identifying and connecting those needed elements. It recognizes that many organizations may lack not only many of the requisite skills but also “absorptive capacity,” the expertise needed to recruit potentially useful help (13). As a result, it concludes with a discussion of boundary organizations that can facilitate these matches (14–16).

Thus, this proposal offers practical ways for organizations with limited material resources and expertise to use the sciences of science communication. It assumes that a systems approach needs both bounded rationality and satisficing. For the former, the approach asks whether a system has the right set of bounded parts (disciplinary sciences). For the latter, it asks whether those parts are connected so as to produce satisficing solutions. It could be used to design systems or to audit them. Its methods are, in part, simplified versions of those used in scientific research. It illustrates the approach with two cases studies, showing variants of these decisions and consultations.

Communication Goals

Communication programs that seek to change observable behavior (e.g., voter turnout, energy consumption, immunization rates) might be judged by their outcomes. However, as Communicating Science Effectively: A Research Agenda (1) notes, sometimes outcomes are hard to observe (e.g., changes in values, aesthetics, feelings of self-efficacy). Moreover, sometimes the goal is not to effect specific changes but to empower people to make better informed choices. In such cases, successful communications might lead to different behaviors for people with different values or circumstances.

In principle, a science communication’s success might be evaluated by whether it produced choices closer to the ones that fully informed decision makers would make. Satisfying that condition is the goal (and ethical commitment) of libertarian paternalist interventions, which try to manipulate individuals’ “choice architecture” toward such choices (17). Thus, employees are nudged to place their retirement savings in stocks only when it has been determined that the expected financial returns outweigh the psychological costs of experiencing market corrections and the economic risks of being in one when funds are needed. Organ donation is made the default only for individuals whose survivors will accept that choice without having had a family consultation (18). Social norms are invoked for health behaviors (e.g., exercise, vaccinations, diet) only when people have the resources to follow them and a safety net should things go wrong.

Analysis and Its Limits

In practice, however, such personalized decision analyses are rare outside of medicine, where they have been conducted for many conditions and treatments (19). So that these analyses can capture the full range of patient concerns, the field has invested heavily in validated quality-of-life measures, patient cost estimates, and utility assessments (20, 21). That research has guided the design of interactive aids, which allow patients to select information and examine its implications for their personal decisions (22). It has also guided policy analyses, evaluating treatment protocols (23, 24).

Such formal decision analyses can reveal issues that more casual analyses might miss. For example, they may find that a generally valued outcome (e.g., money, pain) does not matter in a specific decision, because it is similar for all options (25). Formal analyses can show how seemingly technical definitions (e.g., “risk”) affect choices (26). They can frame ethical issues, such as whose preferences should shape policies—those who have a condition (e.g., paralysis) or the general public, which supports the health care system (27). They can ask when people lack the cognitive competence or emotional strength for making decisions (28). However, formal analyses require expressing all outcomes in numeric terms (e.g., costs, risks) in order to compute expected outcomes. As a result, they privilege outcomes that are readily quantified. They also require skills that few organizations have or can afford.

A more feasible aspiration is to adopt the logic of analysis but not its mechanics. That is, define the terms of a decision precisely enough that a technically adept analyst could “run the numbers” were the data available, but not require that to happen. Creating an analytical model that is “computable,” in this sense, demands clear thinking but no calculations. As a result, anyone should be able to create and understand one (29). Such qualitative formal models can serve parties with different needs. Scientists can see where their boundedly rational evidence fits into the big picture. Practitioners can look for satisficing solutions, addressing the overall problem. Decision makers can readily check that their concerns have been addressed. Having clearly defined variables and relationships helps ensure that the parties are talking about the same things.

One example of such a model is the benefit–risk framework of the Food and Drug Administration (FDA) (30, 31), created to improve communication among the parties involved in evaluating pharmaceuticals and biologics. Those parties include FDA technical reviewers and regulators, industry researchers and managers, patients, and advocates. The framework has rows for the five inputs to FDA’s regulatory decisions (medical condition, existing treatments, expected benefits, expected risks, and how risks will be managed if the product is approved). The framework has columns for FDA’s two bases for evaluating those inputs (science and law) and a narrative summary. Cell entries can be words or numbers and are meant to capture disagreements and uncertainties, reflecting FDA’s philosophy of having analysis inform rather than replace judgment (32).

Similarly spirited qualitative formal models are central to implementing the present proposal.

A Theory of Change for Science Communication

Social programs often reflect a theory of change (33, 34), identifying the elements deemed necessary for success. Although each element might be the subject of boundedly rational theories, a theory of change is not a scientific theory. Rather, it is a satisficing proposal, expressing an integrative vision of how a complex process works. The present proposal is a theory of change for science communication. It asks three questions.

• Staffing: Are the right people involved?

• Internal consultation: Are they talking effectively with one another?

• External consultation: Are they talking effectively with other stakeholders?

The next section describes one complex decision of the sort that effective science communication could inform. Subsequent sections offer ways to answer these three questions. Although the questions are addressed in the order above, the process itself could start anywhere and is inherently iterative. External stakeholders could ask for help, triggering internal consultation that reveals missing skills. Staff could perceive a need, initiative a campaign, and, then, find themselves welcomed, rejected, or redirected (1, 9, 35).

One Complex Decision

In a project that was pivotal to my own thinking, Lita Furby, Marcia Morgan, and I sought to aid decisions related to sexual assault by better communicating relevant scientific evidence (36). The prompt for our work was observing the confident, universal, and contradictory advice offered to women regarding whether to resist physically when attacked. After reviewing the (limited) evidence on the efficacy of self-defense measures, we concluded that, if they knew research, women might differ in the strategies that they chose. As a result, we needed to understand those differences. As befits any communication project, we began by listening. Here, that entailed semistructured interviews with diverse groups of women, men, and experts, eliciting their perspectives on both personal decisions (e.g., how to respond to an assault) and societal ones (e.g., how to make assaults less likely).

These interviews revealed a rich decision space, with many possible options, outcomes, and uncertainties (37, 38). To structure that space, we created categories of options and outcomes, seeking a level of granularity that would be useful to decision makers. Within that structure, we summarized available research on the effects of the options on the outcomes. Where the evidence was limited, as was usually the case, those limits were part of the story. When uncertainty is great, advice in unproven. Unless those limits are acknowledged, if things go badly, then decision makers may bear the insult of blame and regret in addition to the injury that occurred. Whatever they did, some “expert” had advised otherwise.

Given that uncertainty and the diversity of decision makers’ circumstances, our project had no theory of change for encouraging specific behaviors among those concerned with sexual assault. However, we did have a theory of change for ourselves, structuring our efforts to inform those decisions. That engagement and subsequent ones have led to the theory of change that guides this proposal for implementing the recommendation of a systems approach in Communicating Science Effectively: A Research Agenda (1).

Staffing: Are the Right People Involved?

Effective science communications must contain the information that recipients need in places that they can access and in a form that they can use. Achieving those goals requires four kinds of expertise (39).

Subject Matter Experts.

The core of any science communication is authoritative summaries of evidence relevant to decision makers’ needs. That evidence may come from many sciences. For example, sexual assault decisions might be informed by results from psychology, sociology, criminology, and economics. Unless staff have expertise in an issue or the capacity to absorb it (13), they will have to ignore or guess at it (26, 40).

Decision Scientists.

Eager to share their knowledge, subject matter experts may drown decision makers in facts that it would be nice to know. Decision scientists can identify the facts that decision makers need to know. They can also characterize evidence quality, estimate decision sensitivity, and reveal hidden assumptions (16, 17, 30, 31). For example, a sensitivity analysis of the decision faced by a young academic might conclude that “there is no sure way to prevent a powerful figure in your field from destroying your career.” A decision analysis of sexual assault advice might conclude that “it ignores restrictions on your freedom.”

Social, Behavioral, and Communication Scientists.

Knowing what to say does not guarantee knowing how to say it. Coupled with the normal human tendency to overestimate mutual understanding (41), scientists’ intuitions can be a poor guide to effective communication. Indeed, scientists’ success in the classroom may produce unrealistic expectations for being able to communicate with general audiences, with no examinations providing feedback on their success. Communicating Science Effectively: A Research Agenda (1) identifies the diverse expertise available for understanding audiences, crafting communications, and evaluating success (2–4, 35, 42, 43).

Program Designers and Managers.

Finally, science communication needs practitioners to create channels, recruit stakeholders, disseminate messages, mind legal constraints, anticipate cultural sensitivities, and collect feedback. Practitioners are also needed to manage the process, secure the relevant experts, and get them talking with one another and external stakeholders. Without a firm hand, normal group dynamics can lead to recruiting, rewarding, and retaining people with similar backgrounds and blind spots, who are overly comfortable talking to one another. A firm hand is also needed to let everyone offer opinions, while leaving ultimate authority to those most expert in a topic. That will keep subject matter experts from editing for style rather than accuracy, social scientists from garbling the facts when trying to clarify them, and practitioners from spinning messages when the facts are needed.

Internal Consultation: Are They Talking Effectively with One Another?

Experts must combine their knowledge to realize its value. That means jointly examining issues, connections between issues, and the assumptions underlying those interpretations. Fig. 1 illustrates a decision science tool for structuring such consultations (44). A computable (i.e., nonnumeric) version of an influence diagram (45), it depicts actions as rectangles and uncertain variables as ovals (gray if valued outcomes; white if intermediates). It was created to structure discussions at a meeting about the then-pending threat of H5N1 (avian flu). It has places for the science that could inform decisions faced by health officials, employers, parents, suppliers, and others, each wondering if and how to prepare for a possible pandemic. What should they expect regarding quarantine, home schooling, rationing, hospital closures, telecommuting, drug shortages, and social solidarity (or fracture)?

Fig. 1.

Risk model for pharmacological interventions for a pandemic. Ovals indicate uncertain variables, which need to be predicted. Rectangles indicate actions, which need to be planned and implemented. Reprinted by permission from ref. 29, Springer Nature: Journal of Risk and Uncertainty, copyright 2006.

Translating science into such a decision-relevant form requires consultation on three levels. One is summarizing the science at each node (e.g., what quantities of antivirals will be available) and link (e.g., how effectively will vaccines reduce morbidity). The second is estimating interactions (e.g., how will morbidity and mortality combine to affect social costs). The third is identifying contextual factors (sometimes called “index variables”) that affect many model elements (e.g., is the society developed or developing) (40).

Quantifying such models demands technical training and material resources. However, sketching a model well enough to facilitate consultations only requires clear thinking and candid conversation. To that end, before the H5N1 meeting, participants completed a survey eliciting their beliefs about the issues in Fig. 1 and several related models (46). The models were instantiated with scenarios to make their abstractions concrete. The meeting and survey were anonymous so that participants could work the problem without pressure to represent the firms or agencies in which many held senior positions. They were drawn from public health, technology, and mass media; hence, they could offer their views on the needs and responses of publics that they might support in a pandemic, but typically know in more benign circumstances.

These scientists and practitioners were brought together because interpreting such evidence requires actual conversation. It is not enough for members of one field to read the publications of another. Publications reflect their authors’ perspectives and not those of their entire field. They omit assumptions that go without saying when scientists or practitioners write for colleagues or clients. For scientists, those assumptions include bounds on their discipline’s rationality. For practitioners, they include accepted limits to satisficing solutions.

One practical method for describing these internal consultations is social network analysis, created by asking members of a communication team to describe their relationships (47, 48). A successful team will have the requisite connections among those associated with each link and node in the relevant models. Fig. 2 shows such relationships as revealed in self-reports of “close and collegial relations” in a study of six interdisciplinary research centers (49). For this center, the study concluded that “most…interactions are concentrated in a small core of researchers…Disciplines from the physical sciences dominate the core…, environmental scientists/social scientists dominate the periphery” (ref. 49, p. 57). A theory of change for a communication team would specify which members must talk with one another, with a diagram measuring its success. Of course, even parties who view their relationships as close and collegial may not identify and correct all misunderstandings.

Fig. 2.

Network diagram of self-reported close and collegial relationships among members of an interdisciplinary research program supported by the National Academy of Sciences. Reprinted with permission from ref. 49.

External Consultation: Are They Talking Effectively with Other Stakeholders?

Scientific communicators seek to be trusted partners of people making decisions where science matters. In this issue of PNAS, those include decisions about gene drives (50), autonomous vehicles (51), employment (52), and energy (53). Earning that trust means providing the science most relevant to decision makers’ valued outcomes in comprehensible form and accessible places. Existing research is the natural source of initial guidance for accomplishing those tasks (1–4, 41–43, 54).

However good the research, communicators must still consult with their external stakeholders. It is unrealistic to expect them to know how people very different from themselves view their world or the communicators (8, 41, 55). Even if those consultations only affirm what the research says, they are important as “speech acts.” They show respect for the stakeholders as individuals worthy of knowing and hearing. They need to occur throughout the process to maintain the human contact and so that communicators know what is on stakeholders’ minds and stakeholders know what communicators are doing (1, 9, 35, 39). However, making that happen can be challenging, especially with diverse, dispersed, and disinterested publics.

Our sexual assault project adopted one imperfect approach (36–38). It asked nonrepresentative samples of individuals recruited from diverse groups to complete open-ended surveys, allowing them to choose the issues and describe them in their own terms. These surveys were followed by confirmatory structured ones with similarly sampled individuals. Although we engaged diverse individuals who revealed a wide range of views, the consultation was indirect. Our avian flu project involved two days of intense direct consultation, building on a preparatory survey. However, it was with a highly select group experienced in sampling public opinion but not authorized to represent it.

The medical world, with its traditions of informed consent and shared decision making (22–24, 28), offers examples that might be adapted to other settings. For example, to secure patient input to its benefit–risk framework (30, 31), FDA created the Voice of the Patient Initiative (56), with daylong exchanges on critical issues (e.g., chronic fatigue syndrome, sickle cell disease)—a model that some patient groups have adopted. The desire to include patient experiences in clinical trials led to including self-reported quality-of-life measures as outcomes. However, the result was a proliferation of measures with varying content and quality that undermined the research effort. In response, NIH created an inventory of psychometrically validated measures, freely available online, with adaptive testing for efficient administration (20, 21). Recognizing the importance of evaluating communications, Communicating Risks and Benefits: An Evidence-Based User’s Guide published by FDA (43) ends each chapter with guidance on evaluation for no resources, modest resources, and resources commensurate with the personal, economic, and political stakes riding on good communication. It simplest method is the think-aloud protocol, asking people to say whatever comes into their minds as they read draft materials (57, 58).

Any communication that goes to a broader audience constitutes an indirect consultation, as recipients assess their relationships with its source, based on what they infer about its competence and trustworthiness. Fig. 3 shows section headings from an attempt to create a relationship that neither abandons recipients nor provides unsupportable advice. It was written when editors of a journal (American Psychologist) refused to publish a review that was critical of much customary advice without providing an alternative. It was my hope that it would be seen as respecting and empowering recipients.

Fig. 3.

Advice for reducing the risk of sexual assault (36).

Conclusion

Science communications succeed when recipients make better decisions. Applying that standard means evaluating the optimality of choices made with and without the communications. With complex decisions and diverse decision makers, such evaluation is typically infeasible. The alternative is asking how well the communication process has followed a theory of change. The present proposal offers a theory of change embracing a systems approach as advocated by Communicating Science Effectively: A Research Agenda (1). It entails staffing with the right people, internal consultation among them, and external consultation with those whom they seek to serve. It embraces both the bounded rationality of disciplinary scientists and the satisficing of practitioners.

Its proposed procedures rely on simplified versions of scientific methods adapted for use by organizations with limited resources. They include think-aloud protocols, network analyses, and qualitative formal analyses, precise enough to allow quantitative analysis were data requirements met, but not requiring it. The proposal assumes that anyone can create, critique, and discuss a decision space with options and valued outcomes; an influence diagram with the factors determining those outcomes (Fig. 1); and a social network depicting work relations (Fig. 2). Adopting science-like methods should also facilitate commissioning analyses from professionals when circumstances warrant and resources allow (59).

These methods all assume a world where, in the words of Communicating Science Effectively: A Research Agenda (1), “researchers and practitioners…form partnerships” and “researchers in diverse disciplines…work together” (ref. 1, p. 9). There are precedents for creating boundary organizations hospitable to such partnerships (14–16). The American Soldier project during World War II brought together social scientists and practitioners (60). The Medical Research Council Applied Psychology Unit did the same in the United Kingdom (61), as have the Department of Homeland Security Centers of Excellence (https://www.dhs.gov/science-and-technology/centers-excellence). The latest US National Climate Assessment was developed in consultation with stakeholders (e.g., in agriculture and transportation) (https://nca2014.globalchange.gov). The National Science Foundation has supported multidisciplinary centers in many domains, with strong public outreach requirements.

A feature common to these ventures is a crisis that united scientists and practitioners against a common adversary, sometimes identifiable (e.g., the Axis Powers) and sometimes diffuse (e.g., threats to the environment). The urgent tone of Communicating Science Effectively: A Research Agenda (1) implies a crisis in communicating science that is deep enough to impel partnerships. That crisis threatens not only the usefulness of scientific results and society’s return on investment in them, but also faith in the scientific enterprise and its place in public discourse (11, 55, 62). Scientists who share that sense of urgency may change the reward structure in their disciplines, treating science communication as a professional responsibility and valuing colleagues who cultivate the commons of public goodwill on which science and society depend. Those scientists will promote the most relevant science, even it is not their own. They will allow evidence from the sciences of communication to inform, and perhaps even refute, their intuitions regarding what to say and how to say it, thereby embracing the vision of Communicating Science Effectively: A Research Agenda (1).