Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 Apr 7.
Published in final edited form as: J Health Commun. 2023 Apr 7;28(sup1):13–24. doi: 10.1080/10810730.2023.2220668

The Need for Systems Approaches for Precision Communications in Public Health

Bruce Y Lee 1,2,3, Danielle Greene 3, Sheryl A Scannell 1,2,3, Christopher McLaughlin 3, Marie F Martinez 1,2,3, Jessie L Heneghan 1,2,3, Kevin L Chin 1,2,3, Xia Zheng 4,5, Ruobing Li 4,5, Laura Lindenfeld 4,5, Sarah M Bartsch 1,2,3
PMCID: PMC10373800  NIHMSID: NIHMS1906259  PMID: 37390012

Abstract

A major challenge in communicating health-related information is the involvement of multiple complex systems from the creation of the information to the sources and channels of dispersion to the information users themselves. To date, public health communications approaches have often not adequately accounted for the complexities of these systems to the degree necessary to have maximum impact. The virality of COVID-19 misinformation and disinformation has brought to light the need to consider these system complexities more extensively. Unaided, it is difficult for humans to see and fully understand complex systems. Luckily, there are a range of systems approaches and methods, such as systems mapping and system modeling, that can help better elucidate complex systems. Using these methods to better characterize the various systems involved in communicating public health-related information can lead to the development of more tailored, precise, and proactive communications. Proceeding in an iterative manner to help design, implement, and adjust such communications strategies can increase impact and leave less opportunity for misinformation and disinformation to spread.

Keywords: Public Health Communications, Communication Systems, Systems Approaches, Pandemic, Public Health Emergency

INTRODUCTION

Science and health misinformation and disinformation have become major threats to our public ability to identify, understand, and take action in response to complex information, especially when public health emergencies occur (Gisondi et al., 2022; Naeem et al., 2021). The terms misinformation and disinformation both refer to information that is inaccurate and they are often used interchangeably. The difference is in the intent behind sharing the incorrect information. Misinformation refers to false information that is shared without the user knowing it is wrong and without intending to mislead others. Disinformation, however, is the intentional sharing of false information in order to mislead or cause harm. In times of uncertainty or fear, both misinformation and disinformation increase in prevalence (Banerjee & Sathyanarayana Rao, 2020; Wang et al., 2019). For example, during the COVID-19 pandemic, claims that COVID-19 vaccines install microchips in people’s bodies, cause people to become magnetic, alter people’s genes, result in many people dropping dead suddenly, and even turn humans into non-human creatures seemingly appeared every few weeks (Diseases, 2020; B. Y. Lee, May 9, 2021, May 18, 2021; B. Y. Lee, November 2, 2022, November 22, 2022; Scheufele et al., 2021). These factoids quickly went viral on social media or were promulgated by media personalities, which increased the number of people who heard/saw the misinformation and disinformation and added credibility to false claims. Further, the spread of vaccine misinformation and disinformation may be contributing to the lower uptake of COVID-19 vaccines and declining vaccination rates against measles and polio (S. K. Lee et al., 2022; Neely et al., 2022), which has recently led to the resurgence of such viruses in the United States (U.S.).

The complexity of how information is created, communicated, understood, and used makes overcoming misinformation and disinformation a vast challenge. Each of these components is a complex system. For information to do what it is supposed to do – provide facts, explain the impact of these facts, and then provide guidance on how to use these facts – these complex systems must then work together in an even larger system. For example, health and science information itself is complex, involving subtleties and considerable context. That information can then be communicated to groups and individuals through many more means than ever before, at times in 140 characters that do not provide necessary context. Then those that receive the information understand and use it within their own complex systems as individuals, family members, professionals, people at risk, and people whose experience is informed by multiple other factors.

To date, approaches to public health communications have not fully accounted for these complexities. For example, during the COVID-19 pandemic, the U.S. Government required all eligible federal workers to get the COVID-19 Emergency Authorized Use vaccine without tailoring the justification of the benefits and potential risks of the vaccine to different populations. This resulted in psychological reactance among federal workers and health workers (Sprengholz et al., 2021) as well as low vaccination intentions among African Americans, populations that perceive low benefits and high risks of vaccination, and among those with lower educational levels (Ciardi et al., 2021; Guidry et al., 2021), exacerbating health disparities. Additionally, many of these simplified communications approaches have stuck to traditional communications channels (e.g., national and local TV, newspaper, and radio) (Piltch-Loeb et al., 2021), which have been shown to have limited effectiveness with the continuing rise of misinformation and disinformation.

It is worth noting that in recent years, public health communications have moved beyond simplified means and begun to incorporate social marketing to account for the complexities of the communication process. Governmental agencies, nonprofit organizations, as well as state and local communities have adopted social marketing to communicate a wide range of health topics, such as promoting fruit and vegetable consumption, breastfeeding, physical activity, and immunizations. These efforts have led to important successes. For example, the National Institutes of Health (NIH) initiated the National High Blood Pressure Education Program (NHBPEP), which adopted elements of marketing and health-education, including patient education, media development, social networking, as well as program evaluation and measurement to achieve their goals and resulted in a significant increase in the public’s knowledge of high blood pressure (Roccella & Horan, 1988). However, marketing experts have noted that many public health professionals have an incomplete understanding of social marketing. By viewing it predominantly as a promotional or simple communication activity, they often fail to integrate the complete marketing mix when planning program interventions (Grier & Bryant, 2005; Hill, 2001). Although social marketing has been defined in various ways, its defining features stem from marketing’s conceptual framework and include key aspects such as audience segmentation, consumer orientation, sales and distribution, tailored communications, and continuous monitoring (Firestone et al., 2017; Grier & Bryant, 2005). During the recent COVID-19 pandemic, efforts have been made toward tailored and strategic communications. For example, Zhou et al. (Zhou et al., 2023) used latent profile analysis to analyze vaccine hesitant individuals and segmented them into heterogeneous groups based on their psychographic profiles, providing basis to tailored communication to each segment. Public health experts also raised the importance of applying multilevel, evidence-based strategies to influence behavior change and address vaccine hesitancy (Chou, 2020; Finney Rutten et al., 2021).

However, traditional methods like surveys, experiments, and content analysis often only allow researchers to focus on one aspect of the communication process (e.g., audience, source, channel, message), failing to provide a comprehensive view of the entire system or account for all its complexities. Therefore, there is a need to build off prior systems communication work (Ford et al., 2009; Hawe, 2015; Lang, 2014) and establish more formal systems approaches that understand and address the different components of the systems within the current public health communication landscape. This in turn will help develop and implement more tailored, precise, and proactive communications strategies.

WHAT ARE THE CHARACTERISTICS OF A SYSTEM?

A system is a group of different components that are not independent from each other but rather are interconnected (B. Y. Lee, Bartsch, et al., 2017; B. Y. Lee et al., 2021; B. Y. Lee, Mueller, et al., 2017; Mabry et al., 2022; McGlashan et al., 2019). They interact with and affect one another in different ways. The universe is comprised of many different naturally occurring and human-made systems, such as meteorological, biological, transportation, manufacturing, financial, health care, and food systems. In fact, nearly everything in the universe is a complex system and part of complex systems (B. Y. Lee, Bartsch, et al., 2017; B. Y. Lee et al., 2021; B. Y. Lee, Mueller, et al., 2017; Mabry et al., 2022). It is not surprising that scholars have characterized challenges in human-made systems as “wicked problems,” because of how challenging it is to characterize, address, and predict the unintended consequences of actions taken to address multifaceted and difficult to define challenges that are fundamentally embedded in complex ecological, economic, social, and cultural systems (Brown et al., 2010; Frame, 2008; Kreuter et al., 2004). Without attention to communications as a system within these systems we lose site of the very cultural, economic, and social issues that must be considered alongside scientific and technical ones (Funtowicz & Ravetz, 1991; Limoges et al., 1994; Nowotny et al., 2001).

Since none of the parts in a system are completely independent, what affects one part of a system could potentially affect many other things in that system. There are rarely single cause, single effect relationships. Instead, the properties of a system are interconnected, influencing one another in a variety of manners, both obvious and subtle. Significant perturbation of one part tends to have reverberations throughout other parts of the system. Therefore, unless the system is well understood, it can be challenging to identify the root cause of an observed phenomenon or to predict the effects of a change or an intervention (B. Y. Lee, Bartsch, et al., 2017; B. Y. Lee et al., 2021; B. Y. Lee, Mueller, et al., 2017; Mabry et al., 2022).

Systems rarely are static, either. Instead, they tend to be dynamic, changing and evolving over time. For example, people’s tendencies, preferences, and behaviors tend to evolve with experience and learning over time. The ways people communicate have changed with the introduction of different types of technology and the rise and fall of different media presences.

Additionally, systems and effects can cross different scales. A scale is the size and scope of something. Something that occurs at a cellular or biological scale is smaller in size and scope than something that occurs at an individual behavioral scale, which is, in turn, smaller in size and scope than what happens at a population level. These different scales can interact with each other in various complex ways (Huang et al., 2011). The variability between scales can be challenging to fully appreciate since traditional academic disciplines and industry sectors oftentimes focus on particular scales. Additionally, different systems interact with and influence each other.

WHY IS COMMUNICATIONS A SYSTEMS ISSUE?

Communications crosses multiple systems that, in turn, can cross multiple scales. To communicate most effectively, one must appreciate and understand these different systems. These systems include the information users that are supposed to receive the communications, the communications channels, and the information itself. Conceptualizing communications as a complex system requires that we account both for the transactional nature of communication - that it, is how we transfer information from a sender to a receiver – but also for its constitutive nature, that is that communication is the process through which we co-create meaning and (re)establish individual, social, and cultural norms. When we think only of communication as the message that gets sent, we miss the greater opportunity to understand how communication involves a dynamic system that produces structures, processes, and outcomes that enables or disables our ability to characterize problems and generate solutions (McGreavy et al., 2015). Conceptualizing communications as complex systems thus becomes critical to our understanding of public health challenges and opportunities.

Different populations and communities are comprised of complex systems

Populations and communities are not homogenous blocks. Instead, they are very diverse and extremely heterogeneous. This diversity and heterogeneity extend beyond simplified demographic categories such as race, ethnicity, immigration status, socioeconomic status, and geospatial location. For example, it is not reasonable to assume that all Black Americans or all Asian Americans are the same or conform to some set of stereotypes. Not only are there many varied subcategories within these categories (e.g., Asian Americans encompass Chinese-Americans, Taiwanese-Americans, Korean-Americans, Japanese-Americans, Indian-Americans, and many others), there is tremendous diversity and heterogeneity within each of these subcategories. The diversity within each of these demographic subcategories is at least as great as the diversity within other demographic categories. Within each demographic subcategory, there is a broad range of personalities, personal histories, interaction styles, and other characteristics that affect the ways that the people in those groups receive, process, and exchange information. In fact, two people of different demographic subcategories may be even more similar to each other than two people within the same demographic subcategory. Additionally, different people within and across different communities interact, accept information, and share information with each other in very different and complex ways. Table 1 shows examples of mechanisms that make populations and communities complex systems and thus make the receipt and flow of information non-linear and not straightforward.

Table 1.

Mechanisms that make populations and communities complex systems

Mechanism Some of the complexities involved Examples
How people directly interact and mix with each other People do not interact equally with each other, but instead, their interactions (e.g., frequency, intensity, timing, nature) are guided by the time of day/year and their relationships (e.g., family versus friends versus co-workers), shared backgrounds, locations, ages, immigration/language status, and other factors. College students can have more frequent interactions and interact in different ways than older adults; Certain recent immigrant communities may have more limited interactions with others due to language barriers and discrimination
What platforms/Information people are exposed to Different people can get information from different sources and platforms in different ways. The sources and platforms that they use depend on a variety of factors such as what their close contacts are using, the field of work, age, education, cultural background, main spoken language. Teenagers and young adult may be more likely to use TikTok and some newer social media platforms than older adults; Location determines the local news channels that are available; Internet access, cable and satellite TV subscriptions affect what channels/platforms are available
What platforms/information people prioritize/favor People have favorite platforms through which they receive information (e.g., television, radio, social media, podcasts). These platforms can deliver different information in which people will prioritize particular sources (e.g., Fox News, CNN, Facebook, iHeart Radio, Twitter) they find to be most trustworthy or favorable. Different social media platforms are favored by those of certain ages and political affiliations; Certain English as a second language communities may have their own newspapers that use a different language; Those in particular religious groups may view religious leaders as a primary source of information; Different people may view particular newscasters as more or less reliable.
How people’s histories affect how they process and interpret information Different people have different personal histories that can affect their pre-conceived notions and how they accept, process, and interpret information. Those who have suffered racial discrimination or oppression may be less likely to trust institutions such as healthcare systems and governments; Different people are more or less trained in interpreting scientific studies
What actions people take based on information Different people will use a given piece of information in different ways and exhibit different behaviors as result. Upon hearing or reading the same information, one person may follow CDC or WHO recommendation to wear face masks, another person may not.
What is the likelihood of people disseminating information Different people may decide to further transmit information to others in different ways. People can share information on social media, during direct conversation, or through their work positions and channels. For example, the CEO of a company can influence his/her employees.

As populations and communities continue to grow and become more and more diverse, so will their complexity. Communication strategies need to be tailored to resonate with a complex array of information users. There is significant research on different audience preferences, belief and value systems, and levels of trust that can help inform strategies, and we should be drawing on this research whenever we are designing communications to ensure the best possible alignment.

Communications channels are complex systems

Similarly, the landscape of communications channels (i.e., ways by which people get their information) has become increasingly complex. Channels now include a variety of more traditional channels and many newer and emerging channels. Some examples of traditional channels have included:

  • Direct communications from governmental agencies: e.g., press releases, press conferences, and speeches

  • Print media: e.g., newspapers and magazine

  • Radio and television: e.g., news and talk shows

  • Advertising: e.g., in newspapers, on billboards, and during television/radio shows

  • Entertainment: e.g., movies and live shows

  • Workplace-based: e.g., statements by organizational leaders and newsletters

  • School-based: e.g., announcements and lectures, flyers, posters

  • Community-based: e.g., clubs, religious gatherings, concerts, and sporting events

  • Direct person-to-person interactions: e.g., in schools, at workplaces, within families and friendship circles, and in the community

Each of these have been growing more complex with time. For example, there are now many more TV channels than there were several decades ago when only three network channels (NBC, ABC, and CBS) dominated. Also, advertising has become more and more pervasive over the years.

In addition to each of the more traditional channels becoming more complex, the past couple decades have seen exponential growth in newer communications channels, such as:

  • Email: e.g., personal, school, and company emails

  • Direct messaging apps: e.g., WhatsApp, Signal, Slack, and Skype

  • Podcasts: e.g., The Daily, NPR News Show, and Global News Podcast

  • Websites: e.g., Wikipedia, and YouTube

  • Internet search engines: e.g., Google, Bing, and Duck Duck Go

  • Internet discussion boards: e.g., Reddit, and Quora

  • Social media: e.g., Twitter, Facebook, LinkedIn, and TikTok

  • News aggregators: e.g., Apple News and Google News

Each of these communications channels work via different mechanisms (Table 2 lists some examples) that in turn make the flow of information non-linear and quite complex.

Table 2.

Mechanisms that make different communications channels complex systems

Mechanism Some of the complexities involved Examples
What is the format of the information on the channel Different channels have specifications in terms of the type (e.g., text vs graphical vs. audio vs. video) and amount of information that can be shared at a given time. Twitter has a character limit for given post whereas Facebook and LinkedIn do not; A radio broadcast does not allow text, graphical, or video information
What information is being presented to whom Many of the channels, especially the newer ones, do not present the same information to everyone; what one sees can be based on one’s demographics and histories Social media channels use algorithms to determine what to show a given person; Advertising can be personalized and targeted; Internet search histories and location can determine what an Internet search returns
What is the history of the channel The length of time that a channel has been in existence and the events that have occurred can affect the reputation of the channel and who trusts the channel Major network news such as NBC, ABC, and CBS have much longer histories of being go-to sources than cable news; Scandals can affect the trustworthiness of an information source
When is the information being disseminated The timing of the release of information can affect who sees the information and what they end up doing with it. Daytime TV shows may be more likely to reach those who are not at work; Information that comes out soon after a major event may be more likely to affect the reaction to that major event
Who is placing the information on the channel The information shared on the channel may reflect who runs the channel and that person’s background, personality, intentions, and oversight A governmental organization has the obligation to serve the taxpayers; A public corporation may have the obligation to serve its shareholders
Who are the target populations for the channel Different platforms and their messaging may target different populations and communities. TikTok tends to attract a younger demographic compared to Facebook; Certain news organizations appear to be targeting people of a certain political orientation

Health-related information involves complex systems

Health-related information can be quite complex. It can often cross multiple scales, and the line between science communication and health communication is often blurred. For example, face mask use has a biological basis, preventing pathogens from leaving one’s nose and mouth and entering one’s respiratory tract. It has a behavioral basis as the timing of face mask use and the way the face mask is worn can impact its effectiveness. It has a sociopolitical basis as well, since the effectiveness of face mask use depends heavily on the percentage of people in an area wearing face masks. It also has an environmental basis since air ventilation and filtration can affect the need for face masks. Finally, it has an economic basis since preventing the spread of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) can affect productivity losses and medical costs.

Determining what health information to communicate and through which communication channels depends on the synthesis of many considerations and pieces of evidence from a variety of sources of varying quality. Table 3 lists some of these common sources and their strengths and limitations. Likewise, the messenger matters as well, as different communities have higher or lower levels of trust that ultimately enable or disable effective communication.

Table 3.

Examples of sources that comprise health-related information

Source of Evidence Strengths Limitations Example
Materials studies in the laboratory Can determine the physical properties of something May not incorporate various human behavioral and social factors Studies that show how many different particles face masks can filter out but do not account how the person may wear the mask
Laboratory molecular and cellular studies Isolate specific mechanisms Situation may be very different when molecules or cells are viewed in isolation vs. part of the greater system. Laboratory studies show how well a virus can enter a cell, but the virus has to travel down through the respiratory tract to reach the cell
Laboratory animal studies May be done when human studies are not yet possible Results may not necessarily be applicable to humans Ferret studies that have shown that humidity may affect transmission of respiratory viruses does not necessarily apply to humans
Anecdotes Can raise questions for further exploration Typically, does not qualify as scientific evidence; Not verifiable or validated Stories of a person having an adverse event after a vaccine may not be verifiable
Case studies Can raise questions for further exploration Unclear how unusual the findings may be. Case studies showed that a majority of people infected with SARS-CoV-2 developed some sign of myocarditis but is it actually not that common
Observational cohort studies Evaluate what’s happening in the real world under real world conditions; Can show trends, patterns, and associations Cannot demonstrate cause and effect; May overlook many factors and complexities; can oversimplify situations Observational cohort studies suggested that Vitamin D intake may be associated with lower Covid-19 severity but there were many potential confounding factors such as geographic location, use of other COVID-19 precautions, and ability to social distance
Randomized controlled trial (RCT) Good at evaluating medications and treatments for an individual Not as effective at evaluating more complex interventions that are affected by many factors external to an individual; Findings may not hold in the real world An RCT of an antiviral can show the medication may affect the duration and severity of symptoms; however, an RCT of face mask use may not be able to measure and control for factors that affect efficacy such as where the person moved, how the person wore the mask, and how many others were wearing face masks
Data-driven statistical and machine learning studies Can help identify trends, patterns, and associations in historic data that may not be obvious; Can help extrapolate these trends; Can help identify areas that need further exploration Cannot determine cause-and-effect relationships; can oversimplify situations; Often assumes that historical trends will hold; The past does not necessarily predict the future Studies in early 2020 tried to predict the subsequent course of the pandemic and the resulting mortality, however, conditions changed in the latter half of 2020, and again in 2021 and 2022.
Mechanistic computer modeling studies Able to pull together, integrate, and synthesize data and evidence from different sources; Attempts to represent the underlying mechanisms; Can run hypothetical scenarios; Can help determine what may happen should circumstances change; May forecast the future; Can help evaluate multiple layered interventions Can be substantial variation in types and quality of models; The construction of the model may not always be readily understood; Dependent on how well the actual mechanisms are represented; Ability to replicate real-world situations can vary based on the construction of the model and accompanying assumptions Computer models from 2020 showed the anticipated course of the pandemic and impact of Covid-19 vaccines but did not always account for emergence of new variants

Additionally, such information can evolve as new evidence emerges. For example, in early 2020 during the first month of the COVID-19 pandemic, it was not yet clear that SARS-CoV-2 particles could be airborne and remain for so long (Lewis, 2022). Thus, initial guidance did not include protection against airborne pathogens. However, as studies emerged, it became clearer that such protection would be helpful. This finding resulted in policy changes promoting the use of face masks and air purification, filtration, and ventilation. However, the reasons for why such changes in guidance were made may not have been completely clear to everyone. Therefore, communicating the underlying context behind every update is important.

Public health policies and interventions entail complex systems

It is rare that one policy or one intervention can solve a public health problem. Typically, a suite of different policies and interventions that can interact and synergize with each other is necessary. For example, studies have clearly shown how there is a need to layer different policies and interventions such as testing, social distancing, face mask use, vaccination, and treatment to more effectively control the spread of influenza or SARS-CoV-2 during a pandemic (Bartsch et al., 2022; Bartsch, O’Shea, Ferguson, Bottazzi, Cox, et al., 2020; Bartsch, O’Shea, Ferguson, Bottazzi, Wedlock, et al., 2020; Bartsch, O’Shea, et al., 2021; Bartsch, Wedlock, et al., 2021; Chu et al., 2020). Moreover, the impact and value of interventions are rarely linear and instead can be very complex. For example, a study conducted during the 2009 H1N1 influenza pandemic suggested that when there is a limited supply of vaccines, prioritizing lower income neighborhoods for vaccination provided substantially greater benefits to society than randomly vaccinating individuals (B. Y. Lee, 2011). This was because lower income neighborhoods tend to have higher population density and those in lower income neighborhoods are more likely to travel and tend to travel to wealthier neighborhoods for work and other activities than the opposite.

Additionally, the optimal combination of interventions and the resulting impact can change as the pandemic evolves, scientific knowledge grows, and different options become more or less available. For example, face mask guidance changed in the Summer of 2020 when studies began to show that SARS-CoV-2 could remain suspended in the air for longer periods of time and travel further distances in the air than influenza (B. Y. Lee, May 30, 2020).

Public health emergencies involve complex systems

Public health problems are rarely simple problems – indeed, they frequently are wicked problems for which there is no simple, straightforward solution. Typically, they involve many different factors and processes that cross multiple scales. For example, SARS-CoV-2 has had its biological effects (e.g., causing organ damage (Jain, 2020)), behavioral effects (e.g., changing work schedules and limiting where people could go (Wontorczyk & Roznowski, 2022)), social effects (e.g., altering how people interact with each other (Calbi et al., 2021)), and economic effects (e.g., resulting in direct medical costs and productivity losses (Bartsch, Ferguson, et al., 2020)).

Additionally, public health emergencies like pandemics are not static situations. They do not stay the same from day-to-day. Instead, they are dynamic, evolving with time. The needs and nature of communications may change dramatically as public health emergencies cross into different phases (Vaughan & Tinker, 2009). For example, Table 4 lists the different phases of a pandemic and how the accompanying circumstances may call for different types of communications.

Table 4.

How the communications needs change over the phases of a pandemic

Phase Examples of Communication Issues/Needs
Pre- or Inter-pandemic/epidemic/outbreak
Surveillance Help people understand the rationale/need for surveillance and investment into such activities
Threat identification Help people distinguish between different levels of concern (e.g., how likely is a pathogen going to be a threat)
Prevention/preparedness Help people understand need for prevention/preparedness measures even through the threat has not yet appeared
During outbreak/epidemic/pandemic
Discovery/characterization of pathogen Help people realize the different characteristics of the pathogen and how these different characteristics may evolve over time
Intervention design/prioritization Help people know why a given intervention is necessary and the relative strengths and limitations of the intervention and how these may change as more information emerges
Implementation/ramp up Help people understand what is involved in the use of an intervention (e.g., face mask efficacy depends on the types of masks used, how they are worn, and how many other people are wearing them)
Maintenance Help people realize the importance of continuing to employ an intervention even when fatigue may occur
Policy/intervention troubleshooting Help people interpret new emerging information and how that may affect use of an intervention (e.g., how should rare myocarditis cases affect trust in the Covid-19 mRNA vaccines)
Transition phase/new “chapters” and adjustment Help people anticipate what changes in goals, policies, and interventions may be needed to usher in a new phase of the emergency (e.g., setting up a routine vaccination schedule to help transition a pandemic to a seasonal virus situation; implementing face mask use to prevent another surge)
De-escalation Help people know when the intensity and use of an intervention can be decreased (e.g., when can face mask use be decreased)
Post-outbreak/epidemic/pandemic
Hot wash/post-outbreak/epidemic/pandemic review Help people understand what went well and what did not and how to use this information in future outbreaks/epidemics/pandemics
Post-crisis steady state (e.g., “endemic”/seasonal) Help people realize what policies and interventions need to be continued to prevent new and additional emergencies

Intentions behind misinformation and disinformation are complex

The spread of misinformation can happen with the best intentions. The sharer may genuinely not know the information is incorrect. When trusted leaders promote misinformation, they attach their social capital to the things they are saying. When other professionals try to provide corrected information, they are not believed because they do not have the same trusted relationship with those receiving the information. Yet, if the trusted leader corrects their mistake, they may lose the trust of the people who previously listened to them. As a result, three aspects of the complex system are compromised – information, information sources, and information users. Too often, misinformation is not noticed until it has been widely shared and accepted as “fact”.

Contrastingly, the goals of those who share disinformation are nefarious from the start. False information can be used to make a profit on harmful or unproven “treatments,” discredit scientists, reduce restrictions, reduce trust in government entities, or influence public opinion. Political leaders may use disinformation to distract from their performance or support other priorities. For example, a study published in the American Journal of Public Health revealed that bots from the Internet Research Agency based in Russia were responsible for a large percentage of pre-COVID anti-vaccination posts on Twitter (Broniatowski et al., 2018). People may advance disinformation to promote hate and discrimination or to enact revenge for a perceived wrong. These motivations can be hidden and difficult to uncover. Disinformation damages the entire system of communications.

Many aspects of communications historically have not fully accounted for these systems

It is not clear how well these various systems have been characterized and whether communications approaches have been tailored to fit these systems. For example, how much effort has been made to fully characterize the diversity of communications channels that exists for different communities? How effectively did communications relay to everyone the need for layered COVID-19 interventions, rather than single interventions? How well did communications approaches adapt with the changing conditions of the COVID-19 pandemic? Did communications use the full array of communications methods? To what degree were there generalized communications versus more precise approaches, better tailored to the specific information users?

Attempting to communicate without fully understanding the systems involved can be like trying to coach or play a game without really seeing the field or opponents. The most effective communications strategies are tailored to fit the information users and systems involved. Therefore, it is nearly impossible to come up with very effective communications strategies and methods without better characterizing the systems.

There have been some attempts to do this, but traditional research methods are limited in the view of the systems that they can provide. For example, traditional epidemiologic methods such as multivariate regression can show some general associations, trends, and patterns, but they cannot really show cause-and-effect and specific mechanisms and details of the system (Mabry et al., 2022). Traditional randomized controlled trials can focus on some specific mechanisms within one aspect of a system, but may not provide a broader, more comprehensive view. Traditional survey methods can give a sampling and certain views of the system, but they may not show how everything fits together.

There are systems approaches and methods that can take information gathered by more traditional approaches and help reconstruct the systems involved.

WHAT ARE SYSTEMS METHODS AND HOW CAN THEY BE USEFUL FOR COMMUNICATIONS?

Unaided, it is difficult for humans to see and fully understand complex systems (B. Y. Lee, Bartsch, et al., 2017; B. Y. Lee, Mueller, et al., 2017; Mabry et al., 2022). They may be able to see direct cause-and-effect relationships. But it is more difficult to see secondary, tertiary, and other more indirect effects. The more complexity involved, such as with feedback loops and reverberating effects, the more difficult it is for humans to comprehend what is happening. Various methods and tools, including computer-aided ones, may help illuminate what is occurring.

Systems Mapping

One common set of systems methods are systems maps/diagrams that visually represent components of the system and their relationships with each other. A systems map represents all the components of a system and how they may interact with and affect each other. Such maps typically make use of shapes to represent the different components and lines to show their relationships with each other. These can show how different people’s conceptualizations or mental models of the system may be similar versus different (Cox et al., 2021; B. Y. Lee & Bartsch, 2017). This can then identify a more comprehensive representation of the system. Systems maps can also serve as blueprints to develop subsequent systems models.

One way to develop systems maps is through mapping workshops. This involves convening diverse representatives with expertise in the various related fields and topics, including a wide range of disciplines and sectors, which can help ensure that the maps are robust. During a workshop, participants work to identify factors and mechanisms that contribute to the system of interest (e.g., communications and the spread of information). Such sessions begin by brainstorming different components that are relevant to the systems map by posing clear, open-ended but bounded questions pertaining to the topic of the session. Then, after the initial open-ended brainstorming about different factors and characteristics to depict in the map, open discussion helps further build out and develop the map. Additional iterations help refine and augment the systems map.

Systems Modeling

A systems map becomes a systems model when one adds quantitative representations (e.g., mathematical equations) of the relationships and processes that link the different components in the system (Mabry et al., 2022). Thus, the model begins with the understanding or conceptualization of the system. After equations are established, data are used to populate, calibrate, and validate the systems model. These equations can represent a situation at a particular point in time or simulate what happens over time (i.e., a dynamic simulation model). The equations can use specific values for a deterministic model or incorporate variability and uncertainty in data values, thereby making it stochastic.

Since systems models aim to recreate the system, they are quite different from traditional statistical models that try to identify associations and trends and potentially extrapolate them. Statistical models start with data and then identify patterns or trends according to statistical properties that are subject to pre-identified assumptions. Systems models are also different from other computer-driven approaches that start first with the data and then try to identify associations and trends, such as machine learning categorization.

Iterative Approach to Systems Methods

Ideally, systems mapping and modeling should proceed in an iterative manner. As Figure 1 shows, one does not need to come up with a perfect representation of the system at the beginning. Instead, an initial systems map and/or model can be a rough approximation. This can in turn help identify the knowledge and data gaps to then guide study designs and data collection. Once such studies and data collection yield more insights and data, the systems map and model can be updated accordingly – leading to more cycles of further refining both the systems map and model – as well as the studies, data collection, and insights. Such an iterative process can help move toward a better understanding of the system.

Figure 1.

Figure 1.

Modeling is an iterative process

HOW SYSTEMS APPROACHES CAN LEAD TO MORE PROACTIVE PRECISION COMMUNICATIONS FOR PUBLIC HEALTH

Improved understanding of the complexities involved can lead to more impactful, proactive, and precise communications strategies. Most sports teams have players that specialize in either offense or defense, a head coach plus assistant coaches, recruiters, as well as trainers and those providing equipment, uniforms, and even food. The analogy is that the team plays best when everyone involved can see the field and the opponent in advance and they all share the same vision and understanding of the strategy. It enables each person in the complex sports system to anticipate what may happen and adjust their role to play a more effective game. An approach that accounts for the complexities of all these systems can create precise, proactive communications within, across, and throughout each system.

The past decade has seen the initiation of other precision efforts, such as precision medicine and precision nutrition. These have aimed to develop treatments, diets, and other approaches that better account for differences among different people and their circumstances. They aim to eschew traditional approaches that do not account for the complexity of the systems involved. The time has come for more precision communications in public health.

CONCLUSIONS

The COVID-19 pandemic has shown that previous paradigms of public health communications have not worked. There is a need for new approaches that are more anticipatory and precise. The only way to move towards this is to better understand the systems and complexities involved. Otherwise, those trying to communicate evidence-based information may miss factors and processes that could make a big difference.

Acknowledgments

The authors would like to acknowledge Danielle John, a Scientific Coordinator for the NYC Pandemic Response Institute (PRI) at the CUNY Graduate School of Public Health and Health Policy and an Analyst for PHICOR, for her assistance in proofreading and editing the manuscript.

Funding Statement

This work was supported by the Agency for Healthcare Research and Quality (AHRQ) via grant 1R01HS028165-01, the National Institute of General Medical Sciences as part of the Models of Infectious Disease Agent Study network under grants R01GM127512 and 3R01GM127512-01A1S1, and by the National Science Foundation via proposal number 2054858, the National Center for Advancing Translational Sciences of the National Institutes of Health via award number U54TR004279, and by the City University of New York (CUNY) in support of the NYC Pandemic Response Institute (PRI). Statements in the manuscript do not necessarily represent the official views of, or imply endorsement by NIH, AHRQ, the US Department of Health and Human Services, CUNY, or the NYC PRI.

Footnotes

Disclosure Statement

The authors report there are no significant competing interests to declare.

References

  1. Banerjee D, & Sathyanarayana Rao TS. (2020). Psychology of Misinformation and the Media: Insights from the COVID-19 Pandemic. Indian Journal of Social Psychiatry, 36(Suppl 1), S131–S137. 10.4103/ijsp.ijsp_112_20 [DOI] [Google Scholar]
  2. Bartsch SM, Ferguson MC, McKinnell JA, O’Shea KJ, Wedlock PT, Siegmund SS, & Lee BY. (2020). The Potential Health Care Costs And Resource Use Associated With COVID-19 In The United States. Health Aff (Millwood), 39(6), 927–935. 10.1377/hlthaff.2020.00426 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bartsch SM, O’Shea KJ, Chin KL, Strych U, Ferguson MC, Bottazzi ME, Wedlock PT, Cox SN, Siegmund SS, Hotez PJ, & Lee BY. (2022). Maintaining face mask use before and after achieving different COVID-19 vaccination coverage levels: a modelling study. Lancet Public Health, 7(4), e356–e365. 10.1016/S2468-2667(22)00040-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bartsch SM, O’Shea KJ, Ferguson MC, Bottazzi ME, Cox SN, Strych U, McKinnell JA, Wedlock PT, Siegmund SS, Hotez PJ, & Lee BY. (2020). How Efficacious Must a COVID-19 Coronavirus Vaccine be to Prevent or Stop an Epidemic by Itself. medRxiv. 10.1101/2020.05.29.20117184 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bartsch SM, O’Shea KJ, Ferguson MC, Bottazzi ME, Wedlock PT, Strych U, McKinnell JA, Siegmund SS, Cox SN, Hotez PJ, & Lee BY. (2020). Vaccine Efficacy Needed for a COVID-19 Coronavirus Vaccine to Prevent or Stop an Epidemic as the Sole Intervention. Am J Prev Med, 59(4), 493–503. 10.1016/j.amepre.2020.06.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bartsch SM, O’Shea KJ, Wedlock PT, Strych U, Ferguson MC, Bottazzi ME, Randall SL, Siegmund SS, Cox SN, Hotez PJ, & Lee BY. (2021). The Benefits of Vaccinating With the First Available COVID-19 Coronavirus Vaccine. Am J Prev Med, 60(5), 605–613. 10.1016/j.amepre.2021.01.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bartsch SM, Wedlock PT, O’Shea KJ, Cox SN, Strych U, Nuzzo JB, Ferguson MC, Bottazzi ME, Siegmund SS, Hotez PJ, & Lee BY. (2021). Lives and Costs Saved by Expanding and Expediting Coronavirus Disease 2019 Vaccination. J Infect Dis, 224(6), 938–948. 10.1093/infdis/jiab233 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Broniatowski DA, Jamison AM, Qi S, AlKulaib L, Chen T, Benton A, Quinn SC, & Dredze M. (2018). Weaponized Health Communication: Twitter Bots and Russian Trolls Amplify the Vaccine Debate. Am J Public Health, 108(10), 1378–1384. 10.2105/AJPH.2018.304567 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Brown V, Harris J, & Russell J. (2010). Tackling Wicked Problems through the Trandisciplinary Imagination. [Google Scholar]
  10. Calbi M, Langiulli N, Ferroni F, Montalti M, Kolesnikov A, Gallese V, & Umilta MA. (2021). The consequences of COVID-19 on social interactions: an online study on face covering. Sci Rep, 11(1), 2601. 10.1038/s41598-021-81780-w [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Chou WYS, Burgdorf CE, Gaysynsky A, & Hunter CM. (2020). COVID-19 vaccination communication: Applying behavioral and social science to address vaccine hesitancy and foster vaccine confidence. https://policycommons.net/artifacts/2438492/obssr_vaccinewhitepaper_final_508/3460216/ [Google Scholar]
  12. Chu DK, Akl EA, Duda S, Solo K, Yaacoub S, Schunemann HJ, & authors C.-S. U. R. G. E. s. (2020). Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis. Lancet, 395(10242), 1973–1987. 10.1016/S0140-6736(20)31142-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Ciardi F, Menon V, Jensen JL, Shariff MA, Pillai A, Venugopal U, Kasubhai M, Dimitrov V, Kanna B, & Poole BD. (2021). Knowledge, attitudes and perceptions of COVID-19 vaccination among healthcare workers of an inner-city hospital in New York. Vaccines, 9(5), 516. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Cox SN, Wedlock PT, Pallas SW, Mitgang EA, Yemeke TT, Bartsch SM, Abimbola T, Sigemund SS, Wallace A, Ozawa S, & Lee BY. (2021). A systems map of the economic considerations for vaccination: Application to hard-to-reach populations. Vaccine, 39(46), 6796–6804. 10.1016/j.vaccine.2021.05.033 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Diseases TLI. (2020). The COVID-19 infodemic. The Lancet. Infectious Diseases, 20(8), 875. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Finney Rutten LJ, Zhu X, Leppin AL, Ridgeway JL, Swift MD, Griffin JM, St Sauver JL, Virk A, & Jacobson RM. (2021). Evidence-Based Strategies for Clinical Organizations to Address COVID-19 Vaccine Hesitancy. Mayo Clin Proc, 96(3), 699–707. 10.1016/j.mayocp.2020.12.024 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Firestone R, Rowe CJ, Modi SN, & Sievers D. (2017). The effectiveness of social marketing in global health: a systematic review. Health Policy Plan, 32(1), 110–124. 10.1093/heapol/czw088 [DOI] [PubMed] [Google Scholar]
  18. Ford LA, Crabtree RD, & Hubbell A. (2009). Crossing borders in health communication research: toward an ecological understanding of context, complexity, and consequences in community-based health education in the U.S.-Mexico borderlands. Health Commun, 24(7), 608–618. 10.1080/10410230903242218 [DOI] [PubMed] [Google Scholar]
  19. Frame B. (2008). ‘Wicked’,’messy’, and ‘clumsy’: long-term frameworks for sustainability. Environment and Planning C: Government and Policy, 26(6), 1113–1128. [Google Scholar]
  20. Funtowicz SO, & Ravetz JR. (1991). A new scientific methodology for global environmental issues. Ecological economics: The science and management of sustainability, 10, 137. [Google Scholar]
  21. Gisondi MA, Barber R, Faust JS, Raja A, Strehlow MC, Westafer LM, & Gottlieb M. (2022). A deadly infodemic: Social media and the power of COVID-19 misinformation (Vol. 24, pp. e35552). JMIR Publications Toronto, Canada. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Grier S, & Bryant CA. (2005). Social marketing in public health. Annu. Rev. Public Health, 26, 319–339. [DOI] [PubMed] [Google Scholar]
  23. Guidry JP, Laestadius LI, Vraga EK, Miller CA, Perrin PB, Burton CW, Ryan M, Fuemmeler BF, & Carlyle KE. (2021). Willingness to get the COVID-19 vaccine with and without emergency use authorization. American journal of infection control, 49(2), 137–142. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Hawe P. (2015). Lessons from complex interventions to improve health. Annu Rev Public Health, 36, 307–323. 10.1146/annurev-publhealth-031912-114421 [DOI] [PubMed] [Google Scholar]
  25. Hill R. (2001). The marketing concept and health promotion: a survey and analysis of recent health promotion literature. Social Marketing Quarterly, 7(1), 29–53. [Google Scholar]
  26. Huang TT-K, Grimm B, & Hammond RA. (2011). A systems-based typological framework for understanding the sustainability, scalability, and reach of childhood obesity interventions. Children’s Health Care, 40(3), 253–266. [Google Scholar]
  27. Jain U. (2020). Effect of COVID-19 on the Organs. Cureus, 12(8), e9540. 10.7759/cureus.9540 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kreuter MW, De Rosa C, Howze EH, & Baldwin GT. (2004). Understanding wicked problems: a key to advancing environmental health promotion. Health Educ Behav, 31(4), 441–454. 10.1177/1090198104265597 [DOI] [PubMed] [Google Scholar]
  29. Lang A. (2014). Dynamic Human-Centered Communication Systems Theory. The Information Society, 30(1), 60–70. 10.1080/01972243.2013.856364 [DOI] [Google Scholar]
  30. Lee BY. (2011). The Benefits To All Of Ensuring Equal And Timely Access To Influenza Vaccines In Poor Communities. Health Affairs, 30(6), 1141–1150. 10.1377/hlthaff.2010.0778 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Lee BY. (May 9, 2021). As Covid-19 Vaccine Microchip Conspiracy Theories Spread, Here Are Responses On Twitter. Forbes https://www.forbes.com/sites/brucelee/2021/05/09/as-covid-19-vaccine-microchip-conspiracy-theories-spread-here-are-some-responses/ [Google Scholar]
  32. Lee BY, (May 18, 2021). Covid-19 Vaccine Magnet Challenge: Videos Claim Magnets Stick To Arms After Vaccination. Forbes https://www.forbes.com/sites/brucelee/2021/05/18/covid-19-vaccine-magnet-challenge-videos-claim-magnets-stick-to-arms-after-vaccination/
  33. Lee BY. (May 30, 2020). Where Coronavirus Is More Likely To Be Airborne, 5 Places To Avoid https://www.forbes.com/sites/brucelee/2020/05/30/where-coronavirus-is-more-likely-to-be-airborne-5-places-to-avoid/?sh=49310abd39ab
  34. Lee BY. (November 2, 2022). ‘Homoborgenesis’: Here Are Unfounded Claims Covid-19 Vaccinated Are No Longer Human. Forbes https://www.forbes.com/sites/brucelee/2022/11/02/homoborgenesis-here-are-unfounded-claims-covid-19-vaccinated-are-no-longer-human/
  35. Lee BY. (November 22, 2022). New ‘Died Suddenly’ Film Pushes Unfounded Depopulation Claims About Covid-19 Vaccine. Forbes https://www.forbes.com/sites/brucelee/2022/11/22/new-died-suddenly-film-pushes-unfounded-depopulation-claims-about-covid-19-vaccine/?sh=fb49f7149d06
  36. Lee BY, & Bartsch SM. (2017). How to determine if a model is right for neglected tropical disease decision making. PLoS Negl Trop Dis, 11(4), e0005457. 10.1371/journal.pntd.0005457 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Lee BY, Bartsch SM, Mui Y, Haidari LA, Spiker ML, & Gittelsohn J. (2017). A systems approach to obesity. Nutr Rev, 75(suppl 1), 94–106. 10.1093/nutrit/nuw049 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Lee BY, Ferguson MC, Cox SN, & Phan PH. (2021). Big Data and Systems Methods: The Next Frontier to Tackling the Global Obesity Epidemic. Obesity (Silver Spring), 29(2), 263–264. 10.1002/oby.23062 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Lee BY, Mueller LE, & Tilchin CG. (2017). A systems approach to vaccine decision making. Vaccine, 35 Suppl 1, A36–A42. 10.1016/j.vaccine.2016.11.033 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Lee SK, Sun J, Jang S, & Connelly S. (2022). Misinformation of COVID-19 vaccines and vaccine hesitancy. Sci Rep, 12(1), 13681. 10.1038/s41598-022-17430-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Lewis D. (2022). Why the WHO took two years to say COVID is airborne. Nature. https://www.nature.com/articles/d41586-022-00925-7 [DOI] [PubMed] [Google Scholar]
  42. Limoges C, Scott P, Schwartzman S, Nowotny H, & Gibbons M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. The New Production of Knowledge, 1–192. [Google Scholar]
  43. Mabry PL, Pronk NP, Amos CI, Witte JS, Wedlock PT, Bartsch SM, & Lee BY. (2022). Cancer systems epidemiology: Overcoming misconceptions and integrating systems approaches into cancer research. PLoS Med, 19(6), e1004027. 10.1371/journal.pmed.1004027 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. McGlashan J, de la Haye K, Wang P, & Allender S. (2019). Collaboration in complex systems: Multilevel network analysis for community-based obesity prevention interventions. Scientific Reports, 9(1), 12599. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. McGreavy B, Lindenfeld L, Bieluch KH, Silka L, Leahy J, & Zoellick B. (2015). Communication and sustainability science teams as complex systems. Ecology and Society, 20(1). [Google Scholar]
  46. Naeem SB, Bhatti R, & Khan A. (2021). An exploration of how fake news is taking over social media and putting public health at risk. Health Information & Libraries Journal, 38(2), 143–149. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Neely SR, Eldredge C, Ersing R, & Remington C. (2022). Vaccine Hesitancy and Exposure to Misinformation: a Survey Analysis. J Gen Intern Med, 37(1), 179–187. 10.1007/s11606-021-07171-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Nowotny H, Scott P, & Gibbons M. (2001). Re-thinking science: Knowledge and the public in an age of uncertainty. SciELO Argentina. [Google Scholar]
  49. Piltch-Loeb R, Savoia E, Goldberg B, Hughes B, Verhey T, Kayyem J, Miller-Idriss C, & Testa M. (2021). Examining the effect of information channel on COVID-19 vaccine acceptance. Plos one, 16(5), e0251095. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Scheufele DA, Hoffman AJ, Neeley L, & Reid CM. (2021). Misinformation about science in the public sphere. Proceedings of the National Academy of Sciences, 118(15), e2104068118. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Sprengholz P, Betsch C, & Böhm R. (2021). Reactance revisited: Consequences of mandatory and scarce vaccination in the case of COVID‐19. Applied Psychology: Health and Well‐Being, 13(4), 986–995. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Vaughan E, & Tinker T. (2009). Effective health risk communication about pandemic influenza for vulnerable populations. American journal of public health, 99(S2), S324–S332. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Wang Y, McKee M, Torbica A, & Stuckler D. (2019). Systematic literature review on the spread of health-related misinformation on social media. Social Science & Medicine, 240, 112552. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Wontorczyk A, & Roznowski B. (2022). Remote, Hybrid, and On-Site Work during the SARS-CoV-2 Pandemic and the Consequences for Stress and Work Engagement. Int J Environ Res Public Health, 19(4). 10.3390/ijerph19042400 [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Zhou Y, Li R, & Shen L. (2023). Psychological profiles of COVID vaccine-hesitant individuals and implications for vaccine message design strategies. Vaccine X, 13, 100279. 10.1016/j.jvacx.2023.100279 [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES