Enhancing Dissemination and Implementation Research Using Systems Science Methods

Abstract

PURPOSE

Dissemination and implementation (D&I) research seeks to understand and overcome barriers to adoption of behavioral interventions that address complex problems; specifically interventions that arise from multiple interacting influences crossing socio-ecological levels. It is often difficult for research to accurately represent and address the complexities of the real world, and traditional methodological approaches are generally inadequate for this task. Systems science methods, expressly designed to study complex systems, can be effectively employed for an improved understanding about dissemination and implementation of evidence-based interventions.

METHODS

Case examples of three systems science methods – system dynamics modeling, agent-based modeling, and network analysis – are used to illustrate how each method can be used to address D&I challenges.

RESULTS

The case studies feature relevant behavioral topical areas: chronic disease prevention, community violence prevention, and educational intervention. To emphasize consistency with D&I priorities, the discussion of the value of each method is framed around the elements of the established Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework.

CONCLUSIONS

Systems science methods can help researchers, public health decision makers and program implementers to understand the complex factors influencing successful D&I of programs in community settings, and to identify D&I challenges imposed by system complexity.

BACKGROUND

Dissemination and implementation (D&I) research seeks to understand and overcome barriers to adoption of interventions that address complex problems; specifically interventions that arise from multiple interacting forces crossing socio-ecological levels. [1]. Delays between cause and effect, nonlinear relationships between variables, and unanticipated system behavior are common hallmarks of complexity present in D&I. For example, implementation of interventions aimed at preventing tobacco use among youth without consideration of the mix of factors that influence use, such as product appeal (e.g., flavors and packaging), pricing, point of purchase displays and access (e.g., placement and density of outlets near schools) may have limited impact. While these challenges are documented across many areas of public health [2–5], resolving them requires new methodological approaches that can capture the complexity of the environment in which D&I research takes place.

Systems science methods have been developed to understand connections between a system’s structure and its behavior over time. Many such methods exist including (but not limited to), System Dynamics (SD), Agent-Based Modeling (ABM), Network Analysis (NA), microsimulation, discrete event modeling, Markov modeling, many operations research and engineering methods, and a variety of other modeling and simulation approaches. While still somewhat novel in D&I research, the utility of systems science methods for addressing health questions has been demonstrated [6]. In fact, the National Institutes of Health, the agency of the United States government responsible for health-related research, has highlighted the utility of systems science methods in D&I research in several Funding Opportunity Announcements, including PAR-11-314 and PAR-11-315 Systems Science and Health in the Behavioral and Social Sciences and PAR-13-054, PAR-13-055, and PAR-13-056, Dissemination and Implementation Research in Health.

In this paper, we present three system science methods (system dynamics modeling, agent-based modeling, and network analysis) which can be used to conduct research to improve one’s understanding about real world systems and how best to translate evidence into practice. For each method, we present a case study and discussion of the contribution of each method around elements of the widely used Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) conceptual framework [7], often applied to improve success in D&I research[8] (see Table 1).

Table 1.

RE-AIM Model dimensions and potential role of system science methods for each

MODEL DIMENSIONS	SYSTEMS SCIENCE METHODS
System Dynamics	Agent-Based Modeling	Social Network Analysis
Reach: What proportion and how representative are participants of the target population?	Explicate determinants of less-than-optimal reach and potential solutions.	Critically synthesize existing knowledge and data to determine model specifications and settings; Articulate multi-level key influencing factors, the theoretical mechanistic relationships and establish parameters.	Identify peripheral and isolated members of the target population who may be difficult to reach.
Effectiveness: What is the success rate if implemented according to intervention protocol?	Study dynamic determinants of the impact of intervention, intended and otherwise; may quantify if value sufficient.	Simulate the impact of the intervention on agent behavior.	Test proximal and distal relational outcomes of programs (e.g., increased communication; increased social capital).
Adoption: What proportion of people/settings/practices will adopt/participate in the intervention?	Bring together potential adopters to study challenges to adoption	Explore spread of behaviors and reach of intervention by varying influencing factors.	Examine the role the relationships play in access to information about and decisions to adopt new interventions; Understand how individuals’ positions in the network influence their propensity to adopt the intervention; Identify influential community members who can assist in encouraging adoption.
Implementation: To what extent is the intervention implemented as intended in the real-world?	Diagram potential short term threats to successful implementation and indirect consequences of implementation that might undermine impact	Simulate “real-world” setting and explore impact of multiple intervention configurations (e.g. community wide verses spatially focused).	Understand how individuals’ positions in the network influence their frequency of implementation; Identify influential community members who can assist in implementation.
Maintenance: To what extent is the program sustained over time? What happens to the program over time?	Diagram potential threats to longer-term maintenance, and how each might be avoided/addressed	Run multiple iterations of the models to explore what happens to the spread and dynamics of targeted behaviors.	Identify areas of the network to strengthen for D&I efforts and create recommendations; Identify influential community members who can sustain the program over time

System Dynamics Modeling (SDM)

SDM offers a methodology for framing, understanding, and discussing challenges embedded within complex systems. This method seeks to improve the ability to anticipate likely trajectories of intervention effects (or problems in the absence of intervention) over a defined time horizon, where the pathways from interventions to outcomes can be complicated, slow, and best understood via computer simulations. SDM has been used effectively since the 1970s to examine a range of health areas [9, 10]. SDM helps invested stakeholders, specifically individuals engaged in the intervention implementation and dissemination (e.g. policy makers), transform their mental models into explicit causal diagrams (graphical depiction of the salient variables and their cause-effect relationships). If a full simulation model is desired, the causal diagrams are quantified by consulting the extant literature and other sources of evidence to confirm directionality and to estimate effect sizes. Differential equations are used to express the relationships between variables as rates of change over time. Computer programs are employed to perform the calculations and then display the dynamic model graphically. This iterative process of brainstorming, critiquing, and learning helps develop a shared understanding of the problem under study. Once quantified models are built, facilitating live computer experiments (“What If” questions) with stakeholders can be a powerful tool for communicating across a multitude of sectors, establishing feasible targets for change, and motivating collaboration for action. Many dissemination and implementation researchers may be familiar with program logic modeling. Note that SDM is similar at the initial step – both produce a graphical depiction of the causal pathways. But if a goal of the research is to better understand and explore the dynamic aspects of the relationships within the system, only SDM provides the tools for dynamic simulation. SDM is a compartmental model, differing from other microsimulations that model individuals within their context [11]. Compartmental models represent groups of people in categories, segmented by sex and age groups, and other defined subgroups.

Case Study 1: Using an SDM to inform community-level policy decisions

The Centers for Disease Control and Prevention (CDC), with co-funding from the National Institutes of Health (NIH) developed the Prevention Impacts Simulation Model (PRISM), an interactive health policy simulator that can support local community strategic planning and evaluation. PRISM brings greater structure and evidence to the challenge of reducing the burden of chronic diseases [12–14]. Working closely with community members and subject matter experts at the CDC and the NIH, PRISM was developed to address questions such as 1) How does local context affect the major risk factors for cardiovascular disease (CVD), population health, and costs?, and 2) How might local health leaders better choose their policy efforts given limited resources?

PRISM (Figure A) is a learning tool for stakeholders to experiment and see for themselves what future health trajectories might look like over a 20-year time horizon, based on a careful integration of science. It depicts multiple steps of causation, accumulation, and feedback that result in changes in risk factor prevalence, acute events, and health and cost outcomes. PRISM tracks health events, disability and death, and costs attributable to risk factors and risk factor management. Its scope encompasses CVD, diabetes, obesity, blood pressure, cholesterol, smoking, secondhand smoke exposure, physical activity, diet, air pollution, and emotional distress, and this version of the model simulates 34 interventions targeting health behaviors, environmental exposures, and disease progression through a range of channels.

Figure.

Figure A. Systems Dynamics Example

Figure B. Agent Based Model: Spatially focused intervention

Figure Description: Each agent is represented as the silhouette of a person (adults and juveniles). The colors of the agents change according to their status (e.g. juveniles committing crimes are red). When an adult becomes activated to report crimes the agents is represented as squares. The purple clouds indicate spatial areas of high crime.

Figure C. PAS Teacher Instructional Methods Advice Network in a School

Figure Description: Each circle represents a teacher and the lines represent advice about instructional methods, with arrows pointing from advice giver to advice receiver. Here, the lead teacher selected by the principal to endorse the PAS program (#2) and her structurally similar teachers (i.e., those that share at least 1/3 of the same relationships) are shaded in dark green. An alternative teacher (#11) and her structurally similar teachers are shaded in light green. This illustrates that the lead teacher selected by the principal may not have the largest region of influence for encouraging adoption and implementation by peers.

Figure D. PAS Teacher Behavior Management Advice Network in a School

Figure Description: Each circle represents a teacher and the lines represent advice about behavior management, with arrows pointing from advice giver to advice receiver. Red arrows indicate reciprocal advice relationships and the circled section of the network contains teachers with fewer advice relationships. These features of the network indicate potential barriers to reach.

Travis County (Austin) Texas was the first community to use the original version of PRISM [12]. Members of both the local public health department and community members participated in building the model. Engaging decision-makers and community leaders in both the development and use of the model had a strong positive impact on adoption of the chosen intervention program. In Austin, PRISM offered a catalyst for multiple stakeholders to align and develop a comprehensive strategy for reducing chronic diseases and associated costs that all at the table could really support [14]. There may be several reasons for this: SDM provides an opportunity for stakeholders to visually see the intervention choice set and, together from a systems perspective, discuss the rational for each. By participating in model building, stakeholders see how their own work fits within the larger scope of other stakeholders’, thereby offering opportunities for partnerships. They can also test their own mental models, and see the relative power of policy options [12].

A model such as PRISM can be used to directly simulate and compare the reach of alternate interventions under consideration. For example, Travis County had not implemented the maximum level of air quality restrictions. PRISM simulations demonstrated that doing so was among the most powerful interventions, due to its broad reach. Furthermore, when the potential reach of a simulated policy is uncertain, the model can be used to quantify how sub-optimal reach of implemented interventions might compromise the relative power of the intervention – improving decision-making. One could also track disparities explicitly in an SDM, and identify those policies that will best reach the most disadvantaged.

One of the strengths of SDM is that it can help estimate the effort required to implement and achieve identified goals in a specified time frame. For example, Levy et al. [13] used a similar model to estimate what evidence-based policies would need to be implemented to reach the Healthy People 2010 goals for smoking prevalence. They found that no combination of existing policies would work. This result pointed to the need for new innovative evidence-based policies. In other cases, multiple paths might be identified to reach established goals, and the model can help quantify the scale of successful implementation required for each.

SD can improve effectiveness by helping decision makers select from among the available evidence-based interventions the combination that is best suited to the local context. PRISM can be used to determine how many people are affected by a policy (in both desirable and undesirable ways) and the extent to which they are affected. By taking into account the interventions already in effect and/or the demographic characteristics of the local population, resources can be spent on those additional programs that are most locally effective. Moreover, synergistic effects that appear when policies are combined can be identified and can be used to make the most impactful decisions possible with a given set of resources. SDM can help uncover what otherwise might be unintended consequences of favored intervention approaches that might threaten their ability to reach key subpopulations or to produce lasting improvements. When PRISM was used in the Mississippi Delta, a first-line strategy advocated by many was to improve the health of local disadvantaged populations a priori was to increase their access to care. Contrary to stakeholders’ initial opinions, the PRISM model demonstrated how increasing access to care only, without increasing capacity for health care delivery, would result in poorer quality of care for everyone. By providing more people with access to care, the system would be taxed and not be able to keep up with demand. Providers might have to delay services or spend less time with patients, which would result in lower quality of care and ultimately worse health outcomes. By observing the quantified systems-level impact, a different decision was made.

SDM allows users to see how the consequences of their actions are likely to unfold over time. Realistic expectations support maintenanceparticularly important in public health where many interventions take time to impact key outcomes. SDM allows the user to remain committed to interventions that may make things worse before better (a common phenomenon), and offers shorter-term expectations against which to track performance. Additional information on SDM and PRISM can be found elsewhere [12–15].

Agent-Based Modeling (ABM)

ABM is a computational method used to examine the actions of agents (e.g. individuals) situated in environment (e.g. neighborhood). Unlike equation-based models, ABMs specify decision rules controlling dynamics, such as If-Then statements and mechanistic interactions among agents, and simulate them using computational software. This allows for a more flexible modeling approach [16]. When the program is run, agents interact with one another and their environment often resulting in surprising insights about behavior of agents and the system. Much of the ABM and public health research has focused on infectious disease dynamics and epidemic containment [17–19], and increasingly public health scientists are using the method to examine social and behavioral health issues [20–25].

Case Study 2: Using ABM to inform context-specific, cost effective community violence prevention interventions

A interdisciplinary study team affiliated with the Public Health Dynamics Lab (PHDL) at the University of Pittsburgh and including individuals with backgrounds in community health, intervention development, translational research, computational modeling, and violence, worked together to develop a conceptual ABM to explore the impact and effectiveness of community crime interventions [22, 26]. The ultimate goal of the work is to provide a computational tool that can be used to assist in strategic and implementation planning to prevent or reduce crime in local communities. This project has successfully generated a conceptual ABM to assist with such a process, yet it is still in a formative stage and the model is being refined in order to serve as a more accurate and useful tool. Model building such as this offers the model as a transdisciplinary object integrating best evidence and supporting ongoing decision-making.

Community crime and violence, like most other behavioral and community health issues, is a complex problem that is influenced by a range of individual and community-level factors. . This case example illustrates the complex behavioral dynamics and differential cost and effectiveness of alternative community-level crime intervention approaches. Building upon existing community connections and prior research, the project’s academic lead invited the executive director of a local community violence prevention agency to join the team. Over a year of bi-weekly meetings led to the development of the conceptual model. The meetings included discussion and input from team members, a review of published literature, and input from additional content experts on key multi-level factors influencing the behaviors of the model’s agents.

The ABM was developed using NetLogo, where agents representing individual residents move and interact on a two-dimensional grid simulating a neighborhood. Juvenile agents are assigned initial random probabilities of perpetrating a crime and adults are assigned random probabilities of witnessing and reporting crimes. The agents’ behavioral probabilities are modified over time depending upon exposure to other agents’ crime perpetration and/or crime reporting behaviors. Juvenile and adult agents interact within the simulated neighborhood. The Theory of Reasoned Action guided the behavioral parameters of the agents [27]: if perceived reward > perceived risk, then action is taken. Each juvenile’s initial perceived reward was assigned randomly to individuals and declined with age. Likewise, perceived risk depended on the individual’s own experience and exposures as the model is run. Findings from the Pathways to Desistence Study (PDS) contributed to the behavioral probabilities randomly assigned randomly to juveniles in the model [28]. PDS is a study of 1,355 serious adolescent offenders in Philadelphia and Phoenix and was designed to focus on the factors that contribute to adolescents’ engagement with crime and the justice system.

Figure B presents a screenshot and bird’s eye view of the NetLogo agent-based model of the spatially focused intervention. A geographic community is represented by the entire two-dimensional grid which is further subdivided into square blocks. Each agent is represented as the silhouette of a person (adults and juveniles). Colors and shapes are used to show change over time, for example, if an adult agent becomes activated to report crimes or a juvenile commits a crime. Theses colors are assigned to the agents help illustrate that the agents are performing according to the program’s design. For example, adult who witness an offense and report it are green and an arrow points to the offender who has been reported; adults who witness an offense but do not report it are yellow; adults that have not witnessed an offense on the current time step are blue. Juveniles are purple unless they have offended in the last step. Offenders who have not been reported are red; offenders who have been reported and will be punished are orange. The clouds indicate spatial areas of high crime. Additional details and figures can be found in the original article [26].

Much like with SDM, engaging relevant community, law enforcement and policy stakeholders in the ABM process contributes to the practical and valid application of the model, helping to increase buy-in and adoption of the interventions examined in the model. Through the team’s diverse academic and community involvement they have begun discussions about how to expand facilitated interaction with an ABM to improve strategic and implementation planning around problems related to community crime and violence with local law enforcement, public housing, and community and policy leaders. Expansion of the team’s stakeholder base provides further opportunities to test and refine the model and to explore new opportunities for direct community engagement with this method.

As illustrated in this case example, ABM can allow for the efficient investigation of a complex problem, like community crime interventions, that otherwise would be costly and time intensive. The spread of the crime behavior and reach of the community interventions is directly observable in the ABM. The results from this ABM provide valuable insight into the spread and containment of crime behaviors. Such information can be used to direct resources towards the intervention strategy shown to have the greatest impact and reach within the at-risk populations.

The process of reviewing and refining the model contributed to the active engagement of project team members who wanted to see their perspectives reflected in the model. In addition to increasing the likelihood that the model reflects reality, this provides much needed insight on the requirements for and the likelihood of successful implementation of different intervention strategies.

The ABM, by simulating the impact of various community interventions on crime-related behavior, helped the team explore the likely effectiveness of each. When a simulated community intervention occurred, a fraction of adults became activated to report the observed crime. Two kinds of community-based interventions were modeled. In a community-wide crime intervention (i.e., a community-wide community block watch program), a segment of the adults in the community were randomly selected from the entire community to be activated. In a spatially-focused community-based crime intervention (i.e., a targeted block watch intervention), a segment of the adults were activated, but the activated adults were all selected from the block having the highest prevalence of crime. While spatially-focused intervention yielded strong localized reductions in crimes, such interventions move crime to nearby communities, dampening the overall effectiveness. Community-wide interventions reduced overall community crime offenses in the model to a greater extent.

Multiple iterations of the ABM were run to explore and examine the extent to which a community program could be sustained and maintained over time and how the crime behavior would be contained or spread. This type of simulation ensures a better balance of long-term sustainability with short-term feasibility and impact. Additional information about the ABM model of community violence prevention addressed in this section can be found elsewhere [22, 26].

Network Analysis (NA)

NA examines the structure of relationships between a set of nodes (e.g., people, organizations). NA moves beyond studying individual attributes, groups, or dyadic interactions to consider relational patterns within a system. Network data can be collected using a multitude of approaches including surveys, interviews, observations, and archival methods [29–31]. Relationships can be operationalized as discrete (e.g., does a relationship between organizations A and B exist?) or valued (e.g., how frequently does information flow between organizations A and B?). Moreover, relationships can be symmetric (e.g., do heath care providers A and B socialize with one another?) or directional (e.g., does health care provider A give advice to provider B?). Computational advances have increased the utility of NA in measuring dynamic relational patterns [32], and have spurred growth in the use of NA to understand systems. Recent work has highlighted how NA can be used to support D&I efforts. For example, recent studies have illustrated how NA can be used to identify key stakeholders such as school personnel who are best positioned to influence the successful dissemination and implementation of behavioral interventions among their peers [33–35].

Case Study 3: Using NA to understand pathways for dissemination and implementation of teacher practices to support academic success

Here, we illustrate the use of NA to understand the dissemination and implementation of the Promoting Academic Success Project (PAS), an intervention designed to improve educational outcomes for minority boys in elementary school. One major component of PAS is teacher professional development. In each school, principals selected “lead teachers” to encourage teacher attendance at professional development sessions and promote teacher implementation of practices learned in these sessions to improve minority boys’ educational outcomes. The research question was"What are the pathways by which teacher’s practices, once learned at the professional development sessions, might spread to other teachers in the school?” The answer to this question could help school system officials figure out how to maximize the number of teachers following recommended practices.

Researchers at Michigan State University collaborated with principals and teachers to examine teacher advice networks in five elementary schools implementing PAS. Drawing on diffusion theories [36–38], the study team expected teachers’ existing advice relationships would be critical to the spread of PAS practices from lead teachers to other teachers. Specifically, the study team conducted brief interviews with all teachers and principals in each school [39]. Teachers and principals identified other teachers in their school from whom they received advice around four different issues related to minority boys’ education: (1) instructional methods (2) promoting positive relationships (3) family involvement and (4) behavior management. Answers to these questions were used to create four separate teacher advice networks for each school. The study team employed NA to examine these four teacher advice networks, and to understand how information about PAS practices could spread from lead teachers to other teachers in each school.

As the PAS project illustrates, NA can be a valuable method for illuminating processes of dissemination and implementation, including each element of the RE-AIM framework (see Table 1). Previous research suggests that the adoption of new practices spreads among individuals who are structurally equivalentoccupying similar positions in the network’s structure [40]. Thus, NA can be useful for understanding how individuals’ network position influences their propensity to adopt new intervention strategies. The study team was interested in whether “lead teachers” selected by principals were best positioned in the network to facilitate teacher participation in strategies to enhance minority boys’ education. In most cases, results of the NA suggested that they were not. The instructional methods advice network in one PAS school is provided in Figure C, with the principal’s lead teacher (#2) coded in dark green. NA suggests that this lead teacher is structurally similar (i.e., shares at least 1/3 of the same relationships) with only one other teacher (#14). Thus, her region of influence, represented by the dark green shading, is minimal. In contrast, teacher #11 coded in light green is structurally similar to five of her colleagues, and possesses a much larger region of influence, represented by the light green shading. These findings suggest that teacher #11 is better positioned to encourage teacher adoption of PAS strategies, and highlight that principals do not possess the bird’s eye view of the school’s relational structure that NA can provide.

The study team used NA to identify potential barriers to reaching teachers targeted by PAS, which can also help understand which students will and will not be reached overall. Teacher advice networks in PAS schools exhibited relatively low levels of density (i.e., the proportion of present to possible relationships) and reciprocity (i.e., the proportion of relationships in which each party nominates the other). In Figure D, low levels of density and reciprocity in the behavior management advice network result in barriers to reaching teachers including isolated, peripheral teachers (#1, 7, 12, & 13) and little two-way communication. This may hinder the flow of information across the school about effective behavior management strategies and the PAS professional development component. Using these results, the study team was able to make recommendations for improving reach. For example, to improve reach to isolated and peripheral teachers and to increase two-way communication, the study team recommended creating formal (e.g., pairing more and less experienced teachers) and informal (e.g., coffee dates) mentoring opportunities.

Like adoption, individuals’ consistent and regular implementation of new practices is influenced by their structurally equivalent peers in the network [39, 40]. Thus, NA can be used to track how the social relationships in a system influence the frequency and quality of implementation. Additionally, NA can highlight influential individuals who can assist in implementation efforts. In the PAS project, Figure C suggests that teacher #11 is structurally equivalent to more of her colleagues than the principal’s chosen lead teacher (#2). Thus, teacher #11 is better positioned to encourage the implementation of PAS instructional strategies – if she is (or can be made into) a successful implementer.

NA can also be helpful for understanding and improving program effectivenesswhen the intervention is directed at networks. Many interventions have proximal or distal relational outcomes such as increased communication or social capital. Longitudinal NA (i.e., pre- and post-intervention NA) should be used to evaluate the effectiveness of community capacity building or other efforts over time [35]. For example, NA can be used to see if organizations increase their resource sharing ties. Better understanding a given network shaping D&I success can also support the selection of more effective interventions.

NA can provide information about areas of the network that can be strengthened to facilitate the maintenance of interventions over time. The study team provided feedback to teachers and principals on the features of teacher networks in each school that created barriers to the D&I of PAS practices and brainstormed recommendations for improving the capacity of teacher networks to enhance D&I efforts. For example, the study team noted trends of low density and reciprocity in teachers’ advice networks and were able to work with teachers and principals to identify several recommendations for improving communication (e.g., hosting informal mentoring meetings). Finally, NA can be used to pinpoint influential community members who are best positioned to help sustain the program in the absence of researchers. For NA models of educational intervention similar to the one addressed in this section, please see [34, 39, 40].

DISCUSSION

Systems science methods can help researchers, public health decision makers, program implementers, community members and other stakeholders to understand the complex factors that potentially challenge successful D&I of interventions. Our application of the RE-AIM components demonstrates how systems science approaches like SDM, ABM, and NA can enhance efforts to assess barriers or facilitators to D&I and their potential impact on desired outcomes (Table 1). The three case studies illustrate how the modeling process can serve as a tool to present opportunity to select the most locally effective interventions, concretely understand the likelihood that the intervention could be adopted, reach the target population and be implemented and maintained with intended effects in a dynamic real-world context. Our examples feature a variety of topical areas relevant to behavioral medicine and many other examples of these methods can be found in the literature. There is great potential for systems science methods to further D&I research efforts across different health conditions, research questions, and settings. We have attempted to give a brief introduction to three of the most useful systems science methods for D&I research. However, a full understanding of these methods, including comparisons and contrasts between them requires further reading and study on the part of the interested reader. Many resources to support learning about systems science are available through the NIH Office of Behavioral and Social Sciences Research http://obssr.od.nih.gov/scientific_areas/methodology/systems_science/index.aspx