Data-powered participatory decision making: Leveraging systems thinking and simulation to guide selection and implementation of evidence-based colorectal cancer screening interventions

Abstract

A robust evidence-base supports the effectiveness of timely colorectal cancer (CRC) screening, follow-up of abnormal results, and referral-to-care in reducing CRC morbidity and mortality. However, only two-thirds of the U.S. population is current with recommended screening, and rates are much lower for those who are vulnerable due to their race/ethnicity, insurance status, or rural location. Multiple, multilevel factors contribute to observed disparities, and these factors vary across different populations and contexts. As highlighted by the Cancer Moonshot Blue Ribbon Panel group focused on Prevention and Screening Implementation of Evidence-based Approaches, inadequate CRC screening and follow-up represents an enormous missed opportunity in cancer prevention and control. To measurably reduce CRC morbidity and mortality, the evidence base must be strengthened to guide the identification of: (1) multi-level factors that influence screening across different populations and contexts, (2) multi-level interventions and implementation strategies that will be most effective at targeting those factors, and (3) combinations of strategies that interact synergistically to improve outcomes. Systems thinking and simulation modeling (systems science) provide a set of approaches and techniques to aid decision makers in using the best available data and research evidence to guide implementation planning in the context of such complexity. This commentary summarizes current challenges in CRC prevention and control, discusses the status of the evidence base to guide the selection and implementation of multilevel CRC screening interventions, and describes a multi-institution project to showcase how systems science can be leveraged to optimize selection and implementation of CRC screening interventions in diverse populations and contexts.

The Promise of Timely Colorectal Cancer Screening and Linkage to Care

Deaths from colorectal cancer (CRC) can be prevented by timely screening, follow-up of abnormal findings (henceforth referred to as “follow-up”), and referral-to-care. Screening for CRC is highly effective and cost-effective in reducing CRC incidence and mortality.(Guy, Richardson, Pignone, & Plescia, 2014; Knudsen et al., 2016), However, uptake of CRC screening tests remains suboptimal, particularly among racial and ethnic minorities and those with poor access to care.(White et al., 2017) Based on accumulated evidence, the U.S. Preventive Services Task Force has given routine CRC screening its highest recommendation.(U.S. Preventive Services Task Force, 2016) Screening can be completed via multiple modalities, including fecal tests or colonoscopy.(U.S. Preventive Services Task Force, 2016) Despite this recommendation, fewer than two-thirds of US adults ages 50-75 are up to date with recommended CRC screening.(Centers for Disease Control and Prevention, 2013; White et al., 2017) In minority, low-income, uninsured, and rural populations, screening rates are considerably lower and CRC mortality rates are considerably higher.(Centers for Disease Control and Prevention, 2013; Mokdad et al., 2017; Siegel, Sahar, Robbins, & Jemal, 2015; White et al., 2017) In addition, ensuring timely follow-up of abnormal findings and referral for recommended care is essential, but remains problematic.(Selby et al., 2017) Given that death from CRC is potentially preventable with routine screening,(Bibbins-Domingo et al., 2016; Meester et al., 2015) early diagnosis, and timely treatment, there is an urgent need to improve CRC screening and follow-up nationally and in specific vulnerable sub-populations, including racial and ethnic minority, low-income, uninsured, and rural Americans. (Cole, Jackson, & Doescher, 2012, 2013; Klabunde, Joseph, King, White, & Plescia, 2013; Koroukian, Xu, Dor, & Cooper, 2006; Liss & Baker, 2014; Singh, Williams, Siahpush, & Mulhollen, 2011; White et al., 2017; Wilkins et al., 2012) The Cancer Moonshot Blue Ribbon Panel identified CRC screening and follow-up as an enormous missed opportunity in cancer prevention and control. Former Vice President Biden’s Cancer Moonshot Report included the “80% screened for CRC by 2018” national target, under Strategic Goal 4, Strengthen Prevention and Diagnosis, and the National Cancer Institute recently launched dedicated Moonshot funding opportunities to increase implementation of CRC screening interventions broadly. In addition, the Centers for Disease Control and Prevention supports a range of research funding and programmatic activities focused on CRC screening implementation,

Multilevel Factors Influence Timely CRC Screening and Linkage to Care

Understanding multilevel contexts is critical to improving CRC outcomes. A growing body of research has documented predictors of, and barriers to, CRC screening, follow-up and referral-to-care.(Davis et al., 2017; Holden, Jonas, Porterfield, Reuland, & Harris, 2010; Martens et al., 2016; Peterse et al., 2017; Pignone et al., 2014a; Plumb et al., 2017; Steffen et al., 2015; Wheeler et al., 2014) Correlates of CRC screening service delivery include factors at all levels of the socioecological model.(Mobley, Kuo, Urato, & Subramanian, 2010) For example, at the patient level, barriers to screening and follow-up include insufficient health insurance, concerns about healthcare cost, lack of knowledge of screening recommendations and benefits from screening, fear, fatalism, medical mistrust, and competing demands.(Beydoun & Beydoun, 2008; Chou, Rose, Farmer, Canelo, & Yano, 2015; Liss & Baker, 2014; Martens et al., 2016; Smith et al., 2016) At the provider, health system, and community level, barriers include lack of access to information and knowledge, leadership engagement, transportation, and access to diagnostic colonoscopies, among others.(Liang et al., 2016; Mobley et al., 2010) Direct and iterative communication with key stakeholders is essential to understanding how these multilevel factors manifest within a specific context.(Coronado et al., 2017; Coury et al., 2017; Schiff et al., 2017) Approaches that engage stakeholders meaningfully to better understand factors most relevant in their context are urgently needed to ensure appropriate matching and selection of evidence-based interventions (EBI) to increase CRC screening, follow-up, and referral-to-care.(Coronado et al., 2017; Coury et al., 2017; Liang et al., 2016) Once the context is understood, EBIs and implementation strategies can be selected to fit the identified multilevel determinants targeted, optimally, maximizing the potential for intervention success.

The Evidence-Based Pathway to Achieving Better CRC Outcomes

Multiple, multilevel EBIs have demonstrated effectiveness at targeting the aforementioned barriers and increasing CRC screening, follow-up, and referral rates across different populations and practice settings.(National Cancer Institute, 2017.; The Community Guide, 2016) EBIs are commonly disseminated in two ways, as EBI programs and EBI strategies. EBI programs include a combination of intervention and implementation strategies that have been tested and found to be effective in one or more research studies. The National Cancer Institute’s Research Tested Intervention Program website (rtips.cancer.gov) disseminates over a dozen CRC screening EBI programs. EBI programs offer the advantage that they may provide details on how the intervention was implemented and delivered, and also may provide intervention protocols and other materials to support implementation.(National Cancer Institute, n.d.) EBI programs have the disadvantage of being developed for a specific population and context and may be difficult to transfer to new settings.(Leeman et al., 2013)

In contrast, EBI strategies are typically disseminated in the form of recommendations from systematic reviews of the literature. The Centers for Disease Control and Prevention’s Guide to Community Preventive Services (Community Guide) website (https://www.thecommunityguide.org/) disseminates CRC screening EBI strategies. EBI strategies have the advantage that they are derived from multiple studies across different populations and contexts. Although they lack the specific guidance provided by intervention programs, they offer public health and cancer control decision makers the opportunity to mix and match EBI strategies to target multiple, multilevel determinants of CRC screening specific to their context.(Lipsey, 2005) This ability to more precisely target multilevel determinants is key to improving the implementation, effectiveness, and cost-effectiveness of CRC screening-focused interventions in populations and settings with disproportionately low rates of screening, follow-up, and referral-to-care. However, research reporting on the use of EBI strategies often lacks detail on contextual factors or implementation strategies that end users need to operationalize interventions to increase CRC screening in practice.(Davis et al., 2018)

Roadblocks along the CRC Screening Implementation Pathway

Although CRC screening interventions combined to target key multilevel factors (“multilevel EBIs”) have been effective, they have yet to achieve broad-scale implementation.(Escoffery et al., 2015; Hannon et al., 2013) Research therefore is needed to identify how best to disseminate, implement, and support the broad-scale use (i.e., scale-up) of these interventions. Efforts to scale-up multilevel CRC screening EBIs will be most successful when they align with the needs of the clinical, public health, and patient stakeholders involved in EBI adoption and/or implementation.(Leeman, Birken, Powell, Rohweder, & Shea, 2017) Adopting and implementing multilevel EBIs is complex and involves searching for, selecting, adapting, and combining EBIs to target multiple levels synergistically.(Leeman et al., 2015; Leeman, Calancie, et al., 2017; Leeman, Myers, Ribisl, & Ammerman, 2014; Leeman, Sommers, Leung, & Ammerman, 2011; Leeman, Teal, et al., 2014) The number of factors and system levels targeted, as well as stakeholders involved, contribute to the complexity of and uncertainty in optimal implementation.(Damschroder et al., 2009; Elwyn, Taubert, & Kowalczuk, 2007; Lanham et al., 2012) Therefore, EBI dissemination alone is not sufficient and needs to be coupled with training and tools to build public health and clinical providers’ and decision makers’ capacity to adapt EBI programs and select and integrate multilevel implementation strategies to address the multilevel factors influencing CRC screening efficiently (i.e., leveraging strengths and resources) in their specific context. Additionally, many stakeholders select interventions based on personal knowledge and opinion, feasibility, and basic opportunity and convenience, not based on data regarding effectiveness based on local contextual factors.

Understanding System Complexity and Intervention Interactions in Specific Contexts

Systematic approaches are needed to synthesize and harness the evidence base to guide multilevel intervention planning and implementation in specific contexts. Namely, research is needed to help end users determine not just what works, but what EBI strategies and what implementation strategies are best where.(Davis et al., 2018) In a recent review of interventions to increase CRC screening, the Community Guide found strong evidence in support of the effectiveness of multi-componentinterventions, particularly when they targeted factors at the community (e.g., addressing demand for services, access to services) and provider levels (e.g., improving offering of services).(The Community Guide, 2016) Generalizing these interventions to new contexts, however, is constrained by the lack of evidence about how multilevel factors influence EBI implementation and effectiveness in specific contexts. As Weiner et al. (2012) observed, in the absence of this understanding, “multilevel intervention designers run the risk of combining interventions that produce scattered, redundant, or contradictory effects.”(Weiner, Lewis, Clauser, & Stitzenberg, 2012)

Interventions to increase CRC screening, like many challenges in health care and public health, are “wicked” problems that are multilevel, complex, and interactive.(Kessler & Glasgow, 2011) Linear reductionist methods cannot adequately account for the emergent and contextual results in this case. Additionally, processes are needed to support the selection, adaptation, and implementation of EBI strategies into diverse settings. Increasing attention is being directed toward research strategies that blend rigor, relevance and are designed with scalability in mind.(Kessler & Glasgow, 2011) Novel methods are emerging to address this need. For example, participatory implementation science is one approach that supports “iterative, ongoing engagement between stakeholders and researchers to implement research into practice, create system change, and to address health disparities”.(Ramanadhan et al., 2018) Researchers in residence models and learning health care systems are other strategic approaches to blend knowledge and action. Work underway suggests a need to harness the synergy between improvement science and implementation science in order to improve cancer care delivery.(Koczwara et al., 2018)

Harnessing the Power of Data

As computing power has increased and data analytics have grown rapidly in sophistication, the era of ‘big data’ has presented unprecedented opportunities for improving population health and transforming healthcare delivery. Characterizing cancer screening trends and predictors of cancer outcomes regionally and nationally has become much easier due to increasingly available cancer registry linkages, all-payer health insurance claims data, longitudinal cohort studies, and other data gathering and harmonization efforts. CDC- and NCI-funded studies have identified considerable geographic variation within states in CRC screening patterns.(Davis et al., 2017; Wheeler et al., 2014; S. B. Wheeler et al., 2017) Other studies have illustrated stark geographic and sub-population differences in CRC screening follow-up and resolution, CRC treatment, and CRC mortality.(Amri, Bordeianou, Sylla, & Berger, 2013; Beyer, Comstock, Seagren, & Rushton, 2011; Cole et al., 2012, 2013; Davis et al., 2017; Klabunde et al., 2013; Koroukian et al., 2006; Liss & Baker, 2014; Mokdad et al., 2017) These studies have been made possible by considerable federal, state, and private investment in developing a diversity of data-powered resources, which integrate data from multiple sources, permitting identification and tracking of geographic “hotspots” (e.g., areas or populations where CRC burden is high and where screening rates are low) which can be targeted for intervention.(Siegel et al., 2015) In addition, multilevel data structures and analyses can facilitate a more nuanced understanding of the complex determinants of CRC screening, follow-up and outcomes.

Enhancing Implementation Planning through Participatory Systems Science Approaches

Systems science approaches are ideal complements to big data analytics in enhancing intervention and implementation planning. Once the multi-level determinants of screening and CRC outcomes are better understood and opportunities for intervention identified through the analysis of big data, stakeholders need tools to facilitate comparing, selecting, and anticipating the effects of combinations of potential candidate interventions and implementation strategies. In essence, stakeholders need technical assistance to understand how to interpret data and direct action, which requires participatory approaches. Participatory approaches involve co-learning and capacity building between stakeholders and researchers through collaborative selection of the issue/EBI, study design and execution, and analysis, dissemination, and extension of the evidence-base.(Ramanadhan et al., 2018) Participatory systems science approaches can aid stakeholders in interpreting quantitative data, understanding the larger context as well as appreciating contextual nuance qualitatively, specifying theories of change, and designing next step solutions. Participatory systems sciences approaches are inherently designed to anticipate and plan mindful of system complexity, build mental models to anticipate program effects and ultimately with sustainability in mind, and quantify the role of uncertainty; therefore, they are well-suited to planning the design and implementation of multilevel intervention programs. Systems science approaches are generally mixed-methods approaches in nature; for example, systems science tools can help transform diagrams of individuals’ mental models of change into quantified models that can be analyzed or used to estimate intervention impact. Table 1 summarizes several relevant quantitative and qualitative methods from system science. Importantly, these methods overlap and extend into each other; they are not categorically distinct. The extent to which these participatory system science methods are used and combined with each other or more traditional methods depends on project needs and can have utility in both fairly limited, discrete interactions with stakeholders, as well as more intensive ‘workgroup’ stakeholder sessions over time.

Table 1.

Summary of selected systems science approaches than can enhance implementation science

Approach	Description
The Five Rs	A structured framework to help stakeholders develop a richer understanding of the “system” around a focal issue.(United States Agency for International Development, 2014) The system is defined by meaningful results (in this case, related to CRC screening), individuals with a role in affecting those results, resources used or available to be used to improve results, rules governing action related to results (formal or informal), and key relationships between individuals, organizations, actions and/or contextual factors.
Discrete Choice Experiments	A technique used to quantify tradeoffs in preference or utility for particular features of a choice, for example, the tradeoffs that providers may make between cost, convenience, scheduling complexity, and test sensitivity/specificity when recommending colonoscopy versus stool tests for CRC screening. Discrete choice experiment data allow a range of stakeholders to make more explicit the relative influence that particular EBI attributes and levels of an attribute has on their decisions. This methodology can be very useful in understanding features of EBIs that stakeholders are most responsive to or how they value particular components of EBIs over other components.
Process Flow Diagramming (or swim-lane diagramming)	Process flow diagramming documents an intervention program’s implementation plan in terms of the processes and conditions affecting variation in pathways through which the intervention program is implemented.(Madison, 2005) It is particularly useful when the intervention’s implementation crosses system boundaries – individual stakeholder roles, disciplines, departments, organizations. A swimlane is a more detailed process flow diagram, which places process steps undertaken by each stakeholder in their own lane, while interconnecting processes and contextual factors affecting pathways across lanes over time. It is particularly helpful for identifying gaps, inefficiencies, or for strengthening handoffs between stakeholders.
System Dynamics Causal Loop Diagramming	Causal loop diagramming is a method that can be used to engage stakeholders in the identification of gaps, synergies, and lessons learned during intervention implementation. It can be used to generate a complexity-aware theory of change diagram, integrated to depict qualitatively the expected impacts of full multi-level interventions. System Dynamics methods are designed to improve intuition and uncover complex dynamics that can lead to “policy resistance” – when intervention impacts are “diluted, delayed, or defeated” by reactions of the system into which they are implemented.(Meadows, 2008; Sterman, 2006) System Dynamics methods include rich support for efficiently engaging stakeholders in iterating diagrams to anticipate, before implementation plans are final, the most likely sources of resistance.(P. Hovmand, 2014; P. S. Hovmand et al., 2012; Vennix, 1996) Facilitation will then extend stakeholders in adapting implementation plans to increase their fit (or match) to the context in which they will be implemented. Contextual factors that might support or undermine intervention program implementation and/or impact can then be considered to produce an explicit shared understanding of potential intervention effects. Also referred to as “dynamic hypotheses” of how interventions are likely to manifest, these diagrams serve as the foundation for both qualitative and quantitative testing, which supports “double loop learning,” in which assumptions are revisited, as interventions are implemented and evaluated in a given or varied contexts over time.
Systems Support Mapping	A structured systems thinking activity that guides stakeholders efficiently through a “deep dive” to reflect on how they see their responsibilities with an initiative, what they need to accomplish those responsibilities, resources around them they currently use, an assessment of how well they support the work, and, ultimately, what they wish for to be better supported in their work. Individual maps are shared to enrich the shared understanding of who does what, and how individuals’ effort and resources might be reallocated to support better implementation and/or impact. This is a method that can be used to facilitate needed and otherwise missing conversations to strengthen teams addressing complex problems.
Simulation Modeling	A mathematical representation of a complex set of interrelated variables and their functional relationships. Simulation modeling can be useful when projecting the health impact, cost, and/or cost-effectiveness of an intervention to a larger scale or further forward in time. For example, simulation models can estimate the total incremental costs required to coordinate and administer EBIs above and beyond what would be considered usual care. Analyses can take the perspective of public payers/health systems, private payers, healthcare providers, employers, or government entities. Cost assessments are absolutely vital to implementation planning to ensure that constrained resources are invested efficiently and responsibly, not only to ensure the greatest value for stakeholders and payers, but also to enable broader reach across more individuals in need. Simulation can also be useful for exploring potential effects of EBI adaptations, projecting “theories of change” quantitatively, and evaluating uncertainty. Simulation models can also be used as part of integrated Plan Do Study Act (PDSA) cycles and model-informed double loop learning (updating models and using them to adapt implementation plans).

Quantitative and qualitative participatory systems science methods can complement and extend each other considerably. For example, at the early stages of implementation planning, participatory focus groups can help stakeholders more concretely identify their policy, practice, and intervention questions. Process models can help clarify and improve stakeholders’ understanding of the model structure, and variable and structure elicitation exercises can help explore potential inputs and outputs to consider. A proposed mathematical simulation modeling plan with the following elements could then be presented to stakeholders for consideration: (1) description of problem statement, (2) description of the target population demographics, (3) description of potential intervention and implementation strategy scenarios (e.g., mail-out FIT kits, patient navigation), (4) model assumptions (e.g., reach of interventions, rate of adoption of interventions, etc.), and (5) illustrative results (so the group can react to and request different information from model analysis). Feedback on these elements are obtained and refined as appropriate, consistent with local realities, demands, and constraints (e.g., we might not model endoscopy facility expansion in a rural, sparsely populated area with little demand for, or likelihood of attracting, a new endoscopy center).

In later stages of implementation planning, the mathematical simulation model is modified as needed to simulate all stakeholder-driven, selected intervention scenarios. Then, stakeholders can interact with models and model outputs and interpret analysis findings. During these sessions, stakeholders can change parameters of interest (e.g., what happens if we decrease the rate of uninsured men) and see outcomes (e.g., percent of men up-to-date with screening) in real time. Simulated results can also be interrogated to gage stakeholder impressions and refine model assumptions, as needed. Sensitivity analysis also can used to explore the impact of uncertainty on outcomes. For example, easy-to-use web-based platforms can be used with stakeholders to examine how different levels of implementation success affect outcomes.

Case Example - Integrating Systems Science with Implementation Science to Improve CRC Screening and Outcomes

Development and use of systems science approaches and simulation for CRC program planning decision making has been used by our team through the Modeling Evidence-Based Intervention Impact workgroup within the CDC- and NCI-sponsored Cancer Prevention and Control Research Network (CPCRN). This workgroup is tasked with understanding the anticipated economic and health impacts of implementing various EBIs to improve CRC screening within specific geographic regions and sub-populations.(Hassmiller Lich et al., 2017) This workgroup has used big data analytics to understand screening trends and predictors, discrete choice survey techniques to understand underserved patients’ preferences for different CRC screening programmatic features, and simulation modeling to evaluate the cost-effectiveness of alternate EBI approaches to increase CRC screening on a population level.(Davis et al., 2017; Leah Frerichs et al., 2017; Hassmiller Lich et al., 2017; Pignone et al., 2014b; Wheeler et al., 2014; S. B. Wheeler et al., 2017) We previously tested the effectiveness and cost-effectiveness of implementing several interventions in the entire state of North Carolina including: mailed reminders for Medicaid enrollees; expansion of endoscopy facilities to increase access to colonoscopy in underserved areas; mass media campaigns targeting African Americans; and a voucher program providing free colonoscopies to uninsured individuals.(Hassmiller Lich et al., 2017) Findings suggested that stool-based testing was a preferred screening modality among populations experiencing screening disparities and that mailed reminder programs targeting low-income populations were particularly cost-effective.(Hassmiller Lich et al., 2017; Pignone et al., 2014b) These findings were recently used to inform a pragmatic quality improvement effort with NC Medicaid, Community Care of North Carolina, and the Mecklenburg County Public Health Department, which proactively mailed screening reminders and stool testing kits to unscreened Medicaid beneficiaries in a large, urban area in NC with relatively low screening rates.(Wheeler & Basch, 2017; Wheeler et al., 2014; S. Wheeler et al., 2017) The simulation model also has shown that increased access to health insurance through Medicaid expansion would be expected to reduce racial disparities in CRC outcomes and to generate cost-savings in the long term at the population level.(L. Frerichs et al., 2017) This existing microsimulation model has been adapted and is also being used to estimate CRC-related health and cost impacts of health insurance expansion in Oregon and to compare multiple EBIs that Oregon’s Coordinated Care Organizations are considering as options to increase CRC screening. Importantly, once developed, simulation models can be re-parameterized, recalibrated, and reanalyzed as needed to understand how different population dynamics, different intervention designs and strategies, different assumptions, and different levels of uptake affect programmatic success and ultimate return on investment.

Our state-specific approach to input data parameterization allows us to incorporate an understanding of efficiency of specific interventions and policies, taking into account the local nuances of population heterogeneity, setting-specific healthcare resources, and differential impact of interventions on individuals in different settings. We have gone to great lengths to characterize individual screening behavior based on an understanding of the association between key individual and community-level variables. This will more accurately reflect the impact of policy and practice changes on actual screening outcomes. In addition, our ‘real-world’ approach can help to identify unintended consequences of specific interventions on populations of heterogeneous individuals (e.g., to assess whether there is enough endoscopy capacity to absorb demand without creating overly long wait times in different regions of the state).

CRC simulation model structure

Our existing simulation model is geographically explicit to the census block level and its input parameters can be modified and updated easily to estimate outcomes from a variety of analytic perspectives. We have the ability to simulate the full spectrum of CRC outcomes, including: health behaviors (such as % of persons screened/up to date with screening recommendations), incident cancers, stage at diagnosis, cancer deaths, quality adjusted life years (QALYs), expected costs, cost per person screened, cost per cancer case averted, cost per cancer death averted, cost per QALY gained, effects of policies and interventions on disparities, effect of policies and interventions on local healthcare service demand, and more. The exact outcomes to be assessed are driven and prioritized according to stakeholder needs and interests, balancing time and resource constraints. This individual-level simulation environment has 6 modules: the population module, the natural history module, the healthcare infrastructure module, the “screening, diagnosis, treatment and surveillance module”, the intervention module and the behavior/lifestyle module (Figure 1). We use the population module to specify demographic and geographic characteristics of our hypothetical population, and the natural history module to specify the onset and trajectory of any cancer (including colorectal cancer). We use the healthcare infrastructure module to specify characteristics of healthcare facilities in an area of interest, and the “Screening, diagnosis, treatment and surveillance” module to specify the current screening patterns. We use the healthy lifestyle module to specify behaviors that amplify or mitigate the risk of the cancer of interest. This model has granted us greater insight into the comparative public health impact, costs, and cost-effectiveness of various EBIs to improve CRC screening in specific states and regions (currently, North Carolina and Oregon).

Figure 1.

CRC Simulation Module Schematic

CRC simulation model input parameters

Our existing simulation model uses Census-derived local population data, natural history and epidemiologic data, and healthcare utilization data, to simulate CRC risk, CRC screening behavior and treatment receipt, and, ultimately, cancer outcomes under usual care and a variety of “what if” intervention scenarios (Figure 1).(Hassmiller Lich et al., 2017) Simulation models synthesizing data from different study types are often used to determine both budget impact (Sweet et al., 2011) and cost-effectiveness.(Lee et al., 2010) A variety of the best available input data sources has been collated and integrated to comprehensively evaluate the effects of specific policies and interventions on CRC outcomes to assist with local public health planning and capacity development (Table 2).

Table 2.

CRC Simulation Modules and Relevant Input Data Sources

Module	Input Data Sources
Clinical or health policy intervention scenarios	Literature reviews Stakeholder interviews
Healthy lifestyle	Behavioral Risk Factor and Surveillance System National Health Interview Survey National Health and Nutrition Examination Survey Claims data (Medicare, Medicaid, Commercial/Private)
The population	US census US life tables American Community Survey Public Use Microdata Sample RTI synthetic population
Healthcare infrastructure	Area Resource File State Medical Facilities Plan Health insurance claims data American Hospital Association Federally Qualified Health Centers Area Health Education Centers
Cancer screening, diagnosis, treatment, & surveillance	Behavioral Risk Factor and Surveillance System National Health Interview Survey National Health and Nutrition Examination Survey Claims data (Medicare, Medicaid, Commercial/Private) Clinical guidelines (e.g., ACG and USPSTF) Area Resource File State Medical Facilities Plan
Natural history of cancer	Epidemiologic data/models Clinical evidence Literature reviews Expert and/or stakeholder interviews Cancer registries

Engaging stakeholders in simulation-guided decision support

We have used participatory group-model building to work collaboratively with sponsoring organizations and public health professionals to brainstorm, define, and refine key questions that can be addressed using our simulation tools. The goal is to ensure that model assumptions (strengths and limitations) and analyses are fully transparent and responsive to stakeholders’ needs. Such activities should build confidence and allow adaptations, as appropriate, of model assumptions, research questions, and simulated scenarios. We are interested in providing stakeholders with an unbiased source of quantified decision support regarding investments in, and implementation of, specific interventions and policies in geographically specific areas and populations. To that end, stakeholders could help inform the research questions asked of simulations.

Evaluating the utility of systems science approaches

Throughout this process, mixed methods approaches can be used to understand: (1) stakeholders’ knowledge/familiarity and level of comfort/satisfaction with simulation/systems science approaches; (2) the extent to which systems science approaches enhanced stakeholders’ understanding of the barriers, facilitators, opportunities, and threats to CRC screening; (3) the extent to which this approach affected or is expected to affect decision making; and (4) guidance for future implementation planning using this approach. These domains can be explored via stakeholder surveys and focus groups to be conducted at the end of each stakeholder workgroup meeting.

Developing technical assistance/training materials for using simulation/systems science-supported implementation planning

In addition to detailed modeling documentation, training protocols, written guidance resources, and technical assistance templates are needed about how to use simulation/systems science approaches for implementation planning, and these materials could be archived and broadly disseminated to external audiences. Technically sophisticated modeling approaches that are well-supported by detailed, vetted documentation will support community, state, and national-level learning and decision making, as well as lead to more efficient and sustainable sharing of research evidence.

Conclusion

The Cancer Moonshot Blue Ribbon Panel emphasized implementation of evidence-based approaches to optimize cancer screening and follow-up, noting that inadequate CRC screening and follow-up represents an enormous missed opportunity. To measurably reduce CRC morbidity and mortality, the evidence base must be strengthened to guide the identification of: multilevel determinants of screening across different populations and contexts, multi-level EBIs and implementation strategies that will be most effective and cost-effective at targeting those factors, and combinations of EBIs and implementation strategies that complement each other and interact synergistically to improve outcomes at a reasonable cost.

The CDC, in particular, is well-positioned to influence the process through which EBIs and implementation strategies are selected, adapted, and scaled up. The CDC has implemented the Colorectal Cancer Control Program in 23 states, 6 universities and one tribal organization in which EBIs from the Community Guide are being implemented in clinics within health systems.(Centers for Disease Control and Prevention, 2017) The CDC is currently collecting evaluation data to measure best practices, lessons learned, and costs of implementing the EBIs.(Satsangi & DeGroff, 2016) The hope is that these evaluation data will feedback into the simulation models described in this paper to further project impact and understand longer-term public health implications of these activities. The ultimate goal for the CDC is to produce tools, based on data-driven models that will drive decision making at the health system/clinic level to deliver cancer screening to save lives.

Participatory systems science methods, including systems thinking and simulation, provide a set of approaches and techniques to aid decision makers in using the best available data and research evidence to guide implementation planning in the context of complexity; yet, these approaches are underutilized in implementation science. We argue that systems science methods can enable more data-powered decision making by engaging stakeholders more meaningfully in the science, anticipating intervention impacts and unintended consequences through qualitative and quantitative inquiry, and providing stakeholders and public health practitioners with tools and technical assistance to bring this work outside of academic forums and into boardrooms where decisions are happening.