An Application of the Science Impact Framework to the Cancer Prevention and Control Research Network from 2014-2018.

Abstract

The Cancer Prevention and Control Research Network (CPCRN) is a strategic collaborative effort focused on accelerating the dissemination and implementation of evidence-based cancer prevention and control interventions to communities. In 2014, the CPCRN Coordinating Center began collecting information in alignment with the Centers for Disease Control and Prevention’s (CDC) Science Impact Framework. The Science Impact Framework is a CDC-developed approach to trace and link CDC science to events and/or actions recognized as influential to public health, beyond peer-reviewed publications. The purpose of this paper is to highlight the impact of CPCRN activities using key indicators guided by the CDC’s Science Impact Framework. We reviewed annual progress reports submitted by CPCRN centers from 2014-2019 to identify the impact indicators. The CPCRN activities were linked to four domains from the Science Impact Framework and its key indicators: Disseminating Science (presentations, training, general communication, and other communication reports), Creating Awareness (requests for expertise, and feedback), Catalyzing Action (grant applications, partnerships and collaborations, research & development, advocacy groups, office practice/point of care changes, and technology creating), and Effecting Change (building public health practice, creation of registries/surveillance, legal/policy changes, and change instilled). Overall, CPCRN activities demonstrate impact beyond peer-reviewed publications and thus should continue building scientific impact to ultimately influence health outcomes.

INTRODUCTION

Increased attention is being given in public health to accelerate the translation of research discoveries into real world settings. National strategic efforts and funding initiatives from the National Institutes of Health (NIH), the Centers for Disease Control and Prevention (CDC), the Agency for Healthcare Research and Quality (AHRQ), Health Resources and Services Administration (HRSA), and the Centers for Medicare and Medicaid Services (CMS) have created infrastructure and fostered community and academic collaborations essential to translating research into practice (Blake et al., 2017; Fernandez et al., 2014; Neta et al., 2015; Ribisl et al., 2017).The Cancer Prevention and Control Research Network (CPCRN) is one such effort; the CPCRN is a strategic, dynamic, interdisciplinary collaborative effort (co-funded by the CDC and the National Cancer Institute, NCI, since 2002) focused on accelerating the dissemination and implementation of evidence-based cancer prevention and control interventions (Cancer Prevention and Control Research Network, 2019; Fernandez et al., 2014; Harris et al., 2005; Ribisl et al., 2017). CPCRN collaborating centers are housed within CDC-funded Prevention Research Centers (PRCs) (Green 2007; Centers for Disease Control and Prevention, Prevention Research Centers 2019), which comprise a network of university-based centers that engage state, regional, and local partners in the development, testing, evaluation, and dissemination of health interventions (Harris et al., 2005; National Breast and Cervical Cancer Early Detection Program, 2019). This structural relationship connects CPCRN centers with the Prevention Research Centers’ (PRC) local and state partners and allows them to leverage existing community ties. The CPCRN centers, in turn, engage new partnerships within the PRC network.

The CPCRN centers operate at two levels. First, each collaborating center conducts research on dissemination and implementation science in collaboration with their local and state partners (Cancer Prevention and Control Research Network, 2019; Harris et al., 2005; Ribisl et al., 2017). Second, CPCRN centers collaborate with each other through ‘workgroups’ to scale up local discoveries to the national level by drawing from the expertise, resources, and partnerships of participating centers (Cancer Prevention and Control Research Network, 2019; Harris et al., 2005; Ribisl et al., 2017). This two-level approach enables the CPCRN to disseminate research findings to its community partners while also accelerating the dissemination of local research discoveries to the national stage.

The CPCRN centers have been highly productive in disseminating research findings through peer-reviewed publications and receiving additional grants to pursue new research ideas emanating from the CPCRN. Since the network was established, CPCRN investigators have published 6,534 CPCRN-related papers in peer-reviewed journals, of which 249 report the findings of multi-center collaborations (Ribisl et al., 2017). CPCRN investigators also secured approximately 600 grants in funding for their cancer control work, totaling nearly $640 million dollars (Cancer Prevention and Control Research Network, 2017a; Ribisl et al., 2017).

While this level of academic productivity is important, additional metrics are needed to assess CPCRN impact, that is, how the CPCRN centers are creating change in public health practice and policy and ultimately improving health outcomes. In the 2014-2019 grant cycle, the CPCRN Coordinating Center developed a new system to comprehensively capture CPCRN centers’ impact (Centers for Disease Control and Prevention, 2019). This new system was guided by the Science Impact Framework, which the CDC developed with the goal of moving beyond counting peer-reviewed publications to describe the full impact research has on public health (https://www.cdc.gov/od/science/impact/index.htm). Adapted from the Institute of Medicine (IOM) Degrees of Impact Framework, CDC’s framework captures what publication metrics are unable to do and documents how research knowledge is used to create action or change in public health (Centers for Disease Control and Prevention, 2017; Ruegg and Jordan, 2007). This framework captures information under five domains: 1) Disseminating Science, 2) Creating Awareness, 3) Catalyzing Action, 4) Effecting Change, and 5) Shaping the Future. As depicted in Figure 1, these domains are interrelated and show the transition from outputs, to outcomes, to impact on health outcomes. That is, disseminating science leads to creation of awareness of research knowledge and expertise. Increased awareness, in turn, leads to collaboration that catalyzes action through advocacy, training, practice improvement, and new research. These efforts then effect change in capacities, practices, and policies with the potential to shape future health outcomes. For each of the five domains, the CDC has developed a number of key indicators for tracking and linking scientific discoveries to public health impact. Table 1 shows the five domains and their respective indicators. Guided by the CDC’s Science Impact Framework, this paper highlights CPCRN outputs and outcomes within each domain.

Table 1:

Key Indicators from the CDC Science Impact Framework

Degree of Impact	Potential Measurable Indicators
Disseminating Science	• Scientific publications (open access journals) • Trade publications • Professional meetings/conferences • General communication (social media, web, print) • Presentations • Training, coursework • Other scientific output (e.g., Centers for Disease Control and Prevention Public Health Grand Rounds, Vital Signs, Science Clips)
Creating Awareness	• Continuing Education (Continuing Medical Education, Continuing Education Unit) • Awards • Stakeholder resources, curriculum, training • Feedback (Survey, focus groups, anecdote) • Information sharing and communications among professional societies • Electronic communications (information shared on listservs and other electronic resources, social media, news coverage) • Queries • Requests to contribute to efforts that further the science output
Catalyzing Action	• Technology creation • New funding (pilots/research) • Advocacy groups/Nongovemmental organizations • Congressional hearings • Partnerships and collaborations • Research & Development • Office practice/point of care changes
Effecting Change	• Building public health capacity (e.g., workforce development, funded research, improved staff competency) • Creation of registries/surveillance • Legal/policy changes • Accreditation • Cultural/social change • Behavioral change • Economic change • Centers for Medicare & Medicaid Services reimbursement • Other payer actions • Change instilled • (New) formal guidelines and recommendations (e.g., World Health Organization) • Hospital standards • Funding • Anecdotes/case studies • Sustainable and scalable science translation
Shaping the Future	• New hypotheses/Continuous quality improvement • Implementation of public health programs/initiatives • Health outcomes • Prevalence and incidence • Morbidity and mortality (e.g., frequency of outbreaks, trends) • Life expectancy • Quality of life improvements • Reductions in economic burden

METHODS

The CPCRN Coordinating Center developed and employs an online reporting system to monitor and evaluate network activities and outcomes by collecting, biannually, detailed information from collaborating centers. The specific activities and outcomes that were measured were determined by the CDC, CPCRN Coordinating Center, and the Collaborating Centers. Each collaborating center reports on its own research, community activities, and any cross-center activities for which the center plays a leadership role. In the 2014-2019 grant cycle, the CPCRN centers began collecting new information in alignment with four of the five domains in the CDC’s Science Impact Framework: Disseminating Science, Creating Awareness, Catalyzing Action, and Effecting Change. The fifth domain Shaping the Future was not included as it was not feasible to directly measure the indicators. The online reporting tool included questions about the centers and the cross-center workgroup activities related to the framework’s domains. The centers provided narrative responses to each of the questions. To classify the narratives into key indicators, in June 2019, two coders independently coded the narratives reported within each domain into the key indicators detailed in Table 1. Discrepancies between the coders were resolved via reconciliation.

RESULTS

Table 2 shows examples of activities under the Science Impact Framework domains and their respective key indicators.

Table 2.

Cancer Prevention and Control Research Network Activities within the Centers for Disease Control and Prevention’s Science Impact Framework

Impact	Indicators	N	Examples
Disseminating Science	Presentations	1,848	• Presentation of CPCRN collaborative work, entitled “CPCRN Work with Federally Qualified Health Centers: Findings from FQHC, HPV, and Tobacco/Lung Cancer Workgroups” presented at CDC led Cancer Conference. • Collaboration Between Accountable Care Organizations and Primary Care Practices, entitled Promising Practices for Implementing Interventions to Increase Colorectal Cancer Screening in Medicaid Members, CPCRN FQHC Workgroup
	Training	46	• Putting Public Health Evidence in Action curriculum www.cpcrn.org)
	General Communication (social media, web, print)	28	• Investigators from Oregon Health & Science University provide regular contribution to a blog titled “Cancer Translated.” https://blogs.ohsu.edu/cancertranslated/2017/12/27/knight-cancer-institute-signal-achievements-of-2017/ • An investigator at the University of Pennsylvania was featured in television, radio, and print news coverage of regulations for reef-safe sunscreen • Multiple cross-center workgroups have created reports of their research findings to share with community partners
	Other Communication Reports	78	• Together We Can: Georgia Colorectal Cancer Program Evaluation Report
Creating Awareness	Requests for Expertise	40	• Investigators from the University of Washington were invited to serve as members on CDC’s Breast and Cervical Cancer Early Detection and Control Advisory Committee. • Investigators from the University of Kentucky CPCRN provided technical assistance to state teams at the 80% by 2018 Forum, “Increasing Colorectal Cancer Screening Rates through Enhanced Partnerships between Comprehensive Cancer Control Coalitions and Federally Qualified Health Centers.”
	Feedback	24	• Oregon Health & Science University held a special funding opportunity, “Step It Up! Survivors,” and received positive feedback from cancer survivor participants: “I love what you have brought to our community. We needed this, look how many happy people are walking everywhere.” • “The tool created by (The University of North Carolina) to track FIT distribution is one of the most valuable parts of our CRC screening program. It is useful to have a tool that not only shows areas where we might need improvement specifically…but it also shows this data in real time so that we can address it and make improvements right away.”
Catalyzing Action	Partnerships and Collaborations	6	• Local research project by the University of South Carolina led to collaboration with South Carolina Hospital Association and Self-Regional Hospital to develop a pilot program to address health literacy in Greenwood (South Carolina) and to create a statewide Clinic Readiness Assessment.
	Research & Development	5	• Acting on FQHC provider survey evaluation findings. • Health literacy presentation leads to pilot program.
	Advocacy Groups	3	• The University of Washington participated in the American Thoracic Society’s Research Policy Statement on integrating smoking cessation into lung cancer screening. Recommendations included monitoring of smoking cessation and quit rates to lung cancer screening programs were made and the recommendations were incorporated by the Committee in their official policy statement.
	Office Practice/ Point of Care Changes	3	• The University of Washington local project conducted patient interviews to understand FIT test education material and clinic staff workflow. The patients feedback helped update FIT education materials at the FQHCs and restructure the clinic staff workflow to ensure clinic staff members were not missing opportunities to discuss colorectal cancer screening with patients.
	Technology Creating	2	• As a result of the pilot work conducted at Metrohealth to set up and test the eReferral to the State Quitline, the MetroHealth system has now adopted those changes made to the electronic medical record as permanent.
Effecting Change	Building Public Health Capacity	5	• Collaborative Colorectal Cancer Project: Peer Learning Event. Multiple activities took place to help FQHCs to identified barriers, challenges, and needs to uptake evidence-based strategies to increase colorectal cancer screening.
	Creation of Registries/Surveillance	24	• Case Western Reserve CPCRN investigator led an effort at FQHCs to create a registry of abnormal cervical cytology/path reports of patients seen at the clinic, so the team can better follow-up with patients and in a timely manner.
	Legal/Policy Changes	1	• Kentucky Senate Bill 101 allows pharmacists in Kentucky to administer the guideline-recommended HPV vaccinations for adolescents, pursuant to prescriber-approved protocols with the consent of a parent or guardian. The policy was informed by CPCRN HPV vaccination workgroup.
	Change Instilled	57	• The changes at MetroHealth’s electronic medical records are complemented by significant role changes for the medical assistants and nurses at primary care clinics. These role changes have been adopted as new procedures for addressing tobacco assessment and assistance.

^*FQHC=Federally Qualified Health Center; CPCRN=Cancer Prevention and Control Research Network; CDC=Centers for Disease Control and Prevention; HPV= Human Papillomavirus; FIT= Fecal Immunochemical Test

Disseminating Science

Indicators related to disseminating science include trade publications, presentations at professional meetings, and conferences, the provision of trainings and courses, and general communication (mass media, social media). Collectively, 2,000 activities were identified within the dissemination of science domain. The majority of these activities were related to presentations (n=l,848) at professional conferences focused on cancer, dissemination and implementation science and public health. Forty-six activities involved the provision of training to public health and other practitioners on how to select, adapt, implement, and/or sustain evidence-based interventions. Twenty-eight activities were related to using media to communicate research knowledge to the general public, and there were 78 other activities that fell into a range of categories, such as providing an evaluation report to a state-wide colorectal cancer control screening program in collaboration with the State Department of Public Health.

CPCRN researchers have been featured in several media reports, demonstrating the expansive reach of the Network’s impact in disseminating research findings. For example, the principal investigator of the University of Pennsylvania CPCRN was featured in television, radio, and print news coverage. She was interviewed by the Wharton Business Radio and the Hawaii television station, Hawaii News Now, about regulations for reef-safe sunscreen (Richardson, 2018). Another CPCRN investigator from the University of Iowa published a manuscript in the Journal of the National Cancer Institute that garnered international news coverage, with Reuters publishing an article entitled “Large U.S. farm study finds no cancer link to Monsanto weed killer” (Andreotti et al., 2018). Additionally, the “Putting Public Health Evidence Into Action” training developed by University of North Carolina CPCRN was disseminated on the CPCRN website, with over 1800 views over the course of one year (Cancer Prevention and Control Research Network, 2017b). This training brought in requests for dissemination and implementation science expertise from at least seven organizations, including organizations outside of the U.S.

Creating Awareness

Sixty-four activities were identified within the creating awareness domain. Indicators included requests from local and national public health agencies for CPCRN investigators to provide expert consultation on cancer prevention and control best practices, dissemination and implementation science, and collaborations with local and state partners. For example, CPCRN investigators were asked to serve on CDC’s cancer detection advisory committee and to provide technical assistance for a colorectal cancer screening forum. Another indicator included feedback from CPCRN partners on the impact of the CPCRN training “Putting the Public Health Evidence in Action.” For example, participants (n=252) in a series of CPCRN trainings reported that the training was relevant to their work and they intended to apply what they learned in the practice, thus raising awareness on how to find evidence-based interventions, adapt the intervention to specific settings, and evaluate the evidence-based health promotion activities (Mainor et al., 2018).

Catalyzing Action

Nineteen activities were noted as catalyzing action, including new partnerships and collaborations (n=6), new research projects (n=5), advocacy for public health policy change (n=3), collaboration on changes to practice/point of care (n=3), and creating new technology (n=2). For example, the University of South Carolina (USC) CPCRN delivered a presentation to the South Carolina Hospital Association about developing a uniform health literacy assessment tool, which led to a collaboration with the South Carolina Hospital Association and a local hospital to develop a pilot program that addresses health literacy, as well as a statewide Clinic Readiness Assessment program. The presentation also led the Dorn Veteran Affairs Hospital to collaborate with USC to develop a health literacy program for veterans.

Effecting Change

Effecting change was the second most commonly noted activity in the framework (n=87). Key indicators include building public health capacity (n=5), supporting creation of registries/surveillance (n=24), changing policies (n=1), and changing public health and clinical practice (n=57). For example, an investigator from the Case Western Reserve CPCRN led an effort, with the local Neighborhood Family Practice, to create a registry of abnormal cervical cytology/pathology reports of clinic patients to ensure timely follow-up care. Furthermore, CPCRN work has led to legal/policy changes in public health. For instance, policy recommendations that emerged from the CPCRN HPV vaccination workgroup led to the Kentucky Senate Bill 101. This bill allows pharmacists in Kentucky to administer guideline-recommended vaccines, including the HPV vaccination series to youth aged 9-17 (with parent/guardian consent) pursuant to prescriber-approval (Legiscan, 2017). Lastly, the technical assistance provided by the University of Washington CPCRN to Federally Qualified Health Centers to support their efforts to implement evidence-based colorectal cancer screening program led to changes in colorectal screening practices at Federally Qualified Health Centers. Such efforts include targeted training for clinic staff regarding their clinic’s colorectal cancer screening policies and practices and strengthening clinic workflow to reduce missed opportunities to discuss or offer colorectal cancer screening to patients.

DISCUSSION

This paper describes the CPCRN activities within the domains of the CDC’s Science Impact Framework and highlights how they are moving towards shaping the future of public health. CPCRN activities were aligned with four domains from the framework: Disseminating Science (n=l,922), Creating Awareness (n=64), Catalyzing Action (n=19) and Effective Change (n=87).

Most CPCRN activities aligned with Disseminating Science. The majority of these activities involved presentations at professional conferences. CPCRN centers also reported conducting a number of trainings and using a variety of multi-media platforms, such as television, web, and radio to disseminate evidence-based cancer prevention and control interventions. Although the CPCRN centers heavily engage in publication efforts, this activity was not included in this paper as the goal was to assess impact beyond peer-reviewed publication. Additionally, a report that highlights the scientific papers published by this network has been previously published (Ribisl et al., 2017).

Many CPCRN activities aligned with Creating Awareness and Effecting Change. In Creating Awareness, reports showed that CPCRN investigators are frequently consulted for scientific expertise and technical assistance in dissemination and implementation science. In Effecting Change, CPCRN activities have had the largest impact on creating registries/surveillance and other changes in public health practice such as clinical role, electronic medical records, adopting a new procedure that addresses smoking assessments as well as providing smoking cessation assistance to patients.

Although Shaping the Future was not collected through the annual report, it should be noted that a large part of the CPCRN activities were focused on Effecting Change in practice settings, which were proximal to Shaping the Future. The Science Impact Framework depicts the domains as ‘output/outcome/impact’ and aligns them in succession from Disseminating Science to Shaping the Future. Although we noted that the domains are not always progressive, but more interrelated, the ultimate goal is to improve health outcomes through resources that show measurable indication of Shaping the Future. The clustering of activities in the Effective Change domain validates that the activities are building momentum towards improving health outcomes.

It is important to note that many CPCRN activities are still focused on traditional science metrics (i.e., publications, grants, presentations), which may be due to how the evaluation metrics are being used by the scientific community and the funding agencies. Additionally, investigators may be more experienced with activities that align with ‘Disseminating Science’ as academic institutions are heavily focused on developing trainees that remain productive with scholarly activities on the path to becoming a faculty member at which point they are rewarded based on these metrics. To increase scientific impact, CPCRN may also want to focus on advocacy to instigate change in the scientific and policy communities to ensure that activities beyond peer-reviewed publications are equally valued and academic training and funding are aligned with these activities.

The Science Impact Framework provided a valuable lens for assessing contributions of the CPCRN beyond peer-reviewed citations. As a network, CPCRN centers have remained highly productive in disseminating evidence-based cancer prevention and control interventions and impacting public health practice. As other thematic networks are formed and grow, increased interface with other networks (e.g., Clinical and Translational Science Award) will facilitate the exchange of ideas, resource sharing, and the spread of the CPCRN model. Other thematic networks may also want to assess the impact of their activities using the Science Impact Framework. However, it is important to note that although the Science Impact Framework is useful in capturing broad categories, additional work is needed to develop clear definitions of the domains and key indicators to facilitate consistent application of the framework within and across CPCRN centers, as well as other networks.

CONCLUSIONS

The CPCRN is a strategic, dynamic, interdisciplinary collaborative effort centered around accelerating the dissemination and implementation of evidence-based cancer prevention and control interventions. As illustrated in this summary, CPCRN has engaged in many impactful activities that go beyond peer-reviewed publications and serves as a model for accelerating dissemination and implementation of evidence-based strategies. For CPCRN science to improve health outcomes, the scientific impact should remain a priority in the CPCRN centers.