Skip to main content
NCBI home page
HHS Author Manuscripts logoLink to HHS Author Manuscripts
. Author manuscript; available in PMC: 2021 May 9.
Published in final edited form as: Health Promot Pract. 2018 Mar 22;20(2):214–222. doi: 10.1177/1524839918760557

Evaluating Comprehensive State Tobacco Prevention and Control Programs Using an Outcome Indicator Framework

Erika Fulmer 1, Todd Rogers 2, LaShawn Glasgow 2, Susan Brown 1, Nicole Kuiper 1
PMCID: PMC8106980  NIHMSID: NIHMS1694017  PMID: 29566575

Abstract

The outcome indicator framework helps tobacco prevention and control programs (TCPs) plan and implement theory-driven evaluations of their efforts to reduce and prevent tobacco use. Tobacco use is the single-most preventable cause of morbidity and mortality in the United States. The implementation of public health best practices by comprehensive state TCPs has been shown to prevent the initiation of tobacco use, reduce tobacco use prevalence, and decrease tobacco-related health care expenditures. Achieving and sustaining program goals require TCPs to evaluate the effectiveness and impact of their programs. To guide evaluation efforts by TCPs, the Centers for Disease Control and Prevention’s Office on Smoking and Health developed an outcome indicator framework that includes a high-level logic model and evidence-based outcome indicators for each tobacco prevention and control goal area. In this article, we describe how TCPs and other community organizations can use the outcome indicator framework in their evaluation efforts. We also discuss how the framework is used at the national level to unify tobacco prevention and control efforts across varying state contexts, identify promising practices, and expand the public health evidence base.

Keywords: tobacco, program evaluation, public health practice, adolescent, health knowledge, attitudes, practice

INTRODUCTION

Tobacco use is the single-most preventable cause of morbidity and mortality in the United States, with more than 20 million premature deaths attributable to cigarette smoking since 1964 (U.S. Department of Health and Human Services, 2014). The Centers for Disease Control and Prevention’s (CDC) Office on Smoking and Health (OSH) created the National Tobacco Control Program (NTCP) in 1999 to encourage coordinated state and national efforts to reduce tobacco-related diseases, disabilities, and deaths (CDC, 2015a; U.S. Department of Health and Human Services, 2014). The program provides funding and technical support to all 50 states, the District of Columbia, and U.S. territories/jurisdictions (CDC, 2015a). Preventing Initiation of Tobacco Use is a public health priority and the first of NTCP’s four goals (CDC, 2015a; Office of Disease Prevention and Health Promotion, 2016). Nearly all tobacco use begins during youth and young adulthood (U.S. Department of Health and Human Services, 2012). Comprehensive state tobacco prevention and control programs (TCPs), which are typically led by state and territorial health departments and funded through NTCP and other revenue streams (e.g., tobacco taxes, Master Settlement Agreement [MSA] payments, etc.), prevent the initiation of tobacco use, reduce tobacco use prevalence, and help decrease tobacco-related health care expenditures (Farrelly et al., 2013; Institute of Medicine, 2007; Lightwood & Glantz, 2011; U.S. Department of Health and Human Services, 2012, 2014)

To sustain these comprehensive TCPs, funding and implementing agencies must rigorously evaluate program effectiveness to monitor progress toward goals, recalibrate programmatic efforts, and justify resource allocations (CDC, 2014a; Frieden, 2014; Glasgow, Lichtenstein, & Marcus, 2003; Schell et al., 2013). Theory-driven evaluation is an efficient approach to gathering information about how programs are achieving the intended effects (Chen & Rossi, 1983). This evaluation approach is founded on the construction of logic models that illustrate the relationships among a program’s inputs, activities, outputs, and outcomes (CDC, 2011b; Coryn, Noakes, Westine, & Schroter, 2011; Holden & Zimmerman, 2009; Porterfield, Rogers, Glasgow, & Beitsch, 2015). Evaluation questions, indicators, and data sources are identified to assess progress along specific logic model activity, output, and outcome pathways and to test the hypothesized relationships between program activities and outcomes (CDC, 2001; Porterfield et al., 2015). Because theory-driven evaluations are based on a program’s implicit and explicit stakeholder assumptions about the actions needed to solve a social problem, they are likely to produce relevant findings to improve programs and inform future decision making (Chen, 1994). To support theory-driven evaluation at the state and national TCP levels, OSH developed an outcome indicator framework that consists of a high-level logic model and evidence-based outcome indicators for each tobacco prevention and control goal area.

BACKGROUND

With the MSA of November 1998, the tobacco industry settled Medicaid lawsuits with 46 states for recovery of tobacco-related health care costs. MSA funds provided many states with first-time resources to design and implement comprehensive TCPs. In support of these efforts, OSH developed and disseminated a series of guidance materials for state planning and evaluation efforts, including an introduction to program evaluation (MacDonald et al., 2001) and an outcome indicator guide (Starr et al., 2005). The latter document consisted of a high-level logic model and related outcome indicators for each programmatic goal area in the NTCP (i.e., preventing initiation of tobacco use, eliminating exposure to secondhand smoke, promoting cessation among tobacco users) plus guidance and expert reviewer ratings to help TCPs select appropriate indicators for their theory-driven evaluations. The resulting outcome indicator framework was designed to meet TCP needs for consistent, evidence-informed evaluation measures and comparable data sources. The framework, disseminated via intensive OSH technical assistance and training efforts, helped OSH guide and support evidence-based program and evaluation planning by state TCPs.

To ensure that the outcome indicator framework was informed by the most recent evidence and current tobacco prevention and control priorities, including the elimination of tobacco-related disparities, OSH recently updated each programmatic goal area. The first update, Preventing Initiation of Tobacco Use, was published in August 2014 (CDC, 2014b). The second, Promoting Quitting Among Adults and Young People, was published in February 2016 (CDC, 2015b). The third, Eliminating Exposure to Secondhand Smoke, will be released in March 2018. Each updated guide integrates information on identifying and eliminating tobacco-related disparities—NTCP’s cross-cutting fourth goal—throughout the indicator framework.

In this article, we describe how the outcome indicator framework can be used in evaluation planning. We also discuss the strengths and limitations of this approach in meeting varied program and evaluation needs, such as managing and evaluating public health programs at a national level. The examples are intended to “strengthen the body of literature on the practical application of theory-driven evaluation” (Breuer, Lee, De Silva, & Lund, 2016; Coryn et al., 2011).

DESCRIPTION OF THE OUTCOME INDICATOR FRAMEWORK

The outcome indicator framework provides an evidence-based logic model highlighting short- and intermediate-term outcome constructs, which have historically been absent from public health surveillance systems (Porterfield et al., 2015). Using “Preventing Initiation of Tobacco Use” as an example, the logic model demonstrates linkages between program infrastructure, activities, outputs, and short-, intermediate- and long-term outcomes of preventing tobacco use initiation and, subsequently, reducing tobacco-related morbidity and mortality (Figure 1). The logic model assists stakeholders identify elements of an evidence-informed program that will advance tobacco control goals. The “Preventing Initiation of Tobacco Use” logic model clarifies how comprehensive tobacco control programs can reshape the environment to reduce and counteract protobacco messaging, restrict the availability of tobacco products, increase tobacco price, and disseminate prohealth messaging. It demonstrates that, by advancing these initiatives in both community and schools, TCPs can work to increase tobacco-free norms that reduce tobacco use susceptibility among young people and ultimately reduce the initiation of tobacco use and the related health consequences (CDC, 2014b).

FIGURE 1.

FIGURE 1

Open in a new tab

Preventing Initiation of Tobacco Use Logic Model

Nested under each outcome construct (logic model box) are evidence-based outcome indicators, the measurement of which help assess progress toward achieving specific, observable, and measurable changes in the outcome construct (United Way of America, 1996). Causal pathways connecting logic model boxes reflect the implicit connections that help shape TCP efforts toward evidence-informed tobacco control outcomes within the goal area. Having a breadth of indicators within each outcome box was intended to give TCPs the flexibility to select evaluation-specific outcome indicators that accommodate local context and are most closely aligned with program objectives.

Information gathered via an extensive literature and expert panel review process was used to create detailed indicator profiles that include the following elements (CDC, 2014b):

  • Unique indicator title and number

  • Associated logic model outcome box

  • Description of what to measure to gather data on the indicator

  • Evidence-based rationale statement clarifying why the indicator is useful to measure, and when available, evidence of disparities associated with the indicator

  • Expert reviewer ratings based on six criteria (i.e., overall quality, resources needed, strength of evaluation evidence, utility, face validity, and accepted practice)

  • Example data sources where the indicator has been used

  • Example survey questions that have been used to measure the indicator

  • Population groups from which data related to the indicator are most commonly collected

  • Comments summarizing additional information that may be useful for program planning/evaluation purposes

  • References that provide evidence for linkages between short-, intermediate-, and long-term outcomes

Indicator profiles were designed to assist TCPs in prioritizing selected indicators and operationalizing how information could be collected and used (Figure 2).

FIGURE 2. Example Indicator Profile.

FIGURE 2

Open in a new tab

SOURCE: Centers for Disease Control and Prevention (2014b).

†As identified by indicator number, 1.5.f., this indicator links to National Tobacco Control Program Goal 1, Preventing initiation of tobacco use, Outcome Box 5, Reduced susceptibility. ±KOI 2005 refers to the previous edition of Key Outcome Indicators for Evaluating Comprehensive Tobacco Control Programs. #Numbers indicate citation references included in the indicator documentation.

APPLICATION OF THE OUTCOME INDICATOR FRAMEWORK

The outcome indicator framework provides guidance that assists TCPs with conducting theory-driven evaluation in a manner that aligns with CDC’s (1999) Framework for Program Evaluation in Public Health Practice. The Framework for Program Evaluation involves six interdependent steps for planning an evaluation including: Step 1: Engaging Stakeholders, Step 2: Describing the Program, Step 3: Focusing the Evaluation Design, Step 4: Gathering Credible Evidence, Step 5: Justifying Conclusions, and Step 6: Sharing Lessons Learned.

The elements comprising the outcome indicator framework were designed to facilitate TCP use in mapping activities along an intended outcome pathway, providing an opportunity as part of evaluation Step 1: Engaging Stakeholders and Step 2: Describing the Program, to clarify different approaches to achieve program objectives and, ultimately, to identify one that aligns most favorably with community context, program time lines, and resource constraints. When mapping an intended outcome pathway, TCP stakeholders select an intended, long-term outcome or set of outcomes and then trace a causal pathway back through intermediate outcomes, short-term outcomes, program outputs, program activities, and functioning program infrastructure (Rogers, Chappelle, Wall, & Barron-Simpson, 2011). For example, suppose TCP stakeholders select the long-term outcome, “Reduced initiation of tobacco use” (NTCP Goal 1, Outcome 9). Informed by elements of the outcome indicator framework—such as rationale statements, cited references, and materials recommended for further reading—TCP staff and stakeholders collaboratively select the following short- and intermediate-term outcomes they expect to achieve along the presumed causal pathway to the ultimate outcome:

  • Outcome 1: Increased knowledge of the dangers of tobacco use, attitudes against tobacco use, and support for policies to reduce tobacco use initiation

  • Outcome 4: Increased policy and enforcement efforts to reduce tobacco industry influence

  • Outcome 5: Reduced susceptibility to experimentation with tobacco products

Favorable changes in short- and intermediate-term outcome indicators in the selected pathway can lead to improvements in the selected long-term outcome. Moreover, wide-scale and sustained declines in rates of tobacco use initiation can eventually lead to reductions in youth tobacco use prevalence (U.S. Department of Health and Human Services, 2014) and to population-wide declines in tobacco-related morbidity and mortality (Cowling & Yang, 2010). The pathway suggests several TCP policy and programmatic activities, including work to strengthen licensure requirements for tobacco retailers in the state and related local efforts to restrict the tobacco retail density.

As part of evaluation, Step 3: Focusing the Evaluation, which includes specifying the primary evaluation purpose and scope (CDC, 2011a), TCP stakeholders develop evaluation questions and select indicators using the indicator selection criteria included in the framework (i.e., overall quality, resources needed, strength of evaluation evidence, utility, face validity, accepted practice) that are most important to the program given its stage of development. They then compare indicator ratings and supporting information included as part of the outcome indicator framework and select indicators for each of the previously selected outcome boxes.

For example, in Box 9, stakeholders may select indicator 1.9.c, Proportion of young people who report never having tried a tobacco product. Evidence shows that reducing the number of young people who experiment with tobacco will decrease the number who become established tobacco users (U.S. Department of Health and Human Services, 2012). This indicator has the highest ratings across all indicator selection criteria and is measured in an existing, ongoing national survey that provides state-level estimates. The TCP stakeholders make similar indicator selections from each of the selected outcome boxes.

Selecting indicators in the outcome pathway and identifying relevant data sources are early actions needed for evaluation Step 4: Gathering Credible Evidence. Using existing surveillance systems that provide state-level or local estimates greatly enhances the feasibility of capturing information given limited resources (CDC, 2014c, 2017). However, the TCP must consider whether the available survey items meet their evaluation needs for timeliness of reported data, ability to create subpopulation estimates, and availability of precise estimates at the geographic level of interest (CDC, 2001).

In the example discussed above, the indicator selected (1.9.c, Proportion of young people who report never having tried a tobacco product) may be measured through an existing population-based national survey, such as the National Youth Tobacco Survey (NYTS; https://www.cdc.gov/tobacco/data_statistics/surveys/nyts/). However, a major limitation in the use of a national survey for their evaluation purposes is that it does not provide state-level estimates. Other nationally sponsored youth tobacco use surveys, such as the Youth Tobacco Survey (YTS; https://www.cdc.gov/tobacco/data_statistics/surveys/yts/index.htm), are not implemented frequently enough for evaluation purposes and are not designed to provide substate-level estimates or estimates for subpopulation groups within a state. Therefore, the TCP may decide to supplement this information by adding several cognitively tested questions on use of various tobacco products to a state-based youth survey that provides county-level and subpopulation estimates annually. Although the decision requires substantial resource investment, the prioritized needs of the evaluation may outweigh the added cost.

Evaluation Step 5: Justifying Conclusions involves considering how collected data will be used to inform evaluation findings (CDC, 2011a) and guide program improvement action. TCP stakeholders are engaged throughout this process to apply evaluation standards in the analysis and interpretation of evaluation findings (Yarbrough, Shulha, Hopson, & Caruthers, 2011). The “Preventing Initiation of Tobacco Use” outcome indicator framework provides TCPs with links to the evidence base in the indicator rationale statements and a graphic display of “if-then” connections across the logic model. These linkages help TCPs understand how indicator data are connected across the logic model and where gaps in intended effects may be occurring. This information helps guide planning for conclusions by providing a framework to consider contextual information that will influence interpretation and application of evaluation findings for program improvement.

As part of the final, “Step 6: In the Evaluation Process, Sharing Lessons Learned,” the TCP carefully considers how, when, and with whom information will be shared (CDC, 2011a). Planning for use is directly tied to the identified purposes of the evaluation as well as program and stakeholder priorities. Evaluation findings and lessons learned are tailored to the primary audience. The causal pathway, along with the selected outcome indicators, provides structure to evaluation reports and related recommendations. When combined with appropriate data visualization techniques, these elements can tell a coherent story appropriate for a range of audiences. Supplementing this information with success stories and presenting information in alternative ways—such as via infographics, oral presentations, webcasts, news releases, and social media messaging—further enhance the evaluation accessibility and utility.

DISCUSSION

The outcome indicator framework assists TCPs with conducting theory-driven evaluation. Additionally, the outcome indicator framework serves as a communication-tool for TCPs to engage stakeholders with varied backgrounds, perspectives, and skill sets. These features of the framework facilitate understanding how program activities can be mounted to achieve long-term goals, as well as advance understanding of the role outcome evaluation plays in program management.

The outcome indicator framework consists of a logic model, outcome indicators, and detailed indicator profiles that support a disciplined process for mapping the intended outcome pathway and selecting outcome indicators. This stepwise process for conducting a theory-driven evaluation enhances the transparency of methods and makes evaluation findings accessible to a breadth of stakeholders. At the national level, the outcome indicator framework provides a common language, highlights evidence-based activities as they relate to critical outcomes, and provides a foundation on which to measure progress toward meeting national goals.

The “Preventing Initiation of Tobacco Use” outcome indicator framework was developed to help TCPs conduct theory-driven evaluations to improve their programs. The framework was intentionally designed to include a breadth of outcome indicators with rating information to allow users to select a subset of indicators most pertinent to their local evaluation needs. This approach serves multiple uses at the state and local levels: guiding evaluation planning, focusing TCP priorities through the selection of core indicators, providing an evidence-based resource for informed action and program improvement, and identifying areas for program innovation. The logic models and indicator profiles support data collection, analysis, interpretation, and reporting. Additionally, the indicator selection and evaluation planning process can help focus programmatic efforts toward evidence-based interventions with high probability of affecting key outcomes on the path toward achievement of public health goals.

Many state programs have reported use of the outcome indicator framework to guide not only their evaluation planning but also their strategic planning (Center for Tobacco Policy Research, 2012). For example, Indiana used the guides to ensure their plan was in alignment with the most recent evidence. As they were updating their strategic plan in 2015, they used the “Preventing Initiation of Tobacco Use” update to include indicators related to e-cigarettes. Staff also contacted CDC to ensure alignment with updated Cessation evidence prior to that update being published in 2016. They used the outcome indicator frameworks to choose strategic plan objectives and included specific outcome indicators within their target objective tables, directly referencing the “Preventing Initiation of Tobacco Use” update if partners wanted to learn more about the evidence-base for each indicator (Indiana State Department of Health, 2016).

Expanding on its use of the OSH outcome indicator framework at the state level, the California TCP has implemented a comprehensive planning and evaluation system for all 61 local health departments and dozens of local and regional organizations funded by state tobacco excise taxes (Roeseler & Burns, 2010). California’s Community of Excellence in Tobacco Control system provides local and regional agencies with structured guidance, logic models, indicator lists and other tools to select contextually relevant outcomes, plan policy and programmatic efforts, and implement evaluation activities to assess changes in outcome indicators (cf. California Department of Public Health & California Tobacco Control Program, 2016). Moreover, California’s tobacco control outcome indicator approach has been incorporated into other chronic disease prevention programs being implemented by local health departments, such as the Community of Excellence in Nutrition, Physical Activity, and Obesity initiative (Ghirardelli, Quinn, & Foerster, 2010).

Although the outcome indicator framework has numerous strengths, it also has limitations. As developed by OSH, the outcome indicator framework does not provide infrastructure or process indicators. This process information is helpful in understanding why a program is achieving (or failing to achieve) its intended short-, intermediate-, or long-term outcomes. Complementing the outcome indicator framework with descriptive summaries of program infrastructure, activities, and outputs can help justify conclusions and guide actions when outcome findings are ambiguous. Additionally, the indicators included in the outcome indicator framework are comprehensive but not exhaustive. Therefore, it may be that a TCP must add, modify, or further operationalize an indicator to best fit their programmatic context and needs. This is particularly true in the case of persistent problem areas (e.g., tobacco-related disparities among certain subpopulations, such as those with behavioral health or substance abuse issues) or innovative policy or program approaches (e.g., smoke-free multiunit housing policies). Although OSH has updated the outcome indicator framework since its initial release in 2005 (CDC, 2014b, 2015), revisions and updates will continue to be required in order to address emerging trends and innovative practices. For example, OSH is currently exploring the need to develop more extensive guidance on collecting and analyzing indicator data in a manner that allows for both the identification of populations experiencing tobacco-related disparities and the monitoring of outcomes in these populations.

The outcome indicator framework allows OSH to look across states to build the evidence base for program effectiveness, identify innovative efforts, and detect programmatic and evaluation gaps. For example, many TCPs engage in activities to increase restrictions on tobacco product sales and availability. However, the ability to measure reduced tobacco industry influence at the state level is limited. Because of this gap, OSH initiated work to develop a national and state-specific surveillance system that captures tobacco product sales and price estimates longitudinally. This system is invaluable for monitoring sales of emerging tobacco products, such as electronic cigarettes, and provides critical information for assessing the impact of national and state-level programmatic initiatives.

In summary, the OSH outcome indicator framework helps TCPs plan and implement theory-driven evaluation and has utility at national, state, and local levels. It clarifies the evidence-based connections between a program’s inputs, activities, outputs, and outcomes and provides related outcome indicators that are used to focus program priorities. At the national level, core outcome indicators from the framework are integrated into the OSH performance measurement systems, serving to unify the work of the NTCP and facilitate the identification of practice-based evidence. At the state and local levels, the outcome indicator framework serves as a practical technical assistance resource that helps ensure that TCP program and evaluation efforts are aligned with NTCP goals and priorities. When coupled with rigorous evaluation efforts, the outcome indicator framework also helps expand the evidence base for public health approaches to tobacco prevention and control. Moreover, application of the outcome indicator framework concept, tailored to meet programmatic needs and used in coordination with the CDC Framework for Program Evaluation, could be used in similar ways across an array of public health and chronic disease programs.

Authors’ Note:

LaShawn Glasgow and Todd Rogers were supported by the Centers for Disease Control and Prevention through Contract Nos. 200-2011-F-39606, 200-2014-F-61225, and 200-2016-F-92371. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

REFERENCES

  1. Breuer E, Lee L, De Silva M, & Lund C (2016). Using theory of change to design and evaluate public health interventions: A systematic review. Implementation Science, 11(1), 63. doi: 10.1186/s13012-016-0422-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. California Department of Public Health & California Tobacco Control Program. (2016). Communities of excellence in tobacco control: A communities of excellence needs assessment guide. Sacramento: Author. Retrieved from https://www.cdph.ca.gov/Programs/CCDPHP/DCDIC/CTCB/CDPH%20Document%20Library/Community/ToolKitsandManuals/2016CXweb2.pdf [Google Scholar]
  3. Center for Tobacco Policy Research. (2012). Dissemination and implementation of evidence-based guidelines: Final report [Unpublished report]. St. Louis, MO: Washington University in St. Louis. [Google Scholar]
  4. Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health practice. Morbidity and Mortality Weekly Report, 48(RR11), 1–40.9933120 [Google Scholar]
  5. Centers for Disease Control and Prevention. (2001). Updated guidelines for evaluating public health surveillance systems. Morbidity and Mortality Weekly Report, 50(RR13), 1–35. [PubMed] [Google Scholar]
  6. Centers for Disease Control and Prevention. (2011a). Developing an effective evaluation plan. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health; Division of Nutrition, Physical Activity, and Obesity. [Google Scholar]
  7. Centers for Disease Control and Prevention. (2011b). Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: U.S. Department of Health and Human Services Centers for Disease Control and Prevention. [Google Scholar]
  8. Centers for Disease Control and Prevention. (2014a). Best practices for comprehensive tobacco control programs–2014. Atlanta, GA: Centers for Disease Control and Prevention; Retrieved from www.cdc.gov/tobacco/stateandcommunity/best_practices/index.htm?source=govdelivery [Google Scholar]
  9. Centers for Disease Control and Prevention. (2014b). Preventing initiation of tobacco use: Outcome indicators for comprehensive tobacco control programs–2014. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health. [Google Scholar]
  10. Centers for Disease Control and Prevention. (2014c). Surveillance and evaluation data resources for comprehensive tobacco control programs. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health. [Google Scholar]
  11. Centers for Disease Control and Prevention. (2015a). National Tobacco Control Program. Retrieved from http://www.cdc.gov/tobacco/tobacco_control_programs/ntcp/
  12. Centers for Disease Control and Prevention. (2015b). Promoting quitting among adults and young people: Outcome indicators for comprehensive tobacco control programs–2015. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health. [Google Scholar]
  13. Centers for Disease Control and Prevention. (2017). 500 cities: Local data for better health. Retrieved from https://www.cdc.gov/500cities/
  14. Chen HT (1994). Theory-driven evaluations: Need, difficulties, and options. American Journal of Evaluation, 15, 79–82. [Google Scholar]
  15. Chen HT, & Rossi PH (1983). Evaluating with sense: The theory-driven approach. Evaluation Review, 7, 283–302. [Google Scholar]
  16. Coryn CLS, Noakes LA, Westine CD, & Schroter DC (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32, 199–226. doi: 10.1177/1098214010389321 [DOI] [Google Scholar]
  17. Cowling DW, & Yang J (2010). Smoking-attributable cancer mortality in California, 1979–2005. Tobacco Control, 19(Suppl. 1), i62–i67. doi: 10.1136/tc.2009.030791 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Farrelly MC, Loomis BR, Han B, Gfroerer J, Kuiper N, Couzens GL, … Caraballo RS (2013). A comprehensive examination of the influence of state tobacco control programs and policies on youth smoking. American Journal of Public Health, 103, 549–555. doi: 10.2105/AJPH.2012.300948 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Frieden TR (2014). Six components necessary for effective public health program implementation. American Journal of Public Health, 104, 17–22. doi: 10.2105/AJPH.2013.301608 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Ghirardelli A, Quinn V, & Foerster SB (2010). Using geographic information systems and local food store data in California’s low-income neighborhoods to inform community initiatives and resources. American Journal of Public Health, 100, 2156–2162. doi: 10.2105/AJPH.2010.192757 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Glasgow RE, Lichtenstein E, & Marcus AC (2003). Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. American Journal of Public Health, 93, 1261–1267. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Holden DJ, & Zimmerman MA (2009). A practical guide to program evaluation and planning: Theory and case examples. Thousand Oaks, CA: Sage. [Google Scholar]
  23. Indiana State Department of Health. (2016). Indiana tobacco control 2020 strategic plan, 2016–2020. Indianapolis, IN: Indiana State Department of Health, Tobacco Prevention and Cessation Commission. Retrieved from https://www.in.gov/isdh/tpc/files/2020%20Strategic%20Plan_PDF_FINAL.pdf [Google Scholar]
  24. Institute of Medicine. (2007). Ending the tobacco problem: A blue-print for the nation. Washington, DC: National Academies Press. [Google Scholar]
  25. Lightwood J, & Glantz S (2011). Effect of the Arizona tobacco control program on cigarette consumption and healthcare expenditures. Social Science & Medicine, 72, 166–172. doi: 10.1016/j.socscimed.2010.11.015 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. MacDonald G, Starr G, Schooley M, Yee SL, Klimowski K, & Turner K (2001). Introduction to program evaluation for comprehensive tobacco control programs. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health. [Google Scholar]
  27. Office of Disease Prevention and Health Promotion. (2016). Heart disease and stroke. Retrieved from https://www.healthypeople.gov/2020/topics-objectives/topic/heart-disease-and-stroke
  28. Porterfield DS, Rogers T, Glasgow LM, & Beitsch LM (2015). Measuring public health practice and outcomes in chronic disease: A call for coordination. American Journal of Public Health, 105 (Suppl. 2), S180–S188. doi: 10.2105/AJPH.2014.302238 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Roeseler A, & Burns D (2010). The quarter that changed the world. Tobacco Control, 19(Suppl. 1), i3–i15. doi: 10.1136/tc.2009.030809 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Rogers T, Chappelle EF, Wall HK, & Barron-Simpson R (2011). Using DHDSP outcome indicators for program planning and evaluation. Atlanta, GA: Centers for Disease Control and Prevention. Retrieved from http://www.cdc.gov/dhdsp/programs/nhdsp_program/evaluation_guides/docs/Using_Indicators_Evaluation_Guide.pdf. [Google Scholar]
  31. Schell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, & Bunger AC (2013). Public health program capacity for sustainability: A new framework. Implementation Science, 8, 15. doi: 10.1186/1748-5908-8-15 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Starr G, Rogers T, Schooley M, Weisen E, Porter S, & Jamison N (2005). Key outcome indicators for evaluating comprehensive tobacco control programs. Atlanta, GA: U.S. Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health. [Google Scholar]
  33. U.S. Department of Health and Human Services. (2012). Preventing tobacco use among youth and young adults: A report of the Surgeon General. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health. [Google Scholar]
  34. U.S. Department of Health and Human Services. (2014). The health consequences of smoking—50 years of progress: A report of the Surgeon General. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health. [Google Scholar]
  35. United Way of America. (1996). Measuring program outcomes: A practical approach. Alexandria, VA: Author. [Google Scholar]
  36. Yarbrough DB, Shulha LM, Hopson RK, & Caruthers FA (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage. [Google Scholar]

RESOURCES