Advancing the Use of Evidence-Based Decision-Making in Local Health Departments With Systems Science Methodologies

Abstract

Objectives. We assessed how systems science methodologies might be used to bridge resource gaps at local health departments (LHDs) so that they might better implement evidence-based decision-making (EBDM) to address population health challenges.

Methods. We used the New York Academy of Medicine Cardiovascular Health Simulation Model to evaluate the results of a hypothetical program that would reduce the proportion of people smoking, eating fewer than 5 fruits and vegetables per day, being physically active less than 150 minutes per week, and who had a body mass index (BMI) of 25 kg/m² or greater. We used survey data from the Behavioral Risk Factor Surveillance System to evaluate health outcomes and validate simulation results.

Results. Smoking rates and the proportion of the population with a BMI of 25 kg/m² or greater would have decreased significantly with implementation of the hypothetical program (P < .001). Two areas would have experienced a statistically significant reduction in the local population with diabetes between 2007 and 2027 (P < .05).

Conclusions. The use of systems science methodologies might be a novel and efficient way to systematically address a number of EBDM adoption barriers at LHDs.

Local health departments (LHDs) play a vital role in protecting the health of communities and improving population health by leading and coordinating surveillance and health promotion efforts. There are more than 2800 LHDs in the United States, and they collaborate closely with public and private organizations to develop programs and enhance the capacity of local communities to address public health challenges.¹ To carry out these responsibilities in an effective manner, it is necessary to incorporate scientific evidence into program design and decision-making processes.^2–4

Evidence-based decision-making (EBDM) involves “making decisions on the basis of the best available scientific evidence, using data and information systems systematically, applying program-planning frameworks, engaging the community in decision making, conducting sound evaluation, and disseminating what is learned.”^2(p175),5 Although there is increasing support for the use of EBDM in public health practice, many LHDs use evidence-based approaches inconsistently because of a lack of expertise and resources, as well as the lack of availability of evidence-based programs that are adaptable to local contexts. This is the case despite the general interest of LHD leadership in the systematic use of EBDM to inform program design, development, and adoption.⁵

Systems science methodologies may provide a logical and cost-effective approach to implementing EBDM at LHDs that face resource constraints. A 2010 report from the Institute of Medicine (For the Public’s Health: The Role of Measurement in Action and Accountability) made key recommendations to improve health data analysis and reporting, and proposed that the US Department of Health and Human Services should,

coordinate the development and evaluation and advance the use of predictive and system-based simulation models to understand the health consequences of underlying determinants of health. HHS should also use modeling to assess intended and unintended outcomes associated with policy, funding, investment and resource options.^6(p9)

We assessed how systems science methodologies might be useful to bridge resource gaps at LHDs; these gaps impede the implementation of EBDM to solve local population health challenges. Although there are many systems science approaches that can be used to understand complex systems (e.g., network analysis, system dynamics modeling, discrete-event simulation), we focused on agent-based modeling (ABM) because this methodological framework allowed us to evaluate how individuals behave and naturally evolve based on a set of rules that might be more consistent with reality and the way people think about health progression and human and social interactions. ABM is a relatively new modeling approach compared with other systems science methodologies. Examples of its use in public health include studies of epidemics and health behaviors (e.g., drinking and smoking).^7–10

To demonstrate how ABM could be used to embed EBDM in public health practice at the LHD level, we studied the potential effects—in terms of health outcomes over time—of a lifestyle program or intervention in different local populations and compared the results with the natural progression of outcomes for these populations. More specifically, we looked at how LHDs located in 4 areas of New York State could use EBDM, within a systems science model, to help them understand the potential impact of implementing lifestyle interventions targeting cardiovascular disease (CVD) prevention. These model interventions were designed to be consistent with the goals of the state prevention agenda because they relate to conducting activities and developing programs to prevent chronic disease and improve cardiovascular health.¹¹ LHDs in the state have conducted community health planning and developed improvement plans that are used to track progress toward meeting the health objectives of the Prevention Agenda at the local level.¹¹ As such, systems science methodologies might prove useful to help LHDs achieve their objectives by incorporating EBDM into their approach.

METHODS

The New York Public Health Practice-Based Research Network conducted small group interviews and focus groups in 2010 with participants from 31 LHDs who had decision-making responsibilities.⁵ The study focused on how decisions were made at their LHDs, as well as on knowledge and use of EBDM. Although there was strong support and desire to implement EBDM, experience with EBDM was limited, and it was not systematically applied in public health practice. The study identified strong leadership, workforce capacity, resources, funding, data access, and suitable program models as some of the key factors related to the adoption and use of EBDM at LHDs.⁵

Table 1 lists key EBDM processes together with barriers to implementation and how systems science methodologies can be applied to address public health practice challenges at LHDs. A common barrier to the implementation of EBDM processes at LHDs is that specialized human or financial resources are required for effective implementation, but these resources are not always available at the local level. For example, the first EBDM process involves making decisions on the basis of the best available scientific evidence. LHDs may not have full access to the literature needed to conduct systematic reviews or to carefully assess the latest evidence reported. Appropriately designed systems science models can fill this gap, because many of them are built on the best available scientific evidence on how disease progression and health systems evolve. Thus, a well-constructed systems science model has already internalized the scientific evidence and predictions; therefore, simulations from these models are inherently evidence-based.

TABLE 1—

Evidence-Based Decision-Making Processes, Barriers, and Potential Benefits from System Science Methodologies

Evidence-Based Decision-Making Processes	Barriers to Implementation	Benefits From Applying Systems Science Methodologies
Synthesize best available scientific evidence	Limited access to relevant databases and literature to conduct systematic reviews; latest evidence difficult to interpret and translate at the local level	Reduce the need for literature review by incorporating best available scientific evidence in systems science models
Use data and information systems systematically	Lack of resources for data collection; limited information technology at the local level; difficulties tracking populations over time	Systems science models can be used to simulate what happens to individuals or populations over time, even if data sources are limited
Use program planning strategies	Difficulties prioritizing local public health challenges and identifying process and outcome measures	Facilitate understanding of scientific evidence and impact of interventions on local populations through data visualization and population animation, thus promoting coordination
Engage the community in decision-making	Communicating with different community organizations to identify policy priorities and build consensus is challenging	Systems science models can be used to help audiences visualize different policy scenarios, facilitating stakeholder engagement and strategic alignment
Conduct robust program evaluation	Comprehensive program evaluation difficult because of data needs, particularly for programs with long-term outcomes	Although model simulations cannot be used to conduct robust program evaluation, they can provide useful information about projected or expected outcomes
Disseminate lessons learned	Results difficult to convey retrospectively	Model simulations can be used to show how a program or intervention may work over time under different scenarios and assumptions

Systems science methodologies can also be used to engage the community in decision-making. Interactive system dynamics models have been used to help health planners to convene community groups, identify policy priorities, and support community coalitions.¹² Systems science models are particularly useful to help audiences visualize different policy scenarios, which may be useful to facilitate stakeholder engagement and strategic alignment.

Lastly, systems science can be used to project long-term outcomes of programmatic and policy relevance to LHDs (e.g., informing the likely outcomes of a diabetes self-management program designed to prevent complications like foot ulcers or diabetic retinopathy).¹³

Model

We used the New York Academy of Medicine Cardiovascular Health Simulation (NYAM-CHS) Model to evaluate potential local level outcomes of a lifestyle program that reduced the proportion of the population who smoked and who were overweight, while increasing the proportion of the population who were physically active and followed a healthy diet.¹⁴ The NYAM-CHS model was developed based on a comprehensive analysis of the scientific evidence from peer-reviewed literature. The model can be used to facilitate and inform decision-making for LHDs by predicting population health trajectories, to engage community stakeholders with data visualization and animation, and to evaluate intervention programs and establish evidence for future implementation. It is designed to track population health outcomes and mortality over a user-specified period of time. Through a user-friendly interface to the model, users can select varying combinations of initial population characteristics and interventions to evaluate different questions and scenarios.

The conceptual framework for the NYAM-CHS model builds on the 7 health factors used by the American Heart Association (AHA) to define ideal cardiovascular health (i.e., not having CVD while also not smoking, being physically active, having a healthy diet and a normal body weight, and achieving optimal cholesterol, blood pressure, and blood glucose levels).¹⁵ Each agent (person) in the model is defined according to these 7 health factors, as well as by age, gender, and history of myocardial infarction (MI) or stroke.

ABM is a relatively new approach and has been shown to have many advantages over other systems science approaches, in that it can be utilized with complex systems characterized by heterogeneity, nonlinear dynamics, and randomness.^16,17 Transition probabilities among different health states stratified by age and gender were obtained from published studies.^18–35 The correlations among the health factors and CVD were assessed using risks of benefit and harm following evidence-based medicine and public health. For example, the risks of having an MI or stroke were calculated using the widely accepted Framingham CVD Risk Calculator.¹

The model demonstrated consistent internal validity and face validity through extensive examination of model structure and code among the development group and through consultations with CVD experts. Parameters were calibrated, and desirable predictive validity was confirmed by favorable statistical test results for several health outcomes (e.g., by comparing simulated and actual results using nationally representative data from the Behavioral Risk Factor Surveillance System [BRFSS]).¹⁴ Although none of the systems science methodologies (including ABM) can replace actual comparative effectiveness research, the NYAM-CHS Model might provide insights into how to select effective population-specific interventions to improve cardiovascular health rapidly and inexpensively.

Data

We used data from the BRFSS for 4 areas of New York State to analyze how LHDs could use ABM to inform program implementation.^36,37 The BRFSS is a telephone survey conducted by state health departments across the United States. The target population includes adults ages 18 years and older living in households, and the survey includes standard core questions related to preventive health practices and chronic health conditions. We used data from the 2007 BRFSS to obtain demographic and health information on 4 areas in the state: New York City (NYC; Bronx, King, New York, Queens, and Richmond counties), the Rochester Metropolitan Statistical Area (MSA; Livingston, Monroe, Ontario, Orleans, and Wayne counties), the Suffolk County-Nassau County Metropolitan Division (Nassau and Suffolk counties), and the Buffalo-Cheektowaga-Tonawanda MSA (Erie and Niagara counties). These 4 regions were selected based on sample size and because they are part of the Selected Metropolitan/Micropolitan Area Risk Trends (SMART) project of the Centers for Disease Control and Prevention.³⁸ The SMART project uses BRFSS data to track health outcomes and assess public health priorities in selected areas. The NYC area defined in SMART included other surrounding counties, but we only included the 5 NYC counties (boroughs) because this is where local public health decisions are made for NYC.

The variables selected included age, gender, whether the respondent was a current smoker, had a normal weight (body mass index, or BMI [weight in kilograms divided by height in meters squared]< 25), was physically active (> 150 min/wk of moderate physical activity), had a healthy diet (ate ≥5 fruits or vegetables/d), did not have diabetes, hypertension, or high cholesterol, and had no history of MI or stroke. We estimated means or proportions for each of these variables for the population aged 20 to 79 years in 2007 and then compared the ABM simulation results with the population aged 25 to 84 years in 2012 to validate the predictive ability of the model. After excluding respondents with missing data, the sample sizes for 2007 and 2012, respectively, were 1391 and 1657 in NYC, 393 and 354 in the Rochester MSA, 678 and 712 in the Suffolk County-Nassau County Metropolitan Division, and 427 and 377 in the Buffalo-Cheektowaga-Tonawanda MSA.

RESULTS

Table 2 reports the population characteristics for adults aged 20 to 79 years in the 4 areas selected. Means and proportions were estimated using data from the 2007 BRFSS. The mean age for the 4 areas ranged from 44.09 years in NYC to 48.08 years in the Suffolk County-Nassau County Metropolitan Division. The proportion of nonsmokers ranged from 75.57% in the Rochester MSA to 84.64% in NYC. The proportion of the population with a BMI less than 25 kg/m² ranged from a low of 30.68% in the Rochester MSA to a high of 37.45% in the Suffolk County-Nassau County Metropolitan Division. Approximately one third (33.43%) of NYC respondents were physically active (i.e., were doing > 150 min/wk of moderate physical activity) compared with 38.52% of respondents in the Rochester MSA. Only 25.97% of Suffolk County-Nassau County Metropolitan Division respondents had a healthy diet (i.e., ate ≥ 5 fruits or vegetables/d) compared with 30.97% of Buffalo-Cheektowaga-Tonawanda MSA respondents. The proportion of the population without diabetes was the highest in the Suffolk County-Nassau County Metropolitan Division (93.45%) and the lowest in the Buffalo-Cheektowaga-Tonawanda MSA (90.48%). There was also noticeable variation across the 4 selected areas in hypertension and high cholesterol rates, but there was a relatively lower variation in the proportion of the population having a history of MI and stroke.

TABLE 2—

Population Characteristics by Area: Behavioral Risk Factor Surveillance System, New York State, 2007 and 2012

Characteristic	NYC (n = 1391)	Rochester (n = 393)	Suffolk/Nassau (n = 678)	Buffalo (n = 427)
Mean age, y, ±SD	44.09 ±15.75	46.50 ±15.05	48.08 ±14.75	47.27 ±15.72
Female, %	51.03	52.73	52.30	50.96
Not currently smoking, %	84.64	75.57	81.18	79.02
BMI < 25 kg/m2, %	36.56	30.68	37.45	35.42
Physically active, %	33.43	38.52	36.00	37.40
Have healthy diet, %	29.10	28.30	25.97	30.71
No diabetes, %	92.28	91.97	93.45	90.48
No hypertension, %	75.32	69.46	72.86	69.67
No high cholesterol, %	72.23	72.07	66.60	65.53
History of MI, %	2.45	3.28	2.64	2.95
History of stroke, %	2.00	1.89	1.15	4.05

Note. BMI = body mass index (calculated as weight in kilograms divided by height in meters squared); MI = myocardial infarction; NYC = New York City.

Table 3 reports actual and simulated results of 5 key health indicators for the 4 areas selected. Actual and simulated progression of the health indicators selected over the 2007 to 2012 period suggests that the NYAM-CHS Model was able to track these indicators reasonably well; the actual and simulated differences were very close to each other for most of these variables. More specifically, a 2-proportion z-test that compared actual with simulated normal progression for smoking, BMI of 25 kg/m² or greater, and diabetes from 2007 to 2012 had P values greater than 0.05 (except for diabetes in the case of the Suffolk County-Nassau County Metropolitan Division, which had a P = .002). Thus, in 11 of 12 tests, we failed to reject the hypothesis that the actual and simulated proportions for these 3 health indicators in 2012 were equal. The same results arose for MI in NYC and for MI and stroke in the Suffolk County-Nassau County Metropolitan Division. However, the 2-proportion z-test for stroke in NYC and stroke and MI in the Rochester MSA and the Buffalo-Cheektowaga-Tonawanda MSA had P values less than 0.05.

TABLE 3—

Simulation Results (n = 1000) for Normal Health Progression and Lifestyle Program in 2012 and 2027: New York State, 2007 and 2012

	Smoking		BMI ≥ 25		Diabetes		MI		Stroke
Variable	%	P	%	P	%	P	%	P	%	P
New York City
2012
Actual normal progression (n = 1657)	14.1		62.0		12.8		4.1		3.8
Simulated normal progression	14.6	.721	64.0	.302	12.0	.546	2.8	.082	2.3	.034
Simulated lifestyle program	7.9	< .001	39	< .001	10.7	.359	3.1	.691	2.6	.664
2027
Simulated normal progression	13.8		70.9		29.4		5.7		4.7
Simulated lifestyle program	7.4	< .001	51.0	< .001	27.7	.4	4.6	.266	4.0	.443
Rochester
2012
Actual normal progression (n = 354)	20.4		64.0		12.9		6.0		5.2
Simulated normal progression	23.8	.191	69.6	.052	11.2	.391	3.3	.026	2.9	.043
Simulated lifestyle program	11.3	< .001	40.8	< .001	12	.576	4.1	.343	2.1	.252
2027
Simulated normal progression	23.0		72.4		32.7		7.2		5.3
Simulated lifestyle program	10.9	< .001	55.0	< .001	28.9	.066	6.6	.597	4.0	.167
Suffolk/Nassau
2012
Actual normal progression (n = 712)	15.9		65.8		6.3		3.7		1.6
Simulated normal progression	17.6	.355	65.4	.864	10.6	.002	3.0	.424	2.0	.543
Simulated lifestyle program	8.7	< .001	41.2	< .001	9.7	.505	2.9	.895	1.7	.619
2027
Simulated normal progression	17.4		68.8		29.9		6.3		3.7
Simulated lifestyle program	8.8	< .001	54.9	< .001	25.6	.032	5.6	.508	3.2	.54
Buffalo
2012
Actual normal progression (n = 377)	21.7		69.7		12.7		6.0		1.4
Simulated normal progression	21.3	.872	66.3	.231	13.4	.732	3.1	.013	4.9	.003
Simulated lifestyle program	10.5	< .001	38.8	< .001	12.6	.595	2.8	.692	4.9	>.992
2027
Simulated normal progression	20.5		68.5		31.9		6.5		6.7
Simulated lifestyle program	10.2	< .001	53.3	< .001	27.7	.04	5.2	.215	6.3	.717

Note. BMI = body mass index (calculated as weight in kilograms divided by height in meters squared); MI = myocardial infarction.

Table 3 also shows simulations that compared the normal progression of health indicators with the proposed comprehensive lifestyle program which was designed to reduce, by half, the proportion of the population who smoked, ate less than 5 fruits and vegetables per day, were physically active less than 150 minutes per week, and had a BMI of 25 kg/m² or greater. Five and 20-year endpoints (i.e., 2012 and 2027) were used to evaluate health outcomes. In NYC, the lifestyle program would result in reductions in smoking rates from 14.6% to 7.9% by 2012 (an 85% reduction) and from 13.8% to 7.4% by 2027 (an 87% reduction). Smoking rates would have fallen 111%, 102%, and 203% by 2027 in the Rochester MSA, the Suffolk County-Nassau County Metropolitan Division, and the Buffalo-Cheektowaga-Tonawanda MSA, respectively.

For all the 4 areas evaluated, both smoking rates and the proportion of the population with a BMI of 25 kg/m² or greater would have decreased significantly in 2012 and 2027 (P < .001) with implementation of the proposed comprehensive lifestyle program. The proportion of the population with diabetes, MI, and stroke would not have changed significantly in 2012 (P > .05). However, there was a downward trend in the proportions of most of the health indicators studied. For example, in the Suffolk County-Nassau County Metropolitan Division, the proportion of the local population with diabetes, MI, and stroke would decrease from 10.6%, 3.0%, and 2.0% to 9.7%, 2.9%, and 1.7%, respectively (from 2007 to 2012). The same downward trend was observed in the 2027 simulated results.

Both the Suffolk County-Nassau County Metropolitan Division and the Buffalo-Cheektowaga-Tonawanda MSA would have experienced a statistically significant reduction in the proportion of the local population with diabetes during the 2007 to 2027 period (P < .05). Moreover, there was a pronounced downward trend in the proportion of the local population with MI and stroke in 2027 compared with 2012 under the lifestyle program. However, these differences were not statistically significant (P > .05).

DISCUSSION

More than 2800 LHDs coordinate and provide key public health services across communities in the United States. LHDs share a strong and sustained interest in the use of EBDM in public health practice, but they face substantial challenges in the design and implementation of evidence-based programs and interventions.^2–4 The use of systems science methodologies might help to address a number of these barriers at a cost that is likely to be significantly lower than acquiring the technology and human resources required to effectively utilize EBDM internally.

Our study showed how ABM, a specific systems science approach, could be used to compare outcomes for a population in a given area (e.g., New York City) if a comprehensive lifestyle program was implemented compared with outcomes under the normal conditions. In our specific application, we relied on data from the BRFSS in 4 areas of New York State to compare and contrast the differences in results. We also used the NYAM-CHS Model, because it was validated with national data from the BRFSS, and it had a flexible, easy-to-use interface that could compare different policy scenarios at the local level.¹⁴

Other systems science models are readily available to address similar problems, and there is wide variation related to ease, utility, technical support, breadth, and ability to customize interventions. The Archimedes Model, for example, allows for the modeling of disease progression and health care costs, and the model has been extensively validated with data from clinical trials and longitudinal studies.^39,40 The cost of using the model is particularly low for government agencies because of an agreement between Archimedes Inc. (San Francisco, CA) and the US Department of Health and Human Services. Research organizations across the United States also have systems science models or the capability to construct and customize a model at a cost that is significantly lower than building the local infrastructure required to, for example, collect data to conduct a comprehensive needs assessment.

LHDs can accelerate the systematic implementation of EBDM by building collaborative partnerships with universities, research centers, and businesses that have the technical expertise to develop evidence-based systems science models. These organizations have incentives to engage in the long-term collaboration with LHDs to gain access to data and populations for research, seek new opportunities for internships, training and placement, and develop partnerships with other organizations in local areas to meet synergistic goals.

Limitations

There were some important limitations to our work. First, systems science models like the one we used did not allow for the perfect representations of reality. Although the NYAM-CHS Model was able to predict 5-year outcomes for smoking, obesity, and diabetes reasonably well, it was less reliable for MI and stroke. This was in part because MI and stroke were longer-term outcomes; the proportion of the population with MI and stroke was also much lower than for the other health conditions studied. Moreover, because of the lack of longitudinal data at the local level, we had to use cross-sectional BRFSS data at 2 points in time for the selected 4 areas of New York State. Also, the BRFSS data might underestimate the disease prevalence for health conditions such as diabetes because some people who have chronic health conditions might be undiagnosed at the time of interview. Moreover, collecting survey data using phone interviews might lead to biased estimates related to oversampling or undersampling (e.g., high-risk underinsured populations). Lastly, although the NYAM-CHS Model was based on the peer-reviewed literature, the construction of the model might have missed important studies and interactions that would have made the model more accurate.

Conclusions

We showed promising examples of how systems science methodologies could be used to enhance EBDM in LHDs. We focused on the progression of health outcomes over time, but the approach proposed could certainly be expanded to include visualization tools, a higher level of detail on how agents move across different health states, and interactions among agents to capture peer and neighborhood effects. In the end, the adoption of systems science as a way to promote the systematic use of EBDM in public health practice at LHDs will hinge on the level of interest of leaders in using these types of models as a part of implementing a set of evidence-based approaches to solve challenges in local public health systems.