Process Evaluation and Lessons Learned From Engaging Local Policymakers in the B’More Healthy Communities for Kids Trial

Abstract

Partnerships linking researchers to the policymaking process can be effective in increasing communication and supporting health policy. However, these policy partnerships rarely conduct process evaluation. The Policy Working Group (Policy WG) was the policy-level intervention of the multilevel B’More Healthy Communities for Kids (BHCK) trial. The group sought to align interests of local policymakers, inform local food and nutrition policy, introduce policymakers to a new simulation modeling, and sustain intervention levels of BHCK. We conducted an evaluation on the Policy WG between July 2013 and May 2016. We evaluated process indicators for reach, dose-delivered, and fidelity and developed a SWOT (strengths, weaknesses, opportunities, and threats) analysis. The policy intervention was implemented with high reach and dose-delivered. Fidelity measures improved from moderate to nearly high over time. The number of health-related issues on policymakers’ agenda increased from 50% in the first 2 years to 150% of the high standard in Year 3. SWOT analysis integrated a stakeholder feedback survey to consider areas of strength, weakness, opportunity, and threats. Although the fidelity of the modeling was low at 37% of the high standard, stakeholders indicated that the simulation modeling should be a primary purpose for policy intervention. Results demonstrate that process evaluation and SWOT analysis is useful for tracking the progress of policy interventions in multilevel trials and can be used to monitor the progress of building partnerships with policymakers.

Previous work to improve the food environment in urban settings aimed to increase availability and access to healthy food (Shin et al., 2015; Walker, Keane, & Burke, 2010). However, sustainability of environmental interventions and scale-up is challenging, which may be addressed by engaging local policymakers. These environmental changes are also defined as structural change approaches (Golden, McLeroy, Green, Earp, & Lieberman, 2015). Recent evidence shows multilevel, multicomponent interventions including a policy component may be effective (Bell, Simmons, Sanigorski, Kremer, & Swinburn, 2008; Davison et al., 2015; Folta et al., 2013). Including a policy intervention in a study may also increase policymaker engagement for health policy (Lewis et al., 2011; Otten, Dodson, Fleischhacker, Siddiqi, & Quinn, 2015). Furthermore, partnerships linking researchers to local policymakers can increase sustainability of community interventions, thus maximizing impact on obesity rates (Folta et al., 2013; Porter, 2013; World Health Organization, 2012).

In 2013, the B’More Healthy Communities for Kids (BHCK) trial, a multilevel intervention seeking to improve the food environment in Baltimore, convened a Policy Working Group (Policy WG) that partnered Global Obesity Prevention Center at Johns Hopkins University researchers with more than a dozen Baltimore City and Maryland State policy stakeholders. Central goals of the Policy WG included leveraging the expertise of participants to inform and support city and state food policies and to sustain BHCK intervention components (Gittelsohn et al., 2014).

A critical gap exists in describing the implementation of academic–policymaker partnerships (de Silva-Sanigorski et al., 2010; Fleischhacker, Otten, Dodson, & Siddiqi, 2015; Folta et al., 2013). Additionally, strengths and weaknesses of these relationships should be identified to develop recommendations for future improvement. To our knowledge, no studies have reported the results of process evaluation and SWOT (strengths, weaknesses, opportunities, and threats) analysis of a policy intervention.

Overview of the BHCK Intervention Design

Using previous multilevel, multicomponent interventions as models (Economos & Curtatone, 2010; Foster et al., 2008), BHCK established the Policy WG to bridge the gap between the research team and key city policy stakeholders, and to create dialogue about how research can be effectively translated to policy.

BHCK held initial discussions with the Baltimore City Food Policy Advisory Committee (Food PAC), a collaboration between the Baltimore City government and more than 60 community-based organizations (CBOs) involved in community food access and nutrition policy, to ensure buy-in at the city level. BHCK then convened a stakeholder meeting of nearly 40 stakeholders including Baltimore Food Policy Initiative, Baltimore City Health Department, the Office of Sustainability, City Council members, and public school representatives for an initial discussion of food and nutrition policy interests, followed by an online survey to prioritize those interests. Subsequent meetings focused on successes and challenges of the BHCK intervention, sustainability of interventions, local CBOs’ activities in food policy, and current food policies at the city and state levels. Stakeholder input shaped topics in future meetings. These meetings were also a place for the research team over the course of the entire intervention to report on the progress in all intervention components through process evaluation. This provided stakeholders a way to suggest adjustments to the BHCK intervention as a whole with presentations of process data and discussion with teams from all levels of the intervention. Policymakers also often added food-related issues to their agendas. The internal BHCK weekly meeting served as a platform to discuss policy legislation and strategies to use media and social media to shape policymaker and public opinion.

In addition, the Policy WG introduced Baltimore policy-makers to an agent-based simulation model to test the potential impact of specific obesity prevention policies and interventions on the health of the adolescents (Gittelsohn et al., 2015). BHCK shared updates about the simulation model during the meetings, and informed stakeholders about the utility of the model while gathering feedback from stakeholders. This contributed to advancing the development of the model, which helped ensure its relevance to all stakeholders. In 2013, the first iteration of the simulation model supported after-school food policies in low-income environments (Gittelsohn et al., 2015). The Global Obesity Prevention Center research team further enhanced the model in 2014 to simulate individuals’ interaction with the built environment (Lee et al., 2018).

This article aims to evaluate the policy intervention of the Policy WG using process evaluation and SWOT analysis.

Method

The Policy WG measured process indicators (reach, dose-delivered, and fidelity) throughout the intervention. Later, we used a SWOT analysis to integrate the overall progress of the intervention with stakeholder feedback to determine their view of the partnership’s progress.

Prior to the program starting, standards for implementation were set for each process indicator. The process evaluation indicators included the amount in which the intended audience was exposed to the intended intervention (dose-delivered), how well the intended audience was engaged in the Policy WG (reach), and how well the intervention was carried out as originally planned as a proxy for quality (fidelity; Steckler, Linnan, & Israel, 2002). Additional process indicators for simulation modeling development were created based on prior experience with program evaluations and current literature (Gittelsohn et al., 2014; Seifu et al., 2018). During Year 2 of the partnership, media use standards were added as the working group saw potential in the use of mass media to influence policymakers.

Table 1 outlines the data collection instruments used. The quarterly meeting minutes collected the number of sectors including city departments, CBOs, government agencies, and other stakeholder represented. Increased interaction with local media coverage of food policy issues included any use of mass media (print, TV, or radio) by the Policy WG to advance policies discussed in the quarterly meetings. We used Microsoft Excel 2011 to calculate summary statistics for the quarterly meetings and feedback survey data (Microsoft, 2011). Parameters for the model were any factors that can vary based on the policy, influencing adolescent decision making and, in turn, health. Stakeholders suggested model parameters during quarterly meetings and in-depth interviews (IDIs). Suggesting parameters for the model was another avenue for engagement with policymakers. Stakeholders were identified for IDIs through purposive sampling and were conducted by the research team for about 60 minutes each, who asked questions related to how policies and programs on childhood obesity affected the physical activity of teens and the Baltimore food environment, and development of the simulation model (Seifu et al., 2018). The modeling team then determined and quantified how each parameter affected decision making (Lee et al., 2018).

Table 1.

Evaluation Instruments Used for the B’More Healthy Communities for Kids Policy Component.

Instrument	Process component	Indicators collected	Number of times collected
Quarterly meeting minutes	Reach	No. of attendees	11
	Dose-delivered	No. of meetings
	Dose-delivered	No. of action items set
	Fidelity	No. of health related issues on agenda/introduced
	Fidelity	No. of stakeholders suggesting new parameters/year (Modeling)
		No. of stakeholders using simulation model (Modeling)
	Reach	No. of sectors
In-depth interview	Fidelity	No. of interviews conducted	11
Weekly internal policy meeting minutes	Reach	No. of letters to the editor/op-eds (Media)
	Dose-delivered	Percent print media published (Media)	—
	Fidelity	No. of IDIs conducted (Modeling)
	Fidelity	Percent of action items achieved
Feedback survey		Stakeholder feedback	1

Note. IDI = in-depth interview.

Measures of success included percent of high standard attained and positive stakeholder feedback showing the partnership was achieving its goals. Standards for each year were averaged across all 3 years to find an average percent achieved. These percentages were then aggregated for each process component to give a total indicator of dose-delivered, fidelity, and reach. A process component attained a high level at 100% or more of the standard, moderate at 50% to 99% of the standard, and low at 0% to 49% of the standard. These were defined as the degree to which the standard was achieved, and the cutoffs are a modification following previous studies (Gittelsohn et al., 2010; Shin et al., 2015). Between July 2013 and May 2016, the Policy WG aimed to meet quarterly, while holding weekly internal meetings. Meeting minutes and reports served as a way to collect data. The research team set action items at the end of each quarterly meeting and internal meetings tracked progress on action items. Informal communication (in-person and e-mail) with the current and past Policy WG coordinators provided data related to development of the group and action items completed.

The SWOT analysis described internal strengths and weaknesses as well as external opportunities and threats to the sustainability of the partnership as previously the framework has been shown to be useful in showing the strategic fit of a project for future program and policy planning (Helms & Nixon, 2010; van Wijngaarden, Scholten, & van Wijk, 2012). In this case, the SWOT analysis of the Policy WG synthesized process results, relevant trial results, and WG history in the context of the Baltimore policy environment.

The survey included seven questions asking stakeholders for their opinion on how well the Policy WG met objectives and goals using a Likert-type scale, ranging from strongly agree to strongly disagree. Other questions included one open-ended question asking about a future policy opportunity for the group and closed-ended (yes/no) survey questions with the option to further elaborate about future attendance and whether the group assisted in passing policies or increasing the amount of food and nutrition policy added to stakeholders’ agendas. A BHCK research team member used descriptive analysis to aggregate the survey results and in turn inform the SWOT analysis.

Results

Process Indicators

Reach.

The Policy WG achieved a high reach at 133% of the standard, ranging from 6 to 10 sectors, and averaging 8 sectors in attendance at meetings (Table 2). The Policy WG averaged 26 attendees per meeting, achieving a moderate reach at 94% of the high standard (Table 2).

Table 2.

BHCK Policy WG Standard Indicator Results by Year.

		Percent of high standard met, %a
Indicator	High standard	Year 1 (July 2013–June 2014)	Year 2 (July 2014–June 2015)	Year 3 (July 2015–May 2016)	Three-year average, %	Aggregate indicator, %
Dose-delivered						103
Mean no. of meetings	≥4	100	75	75	83
No. of action items set	≥16	100	125	144	123
Fidelity						71
Percent action items achieved per year	50%	90	111	92	98
No. of health-related issues put on any policymaker’s agenda as a result of BHCK Policy WG	≥2	50	50	150	83
No. of health-related issues introduced by policymaker a result of BHCK Policy WG	≥2	50	0	50	33
Reach						113
Mean no. of attendees per meeting	≥25	92	113	76	94
Mean no. of different sectors represented at meeting (min-max)	≥6	121	156	122	133
BHCK use of media to advance or cover health-related issue (dose-delivered)						46
No. of letter to the editor, op-eds, press releases, etc.	≥3	—	0	67	33
% of print media published	50	—	0	200	100
No. of radio appearances	≥3	—	0	67	33
No. of television appearances	≥3	—	0	33	17
Utilizing the simulation model (fidelity)						25
No. of stakeholders suggesting new parameters/year in meetings	≥11	45	0	9	18
No. of IDIs conducted	≥11	—	36	100	45
No. of stakeholders using simulation model to guide policy	≥6	17	0	17	11

Note. BHCK = B’More Healthy Communities for Kids; Policy WG = policy working group; IDI = in-depth interview.

^aLow: 0% to 50%; moderate: 50% to 99%; high: above 99%.

Dose-Delivered.

The Policy WG attained a high dose-delivered at 103% of the standard (Table 2). Though the Policy WG set a goal of quarterly meetings, during the last 2 years the group only had three meetings per year (Table 2). The group set a consistent number of action items across the 3 years and achieved a high dose-delivered at 123% (Table 2). Policy WG media standards achieved a low dose in Year 2 and improved to a moderate dose the last year as the group published all print media (Table 3).

Table 3.

BHCK Policy WG Average Yearly Standards Met by Percent of High Standard.^a

Process component	Year 1	Year 2	Year 3	Aggregate total
Dose-delivered	100%	100%	109%	103%
Media (Dose-delivered)	-	0%	92%	46%
Fidelity	63%	53%	97%	71%
Modeling (Fidelity)	21%	12%	42%	25%
Reach	106%	134%	99%	113%

^aLow: 0% to 50%; moderate: 50% to 99%; high: above 99%.

Fidelity.

Table 2 shows the percent of action items that achieved 98% of the standard on average per year. The number of food and nutrition-related issues on policymakers’ agendas improved from 50% of the standard in the first 2 years to 150% of the standard in the last year. The number of policies introduced by policymakers achieved a low fidelity with a 3-year average of 33%. In the first 2 years of the Policy WG, the fidelity standards reached a moderate level at 73% and 72% of the standard, and in the last year improved to 109% of the fidelity standard (Table 2).

Simulation Model Fidelity.

Measures of fidelity for the simulation model included the number of IDIs, number of stakeholders suggesting parameters, and number of stakeholders using the simulation model to guide policy. The number of IDIs averaged 7.5 conducted per year, achieving moderate fidelity (Table 2). Both the number of stakeholders suggesting parameters and using the model to guide policy had a 3-year average reaching a low level at 37% and 17%, respectively. In the first 2 years, the fidelity of the simulation model was low though improved to a moderate level at 63% of the high standard in the third year (Table 3).

Overall, the Policy WG achieved an aggregate reach at 113% of the high standard (Table 2). Fidelity reached a moderate level in the first 2 years and improved to achieve nearly high in Year 3, showing improvement on achieving the group’s goals, for an overall 3-year average at a moderate level of 71% (Table 3). Simulation modeling fidelity reached a low level overall at 37% of the high standard (Table 2). Policy WG media standards achieved a 46% dose delivered over the 2 years (Table 2). Modeling standards indicated a low fidelity over the 3-year period at 25% (Table 2).

SWOT Analysis

The second part of the evaluation, the SWOT analysis, shown in Table 4, integrated feedback from 11 stakeholders, overall progress, and history of the group.

Table 4.

SWOT Analysis of Working Group Sustainability.

Internal factors	External factors
Strengths	Opportunities
• Formative process with BFPI, Food PAC, and local stakeholders	• Provides policymakers with a way contribute to research
• Disseminated BHCK program findings quickly	• Researchers can engage policymakers easier
• Provided a platform to find ways to sustain the BHCK trial, discuss specific food policy, and prioritize food policy interests collaboratively	• Convened multiple sectors for regular engagement of policymakers
• Introduced and engaged policymakers to simulation modeling	• If food policies are implemented, opportunity to affect childhood obesity in Baltimore
	• Opportunity to affect various media sources through concerted efforts
Weaknesses	Threats
• Difficult to track policies	• Election year threatened the productivity and availability of the group (election campaigns, different mayor and city council members may have other agendas)
• Mainly public sector membership	• Timeline of researchers and policymakers differ
• No full-time staff person for the group and many leadership transitions	• Overlap of existing policy partnerships with Policy WG as a threat to future attendance
• Simulation model took time to create and policymakers had a learning curve

Note. SWOT = strengths, weaknesses, opportunities, and threats; BFPI = Baltimore Food Policy Initiative; BHCK = B’More Healthy Communities for Kids; Food PAC = Food Policy Advisory Committee.

Strengths of the group ranged from engaging stakeholders through initial discussions with city departments and policy stakeholders who ensured the partnership filled gaps in city coalitions and included collaboration from a wide range of stakeholders. Stakeholders agreed that the Policy WG provided a way to sustain BHCK program activities, support policy initiatives, and network with others in food and nutrition policy. Policy WG helped pass two food policies, and four stakeholders added food and nutrition policy to their priorities. The Policy WG allowed the BHCK research team to disseminate findings quickly to key stakeholders and receive immediate feedback. Other levels of the BHCK intervention were sustained potentially as a result of partnerships developed through the Policy WG. For example, the Baltimore County Health Department Baltimarket program will continue corner store work.

The SWOT analysis highlighted a few weaknesses in the group including membership consisting of mainly public sector organizations and lacking engagement of private sector organizations such as grocery stores and restaurants, as other coalitions have done. Additionally, through the feedback survey, 90% of the stakeholders agreed that one of the primary aims of the Policy WG was to continue discussion of the simulation model. Though modeling standards indicated a low fidelity over the 3-year period (Table 2) the feedback from the survey shows an opportunity for the Policy WG to fully engage stakeholders with an interest in remaining updated and included in discussions about the simulation model.

In terms of opportunities, the SWOT analysis revealed that the Policy WG could potentially affect the food environment in Baltimore by combining expert knowledge and community efforts. The group gave policymakers an avenue to contribute to the focus of future research and provided an opportunity for researchers to engage policymakers in discussions about research findings. The Policy WG expanded current food policy partnerships in Baltimore to include sectors not currently involved in Baltimore Food Policy Initiative, such as wholesalers, as well as corner store and carryout owners. Researchers used social media to communicate about policy issues continuously to policymakers, and the general public. Though using media as a way to communicate was brief, the overall trend showed that this avenue might help to increase engagement with policy stakeholders (Loh et al., in press). Since the group implemented media interventions, these publications have the potential to affect public perception of food policy issues though future research studies will need to study to what degree.

Conversely, the SWOT analysis identified several threats to the group. The duration of terms for some policymakers differed from researcher’s timelines that depend on grant funding. For instance, the membership of the group remained uncertain in the final year of the group (2015–2016) because the mayoral and city council elections began. The political landscape of the city government or capacity of individual organizations caused changes in agenda, and the availability and involvement of members. Last, existing food policy initiatives in Baltimore City may be a threat to the Policy WG as the overlap in the groups may cause some members to choose to attend other groups rather than the Policy WG. However, more than 60% of members surveyed said they would continue to attend the Policy WG.

Discussion

This is the first article to systematically describe the process evaluation and SWOT analysis of the policy component of a multilevel, multicomponent intervention. While other large, multilevel trials have conducted process evaluations, they lacked a policy component or an in-depth evaluation of that component (Chuang et al., 2016; Joseph et al., 2015; O’Connor et al., 2015). The study illustrates the utility of evaluating the Policy WG and reflects on its potential application for other multilevel, multicomponent studies collaborating with policymakers.

The Policy WG achieved high reach, convening on average eight sectors per meeting. This supports qualitative findings that emphasized stakeholders from different sectors want more communication and multisector collaboration, especially in achieving common goals (Ganter et al., 2016).

The Policy WG enabled BHCK to communicate with policy stakeholders and successfully gathered multiple sectors to discuss food and nutrition policy. Though the group mainly informed local city policies, policies at the state level were also discussed. The more the group is sustained, the more likely the reach could extend to state and national levels affirming that face-to-face networks can create policy change (Porter, 2013).

The partnership also provided a forum to increase policy-makers’ awareness of obesity and the complex nature of changing a food system and healthy food availability. The quarterly meetings inspired valuable discussion on food policy impact in Baltimore. The researchers used the Policy WG as a springboard for prioritizing food policy topics to use for engaging media. Though only prioritized in the last 2 years, it resulted in a large improvement in dose-delivered. Previous studies also found that communication at every level is needed, showing the need to prioritize media early especially when trying to affect policy (Hatfield, Sliwa, Folta, Economos, & Goldberg, 2016; Porter, 2013). As partnerships thrive, the number of food policies discussed by policymakers should continue to increase including further opportunities to test policies with the simulation model.

The high reach achieved may likely be due to the large interest in food policy already established in Baltimore prior to the study team’s engagement with stakeholders. Since 2010, the City of Baltimore successfully convened a cross-sectoral community collaboration, the Baltimore Food Policy Initiative (Bedore, 2014). This also may prove to be a threat to the sustainability of the group though over 60% of members surveyed said they would continue to attend the Policy WG. The Policy WG is currently discussing combining efforts with Food PAC in order to decrease the number of meetings stakeholders need to attend.

The high dose-delivered was shown by the number of consistently set action items at meetings. Noting that the lower number of meetings in Years 2 and 3 may be due to the differing timelines in the BHCK intervention versus policy-maker election cycles and may be indicate the time needed to increase buy-in with the stakeholders. These may be indicators of policy resistance in some of the key stakeholders. Not only did the study introduce new strategies to address childhood obesity in Baltimore but also new simulation tools to consider how the environment affected obesity. In the multi-level California–Childhood Obesity Research Demonstration (CORD) and Massachusetts-CORD studies, researchers acknowledged the need to decrease policy resistance in order to have maximum impact and sustainability (Chuang et al., 2015). If sustained, the Policy WG could be a way in which policy resistance can be controlled, as BHCK worked alongside policymakers throughout the trial to ensure they were up to date and integrated any feedback they had on the trial into the other levels.

Introduction of the simulation models to the Policy WG enhanced how policymakers viewed the potential impact of food policies in Baltimore. Continued discussion would likely continue to foster greater stakeholder trust and confidence in the model. Though the IDIs became the main avenue for parameter feedback, Policy WG members still proposed a number of policies for the researchers to explore through simulation models. For example, a city councilman and Policy WG member proposed collaborating on a policy and developing a simulation model that ultimately led to the passage of the Baltimore City Urban Agriculture Tax Credit (Gittelsohn et al., 2015). The application of this model allowed policymakers to see how fostering these urban farms could affect food access and health of the community (Gittelsohn et al., 2015). As the model becomes more robust, policies should take less time to simulate and process evaluation indicators should improve.

Using the indicators of reach, dose-delivered, and fidelity through process evaluation is a way to ensure continual progress toward common goals and group development. Combined with the SWOT analysis, this evaluation gave indication of stakeholder feedback and interest in future collaborations. The SWOT analysis incorporated feedback that was not measured through the process indicators. For example, the SWOT analysis indicated that stakeholders’ perception of using the simulation model was positive though the fidelity was low. It also showed stakeholder’s opinion of the progress of the group. More important, this evaluation showed successful integration of a policy component into the BHCK program to provide continuous stakeholder feedback.

Future policy groups may want to consider what stakeholders would like to see more from the partnership to establish more buy in. For example, if policymakers wanted increased knowledge exchange, the researchers could consider synthesizing results into easy to use briefs and policy notes for easy dissemination (Addy, Shaban-Nejad, Buckeridge, & Dubé, 2015). This will engage policymakers on a deeper level and will work toward ensuring a reciprocal and sustainable relationship.

Limitations

The SWOT analysis relied on past self-reporting of the Policy WG leaders. Combined with the Policy WG feedback, this evaluation provides a way for the partners to help shape the future of the partnership. However, the policy stakeholder feedback survey was only collected once. An annual satisfaction survey to revisit objectives and track perceived efficacy of the Policy WG should be considered. Additionally, the Policy WG did not have a full-time staff member solely dedicated to lead the group. Initial leadership turnover resulted in inconsistent tracking methods, which later inspired the development of a tracking instrument to ensure consistent tracking of indicators and fully capture all indicators and a need for a written set of responsibilities for Policy WG leaders. Documenting this process information can facilitate replication of researcher–policymaker partnerships in new locations. To measure changing group interests year-to-year, BHCK staff added additional standards causing some discrepancies in tracking. These included increasing state and local policies introduced at meetings, which led the weekly internal policy meeting to increase engagement with local media the last 2 years.

There is room to improve the use of media and simulation modeling within this group, as these tools can increase engagement with stakeholders (Hatfield et al., 2016; Lee et al., 2017). Members often self-reported current policies informally through e-mails and conversations to research staff, which may have limited the measurement of agenda items and affected the number of policies recorded. A short self-report form for stakeholders could be used to report food policy activities related to the group biannually, helping document activities outside of the Policy WG, similar to a coalition member report form (Butterfoss, 2006).

Conclusions

Policy stakeholders remained engaged throughout the 3-year partnership and the majority agreed the partnership achieved the goals set. The BHCK Policy WG had high reach, successfully including community stakeholders and multiple sectors, high dose-delivered, and moderate fidelity to intended stakeholders. Process evaluation combined with a SWOT analysis provided a useful framework of evaluating a policy intervention in a multilevel, multicomponent trial. If the Policy WG is sustained, it may become an effective platform to support obesity interventions and local food and nutrition policy efforts from early stages of informing intervention development, to aiding in long-term funding and technical support. Future research is needed to develop tools to discern specific factors influencing policy-related decisions.