Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Mar 1.
Published in final edited form as: Am J Community Psychol. 2013 Mar;51(1-2):232–242. doi: 10.1007/s10464-012-9544-4

Accomplishing structural change: Identifying intermediate indicators of success

Robin Lin Miller 1, Sarah J Reed 1, Vincent Francisco 2; the Adolescent Medical Trials Network for HIV/AIDS Interventions
PMCID: PMC3495085  NIHMSID: NIHMS369568  PMID: 22875684

Abstract

Coalitions are routinely employed across the United States as a method of mobilizing communities to improve local conditions that impact on citizens’ well-being. Success in achieving specific objectives for environmental or structural community change may not quickly translate into improved population outcomes in the community, posing a dilemma for coalitions that pursue changes that focus on altering community conditions. Considerable effort by communities to plan for and pursue structural change objectives, without evidence of logical and appropriate intermediate markers of success could lead to wasted effort. Yet, the current literature provides little guidance on how coalitions might select intermediate indicators of achievement to judge their progress and the utility of their effort. The current paper explores the strengths and weaknesses of various indicators of intermediate success in creating structural changes among a sample of 13 coalitions organized to prevent exposure to HIV among high-risk adolescents in their local communities.

Keywords: Coalitions, success indicators, HIV/AIDS

Efforts to create structural change sit at the leading edge of current efforts to reduce entrenched disparities in public health (Frieden, 2010; Marmot, 2005). Structural changes represent second-order changes to social, economic, political, cultural, and environmental systems that have the potential to reduce or eliminate excess vulnerability to health risks (Blakenship, Bray, & Merson, 2000; Blankenship, Friedman, Dworkin, & Mantell, 2006; Frieden, 2010; Gupta, Parkhurst, Ogden, Aggleon, & Mahal, 2008; Sumartojo, 2000). Partnership strategies, such as coalitions are a commonplace approach to the pursuit of health-related structural change (Mizrahi & Rosenthal, 2000; Roussos & Fawcett, 2001; Watson-Thompson, Fawcett, & Schultz, 2008). Coalitions are routinely employed across the United States as a method of mobilizing local communities to improve local conditions that impact on citizens’ well-being. A growing literature documents coalitions’ efforts to create structural change (Cheadle, Hsu, Schwartz, Pearson, Greenwald, Beery, et al. 2008; Emshoff, Darnell, Darnell, Erikson, Schneider, & Hudgins, 2007; Javdani & Allen, 2011; Nowell, 2009; Roussos & Fawcett, 2000; Zakocs & Edwards, 2006).

Success in achieving community change objectives may not translate into improved population outcomes quickly, if ever (Berkowitz, 2001). Considerable effort by communities to plan for and pursue change objectives, without evidence of logical and appropriate intermediate markers of implementation success could lead to wasted effort or demoralization. According to Kreger and colleagues (2007), assessing progress toward success in achieving structural change is of cardinal importance, as ultimate objectives for community change typically require accomplishing multiple steps and activities along the way. Yet, the current literature provides little guidance on how coalitions might select intermediate indicators of implementation success to judge the progress and the utility of their effort. Indeed, much of the literature centers on measuring the healthy functioning of coalitions rather than on the indicators that link the implementation goals they attain to the health outcomes they intend to impact.

Because the promise of coalitions for creating structural change lies in their ability to identify common interests around which to create alliances across diverse parties and because evaluating structural change endeavors is especially difficult, most of the empirical evidence on coalitions’ ability to function over time and accomplish objectives has focused on their internal functioning, development, and composition (see, for example, Cheadle, Senter, Solomon, Beery, & Schwartz, 2005; Collie-Akers et al., 2007; Crowley, Yu & Kaftarian, 2000; Florin, Mitchell, Stevenson, & Klein, 2000). The tendency toward measuring coalition characteristics and processes is particularly prominent among coalitions pursuing structural change (Clark et al. 2010), which may reflect the complexity of the task and the challenges inherent in evaluating and linking implementation to long-term impacts at multiple levels of analysis (Francisco et al. 1993; Balcazar et al. 1994).

Implementation theories of coalitions also place primary emphasis on coalition characteristics and processes. For instance, Butterfoss & Kegler (2009) synthesized the practice and research literatures on coalitions into a single conceptual framework, Community Coalition Action Theory (CCAT), which depicts how coalitions are theorized to achieve community impact. The CCAT framework identifies a set of factors associated with the successful establishment of a coalition that impact on its ability to engage its members and pool resources. Successful engagement and pooling of resources are theorized in turn to lead to better community assessment and planning, which leads to implementing strategies that produce policy and program changes which will ultimately have positive health impact. The framework suggests that engaging diverse sectors in the coalition leads to better planning and community assessment. In CCAT, diversity allows for the collaborative synergies that a coalition must harness to create community change. Such synergy enables comprehensive, sound action plans to emerge.

The bulk of the research reviewed to inform the model closely attends to indicators that are associated with the coalition’s formation (e.g., structure, membership, size, breadth, cohesion, staffing), maintenance (e.g., member engagement, resources, member satisfaction), and effects on community capacity (e.g., skills, leadership opportunities, partnerships). By contrast, the indicators of the quality of community assessments that coalitions conduct to establish goals, the quality of the plans they develop for achieving change, and the quality and outcomes of implementing planned actions are less fully elaborated, reflecting the state of the literature. Thus, although the model provides a detailed overview of the elements one might measure to indicate the strength of a coalitions’ composition and organization, the model provides less guidance on how coalitions might craft specific indicators of planning and implementation.

Granner and Sharpe (2004) reviewed the literature to identify the measures that were available to assess all aspects of coalition functioning and to assess the outcomes of coalition activities. Although the number and quality of the identified measures varied, they found that implementation and intermediate success indicators had the fewest measures. These authors located six attempts to measure the quality of coalitions’ plans, but only two aimed at measuring intermediate success in implementing those plans. One of these efforts operationalized implementation by counting coalition outputs (Gottlieb et al, 1993) and the other focused on the degree to which plans were executed, resources were generated, and the number of activities completed (Kegler, 1998). Among studies of coalitions, counting completed actions and objectives is a fairly common approach to assessing intermediate implementation success (e.g., Fawcett et al., 2000; Clark et al., 2010).

One recent study of community collaborative groups illustrates the importance of measuring the intermediate implementation outcomes that lead to systems changes in novel ways. Javdani and Allen (2011) examined how the processes engaged by 21 local coordinating councils led to distal changes in interpersonal violence outcomes. Respondents’ perceptions of distal achievements were closely associated with the extent to which the councils were perceived to have enhanced stakeholder knowledge, promoted relationships among stakeholders, and enacted institutionalized change. Though Javdani and Allen’s study was not of coalitions and actual changes in distal outcomes were not examined, the study demonstrates an important advance in attempts to measure the kinds of critical intermediary processes by which partnership and collaborative bodies may produce community-level change.

Following the novelty of Javdani’s and Allen’s work, the current paper explores the utility of various indicators of intermediate implementation success for coalitions seeking to create structural community change and the interrelationships among these indicators. We obtained and examined the records from a multi-year study of 13 coalitions that are part of the Connect to Protect (C2P) initiative of the Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN). These 13 coalitions are charged with creating structural changes to reduce adolescent exposure to HIV infection. Using the records produced by the coalitions over the initial 2.5 years of their mobilization efforts, we investigated the degree to which a range of intermediate markers of implementation success provided similar or discrepant pictures of their performance and examined the value of using diverse indicators and descriptive statistics to determine how well each of the coalitions was performing. We also examined the relationship between each indicator and conventional measures of objective accomplishment. Finally, using CCAT as our guide, we examined the relationship between indicators of planning quality and indicators of intermediate implementation success.

Methods

The study is an exploratory analysis of data collected to monitor coalition activities and accomplishments. This research was approved the by Social Science and Education Institutional Review Board at Michigan State University.

Connect to Protect

C2P was initiated by the ATN to develop community-based partnerships and create an adolescent-focused HIV prevention infrastructure in cities across the United States (see Ziff et al., 2006 for detailed information about C2P mobilization). Through coalition mobilization, C2P’s goal is to stimulate structural change to prevent HIV infections in youth. C2P is therefore grounded in the theoretical principles of community mobilization, community capacity building, and structural change.

To meet C2P’s goal, coalitions develop and carry out locally relevant structural change objectives, which are new or modified laws, policies, settings, and practices meant “to create opportunities or remove barriers to promote HIV prevention” and that could be sustained [in the absence of the coalition and its members] (Chutuape et al., 2010, p. 2). Structural change objectives were developed based on either root cause analysis, a group critical analysis process aimed at arriving at the underlying structural determinants of risk (Willard, Chutuape, Stines, & Ellen, in press) or the “VMOSA” (vision, mission, objectives, strategies, action plan) planning process (Fawcett et al., 2000). Examples of the objectives coalitions established include: “By August 2007, [venue name] will distribute free condoms at album release parties in [neighborhood name]” and “By the year 2008, Law 81 will be amended allowing health professionals or representatives to offer the services of HIV preventive counseling or to perform HIV/STD testing in the clinic and community to youth under 21 years of age without parental consent.” Coalitions are routinely monitored by a National Coordinating Committee that serves the twofold purpose of providing technical assistance to the coalitions’ leadership and ensuring that coalitions are operating consistent with national protocol guidelines. Following the work of Fawcett and colleagues (2000), the national protocol operationalizes success as 1) the number of activities completed; 2) the number of community actors engaged, and 3) the number of structural change objectives accomplished. In 2008, 2.5 years after the C2P implementation began, the C2P study team observed there was wide variation among the 13 participating coalitions in the number of objectives that were proposed and accomplished in each city, as well as in performance on these other indicators.

As part of C2P, coalitions provide monthly quantitative and qualitative data to a central coordinating body. These data include community action logs documenting monthly activities and events going on in the local community that impact C2P’s work, meeting minutes, planning documents, and monitoring reports, each of which were used in the current analyses. In examining the structural change objectives, we identified unique quantitative and qualitative ways to classify the coalitions as more or less successful in accomplishing objectives. Indicators, which are described in detail in the section to follow, focus on planning quality and on implementation success.

Measures

Planning Quality

Consistent with the work of Taylor-Powell and colleagues (Taylor-Powell, Rossing, & Geran, 1998), we coded whether objectives met SMART criteria (e.g., specific, measurable, achievable, realistic, time-bound objectives) (Drucker, 1954). Each objective received a “yes” or “no” rating. Each coalition received a score representing the percentage of SMART objectives established out of the total number of objectives established. We also counted the number of times each objective was revised and created a score representing the percentage of objectives modified out of the total number of objectives. A third quantitative indicator focused on abandoned objectives, specifically the percentage of objectives that were abandoned prior to their completion. Though coalitions may have abandoned an objective for a variety of reasons, such as finding it unfeasible to complete or that particular objectives failed to meet the expectations of the study’s oversight body so had to be dropped, high numbers of abandoned objectives may provide an intermediate indication that planning for success has not proceeded adequately (Butterfoss & Kegler, 2002; Reed, Miller, Francisco, 2012).

Measures of Intermediate Success

Two simple quantitative indicators of intermediate success were developed. Each of these indicators reflect the degree to which the coalitions’ strategies were being implemented successfully and would result from coalition activity outputs and, by necessity, precede observed changes in health outcomes. The first indicator provides a simple count of the number of objectives accomplished from the coalition’s inception through to the end of the study period. As previously noted, this indicator is used by the C2P coalitions and those who oversee the coalitions to gauge their productivity and progress in creating local structural changes, so was of particular interest to evaluate in relationship to other indicators. This indicator is also commonly employed in evaluations of coalitions as predictive of health outcomes. Because this indicator fails to account for the number of objectives each coalition sought to accomplish, limiting comparison among coalitions, and because there was wide variation in the number of objectives each coalition created, we also calculated the percentage of objectives accomplished. Thus, each coalition had two scores reflecting the amount they had achieved during the study period.

In addition to these conventional measures, several indicators were developed from qualitative codes that we applied to the objectives. (For a detailed description of the qualitative coding processes, see Miller, Reed, Francisco & Ellen, in press.) As we previously noted, diversity in coalition membership is presumed to lead to better planning and community assessment (Butterfoss & Kegler, 2009). Thus, by extension, we reasoned that diversity should be reflected in the action plans that successful coalitions developed. Our first two qualitatively derived indicators focus on the degree to which coalition accomplishments were diverse. The social and structural determinants of HIV are multiple and include complex factors such as discrimination and economic disadvantage (Holtgrave, McGuire, & Milan, 2007; Marmot, 2005). Given these complex determinants of HIV risk, seeking to move on a diversity of fronts to accomplish a diverse array of changes would be necessary to create a community in which the probability of exposure to HIV was low (Valdiserri, 2011; White House Office of National AIDS Policy, 2010). Engaging the community broadly, across multiple sectors, and employing diverse strategies reflects the reality of what is required to make HIV infection rare among youth. Our diversity indicators focus on two diversity aspects of pursuing structural change: the breadth of community sectors targeted (Hays et al, 2000; Kegler et al, 1998) and the breadth of change strategies (Florin et al., 2000) that were employed by the coalitions across their accomplishments. Regarding sectors, we counted the number of distinct community sectors (e.g., businesses, faith-based organizations, health care organizations, community organizations, citizen groups, government) that were targeted in objectives that were accomplished. For strategies, we modified the community change strategies codebook developed by Fawcett and colleagues (Fawcett, Francisco, Paine-Andrews, Lewis, Richter, and Harris et al., 1995) to fit the context of the HIV focus of C2P. We then coded and counted the number of distinct approaches to community change that had been employed by each coalition. Codes included forming new alliances, changing physical environments, enhancing skills, leveraging new resources, and fostering new or modified policies. Thus, each coalition had two scores, one reflecting the diversity of sectors impacted by achievements and one reflecting use of diverse change strategies.

The remaining indicators we created reflect distinct ways of operationalizing the structural quality of the objectives that coalitions accomplished and provide measures of the implementation of structural change. Three indicators of quality were created.

Our first measure used Gordon’s (1983) prevention framework in which he characterized preventive interventions as universal, indicated, and selected. By definition, universal objectives would impact on a community at large, whereas selected and indicated interventions would specifically target people who have above average risk. Universal objectives are most consistent with the concept of structural change, whereas selected objectives would typically, though not always, be programmatic, and indicated objectives would typically be clinical in foci. We coded each objective as universal, indicated, or selected and then calculated the percentage of universal objectives that were achieved among all objectives accomplished. We also coded objectives for whether or not they used a change strategy that was structural in nature, such as altering laws or other aspects of the risk environment, or used a change strategy that was not structural in nature, such as developing skills among a group of service providers. We then calculated the percentage of objectives using structural strategies that were achieved among all objectives achieved. Finally, we coded objectives for whether or not they were targeted at individual-level change directly, such as improving some particular groups’ knowledge, or at a community-level condition directly, such as lack of interaction among key local government agencies.

We then calculated the percentage of community-risk condition objectives achieved among all objectives achieved. In considering these codes, it is important to note that it is not necessarily the case that they will overlap in precisely the same manner for all objectives. An objective that is coded as using a selected prevention approach is not also coded as using a non-structural change strategy by default. For example, a selected prevention objective might target transgender youth – a subgroup of all youth at above average risk of exposure to HIV, and use a strategy such as creating safe zones in the local community that would be coded as structural. In summary, each coalition had three quality scores (percentage of universal prevention objectives achieved, percentage of structural strategy objectives achieved, and percentage of community-risk objectives achieved) that provide an indication of the degree to which the quality of their work was consistent with a structural change perspective.

We utilized QSR International’s (2008) NVivo 8 to code the structural change objectives. For each success indicator, two researchers, working independently, coded each objective to assess inter-rater reliability (using NVivo’s code comparison query). After ensuring that inter-rater agreement rates were acceptable (Kappa ≤ .87 for all codes), we compared coalitions on each indicator by quantifying the qualitative codes for each indicator.

Results

From the time of coalition mobilization through to the end of 2008, the 13 coalitions proposed 304 objectives. Each coalition proposed between 14 and 44 objectives. Of all 304 objectives, 58% met our SMART criteria. Exactly half of all objectives required at least one modification. Objectives were modified for many reasons, including imprecision in the originally stated objective, improved understanding of community circumstances, or changes in beliefs and opinions among coalition members about the coalition’s action plans. One hundred thirty nine objectives were completed during the study period. On average, coalitions completed 48% (range = 30% to 65%) of the objectives they proposed. Across all coalitions, 32% of objectives were abandoned during the period we investigated. Of all the achieved objectives, 21% were universal, 62% were selected, 16% were indicated, and 1% could not be classified. Twenty-three percent were coded as using structural change strategies and 40% were coded as addressing a community-level condition.

Figure 1 depicts the proportion of objectives completed over the study period plotted at successive 3-month time intervals using box and whisper plots. Data in the plots are the percentage of objectives accomplished by each coalition at the end of the relevant 3-month period. As would be expected, the proportion of objectives completed increased over time. The figure also shows the wide variation in the overall rate of accomplishment across the coalitions within each time interval.

Figure 1.

Figure 1

Open in a new tab

Completion rates of C2P coalitions across time (N=13 coalitions).

Table 1 displays each of the intermediate success indicators we developed, as well as provides descriptive data on each indicator for this sample of coalitions. In the table, the coalitions are rank ordered by the number of objectives they have accomplished, so that by conventional counts, the most successful coalition is listed first and the least successful coalition is last. Below average scores for each indicator are underlined. Each of the indicators captured variation in progress toward accomplishing the task of creating community change among the coalitions. Among the measures, there is least variation in breadth of sectors and strategies engaged in accomplishments.

Table 1.

C2P Coalitions’ Indicators of Success in Accomplishing Structural Change(N=13 coalitions)

Coalition Number of objectives completed Percentage of objectives completed Number of sectors Number of strategies Percentage of universal objectives completed Percentage of community risk objectives completed Percentage of structural objectives completed
M 17.00 0.65 7.00 5.00 0.47 0.59 0.29
L 14.00 0.61 6.00 4.00 0.07 0.36 0.14
B 14.00 0.32 6.00 5.00 0.21 0.36 0.21
H 13.00 0.54 8.00 6.00 0.23 0.15 0.31
E 13.00 0.42 5.00 4.00 0.00 0.23 0.46
I 10.00 0.56 5.00 3.00 0.10 0.40 0.10
J 9.00 0.56 6.00 3.00 0.00 0.11 0.11
C 9.00 0.32 4.00 5.00 0.56 0.22 0.44
G 9.00 0.53 6.00 7.00 0.00 0.56 0.33
D 9.00 0.38 8.00 2.00 0.11 0.56 0.00
K 8.00 0.57 6.00 4.00 0.38 0.33 0.13
F 7.00 0.44 4.00 4.00 0.29 0.14 0.29
A 7.00 0.30 3.00 3.00 0.00 0.57 0.43
Mean (SD) 10.7 (3.1) 47.7 (11.9) 5.7 (1.5) 4.2 (1.4) 18.6 (18.9) 35.2 (17.4) 24.9 (14.7)
Open in a new tab

Note: Below average scores on each performance indicator are underlined.

The data demonstrate that all coalitions were above average performers on more than one of these intermediate indicators of success, scoring above the mean on at least two indicators. However, only one coalition scored above the mean on all indicators. Importantly, the table shows that coalitions that may be considered high performing by a simple count of their accomplishments may be low performers when the structural quality of their accomplishments is considered. Similarly, coalitions that are below average in the absolute number of their accomplishments may be above average in their ability to accomplish objectives consistent with the concept of structural change. For example, coalition L has an above average number of accomplished objectives, but a below average rate of completing universal objectives and objectives that use structural change strategies. Coalition A, with the lowest performance by a simple count of objectives completed and by rate of achievement, is among the best performing coalitions on achieving objectives that use structural strategies and target community-level risks. Thus, looking at structural change efforts only in terms of the simple count of completed objectives obscured variability across coalitions in the quality of their efforts and the complexity and reach of the structural changes that the coalitions were able to achieve.

To further illustrate this variation, Figure 2 displays the number of criteria on which each coalition was above the mean in its performance. Of particular note is that coalitions M, L, B, H, and E all score above the mean on number of objectives accomplished. Yet, only coalition M was above the mean on all indicators. Coalitions B and H are above the mean on five or more criteria whereas coalitions E, J, D, F and A are below the mean on five or more criteria. The indicators on which each of the low performing coalitions showed below average performance are not of equal value when the broader mission of the C2P initiative is taken into account. Thus, for the majority of coalitions in the sample, being successful or above the average in performance on a single criterion fails to provide a good indication of how well that coalition may do when its success is measured on other indicators or when compared to other coalitions.

Figure 2.

Figure 2

Open in a new tab

Number of criteria on which coalitions have above average performance (N=13 coalitions)

Table 2 shows the correlations among the indicators. Strikingly, an overall count of the number of objectives attained is not associated with indicators of planning quality or with successful structural change qualities. For instance, there is a near zero correlation between the overall rate of objective completion and the percentage of completed objectives that employed structural strategies. There are also no significant correlations among indicators of the quality of the objectives that have been completed and the rate at which a coalition elected to abandon objectives.

Table 2.

Correlations Among Indicators (N=13 Coalitions)

Planning Quality Measures of Intermediate Success
Percent of SMART Objectives Percent of Objectives Modified Percent of Objectives Abandoned Number of Objectives Completed Percent of Objectives Completed Number of Sectors Impacted Number of Change Strategies Employed Percent of Universal Objectives Completed Percent of Community-level Risk Objectives Completed
Percent of Objectives Modified −.426* (−.765–.014)
Percent of Objectives Abandoned −.219 (−.705–.352) −.154 (−.529–.250)
Number of Objectives Completed .192 (−.281–.579) −.150 (−.613–.284) .123 (−.487–.613)
Percent of Objectives Completed .194 (−.200–.577) .026 (−.437–.449) −.256 (−.784–.222) .259 (−.313–.753)
Number of Sectors Impacted .141 (−.445–.667) .070 (−.371–.475) .239 (−.351–.725) .418* (.000–.779) .379* (−.093–.813)
Number of Change Strategies Employed .042 (−.445–.667) −.139 (−.591–.317) .139 (−.400–.575) .282 (−.051–.603) .084 (−.350–.519) .183 (−.427–.685)
Percent of Universal Objectives Completed −.255 (−.725–.190) .107 (−.266–.535) .160 (−.343–.652) .085 (−.387–.576) .107 (−.444–.619) .117 (−.374–.582) .247 (−.195–.654)
Percent of Community-level Risk Objectives Completed .170 (−.294–.590) .130 (−.269–.497) .026 (−.422–.543) .124 (−.429–.629) −.052 (−.548–.420) .142 (−.414–.706) −.056 (−.601–.488) −.108 (−.638–.430)
Percent of Structural Objectives Completed .090 (−.441–.589) −.256 (−.753–.250) −.179 (−.666–.380) −.014 (−.363–.341) −.205 (−.576–.235) −.267 (−.653–.316) .502* (0.59–.917) −.013 (−.504–.524) . −.013 (−.471–.470)
Open in a new tab
*

p<.05 Kendall’s tau-b; 95% confidence interval reported in parentheses.

We examined relationships among quality planning indicators and the indicators of the quality of achievements. Table 3 displays the results for the SMART objectives. SMART objectives were more often achieved (68.3%) or still active (67.7%) than were abandoned objectives (36.9%) [χ2 (2, N = 304) = 27.26, p < .001]. Crafting SMART objectives was also associated with completing universal objectives [χ2 (1, N = 139) = 4.39, p < .05] and completing objectives that targeted community-level risks [χ2 (1, N = 139) = 7.90, p <.01]. About 85% of the completed universal objectives were SMART compared with 64% of the completed objectives that were not universal. Similarly, about 83% of the community-level change objectives that were completed were SMART, compared with about 60% of the individual-level change objectives that were completed. There were no differences in how long it took coalitions to complete objectives that were universal, targeted community-level conditions, or used structural strategies than any other type of objective. Half of all objectives formed by the coalitions required modification (SD = 22.1%) and 32% (SD = 15.7%) were ultimately abandoned. There were no differences in modification or abandonment rates by type of indicator with one exception. Unexpectedly, we found that completed objectives that did not use a structural change strategy had to be modified more often than did the structural strategies [χ2 (1, N = 139) = 5.28, p < .05]. These objectives were often developed early in the coalitions’ planning processes as they were still learning to craft feasible structural change objectives. These objectives often lacked SMART qualities and were not achieved on time, so required reformulation (Reed, Miller, & Francisco, 2012). Overall, these findings are largely consistent with the CCAT framework in which indicators of adequate planning should be associated with indicators of a coalition’s ability to accomplish its mission. Taken together, these findings suggest that quality planning is particularly relevant to accomplishing objectives advancing structural changes.

Table 3.

Associations among Planning and Quality of Achievement Indicators (N = 304 Objectives)

SMART Objectives Objectives Abandoned Objectives Modified
Objective Completion Status
 Achieved 68.3** -- 51.1*
 Still Active 67.7 -- 62.9
 Abandoned 36.9 -- 40.8
Prevention Type Achieved
 Universal 85.2* 41.5** 55.6
 Indicated 76.0 12.5 40.4
 Selected 60.9 35.3 52.9
Risk-level Achieved
 Community-risk 82.7** 28.7* 48.1
 Individual-risk 59.8 37.4 52.9
Strategy Achieved
 Structural 71.4 26.1 34.3*
 Non-structural 67.3 36.2 56.7
Open in a new tab

Note

*

p < .05;

**

p < .01

Discussion

Through these analyses we sought to examine which among a variety of indicators provided useful information on the performance of coalitions seeking to create structural change and to explore how well various indicators related to one another. Consistent with the CCAT framework (Butterfoss & Kegler, 2002), we observed that indicators of quality planning, in particular whether objectives were well crafted, were associated with accomplishing objectives that were consistent with a structural change orientation.

CCAT suggests that diversity may facilitate success. In these data we observed that using a greater number of unique change strategies was positively associated with completing objectives that were structural. This occurred in part because when coalitions pursued non-structural objectives, they often made repetitive use of a limited number of strategies. As coalitions increased the number of structural objectives they pursued, they increased the diversity of the strategies they employed. In these data we also observed that the absolute number of objectives completed was not strongly correlated with the number of diverse change strategies that were employed, though it was correlated with the number of sectors targeted for change. Among these coalitions, seeking to create change in diverse sectors did not equate to using diverse methods of creating change across those sectors, as CCAT would predict. This finding is particularly important in light of the emphasis placed in the literature on engaging diverse sectors as a measure of coalition performance rather than engaging diverse change strategies. Consistent with Trent and Chavis (2009), these findings may also challenge the assumption that engaging diverse people and community sectors will necessarily lead to the creation of a multi-pronged structurally-focused community change plan.

Although these data reflect the early stage of C2P operations, the data highlight discordance between the quality and quantity of performance among these coalitions and the potential limitations of an emphasis on simple counts of accomplishments and activities. However, this discord makes intuitive sense. Structural change is a difficult concept and despite training on structural change and root cause analysis (Willard et al., in press), coalition members may need considerable time to construct high quality objectives. Further, because the protocol monitors progress using simple counts of objectives completed, key sectors engaged, and activities completed, quality of the objectives may at times be sacrificed out of necessity. Ostensibly, the coalitions are trying to impact community change while maintaining a coalition and meeting the demands of a multisite research protocol; to do so effectively may at times push coalitions toward accomplishing conventional objectives and quickly completed activities. Conventional objectives may help build member or programmatic capacity while the coalitions are working toward accomplishing objectives that have structural change qualities (Foster-Fishman, Berkowitz, Lounsbury, Jacobson, & Allen, 2001). Pursuit of these conventional objectives may not facilitate the longer term success of these coalitions if they are rewarded for achieving them without taking into account how well these objectives reflect the larger aim of the C2P initiative. Objectives that had structural properties took no longer to complete than conventional objectives, underscoring the importance of taking quality into account in evaluating coalition performance.

Limitations

These data are from a single initiative, which, while it facilitates comparison, may limit the generalizability of the findings. In particular, these coalitions may favor selected interventions, rather than universal interventions, as a means of advancing structural changes that specifically promote the health of their target populations. Similar work with a larger and more diverse sample of coalitions is needed to understand the value of conceptualizing intermediate indicators of coalition success that reflect the quality rather than quantity of achievements. Second, we were limited in what indicators we could develop by the nature of the data that are routinely collected. We lacked data from which we could create other indicators that might be derived from conceptual models such as CCAT, for instance, indicators of community capacity and development of social capital. Finally, we cannot yet link whether these intermediate success indicators are associated with longer-term health outcomes. Future studies should examine how well these intermediate indicators predict improved health among the targeted communities.

These limitations notwithstanding, this study suggests that coalitions and those who provide them with technical assistance would benefit from developing approaches to monitoring progress that are diverse. Simple counts of the number of items achieved on an action plan may not provide coalitions with a sound indication that they are on their desired trajectory. Moreover, if a simple count is the only indicator used to gauge progress, coalitions may then focus on this indicator to the detriment of other aspects of import to the success of the coalitions. Coalitions and their stakeholders may benefit from developing simple qualitative taxonomies that reflect their structural change goals and classifying their objectives and actions accordingly. Calculating rates of progress in relevant quality domains could provide coalitions with an informative complementary measure of progress alongside conventional indicators. For example, Frieden (2010) developed an impact pyramid which could be used to classify each of a coalition’s objectives. The pyramid’s base reflects interventions with high population impact and structural change while its peak reflects interventions with high individual impact and no structural change. Applying his classifications to a set of coalition objectives would allow coalitions to determine the extent to which the nature of the objectives they were accomplishing was consistent with their intentions and to consider how well the mixture of objectives they had set cover the pyramid’s five tiers. This and other existing frameworks on prevention and structural change provide guidance on ways to assess the quality, scope, and diversity of intermediate outcomes.

Fawcett and colleagues (2000) noted that success in moving the behavior of target audiences and related health outcomes is the result of the overwhelming number of structural changes achieved (Collie-Akers et al., 2007), or some distribution of the changes across sectors, behavior change strategies, levels of change, and durations. We observed that merely counting how many objectives a coalition accomplished did not provide an adequate indication of the quality of what they had accomplished. Our analyses suggest that examination of any one indicator of success in isolation may not provide coalitions with adequate information on their progress toward creating desired change. Additionally, our findings underscore the importance of pairing indicators that convey how much activity has occurred with indicators that reflect the quality of that activity. Simultaneous consideration of multiple indicators could help coalition leaders become more strategic and efficient, leading to increasing success in facilitating community improvement and achieving structural change.

Acknowledgments

The Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN) is funded by Grant No. 2 U01 HD040533 from the National Institutes of Health through the National Institute of Child Health and Human Development (B. Kapogiannis, MD), with supplemental funding from the National Institute on Drug Abuse (N. Borek, PhD), National Institute on Mental Health (P. Brouwers, PhD), and National Institute on Alcohol Abuse and Alcoholism (K Bryant, PhD). The study was scientifically reviewed by the ATN’s Community and Prevention Leadership Group. Network scientific and logistical support was provide by the ATN Coordinating Center (C. Wilson, C. Partlow), at the University of Alabama at Birmingham. The ATN 079 Protocol Team members are Vincent Francisco (University of North Carolina-Greensboro), Robin Lin Miller (Michigan State University), Jonathan Ellen (John Hopkins University), Peter Freeman (Children’s Memorial Hospital), Lawrence B. Friedman (University of Miami School of Medicine), Grisel-Robles Schrader (University of California-San Francisco), Jessica Roy (Children’s Diagnostic and Treatment Center), Nancy Willard (Johns Hopkins University), and Jennifer Huang (Westat, Inc.). Research assistantship was provided by Sarah Reed (Michigan State University), Ella Dolan (Michigan State University), and Greer Cook (University of North Carolina-Greensboro).

References

  1. Balcazar FB, Mathews RM, Francisco VT, Fawcett SB, Seekins TB. Assessing empowerment actions and outcomes with consumer advocacy organizations for people with physical disabilities: Four systematic replications. Rehabilitation Psychology. 1994;39:189–203. [Google Scholar]
  2. Berkowitz B. Studying the outcomes of community-based coalitions. American Journal of Community Psychology. 2001;29:213–227. doi: 10.1023/A:1010374512674. [DOI] [PubMed] [Google Scholar]
  3. Blankenship KS, Bray SJ, Merson MH. Structural interventions in public health. AIDS. 2000;14:S11–S21. doi: 10.1097/00002030-200006001-00003. [DOI] [PubMed] [Google Scholar]
  4. Blankenship KS, Friedman SR, Dworkin S, Mantell JE. Structural interventions: Concepts, challenges and opportunities for research. Journal of Urban Health. 2006;83:59–72. doi: 10.1007/s11524-005-9007-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Butterfoss F, Kegler M. Toward a comprehensive understanding of community coalitions: Moving from practice to theory. In: DiClemente RJ, Crosby RA, Kegler MC, editors. Emerging theories in health promotion practice and research: Strategies for improving public health. San Francisco, CA: Jossey-Bass; 2002. pp. 194–227. [Google Scholar]
  6. Butterfoss F, Kegler M. The community coalition action theory. In: DiClemente RJ, Crosby RA, Kegler MC, editors. Emerging theories in health promotion practice and research: Strategies for improving public health. 2. San Francisco, CA: Jossey-Bass; 2009. pp. 157–193. [Google Scholar]
  7. Cheadle A, Hsu C, Schwartz PM, Pearson D, Greenwald HP, Beery WL, Flores G, Casey MC. Involving local health departments in community health partnerships: Evaluation results for the Partnership for the Public’s Health Initiative. Journal of Urban Health. 2008;85:162–177. doi: 10.1007/s11524-008-9260-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Cheadle A, Senter S, Solomon L, Beery WL, Schwartz PM. A qualitative exploration of alternative strategies for building community-health partnerships: Collaboration- versus issue-oriented approaches. Journal of Urban Health. 2005;82:638–652. doi: 10.1093/jurban/jti120. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Chutuape KS, Willard N, Sanchez K, Straub DM, Ochoa TN, Howell K, et al. Mobilizing communities around HIV prevention for youth: How three coalitions applied key strategies to bring about structural changes. AIDS Education and Prevention. 2010;22:15–27. doi: 10.1521/aeap.2010.22.1.15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Clark NM, Lachance L, Doctor LJ, Gilmore L, Kelly C, Krieger J, Wilkin M. Policy and system change and community coalitions: Outcomes from allies against asthma. American Journal of Public Health. 2010;100:904–912. doi: 10.2105/AJPH.2009.180869. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Collie-Akers V, Fawcett SB, Schultz JA, Carson VL, Cyprus J, Pierle JE. Analyzing a community-based coalition’s efforts to reduce health disparities and the risk for chronic disease in Kansas City MO. Preventing Chronic Disease. 2007;4(3):1–9. [PMC free article] [PubMed] [Google Scholar]
  12. Crowley KM, Yu P, Kaftarian SJ. Prevention actions and activities make a difference: a structural equation model of coalition building. Evaluation and Program Planning. 2000;23:381–388. [Google Scholar]
  13. Drucker PF. The practice of management. New York: Harper & Row Publishers; 1954. [Google Scholar]
  14. Emshoff JG, Darnell AJ, Darnell DA, Erickson SW, Schneider S, Hudgins R. Systems change as an outcome and process in the work of community collaboratives for health. American Journal of Community Psychology. 2007;39:2555–267. doi: 10.1007/s10464-007-9110-7. [DOI] [PubMed] [Google Scholar]
  15. Fawcett SB, Francisco VF, Hyra D, Paine-Andrews A, Schultz JA, Roussos S, et al. Building healthy communities. In: Tarlov A, editor. Society and population health reader: State and community applications. New York: The New Press; 2000. [Google Scholar]
  16. Fawcett SB, Francisco VT, Paine-Andrews A, Lewis RK, Richter KP, Harris KJ, Williams E, Berkley J, Schultz JA, Fisher JL, Lopez CM. Work Group evaluation handbook: Evaluating and supporting community initiatives for health and development. Lawrence, KS: Work Group on Health Promotion and Community Development, University of Kansas; 1995. [Google Scholar]
  17. Florin P, Mitchell R, Stevenson J, Klein I. Predicting intermediate outcomes for prevention coalitions: A developmental perspective. Evaluation and Program Planning. 2000;23:341–346. [Google Scholar]
  18. Foster-Fishman PG, Berkowitz SL, Lounsbury DW, Jacobson S, Allen NA. Building collaborative capacity in community coalitions: A review and integrative framework. American Journal of Community Psychology. 2001;29:241–261. doi: 10.1023/A:1010378613583. [DOI] [PubMed] [Google Scholar]
  19. Francisco VT, Paine AL, Fawcett SB. A methodology for monitoring health action coalitions. Journal of Health Education Research: Theory and Practice. 1993;8:403–416. doi: 10.1093/her/8.3.403. [DOI] [PubMed] [Google Scholar]
  20. Frieden TR. A framework for public health action: The health impact pyramid. American Journal of Public Health. 2010;100:590–595. doi: 10.2105/AJPH.2009.185652. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Gordon RS. An operational classification of disease prevention. Public Health Reports. 1983;98:107–109. [PMC free article] [PubMed] [Google Scholar]
  22. Gottlieb NH, Brink SG, Levenson Gingiss P. Correlates of coalition effectiveness: the Smoke Free Class of 2000 Program. Health Education Research. 1993;8:375–384. doi: 10.1093/her/8.3.375. [DOI] [PubMed] [Google Scholar]
  23. Granner M, Sharpe P. Evaluating community coalition characteristics and functioning: A summary of measurement tools. Health Education & Behavior. 2004;19:514–532. doi: 10.1093/her/cyg056. [DOI] [PubMed] [Google Scholar]
  24. Gupta GR, Parkhurst JO, Ogden JA, Aggleton P, Mahal A. Structural approaches to HIV prevention. Lancet. 2008;372:764–75. doi: 10.1016/S0140-6736(08)60887-9. [DOI] [PubMed] [Google Scholar]
  25. Hays C, Hays S, DeVille J, Mulhall P. Capacity for effectiveness: The relationship between coalition structure and community impact. Evaluation and Program Planning. 2000;23:373–379. [Google Scholar]
  26. Holtgrave DR, McGuire JF, Milan J. The magnitude of key HIV prevention challenges in the United States: Implications for a new national HIV prevention plan. American Journal of Public Health. 2007;97:1163–1167. doi: 10.2105/AJPH.2006.095182. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Javdani S, Allen NE. Proximal outcomes matter: A multilevel examination of the processes by which coordinating councils produce change. American Journal of Community Psychology. 2011;47:12–27. doi: 10.1007/s10464-010-9375-0. [DOI] [PubMed] [Google Scholar]
  28. Kegler M, Steckler A, McLeroy K, Malek S. Factors that contribute to effective community health promotion coalitions: A study of 10 Project ASSIST coalitions in North Carolina. Health Education and Behavior. 1998;25:338–353. doi: 10.1177/109019819802500308. [DOI] [PubMed] [Google Scholar]
  29. Kreger M, Brindis CD, Manual DM, Sassoubre L. Lessons learned in systems change initiatives: Benchmarks and indicators. American Journal of Community Psychology. 2007;39:301–320. doi: 10.1007/s10464-007-9108-1. [DOI] [PubMed] [Google Scholar]
  30. Marmot M. Social determinants of health inequalities. The Lancet. 2005;365:1099–104. doi: 10.1016/S0140-6736(05)71146-6. [DOI] [PubMed] [Google Scholar]
  31. Miller RL, Reed SJ, Francisco V, Ellen J. Conflict transformation, stigma, and HIV-preventive structural change. American Journal of Community Psychology. 2012 doi: 10.1007/s10464-011-9465-7. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Mizrahi T, Rosenthal BB. Complexities of coalition building: Leaders’ successes, strategies, struggles and solutions. Social Work. 2001;46:63–78. doi: 10.1093/sw/46.1.63. [DOI] [PubMed] [Google Scholar]
  33. Nowell B. Profiling capacity for coordination and systems change: The relative contribution of stakeholder relationships in interorganizational collaboratives. American Journal of Community Psychology. 2009;44:196–212. doi: 10.1007/s10464-009-9276-2. [DOI] [PubMed] [Google Scholar]
  34. QSR International. NVivo Qualitative Data Analysis Software (Version 8). [Computer software] 2008. [Google Scholar]
  35. Reed SJ, Miller RL, Francisco V. Programmatic capacity and HIV structural change interventions: Influences on coalitions’ success and efficiency in accomplishing intermediate outcomes. Journal of Prevention and Intervention in the Community. 2012 doi: 10.1080/10852352.2012.660123. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Roussos ST, Fawcett SB. A review of collaborative partnerships as a strategy for improving community health. Annual Review of Public Health. 2000;21:369–402. doi: 10.1146/annurev.publhealth.21.1.369. [DOI] [PubMed] [Google Scholar]
  37. Sumartojo E. Structural factors in HIV prevention: Concepts, examples, and implications for research. AIDS. 2000;14:S3–S10. doi: 10.1097/00002030-200006001-00002. [DOI] [PubMed] [Google Scholar]
  38. Taylor-Powell E, Rossing B, Geran J. Evaluating collaboratives: Reaching the potential. University of Wisconsin–Extension, Cooperative Extension, Program Development and Evaluation. Madison, WI: 1998. Available: http://learningstore.uwex.edu/assets/pdfs/G3658-8.PDF; retrieved: 25, July 2011. [Google Scholar]
  39. Trent TR, Chavis DM. Scope, scale, and sustainability: What it takes to create lasting community change. The Foundation Review. 2009;1:96–114. [Google Scholar]
  40. Valdiserri R0. Thirty years of AIDS in America: A story of infinite hope. AIDS Education and Prevention. 2011;23:479–494. doi: 10.1521/aeap.2011.23.6.479. [DOI] [PubMed] [Google Scholar]
  41. Watson-Thompson J, Fawcett SB, Schultz SA. Differential effects of strategic planning on community change in two urban neighborhood coalitions. American Journal of Community Psychology. 2008;42:25–38. doi: 10.1007/s10464-008-9188-6. [DOI] [PubMed] [Google Scholar]
  42. White House Office of National AIDS Policy. National HIV/AIDS strategy for the United States. Washington, DC: Author; 2010. Retrieved on August 24, 2010 from http://www.whitehouse.gov/administration/eop/onap/nhas. [Google Scholar]
  43. Willard N, Chutuape K, Stines S, Ellen J. Bridging the gap between individual level risk for HIV and structural determinants: Using root cause analysis in strategic planning. Journal of Prevention and Intervention in the Community. doi: 10.1080/10852352.2012.660122. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Zakocs RC, Edwards EM. What explains community coalition effectiveness? A review of the literature. American Journal of Preventive Medicine. 2006;30:351–361. doi: 10.1016/j.amepre.2005.12.004. [DOI] [PubMed] [Google Scholar]
  45. Ziff M, Harper G, Chutuape K, Deeds BG, Futterman D, Francisco VT, Muenz LR, Ellen J for the Adolescent Trial Network for HIV/AIDS Interventions. Laying the foundation for Connect to Protect®: A multi-site community mobilization intervention to reduce HIV/AIDS incidence and prevalence among urban youth. Journal of Urban Health. 2006;83:506–522. doi: 10.1007/s11524-006-9036-7. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES