Measuring Dimensions of Coalition Functioning for Effective and Participatory Community Practice

Abstract

Social work has a long history of community practice, but community practice models have been understudied. An important first step in conducting such studies is the establishment of psychometrically sound measures that are relevant for evaluations of community practice. In this article, data are used from a community-randomized trial of Communities That Care, a coalition-based model of community practice shown to be effective at transforming communities and changing rates of problem behavior in youths. Coalition functioning is reported by coalition members in 12 communities across intervention implementation phases. A four-dimensional model of coalition functioning (goal-directedness, efficiency, opportunities for participation, and cohesion) was confirmed using factor analysis, and the dimensions were found to be invariant across time. One test of validity of these dimensions is conducted by correlating coalition members' ratings of coalition functioning with those of external observers.

Social work has a long history of community practice (Weil & Gamble, 2005). The eight dynamic and intersecting models of contemporary community practice include neighborhood and community organizing, organizing functional communities, community social and economic development, social planning, program development and community liaison, political and social action, coalitions, and social movements (Weil & Gamble, 2002). Although community practice strategies offer great potential for effecting change, they are difficult to measure and evaluate, which limits the evidence of their effectiveness (Cheadle et al., 2010; Hawe & Potvin, 2009; Stevenson & Mitchell, 2003; Wandersman & Florin, 2003). In a review conducted on the community practice literature from 1985 to 2001, Ohmer & Korr (2006) found 269 empirical and theoretical articles on community practice, only 20 of which were intervention studies. Of these intervention studies, only nine used a comparison group or longitudinal design, and only a single study (Wagenaar et al., 1999) used an experimental design to allow an attribution of the effects to the intervention. There is a clear need for the social work profession to study its macro practice strategies to determine whether these models attain their stated objectives.

The use of coalition-based social planning models in communities is of particular interest due to the prevalence of the practice method and its popularity with foundations and federal funders (Butterfoss, 1993; Roussos & Fawcett, 2000). Social planning is considered the most technical of the various models of community practice, “focused on the use of rational planning approaches carried out by experts,” but innovative models of coalition-based social planning have broadened participation to “grassroots groups, citizens' organizations, and a broader range of community voices” (Weil & Gamble, 2005, p. 133). Coalition models of practice are

a strategy for social action that can … advocate for reform in structural arrangements for delivering and accessing healthcare, education, social welfare, and other human services. … Coalitions can influence political, social, and economic forces that affect the development of policies and services … [by] advocating for social change. (Roberts-DeGennaro & Mizrahi, 2005, p. 305)

One important challenge facing evaluators and researchers who are testing community practice strategies is the difficulty in measuring community processes. This article first describes Communities That Care (CTC), a specific model of coalition-based social planning, and then uses data from a randomized trial of CTC to explore the psychometric properties of an instrument designed to assess the functioning of social planning coalitions. The validation of such tools is necessary to facilitate the development of a deeper evidence base for community practice.

CTC

CTC (Hawkins, Catalano, & Associates, 1992) is a coalition-based social planning practice strategy intended to mobilize communities to prevent mental, emotional, and behavioral problems in youths. A rigorous randomized controlled trial of CTC was recently conducted with 24 matched communities across seven states, which were randomly assigned either to use CTC or to conduct prevention services as usual from 2004 to 2008. Results showed that after 1.5 years, key leaders in the CTC communities reported significantly greater increases in their community's use of a scientific approach to prevention and in their community's cross-sector collaboration for prevention relative to control communities (Brown, Hawkins, Arthur, Briney, & Abbott, 2007). A population panel of more than 4,400 students was surveyed annually from grade 5 through grade 8. Three years after implementation of CTC, a survey of this panel revealed that students in CTC communities, compared to those in control communities, who had not yet initiated each behavior by fifth grade were 32% to 33% less likely to have tried smokeless tobacco, cigarettes, and alcohol. They were also 23% to 48% less likely to have used alcohol, used smokeless tobacco, or engaged in binge drinking in the past month. Furthermore, they were 25% less likely to have ever engaged in criminal or deviant behavior and 31% less likely to have recently been engaged in a variety of delinquent acts (Hawkins et al., 2009).

CTC achieves these results through the work of a community coalition, bringing together a diverse and comprehensive group of community stakeholders to identify a vision for youths in their community and to work toward that vision by collecting data from community youths; surveying community resources; prioritizing risk and protective factors (predictors and moderators of undesirable outcomes) to address mental, emotional, and behavioral problems; selecting strategies tested to be effective in targeting those predictors; implementing and monitoring the quality of implementation; and tracking results. Coalition members typically include parents, youths, advocates, residents, local business owners, elected officials, religious leaders, philanthropists, media representatives, and professionals from education, public health, juvenile justice, law enforcement, child welfare, and youth recreation sectors. The randomized trial of CTC provides a unique opportunity to study the characteristics of community coalitions that lead to successful practice outcomes. The first step in this process is to refine a reliable and valid measure of coalition functioning.

Coalition Functioning

Coalition functioning is the way in which coalitions operate to marshal resources and exercise power to influence their objectives. Existing measures of coalition functioning were reviewed (Granner & Sharpe, 2004), and three tools were identified that were designed to measure general coalition functioning (National Network for Health's Coalition Self-Evaluation Instrument, the Art of Coalition Building's Coalition Checklist, and the Taylor-Powell University of Wisconsin-Extension Measure of Group Functioning [Granner & Sharpe, 2004]). None of these instruments has published reports of its psychometric properties. Another promising instrument, the Wilder Collaboration Factors Inventory (Mattessich, Murray-Close, & Monsey, 2001), has come into widespread use, but no research on the factor structure of the measurement tool has been reported to determine whether it consistently measures the domains it is intended to measure (Alfonso et al., 2008). The determination of whether process and outcome measures are psychometrically sound is a necessary step before interpretation of the data generated by such tools can inform social work practice (Coulton, 2005).

Coalition functioning has been examined in an earlier quasi-experimental study of CTC (Feinberg, Gomez, Puddy, & Greenberg, 2008). The Coalition Web-Based Self-Report Questionnaire was used to collect coalition functioning data from 867 coalition members in that study. The instrument used indicators that were measured on four-point or seven-point ordinal scales to study coalition operations (such as directedness and efficiency), social process characteristics (such as group cohesion), coalition leadership (such as an inclusive leadership style), and implementation successes and challenges (such as barriers to acquiring resources). Surveys were scored by taking the average of responses on conceptually derived scales and then averaging these subscale scores to create a global measure of functioning (Feinberg, Gomez, et al., 2008). Although the factor structure of this instrument was not analyzed, early psychometric work revealed that most subscales had satisfactory internal reliabilities (range: .54 to .92). In a pilot test–retest reliability study conducted on a small subset of the sample (n = 38), most subscale correlations across two weeks were deemed “adequate” at r > .60 (Feinberg, Gomez, et al., 2008). Subsequent use of this tool to predict outcomes revealed that coalition functioning predicted coalition sustainability, an important and meaningful indicator of coalition success, even after controlling for the level of funding that the coalition achieved (Feinberg, Bontempo, & Greenberg, 2008).

Although these results are useful, there are some limitations. First, the underlying dimensions of coalition functioning that explain the variance in sustainability are obscured by the use of a single measure of overall coalition functioning. Second, a pure measure of coalition functioning (the way in which the team operates) is conflated with measures of implementation quality, which confounds the predictor and the outcome of the analyses. Although this instrument provides the foundation for the present work, modifications were made to improve the relevance of the tool across different models of community practice and to distinguish and test theorized relationships between various dimensions of functioning and successful practice outcomes.

We theorize that different dimensions of coalition functioning could be driving results that would have vastly different implications for community practice. For example, classic transaction-cost economic theory (Coase, 1937) would suggest that coalitions are effective due to gains in work efficiencies. In contrast, theories emerging from population ecology suggest that coalition success is dependent upon focus and specialization in the context of competition (Aldrich & Ruef, 2006). From another perspective, theories of human ecology would suggest that collective action and the interdependence of members determine coalition success (Astley & Van de Ven, 1983). Network theory suggests that cohesion will facilitate coalition work (Reagans & McEvily, 2003). Based on these theories, we focused on four dimensions (efficiency, goal-directedness, opportunities for participation, and cohesion) of coalition functioning for this examination. These specific constructs are relevant for social work practitioners who often must decide between emphasizing insights and local knowledge garnered from participants and emphasizing insights garnered from technical, often external, sources of information. This tension between self-determination and science shapes much of the professional discussion around the utility of, and resistance to, evidence-based practice. It is important to examine how coalitions function in a way that balances their charge to pursue the goals of the externally developed intervention (exemplified by efficiency and goal directedness) and to build community and increase participation locally (exemplified by opportunities and cohesion). In the current study, we consider these four theoretically derived dimensions of coalition functioning to distinguish among potential explanations for intervention effectiveness.

An understanding of the structural dimensions of coalition functioning is an important prerequisite for interpreting differences in observed scores and deriving implications for social work practice (El Ansari & Weiss, 2006; Hoe & Brekke, 2008). This article provides the foundation for a larger research agenda by performing analyses that can examine (a) whether coalition functioning is a single global construct or can be measured by the related, but unique, dimensions noted previously; (b) whether coalition functioning can be measured in a reliable fashion by examining the invariance of factor loadings of each dimension over time; and (c) whether the concept of coalition functioning has construct validity, the type of validity that examines the degree to which the instrument measures the theoretical construct of interest. One way to establish construct validity is to demonstrate that scores generated by the instrument correlate positively with scores generated by similar measures of the same construct. This study compares the observations of multiple types of informants. To the extent that this measure is found to identify distinct aspects of coalition functioning that can be assessed with reliability and validity, hypotheses can be tested as to whether highly focused, efficient coalitions outperform highly participatory, cohesive coalitions; whether the reverse is true; or whether the best outcomes are achieved when both efficient and participatory processes are used. Such studies have the potential to inform community practice methods and, perhaps, reduce the tension between science and self-determination in social work practice.

Method and Procedures

The Community Youth Development Study (Hawkins et al., 2008), the randomized trial of CTC, provides a useful sample for examining coalition functioning. The Institutional Review Board at the University of Washington approved the study to survey coalition members annually while they were participating in the intervention being tested. This article examines the measurement of coalition functioning across multiple waves of data that were provided by members of the CTC coalitions from 2004 to 2008.

Participants

Survey participants were randomly selected from rosters of active coalition members that were provided by community coordinators in CTC communities. The average number of coalition members per community ranged from 24 to 80 early in the intervention (median = 40), and from 14 to 50 (median = 20) late in the intervention. During the first wave of data collection, approximately 20 coalition members per community were surveyed to achieve adequate representation. Due to high costs of data collection, limited within-community variability, and decreasing numbers of active coalition members, approximately 10 coalition members per community were randomly selected during all subsequent waves. Rates of interview completion ranged from 93.3% to 96.6%. Each of the 12 communities contributed 6.5% to 9.7% of the observations, with the interview completion rate by community above 80% across all waves of data collection. Four hundred thirty-two individuals served as respondents, with about 40% of coalition members participating in more than one annual interview because of their sustained participation in a CTC coalition. By the fifth year of data collection, coalition members reported that they had been working in their communities for a median of 11 years, had been living in their communities for an average of 16.5 years, had been involved with CTC for an average of four years, and had spent an average of 10 hours per month on CTC in the past year. Communities involved in the CTC trial were incorporated towns with population sizes that ranged from 1,578 to 40,787 people. The percentage of White residents ranged from 64% to 98.2%. Race, class, and gender information was not consistently collected from individual respondents.

Instruments

A structured interview was used to collect information from coalition members over the phone. Questions were closed ended with four response options (strongly disagree, disagree, agree, strongly agree) that were scored such that high scores indicated higher levels of the construct. Interviews took an average of 40 minutes to complete, and coalition members were asked a broad array of questions related to their role in CTC.

The coalition member interview was based on the Coalition Web-Based Self-Report Questionnaire (Feinberg, Gomez, et al., 2008) and was slightly revised and renamed the Community Board Interview. These revisions were intended to improve the face validity of the measure to focus more narrowly on how coalitions were functioning rather than on the “outputs” of functioning and to respond to the implications of Feinberg and colleagues' analyses. A list of the indicators used in this analysis appears in Table 1.

Table 1:

Item Completion Rates and Descriptive Statistics Organized by Four Dimensions of Coalition Functioning

	Wave 1 (n = 217)					Wave 3 (n = 113)					Wave 5 (n = 109)
Itemsa	Complete %	M	SD	Skew	Kurtosis	Complete %	M	SD	Skew	Kurtosis	Complete %	M	SD	Skew	Kurtosis
Goal-directedness	a1 = .69					a3 = .61					a5 = .64
The CTC leadership adheres to the decision-making procedures	95.4	3.67	0.60	−2.03	4.74	99.1	3.75	0.56	−2.49	6.52	98.2	3.65	0.58	−1.49	1.24
The CTC leadership has control of meetings	99.5	3.64	0.58	−1.53	2.10	100.0	3.78	0.46	−1.91	2.92	100.0	3.59	0.53	−0.74	−0.68
The CTC leadership has a clear vision for the coalition	100.0	3.68	0.58	−1.78	2.82	100.0	3.79	0.41	−1.43	0.03	99.1	3.54	0.68	−1.34	1.27
Team has agreed how to govern itself and make decisions	97.2	3.71	0.60	−2.37	5.94	95.6	3.81	0.50	−3.18	11.69	97.2	3.78	0.46	−1.98	3.23
The CTC board has developed clear goals and objectives	98.6	3.77	0.52	−2.68	8.36	98.2	3.79	0.43	−1.81	2.25	99.1	3.79	0.47	−2.19	4.18
Efficiency	a1 = .80					a3 = .85					a5 = .82
Board members work very hard	98.6	3.49	0.59	−0.67	−0.52	100.0	3.42	0.62	−0.61	−0.55	99.1	3.41	0.61	−0.51	−0.61
This is a highly efficient, work-oriented board	99.1	3.22	0.65	−0.46	0.27	100.0	3.18	0.62	−0.36	0.61	99.1	3.16	0.74	−0.69	0.46
There's a lot of time wasted because of inefficiencies	99.1	3.42	0.76	−1.21	0.92	97.3	3.46	0.77	−1.39	1.32	99.1	3.10	0.78	−0.42	−0.59
The CTC board is organized and efficient	99.1	3.39	0.57	−0.25	−0.78	100.0	3.32	0.57	−0.13	−0.61	100.0	3.25	0.60	−0.14	−0.47
The group is able to move from talk to action	96.3	3.37	0.60	−0.50	0.13	99.1	3.36	0.63	−0.67	0.62	99.1	3.17	0.68	−0.22	−0.80
This board rarely has anything concrete to show for its efforts	98.2	3.62	0.67	−1.89	3.42	100.0	3.68	0.57	−1.92	4.21	99.1	3.60	0.59	−1.49	2.58
Opportunities for participation		a1 = .65				a3 = .72					a5 = .64
The CTC leadership asks you to assist with specific tasks	100.0	3.55	0.64	−1.32	1.56	100.0	3.59	0.56	−0.99	−0.01	100.0	3.42	0.71	−1.14	1.10
The CTC leadership intentionally seeks out your views	100.0	3.50	0.71	−1.37	1.57	100.0	3.58	0.70	−1.87	3.48	100.0	3.52	0.69	−1.30	1.04
The CTC board uses the abilities of all, not just a few	97.7	3.46	0.64	−1.10	1.56	99.1	3.30	0.68	−0.64	0.05	99.1	3.25	0.66	−0.52	0.23
Everyone is involved in the discussion, not just a few	98.2	3.42	0.64	−0.76	0.06	99.1	3.41	0.65	−0.66	−0.57	98.2	3.36	0.72	−0.96	0.74
Cohesion	a1 = .64					a3 = .62					a5 = .66
There is not much group spirit among this board	97.7	3.54	0.68	−1.34	1.22	98.2	3.58	0.65	−1.28	0.43	100.0	3.58	0.70	−1.70	2.57
There is a sense of unity and cohesion in our board	98.2	3.35	0.69	−0.93	1.00	100.0	3.44	0.61	−0.61	−0.54	100.0	3.50	0.65	−0.92	−0.22
People involved with the CTC board come to the meetings	96.3	3.14	0.60	−0.20	0.27	100.0	3.08	0.63	−0.28	0.40	99.1	3.11	0.59	−0.02	−0.10

Note: CTC = Communities That Care.

^aScale ranges from 1 to 4, with 4 being most desirable.

Additional ratings of coalition functioning were provided by three CTC implementation specialists who provided support to communities to implement CTC during the intervention period but who had no knowledge of the content of the coalition member interviews. Each coalition was rated by a single implementation specialist, precluding interrater reliability estimates of these ratings. Items related to coalition functioning from implementation specialists were used to determine whether different types of informants (coalition members vs. outside observers) agreed about the level of coalition functioning on specific dimensions.

Analysis Plan

The analysis sought to answer four primary research questions regarding the reliability and validity of the measurement of coalition functioning. The first goal was to determine whether the four theoretically interesting and psychometrically promising factors that were retained and slightly modified from the Coalition Web-Based Self-Report Questionnaire (goal-directedness, efficiency, opportunities for participation, and cohesion) adequately fit the data observed in the CTC trial. The second goal was to determine whether coalition functioning is a single global concept or is better indicated by these four unique, but related, dimensions. The third goal was to determine whether the factor loadings of items on each scale are consistent across time. The final goal was to determine whether member-reported coalition functioning correlates with the ratings of coalition functioning of external observers, the University of Washington implementation specialists. Mplus version 6 (Muthén & Muthén, 2010) was used to analyze the factor structure of coalition functioning. A confirmatory factor analysis rather than an exploratory factor analysis was conducted, because the scale was theoretically derived to test the relationships between specific concepts. Items were treated as ordinal categorical, and models were therefore estimated using a weighted least squares means- and variance-adjusted procedure (WLSMV) (Muthén, du Toit, & Spisic, 1997) and the Delta parameterization (Muthén & Muthén, 2010). This approach is appropriate for use with Likert-type ratings where serious violations of normality occur, and it requires the collapsing of categories that are endorsed by respondents with very low frequency. It was suspected that ratings of coalition respondents would likely be skewed in these data, because all of the coalitions studied were involved in a well-resourced efficacy trial where technical assistance was provided to optimize functioning. Items were examined against thresholds for univariate skew and kurtosis (3 and 8, respectively) (Kline, 2005) to identify any serious violations to normality. As suggested by Comrey and Lee (1992), only factor loadings above .32 were interpreted, using .55 as the criterion for a “good” loading and .71 as the standard for an “excellent” loading. Internal reliability, the extent to which the items on the same scale measure the same underlying construct, was measured using Cronbach's alpha (Cronbach, 1951). Noting that Cronbach's alpha is highly influenced by the number of items on a given scale, and the need for practical scales of coalition functioning to be brief, a .6 criterion was used to indicate acceptable internal reliability, with the understanding that constructs with lower coefficients would need to be carefully evaluated for the potential of attenuated relationships with other variables (Schmitt, 1996).

To determine whether the four factors adequately fit the data observed in the CTC trial, goodness of model fit was assessed through the use of a two-index strategy (Hu & Bentler, 1999). Both of these criteria are commonly used to determine how likely it is that the specified model generated the observed data. The comparative fit index (CFI) ranges from 0 to 1, with a value greater than .90 indicating a close fit (Gerbing & Anderson, 1993) and greater than .95 indicating an excellent fit (Vandenberg & Lance, 2000). The root mean square error of approximation (RMSEA) value indicates a close fit when less than .05, a reasonable fit when less than .08, and an unacceptable fit when greater than .10 (Browne & Cudeck, 1993). This test was performed on the data derived from the midpoint of the intervention (wave 3), hypothesized to represent the most stable indicator of team functioning. The analysis was replicated in waves 1 and 5 while the factor pattern was held constant in early and late points of CTC implementation to determine whether all factor loadings remained significant at the most disparate time points. Together, these tests determined the adequacy of the factor structure at the early, mid-, and late points of the intervention.

To determine whether coalition functioning is a single global concept or is better modeled by specifying four unique parts, a four-factor model with correlated factors was compared to a single-factor model. To determine whether the measurement structure of each dimension was consistent across time and ensure that any differences in coalition functioning over time could be interpreted as real differences rather than differences in measurement properties, each factor was isolated in turn such that the model could be estimated simultaneously at five time points. A model with the factor loadings constrained to be equal across time was compared to a relevant model with parameters freely estimated across time. Evidence for a significant difference in model fit was indicated by a greater than .01 change in the CFI value (Cheung & Rensvold, 2002) or a significant (p < .05) robust chi-square difference test (Asparouhov & Muthén, 2006). The DIFFTEST procedure, appropriate for use with the WLSMV estimator, was used to conduct the robust chi-square difference test.

Results

The items used to assess coalition functioning, their scale assignment, and the item-level means, standard deviations, skew, kurtosis, and percentage of missing data at the beginning, midpoint, and end of the intervention as generated through analysis in the Statistical Package for the Social Sciences (IBM SPSS; PASW v.18, 2010) are reported in Table 1. Because the data were collected in the context of an efficacy trial where technical assistance was provided to optimize functioning, high means and low standard deviations were found, as expected. One item, “The team has agreed how to govern itself and make decisions,” exceeded thresholds for normality at the peak of the invention, but it was retained due to an interest in its variability over time. These data suggest that, given resources and support, communities can achieve high levels of all four dimensions of coalition functioning that were assessed.

The four-factor model at the midpoint of the intervention (wave 3) is presented in Figure 1. All 17 items loaded significantly, with only three weak factor loadings of less than .55 (“Leadership usually has control of the meetings,” “The team has agreed how to govern itself and make decisions,” and “Everyone is involved in the discussion, not just a few”). These items were retained to maintain the breath of construct coverage, given that the overall model fit well and that lower loadings are somewhat expected when generated from a homogeneous sample (Cortina, 1993). The fit of the model to the wave 3 data (n = 113) was excellent (CFI .98; RMSEA .04) and was replicated with an acceptable fit at waves 1 and 5 (see Table 2).

Figure 1:

Factor Structure at Midpoint (Wave 3)

Table 2:

Test of Unidimensionality of Four Dimensions of Community Functioning

Time Point	Fit Statistics, 4-Factor					Fit Statistics, 1-Factor					WLSMV Difference Test
	χ2	df	p	CFI	RMSEA	χ2	df	p	CFI	RMSEA	Δχ2a	df	p	ΔCFI
Early implementation (wave 1)	281.47	129	.00	0.94	0.07	344.06	135	.00	0.92	0.09	56.61	6	.00	0.02
Midpoint (wave 3)	157.18	129	.04	0.98	0.04	207.04	135	.00	0.96	0.07	54.57	6	.00	0.02
Late implementation (wave 5)	251.26	129	.00	0.92	0.09	288.32	135	.00	0.90	0.10	40.34	6	.00	0.02

Note: CFI = comparative fit index; RMSEA = root mean square error of approximation.

^aΔχ2 is based on the DIFFTEST function in Mplus, appropriate for weighted least squares means variance-adjusted procedure (WLSMV) estimation.

This four-factor model was compared against a more parsimonious single-factor model to test whether the four dimensions of coalition functioning are distinguishable from each other, given the moderate levels of shared variance (average of 64%) between the four factors. This test was done at three time points (early implementation, midpoint, and late implementation), and at each time point the single-factor model produced a significant deterioration in fit (see Table 2), indicating that the four-factor model is preferable to a single global functioning factor.

Differences in the measurement structure over time were assessed for each of the four factors separately. The procedure of testing for invariance over time within dimension, rather than simultaneously within and across dimensions, was done in consideration of the small sample size. Model fit statistics from a comparison of a model in which the factor loadings were freely estimated across the five intervention phases to a model where factor loadings were constrained to be equal across all five intervention phases are presented in Table 3. Constraining the model across time caused no deterioration in model fit for any of the four dimensions of coalition functioning. This provides evidence of invariant factor loadings in the measurement of each of these dimensions of coalition functioning across time, permitting the comparison of them across different implementation moments with the prerequisite understanding that the same concept is being measured across time.

Table 3:

Test of Measurement Structure Equivalence of Coalition Functioning across Time

Factor	Fit (factor loadings freely estimated)					Fit (factor loadings constrained equal)					WLSMV Difference Test
	χ2	df	p	CFI	RMSEA	χ2	df	p	CFI	RMSEA	Δχ2a	df	p	ΔCFI
Coalition goal-directedness	68.13	41	.00	0.97	0.07	82.42	61	.04	0.97	0.05	21.95	20	.34	0.00
Coalition efficiency	167.71	65	.00	0.97	0.11	167.02	89	.00	0.98	0.08	30.49	24	.17	0.01
Coalition opportunities	72.11	22	.00	0.95	0.13	69.82	38	.00	0.97	0.08	14.59	16	.55	0.02
Coalition cohesion	7.10	8	.53	1.00	0.00	12.70	20	.67	1.00	0.00	10.14	12	.60	0.00

Note: CFI = comparative fit index; RMSEA = root mean square error of approximation.

^aΔχ2 is based on the DIFFTEST function in Mplus, appropriate for weighted least squares means variance-adjusted procedure (WLSMV) estimation.

The analyses described previously indicate that this self-report instrument has promise for reliably measuring four distinct dimensions of coalition functioning. It is also useful to understand whether these ratings are valid indicators of the constructs under study. Implementation specialists from the University of Washington rated each community's coalition functioning along several dimensions in 2007 (wave 4 of 5), which approximated the self-reported constructs. After mean-aggregating the individual self-report scores to the coalition level, rank-order correlations for each community were calculated between self-reported dimensions of functioning and those reported by the external implementation specialists (n = 12) (see Table 4). All associations were in the expected direction, and the magnitudes of the rank-order correlations were notably large. According to Cohen's (1988) criteria, a small effect size was found for goal-directedness, a medium effect size was found for efficiency and opportunities, and a large effect size was found for cohesion. Although the small sample size in this analysis makes the detection of statistically significant relationships difficult, a significant relationship between member-reported and observer-reported cohesion and a marginally significant relationship between member-reported and observer-reported board efficiency were found. When a similar analysis was done in the quasi-experimental study of CTC, comparing the self-report ratings to those of technical assistance providers on three out of four of these dimensions (albeit measured somewhat differently), significant correlations with the magnitudes of .36, .38, and 39 were discovered in this larger sample on directedness, cohesion, and efficiency, respectively (Feinberg, Gomez, et al., 2008). Considered in the context of the smaller sample size in the present study, this analysis indicates a reasonable convergence of perceptions of coalitions between coalition members and external observers and provides some support for the construct validity of these four dimensions of coalition functioning.

Table 4:

Pearson Correlation between Ratings of Coalition Functioning by Coalition Members and External Observers (n = 12 Communities) in Wave 4 (2007)

2007 Self-Report (Wave 4)	Technical Assistant Prompt (Wave 4)	r	p
Average directedness	Board has a clear sense of direction	0.25	.21
Average efficiency	This is a highly efficient board	0.45	.07
Average cohesion	Board has a feeling of unity and cohesion	0.5	.05
Average opportunities	There is a sense of ownership and commitment	0.3	.17

Discussion

These results indicate that coalition functioning is best represented with four unique, yet interrelated, factors: goal-directedness, efficiency, opportunities for participation, and cohesion. Each of these dimensions has an invariant factor structure over time and shows the hypothesized directionality when related to external observations of the same concepts. These results provide a foundation for determining whether these attributes of coalition functioning are related to coalition achievement. In the case of CTC, this will include a study of the relationship between dimensions of CTC coalition functioning and community adoption of science-based prevention strategies; community rates of mental, emotional, and behavioral disorders in youths relative to their baseline and relative to rates in their matched communities; and the sustainability of the CTC coalitions themselves.

The reliability coefficients in this analysis are somewhat lower than those found in the earlier quasi-experimental study of CTC done with the Coalition Web-Based Self Report Questionnaire (Feinberg, Gomez, et al., 2008). To investigate whether these lower reliabilities are based on instrument modifications or sample differences, future studies may want to replicate Feinberg's use of a seven-point scale for the response options in the measurement of coalition functioning.

Although the evidence for the psychometric soundness of the measure of coalition functioning examined in this analysis is promising, it is important to recognize that the analysis was conducted on a small number of community coalitions (12) that exist within small towns. Given the opportunity to replicate this study with a greater number of coalitions, it would be useful to test the invariance of the entire factor structure (rather than sequentially by dimension) over time. Replicating this analysis on a sample with greater variability could overcome the effect of range restriction, likely improving the observed correlations and fit indices. It will be important to examine the generalizability of these results with other communities, with coalitions not implementing CTC, and with a greater number of external observers providing ratings. Such replications may reveal whether the factor structure holds for coalitions that do not achieve such high levels of functioning. Despite these limitations to generalizability, by developing psychometrically sound measures of four dimensions of coalition functioning, this analysis provides a promising tool for social workers implementing CTC or other coalition-based community practice models and provides a foundation for further research and development of an evidence base for social work community practice. Furthermore, the results of this study provide some evidence that coalitions can be efficient and goal-focused while also being participatory and cohesive in their work within communities. Such evidence may advance the long-standing debate in social work about whether professionals and participants can use and benefit from evidence-based practices in a way that simultaneously honors the self-determination of individuals and communities. The integration of external–technical and local–contextual knowledge through high-functioning CTC coalitions may explain the intervention effects demonstrated in prior analyses. Future research should explore which dimensions of functioning are most predictive of the achievements of community transformation, outcomes in youths, and sustainability of the coalitions.