Development, Validation, and Performance of a Scale to Measure Community Mobilization

Abstract

Rationale

Community mobilization approaches (CMAs) are increasingly becoming key components of health programming. However, CMAs have been ill defined and poorly evaluated, largely due to the lack of measurement tools to assess mobilization processes and impact.

Objective

We developed the Community Mobilization Measure (CMM), composed of a set of scales to measure mobilization domains hypothesized to operate at the community-level. The six domains include: shared concerns, critical consciousness, leadership, collective action, social cohesion, and organizations and networks. We also included the domain of social control to explore synergies with the related construct of collective efficacy.

Method

A survey instrument was developed and pilot tested, then revised and administered to 1,181 young people, aged 18-35, in a community-based survey in rural South Africa. Item response modeling and exploratory factor analyses were conducted to assess model fit, dimensionality, reliability, and validity.

Results

Results indicate the seven-dimensional model, with linked domains but no higher order construct, fit the data best. Internal consistency reliability of the factors was strong, with ρ values ranging from .81 to .93. Six of seven scales were sufficiently correlated to represent linked concepts that comprise community mobilization; social control was less related to the other components. At the village level, CMM sub-scales were correlated with other metrics of village social capital and integrity, providing initial evidence of higher-level validity, however additional evaluation of the measure at the community-level is needed.

Conclusion

This is the first effort to develop and validate a comprehensive measure for community mobilization. The CMM was designed as an evaluation tool for health programming and should facilitate a more nuanced understanding of mechanisms of change associated with CM, ultimately making mobilizing approaches more effective.

Introduction

Community mobilizing approaches (CMAs) to engage community members to take action towards achieving a common goal,(1) have gained traction in health promotion. Mobilizing strategies have been employed to decrease urban violence,(2-4) prevent drug, alcohol, and tobacco use,(5-7) and for HIV prevention.(8-11) Mobilizing efforts commonly target community norms to promote healthy behaviors (e.g. decrease smoking), promote social integration and destigmatization of marginalized groups (e.g. sex workers), or shift policy to improve health services access through stakeholder engagement and coalition building, community consciousness-raising, and fortifying community groups. Health researchers, advocates, and donors increasingly understand that engaging communities in change processes represents a key strategy for community health. However, evaluation of CMAs has lagged behind enthusiasm for them, partly due to the lack of frameworks to understand multi-level programming (12) and a dearth of measurement tools to assess mobilization processes. This manuscript describes the development of the Community Mobilization Measure (CMM) and its psychometric properties.

Conceptual framework - elucidating the domains of mobilization

To conceptualize CM we took a deductive approach, reviewing and synthesizing mobilization-related literature from four complementary disciplines: social movements (sociology), community empowerment, community development, and community capacity. We culled key principles from each field, identifying the fundamental and common features hypothesized to be essential to successfully mobilize a community. A detailed review of the literature is published elsewhere. (13) Six domains of mobilization were identified: 1) A shared concern or community issue that may address power imbalances, improve access to resources and services, or promote social inclusion.(14-17) 2) Critical consciousness addresses the requirement that the shared concern be built from collective sensitization processes, or Paulo Freire’s ‘consentização,’ (18) which lies at the heart of the community empowerment literature (15, 19) and the concept of the learning culture (20) from community capacity. The critical consciousness domain is also akin to social movement theory’s cognitive liberation and collective framing process.(21) 3) Organizational structures and networks or vehicles “through which people mobilize and engage in collective action” (21) serve as basic structures to promote dialogue, disseminate messages, and build collective actions.(20) The presence of organizations is also key to building “bridging” social capital – inter-organizational linkages that connect communities and groups to more diverse networks and resources.(22, 23) 4) Leadership, whether it be individual, institutional, or a coalition of activitists, is at the center of community change programs across disciplines.(19, 20) 5) Collective actions or the ‘social movement repertoire’ (14) is a critical component to all reviewed literatures and is primarily the domain associated with public participation in mobilization.(24) Finally, 6) social cohesion represents the idea that there is a “glue” that holds people together, which is akin to the need for collective identity (25) and has most often been operationalized as shared trust (26, 27) or as a sense of community.(20, 28)

Related to the above mobilization domains is social control, which comprises one of two domains of collective efficacy, along with social cohesion. Developed by Sampson and colleagues in studying community violence in urban U.S. neighborhoods, collective efficacy is defined as “the capacity of residents to achieve social control over the environment and to engage in collective action for the common good.”(26, 27) Collective efficacy theorists hypothesize that for communities to tackle social problems, some level of baseline social cohesion among community members should exist based on working trust and mutual expectation to intervene for shared interests. Because social cohesion was included in the hypothesized domains of mobilization, we also explored informal social control (i.e., whether community members intervene on behalf of the common good) in our measurement model in order to explore synergies between CM domains and collective efficacy.

Community-level measurement

Mobilization is theorized at the community-level; the domains are meant to characterize and measure community properties that can create community-wide change. To that end, measurement should also be operationalized at the community-level.(29, 30) Community-level constructs can be envisioned as a shared group property, operationalized as a composite of individual characteristics or perspectives that are aggregated, such as indicators of social norms, or be ‘global’ characteristics that are integral, observable or descriptive (i.e., rural versus urban), and may not be adequately captured by summarizing individual responses.(30, 31) For example, social cohesion is referent to conditions of mutual trust and solidarity and conceptually describes a group/community attribute.(32) If one hypothesizes that cohesion is an integral characteristic, the measure could consist of observed demonstrations of trust (e.g. documentation by video). However, structural measures of community attributes are challenging to develop, laborious to implement, and difficult to standardize, and thus are less often employed.(33) Instead, social cohesion has been most commonly operationalized as an individual’s perception of social cohesion within his/her community. If enough groups are present, the measure is aggregated and mean reported group cohesion is calculated; others have utilized measures of perceived group cohesion in individual-level analyses.(27, 34)

For the CMM, we have operationalized the domains as shared group characteristics, or properties that “originate in experiences, attitudes, perceptions, values, cognitions, or behaviors that are held in common by the members of a team [or community].”(30) We thus created items that elicit individuals’ perceptions and attitudes about their community; these items were compiled into a survey instrument administered to individuals and then can be aggregated to estimate community means. While the CMM is intended to assess community properties, community-level scale validation is difficult to undertake with a small number of communities.(35) It also follows that scale items can first be assessed for item function and reliability at the level of the individual prior to aggregating responses. We therefore report on CMM performance at the level of the individual (i.e., individual perceptions of community characteristics) and briefly explore aggregation and higher-level validity.

Below we describe the development of the CMM, including refinement based on pilot testing; report the properties and performance of the refined CMM in a large community survey across 22 villages in rural South Africa; explore validity; and discuss the utility of this measure to improve the evaluation of large CMAs in health promotion.

Method

CMM item generation

We followed the ‘construct mapping’ method for item generation and assessment,(36) beginning with a deductive approach, complemented by qualitative research, and followed by multiple iterative steps. A pool of items were generated to capture individuals’ perception of their community along a theoretical continuum, including items that were hypothesized to represent less mobilized communities (easier items to endorse) and more mobilized communities (harder items to endorse).(36) The items were adapted from other sources when available (e.g., measures of collective efficacy) and were constructed by the authors based on concepts drawn from the literature (see Figure 1). For example, there are no published measures of critical consciousness. As a result authors created items designed to assess whether communities were undertaking processes of reflection and inquiry, debate or dialogue, collective attribution, and response – all comprise aspects of undergoing what the social movements literature terms “framing processes” or “cognitive liberation.”(21) According to Freire, (18) who coined the term, dialogue is the principle tool needed to encourage community members to collectively understand and solve injustices. As a result, items in the critical consciousness scale ask respondents about whether their community “talks about” or “discusses” village problems; explores village “cooperation,” willingness to “hear different views;” and collective seeking of “solutions” and learning. To ensure the scale covers the spectrum of critical consciousness, items included basic skills (e.g., villagers must engage in discussing problems) as well as behavior indicative of true collective reflection (e.g., thinking about why problems exist in the first place).

Figure 1. Community Mobilization Measure Components, Format, and Items Source or Conceptual Roots.

^a Included to explore intersections between Collective Efficacy and Community Mobilization.

Both newly developed and adapted items were optimized through qualitative work to ensure relevance to rural South Africa and to better understand operationalization of items. (13) For example, our measure of shared concerns focused on the theme of the mobilization intervention in which the CMM was being utilized: HIV prevention. Thus, items assess the degree to which the community perceives HIV as a salient concern that they can work together to prevent. In qualitative research, community members qualified the importance of a topic by whether it was discussed at community meetings; we therefore included an item that assesses whether HIV prevention is discussed at community meetings. Items comprising the other domain scales were not specific to HIV and are meant to capture intrinsic capacity or activity related to mobilization around any topic.

Prior to pilot testing, items were vetted in extensive discussions with local community representatives, including members of the community research liaison office. Items were then examined for face validity with cognitive interviewing.(37) Overall 10 men and 10 women from two age groups (18-25 and 25-35 years) were interviewed to assess item clarity and interpretation of the questions and the response options; adjustments were made when respondent interpretations did not consistently align with the intention of the item. All items referenced the respondents’ village when eliciting information. Items were written in English, translated into the local language of Shangaan and back-translated into English.

Data collection

Data collection took place in the Agincourt study area of Bushbuckridge sub-district, Mpumalanga province, a rural area with high rates of unemployment, temporary labor migration, and extremely high HIV prevalence (i.e., 45% among 35–39 year olds).(38, 39) The area is home to the Agincourt Health and Socio-Demographic Surveillance Site (AHDSS). This site consists of 27 contiguous villages that were established during the resettlement programs of the apartheid government of South Africa in the 1940s, where people were forcibly relocated into villages using the geographical boundaries of the formerly white-owned farms in the area. Villages have their own political and traditional leadership structures, and qualitative work undertaken prior to the survey confirmed that the people in the area defined their geographical village as their community.

The survey was piloted between November-December 2010 among a random sample of 101 participants, between 18 and 35 years of age, from five villages in the AHDSS sampling frame (20 people per village). Villages were selected purposefully to garner responses from inhabitants of communities that varied in size, socio-demographic attributes, and community resources. Data for the main survey were collected between March-June 2012 and included a random sample of 1,181 young adults aged 18–35 years, approximately 55 people in each of 22 villages participating in an HIV prevention trial.(40) Surveys were administered to one eligible respondent per household in English or Shangaan via computer-assisted personal interviews (CAPI) at the respondent’s home. The study protocol was approved by institutional review boards at the University of California, San Francisco and the University of North Carolina at Chapel Hill; the Human Research Ethics Committee of the University of the Witwatersrand, South Africa; and by the Mpumalanga Provincial Ethical Review Committee.

Data analysis

Item response modeling (IRM), exploratory factor analyses (EFA in the pilot) and confirmatory factor analysis (CFA in the main study) were performed for each of the community mobilization instrument's domains. We utilized both techniques as they have complementary strengths.(41) IRM analysis was performed using the software program Conquest 3.0 (ACER, Australia) and the Test Analysis Modules and WrightMap packages in R 3.1.0; EFA and CFA were performed using the latent variable modeling program Mplus (Muthen & Muthen, Los Angeles).

For pilot data, Mplus EFA analyses obtained estimates of factor loadings via a weighted least squares estimator with a mean and variance adjustment (Mplus estimator WLSMV) suitable for use with ordinal survey items.(42) Factor loadings < 0.6 in EFA were considered evidence of misfit. IRM was used to assess coverage and item fit, with an infit statistic <0.75 or >1.33, which implies less or more variance than expected, considered evidence of poor fit.(36) IRM and EFA results were used to identify a subset of well-performing items from each domain. We developed new items when IRM results indicated that a scale was less sensitive (i.e., had more measurement error) at a certain range of response patterns (i.e., usually items capturing responses at the very high or low end of the scale).

Based on the pilot study results, the community mobilization measures were refined and administered in the main study. We investigated the model fit and factor structure (i.e., the dimensionality) in the main study's data by fitting models corresponding to four potential hypotheses of dimensionality: 1) All CM items are indicators of a single latent construct (all items in one model together – no hierarchy), 2) The items designed to measure the two collective efficacy sub-domains represent a single factor and the remaining items are indicators of a second factor, with the two factors allowed to correlate; 3) Each domain is a separate dimension within a seven-dimensional construct (all items were fitted together in a single model but linked only to their parent factor); and 4) The seven dimensions were treated as indicators of a single, latent higher-order factor of general community mobilization. For all CFA models, exact model-data goodness of fit was assessed using the Mplus WLSMV χ² test of exact fit. Because the χ² test is sensitive to trivial departures of model-data fit in larger samples, approximate fit was assessed using a combination of Bentler's Comparative Fit Index (CFI),(43) the Root Mean Square Error of Approximation (RMSEA),(44) and the Weighted Root Mean Square Residual (WRMR).(45) Satisfactory approximate fit occurred when two out of the three following criteria were met: CFI ≥.95, RMSEA ≤.06, and WRMR ≤1.00.(46) Reliability for each subscale was assessed using Raykov's ρ, which is similar to Cronbach's coefficient alpha but relaxes alpha's often-unrealistic assumption of equal factor loadings.(47) Confidence intervals (CIs) for ρ were calculated based on a logit transformation. In IRM, we assessed model fit using the Akaike Information criterion (AIC) and the standardized root mean square of squared residuals (SRMSR), a metric of item correlation unexplained by the measurement model. A threshold of SRMSR ≤0.05 has been proposed to indicate good model fit.(48) All IRM models were single parameter partial credit (multinomial) models (PCM). Finally, we tested Differential Item Functioning (DIF) by language of survey administration to explore unexpected item response patterns for those answering in English (10%) rather than Shangaan. We consider significant DIF over 0.64 logits (log odds) to be of large magnitude.(49)

Validity was assessed first by confirming structural validity at the individual level: the anticipated factor structure was obtained in EFA with pilot data and then confirmed in CFA using the main study data. Validity evidence based on internal structure also includes evaluating whether the observed data are consistent with a priori expectations of item location and rank using the item threshold rankings generated from IRM. We used a Wright Map as a graphical method of assessing construct validity: We assessed the order of item thresholds relative to the theorized development of community mobilization domains. In addition, we evaluated the dispersion of items within and across subscales relative to the distribution of respondents to identify areas of under- or over-coverage.

We assessed validity of the CMM sub-scales as community measures in several ways. We calculated the intra-class correlation (ICC) to quantify within versus between-village variance. We then merged data for two CMM sub-scales (collective efficacy: social control, and social cohesion) with community-level data using the same metrics from a population-based surveillance study conducted in 2014 across 43 enumeration areas in the rural North West province, South Africa. Using this larger data set, we again calculated the ICCs to further explore variance and homogeneity in the aggregated measure. Assessment of convergent validity was undertaken by testing correlations of village mean IRM scores of each CMM sub-scale against village mean scores of theoretically related constructs – civic participation and village disorganization. The civic participation scale assessed involvement in different community groups as well as participation in voting and volunteer work. Voluntary group membership has been used as a proxy measure for social capital, a characteristic of community social relationships that is hypothesized to promote collective action towards shared community goals;(32, 50) as a result we predicted a positive association with CMM. Furthermore, we correlated the CMM sub-scales with a metric of village disorganization calculated from community mapping data collected in each village in 2010. During the mapping process, village representatives rated their villages on three elements of physical disorganization: abandoned structures, graffiti, and trash; we calculated the average village rating across these elements. Deteriorating structures or physical disorder are posited to relate to a lack of neighborhood social organization or shared ownership and monitoring of spaces, and should negatively correlate with the CMM sub-scales.(51)

Due to the complex sampling design for both studies, all reported analyses employed case weights and used cluster-based robust standard errors to account for sampling design and clustering within villages. Standardized factor loadings for community mobilization survey items, interfactor correlations, ICCs, internal reliability estimates for subscales are reported with 95% CIs.

Results

Sample Characteristics

In the pilot study, a total of 186 households (one resident per household) across five villages were sampled for participation; 101 individuals completed interviews out of the 144 households (77%) successfully contacted. In the main study, visits were conducted to 1,826 households among which 1,256 (69%) had an eligible resident selected; the most common reason for ineligibility was non-residence in the last 12 months, as migration in the area is high. Among those eligible, 1,181 people were enrolled into the study (94%), 66 people (5%) refused to participate, and the remaining nine (<1%) did not enroll for other reasons. Study participants were mostly unmarried and had at least some secondary education; the pilot sample was slightly younger than the main study sample (see Table 1). Due to the minimal amount of missing data, software default approaches were used (e.g. pairwise in Mplus; imputation assuming missingness at random in IRM). Of five respondents with missing data in the pilot, four were missing only one or two items; and among the 17 respondents in the main study with missing data (1.4%) only three had more than one missing item.

Table 1.

Characteristics of the study populations in the pilot (2010) and main study (2012).

	Study Samplesa
Characteristics	Pilot Study (N=101)	Main Study (N=1,181)
Villages sampled	5	22
Gender
Male	51 (50.5)	581 (49.2)
Female	50 (49.5)	600 (50.8)
Age (years)
18-20	47 (46.5)	408 (34.6)
21-25	27 (26.7)	331 (28.0)
26-30	11 (10.9)	228 (19.3)
31-35	16 (15.9)	214 (18.1)
Education
Primary or less	6 (6.0)	125 (10.6)
Some secondary	67 (66.3)	663 (56.1)
Completed secondary	28 (27.7)	393 (33.3)
Marital status
Never married	78 (77.2)	863 (73.1)
Married (legal or traditional)	20 (19.8)	264 (22.3)
Separated, divorced, widowed	3 (3.0)	54 (4.6)
Immigrantb	—	104 (8.8)
Earned income within past 3 monthsb	—	410 (34.7)
Experienced food insecurity within  past 30 daysb	—	47 (4.0)

Note.

^aCharacteristics are presented as the number of participants (n) with the percentage of the total study sample in brackets.

^bNot included in pilot study questionnaire.

Pilot Data: Item Response Modeling and Exploratory Factor Analyses

In both IRM and EFA analyses, reverse-coded items performed poorly and were either rephrased or removed.(47) Items that performed only moderately or poorly were reviewed for clarity of meaning and phrasing. Collectively for six of seven domains, seven items were dropped, 15 rephrased, and four items were split apart; 20 items were added to extend scale sensitivity and to discern differences between respondents at the higher and lower ends of the mobilization not shown), the ‘Organizations and Networks’ measure performed poorly and was entirely recreated. The initial measure required respondents to name groups in their village working towards the common good and then respond to questions about group characteristics (i.e., membership, inclusivity, longevity). Only 30 respondents could name any community organizations and fewer could characterize named organizations. For the main survey, items elicited information regarding the presence of specific types of groups and their importance in the community.

Main Study: Confirmatory Factor Analyses and Internal Reliability

Among the four models compared, the 7-dimensional model had the best fit (i.e., all items modeled together but linked to their own dimension) in both CFA and IRM (see Table 2). Although the approximate fit statistics for the 7-dimensional model and the higher-order CFA were similar, a nested model χ² test rejected the null hypothesis of equivalent fit (χ²(14)=42.78, p<.001), indicating that the higher-order CFA fit the data significantly worse than the 7-dimensional CFA. For the 7-dimensional CFA, the χ² test of exact model-data fit was rejected (χ²(1994)=2347.94, p<.001). However, approximate fit statistics indicated that the proposed CFA model fit the data very well on an approximate basis (CFI=.97, RMSEA=.01, and WRMR=1.46). Internal consistency reliability of the factors was strong, with ρ values ranging from .81 to .93 (see Table 3). The revised organizations and networks items performed well, though items addressing the importance of police/safety and men's organizations were less strongly related to the organizations and networks factor than the other organizations listed in Table 3.

Table 2.

Comparison of unidimensional and multi-dimensional model fit to seven community mobilization domains.

	Confirmatory Factor Analysis a			Item-Response Modeling b
Models	CFI	RMSEA	WRMR	SRMSR	AIC	Parameters
A. Unidimensional	0.79	0.03	3.70	0.14	136380	131
B. 2-dimensional	0.78	0.04	3.88	0.13	134017	133
C. 7-dimensional	0.97	0.01	1.46	0.06	123203	158
D. Higher-order c	0.97	0.01	1.55	—	—	—

Note. Bolding indicates the 7-dimensional had the best fit. — =Not applicable. CFI=Bentler's Comparative Fit Index. RMSEA=Root Mean Square Error of Approximation. WRMR=Weighted Root Mean Square Residual. SRMSR=Standardized Root Mean Square of Squared Residuals. AIC=Akaike information criterion.

^aSatisfactory approximate fit if two of three criteria met: CFI≥.95, RMSEA≤.06, and WRMR≤1.00

^bSRMSR≤0.05 considered good model fit; no absolute criterion possible for AIC, but lower values indicate better model fit.

^cHierarchical analysis of partial credit item-response models is not yet possible in most statistical software.

Table 3.

Estimated factor loadings of community mobilization measure scale items and reliability of domain scales (Raykov's ρ) from confirmatory factor analysis: Main study (N=1,181)

Subscales and Items	Reliability (95% CI)
Social Control (very likely-somewhat-unlikely that):	.89 (.87, .90)
Your neighbors would intervene if children were skipping school and hanging out on a street corner?	.78 (.70, .86)
Your neighbors would intervene if children were breaking windows on a local building/destroying public   property?	.87 (.81, .92)
Your neighbors would intervene if children were showing disrespect to an adult?	.76 (.67, .84)
Your neighbors would intervene if a fight broke out at the pension point?	.93 (.89, .96)
Your neighbors would intervene if the local school closed down the feeding scheme?	.92 (.88, .96)
Your neighbors would intervene if a family didn’t have enough food?	.68 (.60, .75)
Your neighbors would intervene if the neighborhood water tank was broken?	.77 (.72, .81)
Your neighbors would intervene if an elderly person was robbed?	.79 (.73, .86)
Social Cohesion (agree-somewhat agree-disagree that):	.81 (.79, .83)
People in this village are willing to help their neighbors?	.69 (.66, .72)
This is a close knit community?	.73 (.68, .79)
People in this village can be trusted?	.81 (.76, .87)
People in this village generally get along well with each other?	.75 (.69, .80)
People in this village share the same values?	.59 (.51, .66)
People in this village look out for each other?	.86 (.81, .91)
Shared Concerns (agree-somewhat agree-disagree that):	.85 (.84, .86)
People in your village are concerned about HIV?	.68 (.64, .71)
People in your village consider HIV/AIDS an important issue?	.70 (.63, .77)
People in your village talk openly about HIV?	.65 (.58, .71)
People in your village believe that HIV impacts the community?	.50 (.41, .60)
People in your village talk about HIV/AIDS at community meetings?	.64 (.59, .69)
People in your village work together to prevent HIV from spreading?	.94 (.91, .97)
People in your village work together to reduce the effects of HIV?	.89 (.87, .91)
People in your village believe they can change the course of the HIV/AIDS epidemic?	.70 (.64, .75)
People in your village exchange information about HIV/AIDS?	.80 (.76, .83)
People in your village take HIV/AIDS seriously?	.84 (.82, .87)
Critical Consciousness (agree-somewhat agree-disagree that):	.93 (.92, .94)
People work together to solve problems in the village?	.85 (.82, .89)
People in your village talk to each other about how to solve village problems?	.88 (.84, .92)
People in your village enjoy discussing different ways to solve village problems?	.84 (.81, .87)
People in your village are open to hearing different views about community problems and solutions?	.87 (.83, .90)
People in your village volunteer to help solve village problems?	.89 (.86, .92)
People in your village think about WHY there are problems so they can address the cause of problems?	.84 (.80, .88)
There is a lot of cooperation between groups in the village?	.75 (.71, .78)
People in this village not only talk about problems but they also try to solve them?	.90 (.87, .93)
If your community fails to resolve a community problem-they will try another-different approach to solving   the problem?	.85 (.83, .88)
If your community fails to resolve a community problem-they will learn from that experience and do a   better job when they try to solve the problem in the future?	.83 (.80, .86)
If leaders in the village fail to resolve a village problem-the villages will work together to find a solution?	.84 (.81, .87)
Leadership (agree-somewhat agree-disagree that):	.92 (.91, .93)
The village leaders represent your opinions?	.71 (.67, .76)
The leaders are responsive to your concerns?	.65 (.60, .69)
When you have a problem in your community-you can go and speak with your leaders about the problem?	.60 (.52, .67)
The leaders in your community work effectively together?	.89 (.85, .93)
The leaders in your village get a lot done for the community?	.79 (.75, .84)
The leaders in your village represent all the different kinds of people who live in your community?	.70 (.63, .77)
The leaders in your village encourage the people to participate in village decision making?	.81 (.75, .87)
The power to make community decisions is shared among leaders and the people in this village?	.72 (.64, .79)
There is strong leadership in my village?	.84 (.81, .87)
The leaders in this village are trustworthy?	.83 (.80, .87)
The leaders in this village act responsibly with the power they have?	.86 (.82, .89)
The leaders in this village put the villagers’ needs first-before their own needs?	.76 (.69, .82)
The leaders in this village can put aside their personal interests to act in the interests of the community?	.81 (.76, .86)
The leaders in this village are honest; there is little corruption here?	.71 (.68, .73)
Organizations and Networks(--very important-a little important-not important /no such organizations):	.81 (.78, .84)
Are there any organizations or groups that help to make your community better in your village? a	.62 (.48, .76)
Are there groups with which you can volunteer to help people in your community? a	.55 (.46, .64)
How important are school organizations in this community?	.76 (.72, .81)
How important are police/safety organizations in this community?	.48 (.37, .59)
How important are youth organizations in this community?	.72 (.65, .80)
How important are sport organizations in this community?	.75 (.65, .85)
How important are women's organizations in this community?	.73 (.66, .79)
How important are men's organizations in this community?	.48 (.38, .57)
How important are religious/church organizations in this community?	.81 (.76, .87)
How important are cultural organizations in this community?	.76 (.68, .84)
Collective Action b	.84 (.81, .86)
How many community meetings were called in the last 3 months?	.74 (.66, .81)
How many community members participate in community meetings on average in the last 3 months?	.82 (.74, .89)
How many community meetings have you attended in the last 3 months?	.64 (.57, .71)
How many times has the community worked together to fix a problem in the past 3 months?	.89 (.83, .95)
How many community members participate in fixing community problems?	.89 (.86, .92)
How many times have you worked to fix a community problem?	.84 (.78, .89)

Note. CI=Confidence interval.

^aOrganization and Networks responses: Yes, no, don’t know.

^bCollective Action responses recoded to categories: 0, 1-2, 3+ (items 1, 3, 4); 0, 1-49, 50-99, 100+ (items 2 and 5); 0, 1, 2+ (item 6).

IRM findings concurred with CFA: The 7-dimensional model had the lowest AIC and was the closest to meeting the SRMSR threshold of 0.05. Separation reliability was high for all dimensions under this model, ranging from 0.75 for organizations and networks to 0.92 for leadership (data not shown). Individual item infit statistics supported acceptable model fit, with only one item of 65 falling outside of an acceptable range of 0.75–1.33 (data not shown).(36) Eight items were identified with large DIF by survey language, including four on the social control sub-scale and three from the organizations and networks sub-scale.

Factor intercorrelations were generally mild to moderate. Leadership and critical consciousness were the most strongly correlated using both CFA (r=.67) and IRM (r=.70); shared variance may express a meta-level of community capacity to respond to shared concerns (see Table 4). The ‘social control’ domain was consistently the least correlated with the other measures, indicating that it may not be a domain of mobilization but measure a separate, related community trait (as initially hypothesized).

Table 4.

Correlations among latent factors and their 95% confidence intervals using item-response modeling and confirmatory factor analysis.^a

	Shared Concern	Critical Consciousness	Leadership	Organizations/ Networks	Collective Action	Social Cohesion	Social Control
Shared Concern		.62 (.59, .66)	.42 (.38, .47)	.34 (.28, .39)	.43 (.38, .48)	.62 (.58, .65)	.34 (.29, .39)
Critical Consciousness	.52 (.47, .58)		.70 (.67, .73)	.43 (.38, .47)	.60 (.56, .64)	.58 (.54-0.62)	.38 (.33, .43)
Leadership	.39 (.34, .44)	.67 (.65, .69)		.42 (.37, .46)	0.45 (.41, .50)	.57 (.53, .61)	.29 (0.24-0.34)
Organizations / Networks	.28 (.20, .37)	.37 (.27, .46)	.36 (.29, .42)		0.55 (.51, .59)	.36 (.31, .40)	.19 (.13, .24)
Collective Action	.37 (.32, .42)	.53 (.48, .57)	.42 (.35, .48)	.45 (.36,.55)		.44 (.40, .49)	.32 (.27, .37)
Social Cohesion	.49 (.45, .53)	.46 (.38, .55)	.48 (.41, .54)	.26 (.13, .40)	.33 (.26, .41)		.32 (.26, .37)
Social Control	.26 (.20, .32)	.36 (.29, .43)	.28 (.19, .37)	.10 (, .10, .29)	.26 (.15, .37)	.23 (.17, .28)

^aCorrelations among factors using item-response modeling appear above the diagonal; correlations among factors using confirmatory factor analysis appear below the diagonal.

Validity evidence: Individual-level

Item location and rankings were found to be consistent with a priori expectations of the domain locations based on our construct map, the literature, and our pre-survey qualitative research. The item thresholds are presented in Figure 2, a Wright Map of the construct combining the location of respondents on each dimension (left) with the approximate probability of endorsing each response level by item (right). The figure demonstrates reasonable item coverage: almost no individuals would have disagreed with all items and relatively few people found all items possible to endorse. The figure also provides a visual map of ascending attainability of the CM domains; the easiest domains to affirm were a base level of community consciousness and basic social cohesion. Leadership and shared concerns followed, with the most difficult items to affirm being organizations and networks (which were scarce) and collective action. We expected that collective action would be the most difficult domain to affirm; in fact, it was easier for most respondents to agree with even the ‘hardest’ items assessing community cohesion and consciousness before the ‘easiest’ affirmation of collective action. Thus, crossing a threshold into collective action would require a strong sense of cohesion, community consciousness to dialogue around the issue, and the presence of a basic shared concern, which is consistent with theory that a collective identity logically precedes collective claims making.(21, 52)

Figure 2. Wright Map of Community Mobilization Measure.

For each of the domains of community mobilization, the distribution of respondents’ locations (perceived level of that domain) is shown in the histograms on the left-hand side. In IRM estimation, person scores were constrained to have a global mean of zero so that each sub-scale did not need to be centered at zero, enabling the dimensions to be aligned using a single metric. The plot is scaled by logits, or log odds of selecting a higher versus lower response option. The right side maps each item threshold, the point at which the probability of choosing all lower response options equals the probability of choosing that response option or a higher one. The lowest points represent the response options to items that were easiest to endorse – even respondents who perceived low levels of community mobilization were likely to agree with these items – and the highest points represent those response options that were harder to endorse. Each item has one fewer threshold than the number of response options; most items had three response options and hence two thresholds; a small number of items had two or four response options. Items are grouped by sub-scale and ordered in ascending order of sub-scale location on the latent construct. Social control is not shown.

Validity evidence: Community-level

Sub-scale ICCs were quite low, ranging from 0.02 (shared concerns) to 0.15 (organizations and networks), indicating little intra-village agreement (see Table 5). Because the study villages in Agincourt are relatively interconnected, we tested the hypothesis that low ICCs were a result of homogeneity by examining data on the two collective efficacy domains (social control and social cohesion) which were included in a surveillance study of 1,044 adults in a larger region of enumeration areas in North West Province, South Africa. ICCs estimated from the combined sample were higher for both social control and social cohesion (0.21 and 0.14, respectively), supporting the hypothesis that CMM subscales are sensitive to community agreement when there is sufficient heterogeneity between groups. Correlation of the CMM subscales with aggregate civic participation provided evidence of convergence, with all sub-scales significantly or near significantly correlated, except for social control. Finally, all sub-scales except social control were negatively correlated with village disorganization, significantly so for cohesion and leadership, despite the small number of villages (Table 5).

Table 5.

Village-level assessment: Means, intra-class correlations, and construct validity assessment of latent factors in 22 study villages and 43 additional South African communities participating in a surveillance study.

	Validity of the CMM sub-scales as Community Measures
	Raw Village Score (M±SD)	ICC (95% CI) a		r (95% CI) of Village Mean Scores c
CCM sub-scales	(22 Study Villages)	22 Study Villages (N=1,181)	Combined Sample (N=2,225) b	IRM with Civic Participation	IRM with Village Disorganization
Social Control	1.98 ±.16	.04 (.02, .09)	.21 (.15, .29)	.21 (−.23, .58)	.02 (−.41, .44)
Social Cohesion	2.34 ±.15	.05 (.02, .11)	.14 (.09, .20)	.46 (.04, .74)	−.48 (−.75, −.07)
Shared Concern	2.14 ±.11	.02 (.00, .05)	----	.54 (.15, .78)	−.26 (−.62, .18)
Critical Consciousness	2.47 ±.14	.04 (.02, .09)	----	.56 (.18, .79)	−.40 (−.70, .03)
Leadership	2.10 ±.18	.07 (.03, .13)	----	.53 (.14, .78)	−.51 (−.77, −.12)
Organizations/Networks	0.97 ±.19	.15 (.08, .25)	----	.42 (.00, .72)	−.38 (−.69, .04)

Note. EA=Enumeration area. CCM=Community mobilization measure. CI=Confidence interval (CI). ICC=Intra-class correlations. IRM=Item-response model. SD=Standard deviation. r=Correlation.

^aQuantified within- versus between-village variance.

^bRepresents 65 groups in total; Data from the 22 study villages were merged with community-level data collected in 2014 (43 EAs, N=1,044) in the rural North West Province, South Africa.

^cCorrelation between village mean IRM score for each CMM sub-scale and village mean score for two theoretically related constructs – civic participation and village disorganization.

Discussion

This study is the first to operationalize a measure of community mobilization that can be used to monitor and evaluate CMAs by assessing changes in defined CM domains over time. The CMM was developed on solid theoretical grounds, included careful tool development, and demonstrated robust reliability and construct validity at the individual level, based on model performance and synergies between the theory guiding the measure, qualitative data, and measurement results. Initial community-level analysis suggests evidence of construct validity, although evidence from larger, more heterogeneous samples is required to fully determine higher-level validity and reliability. The best model fit using complementary measurement approaches was the 7-dimensional model. Our results indicate that six of seven scales for components of mobilization were sufficiently correlated to represent linked concepts that comprise community mobilization. There was only limited evidence that the social control domain was sufficiently related to the other components. Instead, in our analysis, not only was social control the least correlated with the other measures, it also was insufficiently correlated with cohesion to warrant a higher order ‘efficacy’ factor. Because collective efficacy theory originated to describe community abilities to combat social ills in the urban U.S., it is possible that the meaning of the social control concept was not adequately captured in the South African context. Notably, social control was not explored in the qualitative research that informed the other measures and may be more distally related to mobilization as currently conceptualized.

Research considering the relevance of collective efficacy theory in South Africa has noted inconsistencies compared to the original U.S. research.(53) In future applications of the multi-component scale, inclusion of social control may not be warranted, unless research can shed light on the local meaning and result in improved adaptation.

The individual scales performed quite well, with indicators of reliability above .8 in all scales. The most reliable scale was that of critical consciousness. It is, to our knowledge, the first attempt to operationalize this concept, which has been popular in educational, sociological and health promotion research since the publication of Paulo Freire’s seminal work.(18, 54, 55) The least reliable scale was that of organizations and networks, which was conceptualized to capture the vital role of mobilizing structures.(21) In Western literature, these organizations manifest as formal groups that provide human and material resources and dissemination networks; our initial organization and networks items deployed in the pilot study reflected this view. However, the original scale failed based on limited local knowledge of formal organizations. Because in this rural area community activity centers mostly around family-based informal networks, we revised the initially piloted scale and developed items to assess the presence and significance of specific formal and informal affinity groups.(13) While performance was much improved, we believe that the measure could benefit from additional work not only to elicit responses that confirm the presence, visibility, and perceived salience of organized groups in the community, but also to assess whether these groups actually play a role as a mobilizing resource. This domain may also be intrinsically difficult to measure where networks are diffuse.

We did include one topic-specific scale in the measure, shared concerns; the other subscales apply more broadly to any locus of community concern. Whether the topic-specific sub-scale should be assessed separately is open for debate. It is possible that inclusion of a topic-specific section could change the measurement properties of the scale, given that changing the ‘concern’ or focus could elicit different answers from the same community. However, we take the position that while a community’s capacity to mobilize might be a generalized condition, mobilization will not occur unless there is an issue or concern that motivates a response.(20)

We measured domains of mobilization by assessing individuals’ perceptions of their community; the individual-level data were used to evaluate scale performance, accounting for village clustering, and then aggregated to explore village-level validity. With the exception of social control, the CMM scales showed expected patterns of convergence with theoretically related constructs of civic participation and village disorganization at the community-level, initial evidence of higher-level validity. There was little agreement within villages as evidenced by low ICCs for most measures, though even low ICCs may be meaningful.(56) Higher ICCs resulting from merging in group-level data from a different region suggests the current sample may be too homogeneous to note group level differences in CMM scores. Additional research on the community measurement properties of the CMM includes assessing patterns of variability within the villages, including whether within-village variability decreases over time as mobilization proceeds and whether variance patterns could represent ‘configural properties’ that characterize communities.(30)

While our principal objective for developing the scale is evaluation of CMA programming, the CMM and its sub-scales can and should be purposed for other needs. For example, to establish baseline levels of cohesion or concern in groups where policy work or health programming may take place; to monitor perceived accountability among community leaders; and to assess how community traits, such as critical consciousness, predict resilience in response to tragedies or change. The scale and sub-scales can be adapted for other needs and cultural contexts; though we recommend careful consideration and additional validation when adapting the CMM to different cultures.(57) Additionally, as research on the community-level properties of the scale continues, the CMM can also be utilized at the individual-level to assess perceptions of the hypothesized community properties, which may be salient indicators of ongoing social context and resulting health. For individual measurement, utilizing CFA or IRM-based coefficients to determine personal CMM scores may result in less bias than simple summation and can be easily calculated in software such as Stata, SAS, or Mplus.

Limitations

Sample size for this undertaking was small, constraining some options for validation. With 22 communities we were unable to formally assess whether our instrument exhibited the same factor structure at the community-level as it did at the individual-level (psychometric isomorphism); subsequent research with a larger number of communities is needed to fully establish community-level validity.(35) Furthermore, the villages are fairly homogeneous being part of a single demographic surveillance site. While the homogeneity of the villages poses a limitation for measurement validation at the community-level, it strengthens the trial evaluation by ensuring that villages are comparable. However, the size and homogeneity of the sample does not eclipse the need for and utility of this measure.

Conclusion

Overall, there have been few efforts to develop and validate measures for mobilization and, despite the popularity of CMAs, little conceptual work has been undertaken to align theory to programming and evaluation. The need for a mobilization measure is keen: A better understanding of how community processes impact health outcomes over time requires stated CM components and a validated measure. Understanding mechanisms of change associated with CMAs will be instrumental in ensuring effective programming in health promotion. The utilization of the CMM in the context of the HIV prevention trial underway (40) provides an opportunity to explore whether all CM domains change among individuals and within villages; at what pace; whether change is associated with intervention status; and whether domain-specific changes impact health outcomes. We hope that the full CMM and its individual components will be widely used and provide an important means to better understand CMAs and their processes as we move towards improving mobilization approaches in the future.

Highlights.

• We conceptualized and developed the Community Mobilization Measure (CMM)

• The 7-scale CMM was assessed for model fit, dimensionality, reliability and validity

• The CMM performed well, with high reliability and evidence of validity

• This CMM scale will facilitate deeper understanding of CM processes and programming