Scales of Practices and Outcomes for Community-Engaged Research

Abstract

Despite the growth of research on community-engaged research (CEnR), recent reviews suggest there has been limited development of validated scales to measure key contexts, mechanisms, and outcomes, impairing testing and refinement of theoretical models. The purpose of this study is to present the psychometric properties of scales from the Engage for Equity (E2) project, stemming from a long-term research partnership examining community-engaged research projects. This study used a three-stage, cross-sectional format: a) a sampling frame of 413 CEnR projects was identified; b) 210 principal investigators completed a project-level survey and nominated partners for another survey; c) 457 investigators and partners completed a survey about project contexts, processes, interventions, and outcomes. Factorial validity was established through confirmatory factor analysis supporting seven scales: contextual capacity, commitment to collective empowerment, relationships, community engagement in research actions, synergy, partner and partnership transformation, and projected outcomes. Convergent validity was established through examining covariances among the scales. This study largely yielded results consistent with a previous psychometric study of related measures, while demonstrating improved ceiling effects of the items and refined conceptualization of core theoretical constructs.

The tremendous growth of scholarship focused on community engaged research (CEnR) and community based participatory research (CBPR) over the last 25 years has amplified the demands for validated scales and metrics to better understand how community-academic research partnerships function successfully to achieve shared goals and project outcomes. The increase in demand for these scales and metrics is due, in part, to the growth of interest and funding requirements for CEnR/CBPR research projects and Community Engagement Cores from the National Institutes of Health (NIH), the Centers for Disease Control and Prevention (CDC), the Patient-Centered Outcomes Research Institute (PCORI), and Clinical Translational Science Awards (CTSAs) across academic institutions. As community and academic partners develop shared research interests and goals, these individuals and groups must learn to traverse the dynamics of power relationships, community mistrust due to research abuse by academic institutions, and often-changing priorities to achieve shared project outcomes. Broadly validated, equity-informed quantitative metrics of partnering practices have a role to play in contributing to building this shared understanding, particularly for partnerships, in the context of neoliberal higher education and systemic racism, re-committing to power sharing and capacity building in order to meet the continuing social challenges of pandemic health inequities.

This investigation expands on the empowerment and capacity-building literature that the field of community psychology has been building for decades. The capacity literature has focused primarily on identifying and measuring community capacities (Goodman et al., 1998; Liberato et al., 2011), with empowerment defined at three levels: psychological, organizational, and community (Douglas et al., 2016; Peterson & Zimmerman, 2004; Rappaport, 1987; Wallerstein, 2006; Zimmerman, 2000). This measurement literature has been most extensive for psychological empowerment (Barr et al., 2015; Perkins & Zimmerman, 1995; Speer et al., 2019). In a recent systematic review, however, Cyril et al. (2016) identified significant gaps in scales for community and organizational empowerment. Our study has more specifically sought to examine partnership capacity to promote internal empowerment/power-sharing processes among partners as well as partner (inclusive of organizational partners) and partnership transformation and community empowerment-based social transformation as part of outcomes that are correlated with measures of CBPR/CEnR partnering practices. Relatedly, although some claim participatory action research (PAR), an overlapping framework with CBPR/CEnR (Wallerstein et al., 2018), is still underutilized (Kidd et al., 2018; Wallerstein, in press), community psychology has had a long tradition of espousing PAR (Tebes, 2016, 2018). Amidst this activity, though, partnership measures per se remain under-developed.

Indeed, in recent years within the health field, new tools have begun to be developed to evaluate CBPR/CEnR practices and outcomes (Hamzeh et al., 2019; Robles-Schrader et al., 2019; Srinivas et al., 2015). However, these same publications often recognize the limits, absence, and need for psychometric testing and replication of strong, reliable, and valid measurement and evaluation tools of CBPR and CEnR. Recent systematic reviews reinforce these limitations as well. Tigges et al. (2019) conducted a systematic review that identified 44 measures of quality of research collaboration and 89 different coalition outcomes. However, few were psychometrically tested and validated, thus confirming the need for more valid, reliable measures of collaboration and the general lack of tools available. In their mapping review, Luger et al. (2020) identified 28 measures of context (CEnR conditions and supportive factors for academic/community collaboration), 43 measures of process (group dynamics and trust), and 43 measures of outcomes (benefits/challenges of CEnR impacts). The authors found significant variation in how CEnR partnerships defined and conceptualized these domains, with over half of identified measures being applied to more than one domain. The authors suggested partnerships discuss the metrics they actually need and develop measures among partners to increase clarity. Such local specification of CBPR/CEnR knowledge continues to be vital. In addition, developing quantitative CBPR/CEnR science demands testing and validation of tools and measures that can be used across diverse partnerships to demonstrate which partnering processes are, broadly, most effective for community and academic partners to reach their desired collective outcomes.

In concert with psychometrically valid scales and tools, there is a need for strong conceptual models to guide the selection of such scales and tools. The CBPR model (Belone et al., 2016; Wallerstein et al., 2018), developed with support from Native American Research Centers for Health (NARCH) funding, identified four domains that together map out collaborative processes; research Contexts ground Partnership Processes, which in turn shape Intervention and Research actions, and contribute to intermediate System and Capacity Outcomes and long-term Social Justice Outcomes.

The development of the CBPR conceptual model was followed by the Research for Improved Health (RIH) study, which tested the CBPR conceptual model and identified measures that map along the domains of the CBPR model. This critically important step allowed us to visually display pathways and generate hypotheses about how and if outcomes were impacted by the relational processes within partnerships. Briefly, earlier work in the RIH study resulted in psychometric validation of 22 measures, falling in seven categories, across the four domains of the CBPR model in a diverse, nationally representative sample of federally-funded CBPR/CEnR partnerships (Oetzel et al., 2018; Oetzel et al., 2015). RIH analyses identified promising partnering practices and pathways contributing to outcomes (Duran et al., 2019; Oetzel et al., 2018) and recognized, in particular, practices of culture-centeredness, trust, and challenging power inequities as key to effective practice (Wallerstein, Muhammad et al., 2019; Wallerstein, Oetzel et al., 2019). Specifically, structural equation models identified both relational and structural pathways through partnering practices to outcomes, strengthening the use of the validated CBPR model to theorize how community/academic partners successfully work together to achieve outcomes (Oetzel et al., 2018). A recent scoping meta-review of 100 CBPR/CEnR studies found they also mapped tightly onto the four domains of the CBPR model: context, partnership processes, intervention/research actions, and outcomes (Ortiz et al., 2020).

To build on and expand our nascent understanding of how partnerships successfully worked together to achieve their outcomes, the current study, Engage for Equity (E2), sought to not only refine but extend previous RIH measures. We also conducted an intervention trial of workshops compared to online resources to strengthen power sharing and collective empowerment practices towards partnerships’ desired outcomes (Dickson et al., 2020; Parker et al., 2020; Wallerstein et al., 2020). We hypothesized (and sought to create a new measure of) “Collective Empowerment,” as a key driver of both structural and relational pathways to partnership outcomes, building from the community empowerment definition of people participating collectively, embracing core values for social change, and engaging in Freirian critical reflection and action to gain control over their lives (Allen et al., 1989; Freire, 1970). For our new measure, we added the importance of culture-centeredness, or grounding in community history and ways of knowing, to provide greater support for people’s commitment to working, thinking, and engaging together for social justice (Wallerstein, Oetzel et al., 2019).

In addition, based on psychometric results from the previous study (Oetzel et al., 2015), community consultations (Belone et al., 2016), and consultation with a national advisory group of community and academic researchers (Wallerstein et al., 2020), we included new subscales (n = 5), shortened longer subscales (n = 5), made major revisions to or dropped several subscales (n = 5), and made minor revisions to enhance readability of other subscales (n = 4). The result was seven major scales (with 23 subscales) corresponding to the domains of the CBPR conceptual model: contextual capacity, commitment to collective empowerment, relationships, community engagement in research actions (CERA), synergy, partner and partnership transformation as part of intermediate systems and capacity changes, and longer-term projected outcomes.

Briefly, contextual capacity involves partnership capacity to achieve its goals within the community’s history and bridging capacity of its diverse partners to collaborate across difference. Relationships in the partnership processes domain includes the importance of dialogue and listening for developing and sustaining trust, fostering effective leadership, and resolving conflicts. Commitment to collective empowerment traverses the two model domains of partnership processes regarding partner agency, values, and collective reflection; and community/cultural fit within the intervention and research action domain, building from the community empowerment literature. Also part of the intervention/research domain, the CERA scale identifies the extent of community member participation in each step of the research process, and synergy reflects empowering processes that enable partners to work together effectively. Finally, within the outcomes domain, partner and partnership transformation as intermediate outcomes include personal and agency benefits, sustainability, and growth of community power in the research. Projected outcomes include longer-term changes in policy, social transformation, and health and health equity improvements.

Building on our prior work, this current study provides an analysis of the psychometrics of our current E2 scales with a comparison to previous RIH measures and showcases the inter-relationships among these scales. We hypothesized that the subscales within our seven scales would have factorial validity as evidenced by CFA model fit indices and that the seven scales themselves would have convergent validity as evidenced by positive covariance among the scales. Implications are presented for how these scales and measures may contribute to evolving both collective empowerment theory and the practice of CBPR and CEnR.

Methods

Survey recruitment in the E2 project utilized a three-stage, cross-sectional format. The first stage involved defining the sample of CBPR/CEnR projects with federal funding. The second stage involved principal investigators (PIs) of the selected projects completing a key informant survey and nominating up to six partners to complete a partner-level survey. In the third stage, PIs and the nominated partners completed the community-engagement survey (CES) assessing CBPR contexts, mechanisms, and outcomes. The current study focuses on findings from this third stage, although a brief review of the first two stages is provided. Further details are provided elsewhere (Dickson et al., 2020; Wallerstein et al., 2020). Research ethics approval was provided by the University of New Mexico institutional review board (#16–098).

Research Design and Sampling

Stage one involved an extensive search of databases of federally-funded projects (Dickson et al., 2020). Projects with funding in 2015 from the NIH, CDC-funded Prevention Research Centers, the National Institute of General Medicine-funded Native American Research Centers for Health projects, and PCORI projects were included in the search frame. Projects were included if they had descriptions containing key words related to community engagement (e.g., CBPR, CEnR), had a research grant mechanism (e.g., U or R), and had at least two years of funding remaining. This resulted in a sampling frame of 377 community-academic research partnerships. Subsequently, to further diversify the sampling frame, a convenience sample of 36 newer partnerships mostly associated with three training institutes was added.

Stage two involved the PI responding to a project-level survey called the key informant survey (KIS) (Dickson et al., 2020). PIs were sent an e-mail invitation in 2016–17 (2018 for newer partnerships) to complete this online survey about project structural characteristics (e.g., governance structures, funding, decision-making processes). There was a response rate of 53% from the original sample (179 usable responses), and 86% from the newer partnerships (31 usable responses) to the KIS. For the third and final stage, an invitation to the online community-engagement survey (CES) was sent to 620 PIs and nominated partners in 2016–17. From these invitations, 381 (61%) respondents completed at least 75% of the survey. There were 133 invitations sent in 2018 to the newer partnerships; 76 (57%) respondents completed at least 75% of the survey.

Measures

We measured seven scales in the CES across the CBPR conceptual model domains of contexts, partnerships processes, intervention/research actions, and outcomes. Each of the seven scales incorporates or extends measures from RIH. As mentioned above, revised and newly created subscales for these scales were developed and included in this study based on theoretical and empirical relevance and were derived through community consultation, theorizing about CBPR, and Think Tank engagement (Belone et al., 2016; Wallerstein et al., 2020). Response options for these scales were modified and expanded from RIH to enhance consistency and reduce ceiling effects. In sum, a total of 14 original RIH items were dropped and 24 items were added or changed. Supplemental File 1 provides further detail on these scales, their development, and the specifics of their revision.

In the context domain, we included a capacity scale with three subscales: a) bridging differences—ability to make interpersonal connections across academic and community contexts (3 items; Oetzel et al., 2015); b) partnership capacity—whether the partnership has the skills and resources to achieve its goals (5 items; Khodyakov et al., 2011; Oetzel et al., 2015); c) community history—whether the community has a history of community activism (3 items created for this study).

Partnership processes included two scales: commitment to collective empowerment and relationships. Commitment to collective empowerment included four subscales: a) partnering principles—commitment to follow CBPR principles (7 items; Oetzel et al., 2015 based on Israel et al., 1998); b) community fit—the extent to which the project utilizes community and cultural perspectives, knowledge, and needs in the research (3 items created for this study); c) influence—whether individual partners can affect the work of the partnership (4 candidate items; adapted from Israel et al., 1994 and Oetzel et al., 2015); and d) collective reflection—the extent to which partners reflect on issues around power and privilege and engage in self-evaluation (3 items; created for this study). Relationships included six candidate subscales: a) leadership—whether project has effective leadership (5 candidate items; Oetzel et al., 2015); b) dialogue and listening—the degree to which all partners are actively engaged in the project in a respectful manner (3 items; Oetzel, 2001; Oetzel et al., 2015); c) conflict resolution—whether the partnership uses collaboration to resolve conflict (3 candidate items; Oetzel, 2001; Oetzel et al., 2015); d) trust—whether the partners displayed confidence and trust in each other (3 items; Figueroa et al., 2002); e) partner values—shared understandings of goals and priorities (4 items; Allies Against Asthma, 2003; Oetzel et al., 2015) [later dropped]; and f) resource management—effective use of shared resources (3 items; Khodyakov et al., 2011; Oetzel et al., 2015) [later dropped].

There were two scales for intervention/research actions: community engagement in research actions and synergy. Community engagement in research actions (CERA) was based on the community engagement in research index (Khodyakov et al., 2011) and was extended to include community action and dissemination items. CERA included three subscales: a) background and design—the engagement of community with research design and foundational research (8 candidate items; Oetzel et al., 2015); b) analysis and dissemination—the engagement of community partners with data analysis and sharing research at conferences and in publications (3 items; Oetzel et al., 2015); and c) community action—the involvement of community partners in leveraging research findings for community change (4 items created for this study). Synergy is the extent to which partners are able to integrate work efforts to create more than the sum of the parts. It was measured with five items adapted from a previous scale (Weiss et al., 2002) and adapted in two previous studies (Khodyakov et al., 2011; Oetzel et al., 2015).

For outcomes, there were two scales: partner and partnership transformation and projected outcomes. Partner and partnership transformation had five candidate subscales: a) personal benefits—the degree to which individual partners benefit (4 candidate items; Khodyakov et al., 2011); b) agency benefits—whether the agencies involved benefit or increase capacity and empowerment (3 items; Khodyakov et al., 2011); c) community power in research—the degree to which community partners developed power in engaging in research (5 items; Oetzel et al., 2015 and one new item); d) sustainability—the extent to which the partnership will be maintained over time (3 items; Oetzel et al., 2015); and personal challenges—individual difficulties related to partnership participation [later dropped]. Projected outcomes included four subscales: a) policy—the extent to which the project will likely lead to changes in policy or in the policy environment (4 items; Khodyakov et al., 2011); b) community integration into research—research and practices linked to community needs (3 candidate items created for this study); c) social transformation—degree to which the project will result in broad social changes (3 items; Khodyakov et al., 2011 and Oetzel et al., 2015); and d) health improvement—degree to which the project will result in changes in health behaviors and improved community health (2 items; 1 adapted from Khodyakov et al., 2013 and confirmed in Oetzel et al., 2015 and 1 created for this study).

Data Analysis

Descriptive statistics appropriate to level of measurement were calculated for all measures with scale and subscale scores computed based on means across items. To account for the nesting of partners within partnerships, cluster-robust standard errors were used in the examination of mean differences in scale scores between community and academic partners. Confirmatory factor analysis (CFA) models in this study were fit in two main stages, using consistent estimation techniques and a general strategy seeking to balance a need to fit reasonably parsimonious models with being responsive to the nature of the data. All models were individual level, incorporating standard error estimates that were adjusted upwards to account for the nesting of partners within partnerships, based on polychoric correlation matrices due to the ordinal nature of the response options, and estimated using maximum likelihood estimation due to the desirable asymptotic properties of this estimation technique. In the first stage, each of the seven scales was modeled separately in CFA models including subscales, where applicable, as latent manifestations of a single, underlying, second order construct. Once individual CFA models for each scale were fit, covariances between scales were then examined by fitting paired CFA models that included CFA models for both scales along with a covariance term between the underlying latent constructs. Along with conceptual review, post hoc modification indices were used in a limited way in these models to examine when a model might be substantively improved by adding at most one error covariance term within a scale and when a large modification index for an added path might suggest some lack of discriminant validity. Consistent with Hu and Bentler (1999), “good” CFA model fit was indicated by SRMR < .06, and TLI >.95. Standardized factor loadings in these models were interpreted as correlations and compared across independent samples using Cohen’s q (Cohen, 1988). Missing values were excluded listwise and all statistical analyses were conducted in Stata 16.

Results

Known respondent characteristics of the combined CES sample are shown in Table 1. Both community and academic partners were most likely to be White and female with academic partners substantially more likely to be White than community partners (66% vs. 45%). Aside from Whites, each of the identified race/ethnicity and additional population groups in Table 1 comprised less than 25% of both community and academic partners. Not shown in table, of the 157 of these respondents who identified as PIs or co-PIs, 144 (92%) were academic partners. The demographic disparities present in Table 1 between community and academic partners are consistent with previously observed gaps in concordance between PI demographics and project populations of interest in this sample (Dickson et al., 2020).

Table 1.

Characteristics of survey respondents with known demographics by partner type (N = 446)

	Community partners (N = 182)	Academic partners (N = 264)
Demographic characteristic	n (%)	n (%)
Race/ethnicity
American Indian/ Alaska Native	39 (21)	27 (10)
Asian	6 (3)	30 (11)
Black or African American	41 (23)	33 (13)
Native Hawaiian or Other Pacific Islander	6 (3)	5 (2)
White	81 (45)	173 (66)
Hispanic or Latino	21 (12)	26 (10)
Additional population groups
LGBTQ	18 (10)	18 (7)
Low socio-economic status	36 (20)	15 (6)
Persons with disabilities	18 (10)	9 (3)
Immigrants	19 (10)	33 (13)
Refugees	2 (1)	2 (1)
Gender
Male	51 (28)	66 (25)
Female	131 (72)	198 (75)

In the confirmatory factor analysis models, based upon empirical findings and conceptual review, three candidate subscales and eight candidate items from retained subscales were dropped from scales prior to finalizing models in order improve precision, enhance discriminant validity, and reduce missing-ness. In the relationships scale, partnership values was dropped due to overlap with subscales of Commitment to Collective Empowerment, resource management was dropped due to missing values, the item “resolve conflict among partners” was dropped from leadership due to overlap with conflict resolution, and the item “the dialogue is dominated by the perspectives of the academic partners” was dropped from conflict resolution to enhance precision. Also to enhance precision, the item “suggestions I make within this partnership are seriously considered” was dropped from the influence subscale of Commitment to Collective Empowerment, three items (“integrating community understandings into the research question or approach”, “recruiting study participants”, and “collecting primary data”) were dropped from the background and design subscale of Community Engagement in Research Actions, personal challenges was dropped from Partner and Partnership Transformation, and “changes in clinical practices” was dropped from the community integration into research subscale of Projected Outcomes. The item “enhanced reputation” was dropped from the personal benefits subscale of Partner and Partnership Transformation due to overlap with agency benefits.

Finalized confirmatory factor analysis models presented in Table 2 show, overall, strong fit of items and subscales to scales to support factorial validity of the scales. Values for SRMR were less than .06 for all scales and values for TLI were at least .95 for four out of seven scales and at least .93 for all scales. With a minimum of .88, values for Cronbach’s alpha indicated very good internal consistency within all scales. Nearly all observed factor loadings (96 of 102) were greater than .70, and all factor loadings were greater than .50. Descriptive statistics indicated substantial endorsement of a vast majority of items, with 84 of 110 item, subscale, and scale scores showing means of at least 4.0 on a possible range from 1 to 6, oriented in the positive sense. Only three items, all on the Community Engagement in Research Actions scale, indicated somewhat lower levels of endorsement, showing means of less than 3.0. While some items did show substantial skewness, indicating departures from normality, it should be noted that maximum likelihood estimation produces consistent estimates for factor loadings even in the absence of normality. Overall, mean scale scores were similar between community and academic partners, differing by less than 0.1 for six of the seven scales and by less than two standard errors for all scales. The Community Engagement in Research Actions scale showed the largest difference in mean scale scores with the mean for academic partners exceeding the mean for community partners by 0.17 points on a one to six scale.

Table 2.

Descriptive statistics and confirmatory factory analysis models for items, subscales, and scales of practices and outcomes of community engaged research (N = 457)

SCALE, subscale, or item name with [number of partners, (number of partnerships), Cronbach’s a, and (TLI, SRMR)] included for scales	M (SD)	B (SE)a
CONTEXTUAL CAPACITY [426 (160), .91, (.95, .05)]	4.34 (0.88)	−.06 (.07)
Bridging differences	4.11 (0.99)	.88 (.05)
1. The community partners have the knowledge, skills, and confidence to interact effectively with the academic partners.	4.25 (1.15)	.90 (.04)
2. The academic partners have members who are from a similar background as the community partners.	3.63 (1.32)	.54 (.07)
3. The academic partners have the knowledge, skills, and confidence to interact effectively with the community partners.	4.46 (1.09)	.72 (.05)
Error covariance between academic partners items		.47 (.07)
Community history	4.19 (1.27)	.75 (.05)
1. History of organizing services or events.	4.31 (1.37)	.87 (.02)
2. History of advocating for social or health equity.	4.18 (1.44)	.90 (.02)
3. Previously influenced decisions that affected their communities.	4.08 (1.38)	.88 (.02)
Partnership capacity	4.72 (0.89)	.86 (.05)
1. Skills and expertise	4.78 (1.02)	.76 (.04)
2. Diverse members	4.57 (1.14)	.69 (.04)
3. Legitimacy and credibility in the community	4.75 (1.09)	.87 (.02)
4. Ability to bring people together for meetings/activities	4.72 (1.06)	.94 (.01)
5. Connections to relevant stakeholders	4.77 (1.04)	.89 (.02)
COMMITMENT TO COLLECTIVE EMPOWERMENT [415 (167), .93, (.95, .03)]	4.79 (0.76)	−.07 (.08)
Partnering principles	4.62 (0.95)	.96 (.07)
1. This project builds on resources and strengths in the community.	4.69 (1.08)	.82 (.03)
2. This project facilitates equitable partnerships in all phases of the research.	4.31 (1.19)	.87 (.02)
3. This project helps all partners involved to grow and learn from one another.	4.63 (1.15)	.86 (.02)
4. This project balances research and social action for the mutual benefit of all partners.	4.37 (1.22)	.84 (.03)
5. This project emphasizes the factors that are important to the community (e.g., environmental and social factors) which affect well-being.	4.72 (1.13)	.84 (.03)
6. This project communicates knowledge and findings to all partners and involves all partners in the dissemination process.	4.60 (1.14)	.83 (.03)
7. This project views CBPR or community engaged research as a long term process and a long term commitment.	5.02 (1.11)	.75 (.04)
Community fit	4.65 (1.06)	.85 (.08)
1. This project is responsive to community histories.	4.62 (1.18)	.87 (.02)
2. This project integrates the words and language of the community.	4.69 (1.18)	.89 (.02)
3. This project connects with the ways things are done in the community.	4.64 (1.12)	.94 (.01)
Error covariance between Partnering principles and Community fit		.57 (.29)
Influence in the partnership (Voice)	5.19 (0.79)	.51 (.07)
1. I have influence over decisions that this partnership makes.	5.21 (0.90)	.90 (.02)
2. My involvement influences the partnership to be more responsive to the community.	5.08 (0.93)	.73 (.04)
3. I am able to influence the work on this project.	5.28 (0.87)	.95 (.02)
Collective Reflection	4.70 (0.99)	.78 (.07)
1. Our partnership has discussions about our role in promoting strategies to address social and health equity.	4.92 (1.09)	.82 (.04)
2. Our partnership evaluates together what we’ve done well and how we can improve our collaboration.	5.02 (1.03)	.82 (.04)
3. Our partnership reflects on issues of power and privilege within our partnership.	4.17 (1.44)	.72 (.05)
RELATIONSHIPS [422 (167), .94, (.95, .04)]	5.23 (0.62)	−.04 (.06)
Leadership	4.75 (1.00)	.84 (.04)
1. Encourage active participation of academic and community partners in decision making	4.73 (1.17)	.89 (.02)
2. Communicate the goals of the project	4.78 (1.06)	.89 (.02)
3. Foster respect between partners	4.93 (1.03)	.94 (.01)
4. Help the partners be creative and look at things differently	4.58 (1.18)	.91 (.02)
Dialogue and listening	5.40 (0.61)	.85 (.04)
1. We show positive attitudes towards one another.	5.55 (0.62)	.91 (.02)
2. Everyone in our partnership participates in our meetings.	5.15 (0.85)	.75 (.04)
3. We listen to each other.	5.50 (0.62)	.95 (.01)
Conflict resolution	5.41 (0.67)	.90 (.03)
1. When conflicts occur, we work together to resolve them.	5.39 (0.73)	.93 (.02)
2. Even when we don’t have total agreement, we reach a kind of consensus that we all accept.	5.44 (0.70)	.90 (.02)
Error covariance between Dialogue and listening and Conflict resolution		.67 (.10)
Trust	5.35 (0.63)	.82 (.04)
1. I trust the decisions others make about issues that are important to our projects.	5.30 (0.73)	.79 (.03)
2. I can rely on the people that I work with on this project.	5.44 (0.70)	.88 (.02)
3. People in this partnership have a lot of confidence in one another.	5.30 (0.74)	.94 (.02)
COMMUNITY ENGAGEMENT IN RESEARCH ACTIONS [397 (160), .95, (.94, .04)]	3.50 (1.23)	.17 (.11)
Background & Design	3.27 (1.22)	.92 (.02)
1. Grant proposal writing	2.92 (1.46)	.79 (.04)
2. Background research	2.89 (1.47)	.76 (.04)
Error covariance between grant proposal writing and background research		.50 (.07)
3. Developing sampling procedures	3.11 (1.52)	.79 (.04)
4. Designing and implementing the intervention	3.92 (1.54)	.78 (.04)
5. Designing data collection instruments (such as interviews or surveys)	3.51 (1.52)	.80 (.03)
Analysis & Dissemination	3.33 (1.38)	1.0 (.00)
1. Interpreting study findings	3.46 (1.55)	.87 (.02)
2. Writing reports and journal articles	2.94 (1.51)	.90 (.02)
3. Giving presentations at meetings and conferences	3.60 (1.55)	.86 (.02)
Community Action	3.91 (1.41)	.86 (.03)
1. Informing the community about research progress and findings	4.11 (1.46)	.93 (.01)
2. Informing relevant policy makers about findings	3.62 (1.61)	.89 (.02)
3. Sharing findings with other communities	3.82 (1.56)	.92 (.01)
4. Producing useful findings for community action and benefit	4.08 (1.49)	.92 (.01)
SYNERGY [438 (166), .94, (.94, .02)]	4.65 (0.97)	−.09 (.09)
1. Develop goals that are widely understood and supported in this partnership	4.58 (1.10)	.91 (.02)
2. Develop strategies that are most likely to work for the community or stakeholders as a whole	4.61 (1.07)	.93 (.01)
3. Recognize challenges and come up with good solutions	4.67 (1.03)	.92 (.01)
4. Respond to the needs and problems of your constituency or community as a whole	4.54 (1.11)	.91 (.02)
5. Work together well as a partnership	4.84 (1.04)	.89 (.02)
PARTNER AND PARTNERSHIP TRANSFORMATION [413 (165), .88, (.96, .03)]	4.47 (0.75)	−.01 (.07)
Personal benefits	3.79 (1.21)	.59 (.08)
1. Increased use of your expertise or services by others	4.20 (1.24)	.86 (.03)
2. Increased ability to acquire additional financial support	3.77 (1.40)	.77 (.04)
3. Increased ability to seek formal or informal education	3.39 (1.63)	.75 (.04)
Agency benefits	4.25 (1.07)	.63 (.07)
1. Enhanced reputation	4.33 (1.15)	.88 (.02)
2. Enhanced ability to affect public policy	4.09 (1.23)	.86 (.03)
3. Increased use of the agency’s expertise or services by others	4.34 (1.17)	.93 (.02)
Error covariance between Personal and Agency benefits		.53 (.08)
Community power in research	4.99 (0.73)	.80 (.07)
1. Have increased participation in the research process	4.97 (0.93)	.75 (.04)
2. Are able to talk about the project with groups or in other settings, such as community or political meetings	4.99 (0.87)	.85 (.03)
3. Can apply the findings of the research to practices and programs in the community	4.84 (0.95)	.86 (.03)
4. Can voice their opinions about research in front of researchers	5.08 (0.93)	.79 (.03)
5. Have the capacity or power to promote research that will benefit the community	5.05 (0.83)	.78 (.03)
Sustainability	4.86 (0.91)	.78 (.07)
1. I am committed to sustaining the community-academic relationship with no or low funding.	5.07 (1.07)	.73 (.05)
2. This project is likely to continue forward after this funding is over.	4.58 (1.31)	.77 (.05)
3. Our partnership carefully evaluates funding opportunities to make sure they meet both community and academic partners’ needs.	4.93 (1.06)	.67 (.06)
PROJECTED OUTCOMES [415 (166), .94, (.93, .03)]	4.02 (0.98)	−.06 (.10)
Policy	3.94 (1.00)	.95 (.02)
1. Better coordination between agencies, researchers, and community groups	4.10 (1.11)	.86 (.02)
2. Changes in the nature of debates about important health issues in the community	3.85 (1.17)	.89 (.02)
3. Useful findings for the development of community practices, programs, or policies	4.30 (1.09)	.88 (.02)
4. Changes in policy	3.51 (1.25)	.75 (.04)
Community integration into research	4.43 (1.08)	.81 (.03)
1. Improved academic ability to integrate community perspectives into research design and methods	4.39 (1.12)	.92 (.02)
2. Research better linked to community needs	4.47 (1.12)	.97 (.02)
Social transformation	3.68 (1.29)	.92 (.02)
1. Reinforced cultural identity or pride	3.78 (1.34)	.84 (.03)
2. Broad social impacts	3.66 (1.56)	.91 (.02)
3. Better overall environment in the community	3.59 (1.39)	.89 (.02)
Health improvement	4.05 (1.14)	.79 (.04)
1. How much do you think this project will improve the health of the community?	4.12 (1.17)	.96 (.02)
2. How much do you think this project will improve the health behaviors of community members?	3.98 (1.23)	.88 (.03)

Notes: TLI = Tucker-Lewis Index, SRMR = Standardized Root Mean square Residual; item, subscale, and scale scores have been linearly transformed to range from 1 (e.g., not at all; completely disagree) to 6 (e.g., to a complete extent; completely agree) for descriptive purposes in this table.

^aEstimates represent factor loadings and associated standard errors for subscales and items. For scale scores, estimates represent overall differences in means between academic partners and community partners and associated standard errors.

Standardized covariance estimates between scales in Table 3 show, as expected, a large degree of convergence among these scales of practices and outcomes of community engaged research. Across these correlations, the minimum observed value, between Relationships and Community Engagement in Research Actions, was .47, and the maximum observed value, between Commitment to Collective Empowerment and Partner and Partnership Transformation, was .97. An 11 out of 21 majority of correlations were at least .70, and only three correlations were greater than .90. Commitment to Collective Empowerment showed an especially high degree of convergence with other scales, having the highest or second highest observed correlation with each other scale. These findings support the convergent validity of these scales and confirm Commitment to Collective Empowerment as a key correlate of processes and outcomes of community engaged research. It should be noted, however, that the large values for some of these observed correlations did indicate some lack of discriminant validity.

Table 3.

Standardized covariance estimates between scales of practices and outcomes of community engaged research (N = 457)

	CCE	Relationships	CERA	Synergy	PPT	Projected outcomes
Scale	B (SE)	B (SE)	B (SE)	B (SE)	B (SE)	B (SE)
Contextual capacity	.72 (.06)	.59 (.07)	.55 (.07)	.68 (.05)	.78 (.06)	.56 (.07)
Commitment to collective empowerment (CCE)		.91 (.04)	.68 (.06)	.89 (.02)	.97 (.04)	.78 (.04)
Relationships			.47 (.07)	.85 (.03)	.77 (.05)	.61 (.06)
Community engagement in research actions (CERA)				.49 (.06)	.70 (.06)	.57 (.06)
Synergy					.81 (.05)	.68 (.05)
Partner and Partnership Transformation (PPT)						.94 (.03)

Overall, the results presented in Table 4 show lower ceiling effects and fairly consistent factor loadings when fitting common models across the 57 of 81 current E2 items in Table 1 that were substantially similar to items from the corresponding previous RIH study survey. In nearly all observed cases (53 of 57) the percentage of respondents endorsing the maximum possible response option was lower in the current E2 study than in the previous RIH study. Consistent with a reduction of correlation attenuation due to increased measurement sensitivity, a vast majority (54 of 73) of observed factor loadings were larger in models fit on current E2 data than in models fit on previous RIH data. Overall, however, such differences in observed factor loadings were not large, and Cohen’s q was greater than 0.5 in only 7 of 73 cases.

Table 4.

Comparison of ceiling effects and factor loadings between Engage for Equity (N = 457) and Research to Improve Health (N = 450)

Item within SCALE and subscale	RIH ceiling n (%)	E2 ceiling n (%)	RIH B (SE)	E2 B (SE)	RIH to E2 Cohen’s q
CONTEXTUAL CAPACITY
Bridging differences			.81 (.06)	.85 (.05)	0.12
Item 1	188 (43)	74 (17)	.77 (.05)	.84 (.04)	0.19
Item 2	159 (36)	49 (11)	.52 (.07)	.57 (.06)	0.07
Item 3	229 (52)	88 (20)	.82 (.05)	.77 (.04)	0.14
Error covariance term			.41 (.09)	.41 (.08)	0.00
Partnership capacity			.90 (.07)	.94 (.05)	0.24
Item 1	267 (61)	122 (28)	.70 (.05)	.76 (.04)	0.13
Item 2	215 (49)	108 (25)	.59 (.06)	.69 (.04)	0.18
Item 3	300 (68)	126 (29)	.79 (.04)	.87 (.02)	0.27
Item 4	244 (56)	118 (27)	.77 (.04)	.94 (.01)	0.74
Item 5	195 (44)	125 (29)	.76 (.04)	.89 (.02)	0.43
COMMITMENT TO COLLECTIVE EMPOWERMENT
Partnering principles			.69 (.08)	.75 (.06)	0.12
Item 1	267 (61)	107 (25)	.71 (.04)	.81 (.03)	0.25
Item 2	187 (43)	80 (18)	.83 (.03)	.88 (.02)	0.18
Item 3	246 (56)	114 (26)	.86 (.03)	.86 (.02)	0.02
Item 4	207 (47)	86 (20)	.81 (.03)	.84 (.03)	0.10
Item 5	268 (61)	136 (31)	.81 (.03)	.84 (.03)	0.09
Item 6	226 (51)	113 (26)	.76 (.04)	.83 (.03)	0.19
Item 7	305 (69)	198 (45)	.74 (.04)	.75 (.04)	0.01
Influence in the partnership (Voice)
Item 1	119 (27)	170 (39)	.49 (.06)	.61 (.05)	0.17
RELATIONSHIPS
Leadership			.98 (.03)	.84 (.04)	1.15
Item 1	205 (48)	126 (30)	.90 (.02)	.89 (.02)	0.05
Item 2	217 (50)	128 (30)	.86 (.02)	.89 (.02)	0.15
Item 3	245 (57)	145 (34)	.93 (.01)	.94 (.01)	0.04
Item 4	175 (41)	106 (25)	.84 (.03)	.91 (.02)	0.31
Dialogue and listening			.85 (.03)	.85 (.04)	0.02
Item 1	301 (70)	235 (56)	.92 (.02)	.91 (.02)	0.05
Item 2	167 (39)	134 (32)	.74 (.04)	.75 (.04)	0.03
Item 3	245 (57)	219 (52)	.94 (.02)	.95 (.01)	0.08
Conflict resolution			.65 (.06)	.90 (.03)	0.70
Item 1	181 (42)	197 (47)	.92 (.04)	.93 (.02)	0.09
Item 2	177 (41)	209 (50)	.80 (.04)	.90 (.02)	0.40
Error covariance term			.44 (.10)	.67 (.10)	0.33
Trust			.84 (.04)	.82 (.04)	0.06
Item 1	187 (43)	155 (37)	.81 (.03)	.79 (.03)	0.06
Item 2	244 (57)	205 (49)	.88 (.03)	.88 (.02)	0.02
Item 3	211 (49)	158 (37)	.89 (.02)	.94 (.02)	0.28
COMMUNITY ENGAGEMENT IN RESEARCH ACTIONS
Background & Design			.61 (.07)	.92 (.02)	0.89
Item 1	120 (27)	25 (6)	.66 (.06)	.79 (.03)	0.29
Item 2	106 (24)	26 (6)	.52 (.08)	.76 (.04)	0.44
Error covariance term			.37 (.08)	.50(.07)	0.16
Item 3	171 (39)	36 (9)	.58 (.07)	.79 (.03)	0.41
Item 4	313 (71)	79 (19)	.42 (.08)	.77 (.04)	0.58
Item 5	202 (46)	54 (13)	.73 (.06)	.80 (.03)	0.18
Analysis & Dissemination
Item 1	155 (35)	48 (12)	.79 (.04)	.88 (.02)	0.32
Item 2	136 (31)	31 (8)	.91 (.03)	.91 (.02)	0.02
Item 3	230 (52)	58 (14)	.75 (.04)	.85 (.03)	0.26
SYNERGY
Item 1	270 (61)	91 (21)	.88 (.02)	.91 (.02)	0.15
Item 2	251 (57)	99 (23)	.87 (.02)	.93 (.01)	0.30
Item 3	235 (53)	98 (22)	.92 (.02)	.92 (.01)	0.00
Item 4	217 (49)	90 (21)	.91 (.02)	.91 (.02)	0.03
Item 5	285 (64)	135 (31)	.84 (.03)	.89 (.02)	0.19
PARTNER AND PARTNERSHIP TRANSFORMATION
Personal benefits			.85 (.07)	.60 (.08)	0.57
Item 1	123 (30)	68 (16)	.78 (.05)	.87 (.03)	0.26
Item 2	57 (14)	48 (12)	.64 (.06)	.77 (.04)	0.25
Item 3	103 (25)	53 (13)	.54 (.07)	.75 (.04)	0.37
Agency benefits			.76 (.05)	.64 (.07)	0.22
Item 1	106 (25)	64 (15)	.90 (.02)	.88 (.02)	0.13
Item 2	63 (15)	51 (12)	.77 (.04)	.86 (.03)	0.26
Item 3	108 (26)	69 (17)	.92 (.02)	.93 (.02)	0.01
Error covariance term			.78 (.13)	.53 (.09)	0.45
Community power in research			.85 (.05)	.80 (.07)	0.15
Item 1	170 (41)	110 (27)	.83 (.03)	.75 (.04)	0.23
Item 2	192 (46)	104 (25)	.84 (.03)	.87 (.03)	0.12
Item 3	141 (34)	93 (22)	.73 (.04)	.86 (.03)	0.36
Item 4	193 (46)	140 (34)	.74 (.04)	.76 (.04)	0.05
Sustainability			.87 (.05)	.77 (.07)	0.31
Item 1	169 (41)	162 (39)	.71 (.05)	.73 (.05)	0.04
Item 2	107 (26)	108 (26)	.69 (.05)	.77 (.05)	0.17
Item 3	161 (39)	134 (32)	.79 (.04)	.66 (.06)	0.28
PROJECTED OUTCOMES
Policy
Item 4	33 (8)	23 (5)	.53 (.09)	.74 (.05)	0.36
Social transformation
Item 3	65 (15)	40 (9)	.98 (.14)	.90 (.04)	0.83
Health improvement
Item 1	99 (23)	52 (12)	.41 (.09)	.71 (.05)	0.45

Note: RIH = Research to Improve Health study; E2 = Engage for Equity study

Discussion

The major contribution of this study was to provide empirical data about the psychometric properties of scales of CBPR and community engagement contexts, mechanisms, and outcomes. CBPR and CEnR are frequently used in a variety of research contexts to improve health outcomes and health equity, particularly in underserved, marginalized communities (Wallerstein et al., 2018). Several large-scale initiatives (e.g., PCORI, NCI Comprehensive Cancer Centers, and CTSAs) have mandated or encouraged the use of community engagement to promote the translation of research amidst a noted lack of valid and reliable scales of community engagement to facilitate robust evaluation (Ahmed et al., 2019; Boivin et al., 2018; Sprague Martinez et al., 2019). Indeed, researchers have argued that the science of CPBR/CEnR (i.e., why it works and under what conditions) can only be advanced with strong measurement of key constructs, though reviews of engagement constructs over the past decade have continued to find that many scales employed in CEnR/CBPR studies lack basic psychometric properties (Boivin et al., 2018; Luger et al., 2020; Sandoval et al., 2012; Tigges et al., 2019). There are some exceptions to this finding (Hall et al., 2008; Oetzel et al., 2015; Treitler et al., 2018) and the current study further contributes opportunities to measure and evaluate various aspects of community engagement with robust scales.

The current scales and subscales have been developed over a rigorous, long-term process. The original subscales in the previous RIH study were developed through a thorough review of literature, development of a conceptual model, and engagement with expert Think Tank members (Hicks et al., 2012; Oetzel et al., 2015). The current subscales provide further validation of 16 of the previous RIH subscales: bridging differences, partnership capacity, partnering principles, influence, leadership, dialogue and listening, conflict resolution, trust, community engagement in research actions (background & design and analysis & dissemination), synergy, personal benefits, agency benefits, community power in research, sustainability, and social transformation. One of these subscales was significantly modified from the previous RIH study (influence) and some reflect combinations of previous subscales (partnering principles and community engagement in research actions).

The inclusion of a new large-scale sample of CBPR/CEnR partnerships and generally consistent results on common items between the two RIH and E2 studies provides empirical support for the robustness of these scales. Further, the current E2 study includes seven new subscales: community history, community fit, collective reflection, community action, projected policy outcomes, projected community integration into research, and projected health impacts. The latter three are outcomes-focused, a type of measure which was particularly identified as lacking from a recent review of scales for research collaborations (Tigges et al., 2019). Notably, some lack of discriminant validity among these scales of processes and outcomes, while challenging from a measurement perspective and necessarily impairing efforts to disentangle any singular contributions of current partnership processes to later outcomes, is also consistent with the character of the CBPR conceptual model, which was developed with a recognition of the existence of reciprocal and cross-cutting linkages interwoven throughout its domains (Belone et al., 2016).

The findings from this study also allow for a renewed perspective on the relative importance of potential drivers as well as theories of change embedded in the work of community-academic research partnerships. In particular, the new commitment to collective empowerment scale, which is a strong correlate of the other six scales in this study, reflects theoretical foundations of CBPR in the literature of the Global South, particularly the action-reflection praxis of Paulo Freire (Freire, 1970), and builds from other community empowerment literature, adding fit to culture and community ways of knowing. Crucially, both the commitment to collective empowerment and community engagement in research actions scales are associated with practices of structural community governance (Sanchez-Youngman et al., in press), suggesting these scales as important focal intermediate measures for building an understanding of the impacts of structural changes in partnership practices on eventual outcomes, particularly given the relative gaps in community engagement in research actions scores observed in this study.

Limitations and Future Directions

The current study is not without limitations. The study is based on the perceptions of a small number of partner participants, selected by a PI, leading to possible selection bias, especially in the context of continued gaps between PI demographics and the demographics of focal communities of interest to community engaged researchers (Dickson et al., 2020). Further, self-reports about CBPR processes and outcomes may not reflect actual contexts, processes, and outcomes; we were not able to corroborate reported perceptions with different types of data (e.g., actual evaluation data from each project). As may be suggested by descriptive statistics indicating substantial endorsement across items, social desirability and individual and systemic unconscious biases could have, indeed, led to problematic over-estimation of these measures of equity-based practices and authentic partnering and reduced the ability of these measures to meaningfully differentiate between partnerships. Also, we did not look at issues around measurement invariance of these scales (e.g., between community and academic partners or by privileged vs. marginalized identity categories) in this study and acknowledge the promise of such analyses to furthering our understanding of the dynamics of CBPR/CEnR partnerships in future work.

In addition to these limitations, this study, while meeting an articulated demand for quantitative tools from key large-scale initiatives, is also prone to the epistemological limitations common to its form of science as with any single method. Psychometric measurement is but one way of knowing about the contexts, mechanisms, and outcomes of community-academic research partnerships. Partnerships committed to co-learning embedded in an ecology of knowledges (Santos, 2007) will need to assess, together, the knowledge and possibilities they gain from the use of these tools with the knowledges and connected ways of being that they already possess, but may have yet to acknowledge. Indeed, such partnerships may need to draw on a praxis of honoring syntheses of strengths across knowledge systems, consistent with “Two-Eyed Seeing”, a term coined by Mi’kmaq elder Albert Marshall (Forbes et al., 2020).

There are several future directions from this study. First, additional testing of the CBPR conceptual model is possible with the revised scales. Such testing can help further validate it as a useful and practical model of community-academic research partnerships. Second, additional scales can be developed to further our understanding and theorizing about community engagement. For example, Israel, Brush, and colleagues (2020) are engaging in a study to develop scales for long-term partnerships (those existing for more than six years). Such scales may be helpful in identifying what makes a partnership sustainable. As another example, the work of Miranda Fricker (2007) points to collective hermeneutical resources as a key dimension of epistemic justice, suggesting the importance, alongside commitments to collective empowerment, of testing measures of meaning-making in CPBR partnerships. Third, recognizing some lack of discriminant validity, we are currently developing a shorter instrument based on brief scales which may be more practical for rapid-cycle data collection, analysis, and actionable use for partnerships interested in evaluations geared towards quality improvement and recalibration of actions to meet moving goals. Finally, longitudinal research would be helpful to test how current scales of CBPR partnering processes predict future capacity building, empowerment, and outcomes.

Conclusions

In conclusion, the current study developed and/or tested seven scales and 23 subscales related to the CBPR conceptual model. There was good factorial validity in a majority of these scales and strong convergent validity among the scales along with good internal consistency and reduced ceiling effects. This level of evidence exceeds most of what is found in the CBPR and CEnR literature and thus can be helpful in advancing the science of CBPR. The research design for the study yielded adequate response rates for two stages of surveys across a total of 210 partnerships, supporting external validity of these scales. Overall, this study further establishes measurement validity of scales that have been developed through a long-term and rigorous process to help advance the practice and science of CBPR and CEnR to achieve goals of health and social equity, while highlighting the importance of commitments to collective empowerment. Collectively, our seven scales provide an example of the type of theoretical and empirical foundation that Luger and colleagues (2020) called for to strengthen the measurement foundation of CEnR.