A Tale of Two Community Networks Program Centers: Operationalizing and Assessing CBPR Principles and Evaluating Partnership Outcomes

Abstract

Background

Community Networks Program (CNP) centers are required to use a community-based participatory research (CBPR) approach within their specific priority communities. Not all communities are the same and unique contextual factors and collaborators’ priorities shape each CBPR partnership. There are also established CBPR and community engagement (CE) principles shown to lead to quality CBPR in any community. However, operationalizing and assessing CBPR principles and partnership outcomes to understand the conditions and processes in CBPR that lead to achieving program and project level goals is relatively new in the science of CBPR.

Objectives

We sought to describe the development of surveys on adherence to and implementation of CBPR/CE principles at two CNP centers and examine commonalities and differences in program- versus project-level CBPR evaluation.

Methods

A case study about the development and application of CBPR/CE principles for the Missouri CNP, Program for the Elimination of Cancer Disparities, and Minnesota CNP, Padres Informados/Jovenes Preparados, surveys was conducted to compare project versus program operationalization of principles. Survey participant demographics were provided by CNP. Specific domains found in CBPR/CE principles were identified and organized under an existing framework to establish a common ground. Operational definitions and the number of survey items were provided for each domain by CNP.

Conclusion

There are distinct differences in operational definitions of CBPR/CE principles at the program and project levels of evaluation. However, commonalities support further research to develop standards for CBPR evaluation across partnerships and at the program and project levels.

CE is a powerful vehicle for bringing about changes that can improve community health and well-being¹; engaging community members in the research process is often the missing link to improving the quality and outcomes of health promotion activities, disease prevention initiatives, and research studies.^2,3 CE requires a long-term process that builds trust, values the contributions of all stakeholders, and generates a collaborative framework.⁴ Engaging marginalized communities to address identified health concerns requires establishing a rapport and maintaining a consistent presence.^5,6

CBPR is effective in abating issues of mistrust by engaging minority and underserved communities as true partners in the research process.^7,8 CBPR is an approach to research that focuses on the development of an academic–community partnership and is often used by universities to engage community stakeholders and address priority public health concerns.^7,9–11 Using the principles of co-learning, mutual benefit, and community participation, among others,^11,12 researchers elucidate and address identified public health concerns by working collaboratively with communities as true partners and not on communities as research participants.¹³ Moreover, community-based interventions have demonstrated the potential to be powerful tools in reducing health disparities,^14,15 particularly when they address the systematic, environmental, and community-level factors that impact health.¹⁶ CBPR and CE principles (Table 1) can be used to 1) promote collaboration and participation at each stage of the research process, 2) ensure that research projects are community driven, and 3) and disseminate useful results in a culturally appropriate forum.^{10,12,17–19} Given the importance of CBPR, emphasis must be placed on how to assess adherence to and implementation of CBPR and CE principles.

Table 1.

Community-Based Participatory Research and Community Engagement Principles

Community-Based Participatory Research

Recognizes community as a unity of identity. Build on strengths and resources within the community. Facilitates collaborative, equitable partnerships in all phases of the research. Promotes co-learning and capacity building among all partners. Integrates and achieves balance between research and action for the mutual benefit of all partners. Emphasizes local relevance of public health problems and ecological perspectives that recognize and attend to the multiple determinants of health and disease. Involves systems development through a cyclical and iterative process. Disseminates findings and knowledge gained to all partners and involves all partners in the dissemination process. Involves a long-term process and commitment.

Community Engagement

Be clear about the population/communities to be engaged and the goals of the effort. Know the community, including its norms, history, and experience with engagement efforts. Build trust and relationships and get commitments from formal and informal leadership. Collective self-determination is the responsibility and right of all community members. Partnering with the community is necessary to create change and improve health. Recognize and respect community cultures and other factors affecting diversity in designing and implementing approaches. Sustainability results from mobilizing community assets and developing capacities and resources to make decisions and take action. Be prepared to release control of actions or interventions to the community and be flexible enough to meet its changing needs. Community collaboration requires long-term commitment by the engaging organization and its partners.

Evaluating partnerships to assess their authenticity in relation to application of CBPR principles has been emphasized in the literature.^10,18,20 However, evaluating partnerships for the purpose of understanding the conditions and processes within CBPR that lead to achieving project goals, such as increased capacity for CBPR and successful research implementation and outcomes, is recent in the science of CBPR. Structured reviews of current tools to assess partnerships have identified gaps in the science of CBPR in that they largely focus on group dynamics, with less attention paid to context or outcomes, and many lack rigorous validity testing.²¹ Furthermore, they fail to consider that, across CBPR projects, there may be multiple purposes for assessment depending on the individual project goals and outcomes. For example, one program-level purpose may be focused on understanding whether university and community efforts to work in collaboration to improve community health lead to systems and capacity development; a second project-level purpose may be premised on the assumption that better collaboration contributes to a shared understanding of, belief in, and commitment to a CBPR project, leading to quality implementation of research projects and therefore improved outcomes.

As CBPR/CE programs and projects are challenged to demonstrate the impact of CBPR and CE on health outcomes, the development of standardized measures of adherence to and implementation of CBPR and CE principles is critical; however, there is a definite need to maintain an appreciation for the different contexts within which CBPR/CE measures are needed across sites, projects, and programs. This is especially important for demonstrating the impact of CPBR and CE on health as a result of funding initiatives that specifically call for the use of CBPR and CE in practice, research, and training.

CNP CENTERS

Currently, there are 23 CNP National and Regional Centers for Reducing Cancer Health Disparities, funded by the National Cancer Institute’s (NCI) Center to Reduce Cancer Health Disparities. Using a CBPR approach, CNP centers are focused on eliminating cancer disparities in their respective priority communities by engaging them to address needs for cancer prevention and control, and conducting action research to improve health outcomes. It is within this framework that CNP Centers work to meet three main goals: 1) to increase knowledge, access, and use of prevention measures and treatment options to reduce cancer disparities in priority populations, 2) to perform CBPR interventions to promote prevention and treatment, and 3) to train qualified health disparities researchers in the CBPR approach and promote their career development.

Missouri CNP: Program for the Elimination of Cancer Disparities

The Program for the Elimination of Cancer Disparities (PECaD) of the Siteman Cancer Center at Barnes-Jewish Hospital and Washington University School of Medicine was established in 2003 with institutional funds in response to a known excess cancer burden within the region and the state, particularly in minority and medically underserved populations. PECaD was one of 25 CNPs funded from 2005 to 2010 by NCI CRHD (U01-CA114594) with additional funding from the Siteman Cancer Center (NCI-P30-CA91842). When PECaD received its second round of funding in 2010 (U54 CA153460), new partners were invited to strengthen overall efforts. PECaD’s site-specific (breast, colorectal, and prostate) cancer community partnerships foster ongoing dialogue with community stakeholders, including individuals and community organizations in the region. Each partnership works to refine program strategies that are designed to reduce and ultimately eliminate cancer disparities. The partnerships create an avenue through which community cancer needs and priorities can be reflected in the implementation of PECaD activities. Members of each partnership consist of cancer survivors and advocates, representatives from community health care organizations, representatives of community-based organizations, and academic faculty members and staff. PECaD’s target population is African Americans and low-income communities, and primarily serves St. Louis City, St. Louis County, East St. Louis/St. Clair County, Illinois, and the rural Bootheel region of Missouri.

PECaD’s core research project is a systems-level intervention to increase colorectal cancer screening in community health centers. This project works with safety net health centers in St. Louis City and St. Louis County in Missouri, in East St. Louis/St. Clair County in Illinois, and in the Bootheel region of Missouri. This project tests the effectiveness of community health center-selected systems-level, evidence-based interventions for increasing rates of CRC screening. The control condition is usual care, but the study uses a cluster randomized delayed start so, in concordance with community partner wishes, clinics in the control group will have access to the intervention after the intervention clinics. The primary outcome of this study is CRC screening adherence measured by self-report surveys of a random sample of health center patients. The evaluation of the project is informed by a chart audit to assess screening referral and completion, and interviews with physicians, staff, and administrators at the health centers. This study was developed and is being conducted adhering to principles of CBPR working with the PECaD Colorectal Cancer Community Partnership.

Minnesota CNP: Padres Informados/Jovenes Preparados Multisite Participatory Community Trial

The Minnesota CNP Center for Eliminating Cancer Disparities’ (CECD) was established in 2010 with the start of their CNP funding. The core partners have been working together for 8 years and include Latino medical and mental health providers, social workers, health outreach workers, parent educators, and researchers from the University of Minnesota Program in Health Disparities Research, University of Minnesota Extension, and key organizations serving the local Latino community, namely,– Aquí Para Ti/ Here For You and Centro, Inc. The collaboration adheres to principles of CBPR in its work together by recognizing and emphasizing the unique contributions and perspectives of all partners to the design, implementation, and dissemination stages of the study. At the time of planning for the Minnesota CNP grant, core partners invited five additional community partners into the collaboration. Although these partners were prepared to participate in a CBPR process, they were not required to collaborate to the extent of the core partners. Therefore, the depth of participation in core CBPR processes varied across sites, potentially affecting the quality of collaboration and implementation of the project. CECD’s target population is immigrant Latino parents of adolescent children and serves two areas with large Latino immigrant communities in Minnesota: the Twin Cities of Minneapolis and St. Paul, and the rural areas of Central Minnesota.

Padres Informados/Jovenes Preparados (PI/PJ; “Informed Parents/Prepared Youth”) is the core research project of the CECD. In developmental and pilot phases, PI/JP has been funded through the American Cancer Society (Cancer Control Career Development Award), and Clearway Minnesota (RC–2007–0032). As the core research project, PI/JP is a 5-year multisite participatory CBPR intervention trial testing the effectiveness of an eight-session, collaboratively developed program directed at preventing tobacco and other substance use intentions among Latino youth ages 10 to 14 years old. PI/JP is delivered in community settings by trained community staff. PI/JP aims to develop strong parenting practices and facilitate relationship building between parents and youth while emphasizing Latino cultural values, navigation through multiple cultures, and environmental risks related to socioeconomic circumstances. The design is a randomized, controlled trial with a delayed treatment control condition.² A detailed description of the development of the intervention and protocol for implementation may be found elsewhere.³ The program has been offered at seven community sites across Minnesota for a total of nine cycles.

Our purpose here is to describe how two these National Institutes of Health–funded CNP have developed independent CBPR assessments that address overlapping, but distinct, purposes at the project and program levels. In this case study, we specifically focus on CBPR assessments from the Missouri CNP center (PECaD) and the Minnesota CNP core research project (PI/JP) in response to a call for collaboration based on our respective efforts to evaluate our own adherence to and implementation of CBPR/CE principles. Specifically, we present a comparison of the two independent CBPR surveys administered by the Missouri CNP and Minnesota CNP to demonstrate the operationalization and assessment of CBPR and CE principles at the project and program levels and to identify commonalities.

METHODS

In response to the call for abstracts to be considered for a special issue of Progress in Community Health Partnerships, the PECaD evaluation team and the PI/JP leadership recognized the importance of understanding the differences and similarities in the development and implementation of CBPR/ CE surveys on a project versus a program level. The CBPR/ CE surveys from PECaD and PI/JP provided adequate representations of both project and program level evaluation. The PECaD evaluation team members (B.F.D., V.T.S., M.S.G.) and PI/JP leadership (M.A.A., C.S.D., G.A.H., M.V.S., M.R.L.) held a conference call to solidify the purpose of this collaborative article. We felt it would be particularly important to highlight the rationale and survey development for each CNP survey. Statisticians (C.A.J., M.S.G., C.S.D.) from both CNPs reviewed the surveys to align survey domains with measures used in the survey development process, compare domains across surveys, and identify operational definitions of the overarching domains at the program and project levels. The results of the review were synthesized using effective group characteristics within a conceptual framework for assessing coalitions¹⁸ and compared across CNPs. Descriptive statistics were provided by both CNP teams.

Rationale for the PECaD Collaborative Survey

From 2003 through 2011, PECaD used the CBPR approach without formal evaluation of adherence to and implementation of CBPR. During these years, evaluation processes were informal and formative. In 2011, PECaD’s internal leadership made the decision to formally evaluate its adherence to and implementation of CBPR and CE principles. Exemption was obtained from Washington University School of Medicine Institutional Review Board.

PECaD Survey Development

A review of CBPR and CE literature was conducted to determine best practices in evaluating adherence to, effectiveness of, and implementation of CBPR and CE principles. Based on this review, PECaD’s evaluation team developed a CBPR evaluation survey that would be administered biannually to everyone involved in PECaD activities. The evaluation team adapted questions from published measures on group dynamics, characteristics of effective partnerships, intermediate measures of partnership effectiveness, facilitation of partner involvement and member satisfaction, and medical mistrust of research.^18,22,23 In addition to published measures, questions directly addressing PECaD’s effectiveness in the CBPR principles were included along with partner demographics. For example, the full survey contained 60 items and included both closed- and open-ended questions. A web-based, password-protected survey link was emailed to any person/group with whom PECaD works, past and present. The survey was open from April 2, 2011, to May 2, 2011. Four reminder emails were sent to PECaD partners.

Rationale for the PI/JP Collaborative Survey

For multisite participatory trials such as PI/JP to succeed, they must answer to what degree does the appropriate implementation of collaborative processes known to be important in CBPR projects,^5,6 and utilization of community and university capacities identified as important in implementation science literature,^7,8 contribute to successful partnership and research outcomes. Partnership characteristics are potential moderators that establish under what conditions and in what contexts the intervention is successful.⁹ Therefore, the PI/JP group administered an annual survey to formally measure CBPR group processes and organizational capacity for two purposes: 1) to understand areas to improve participatory processes and strengthen the partnership across a set of geographically diverse organizations with variable participation with the core team and 2) to understand whether variation in perceptions of the partnership existed across sites and determine whether this related to research implementation or outcomes. Approval for PI/JP was obtained from the University of Minnesota Institutional Review Board.

PI/JP CBPR Survey Development

The PI/JP team conducted a review of CBPR and implementation science literature for approaches to evaluating community partnership effectiveness, community and university capacity and readiness for CBPR, organizational characteristics and capacities contributing to successful research/program implementation, and the added value to agencies participating in CBPR projects. Published measures on the topics of collaborative processes, organizational factors and capacity, and value added from participation in a CBPR project were adapted and additional measures were developed by the core collaborative team.^7,10–12 In the second year of implementing the survey, a password-protected link was sent to all PI/JP partners via a confidential but not anonymous online survey. The final survey contained 45 items and included both closed- and open-ended question. The survey was open in May and June 2012. Partners were given reminders via emails and at monthly meetings.

RESULTS

Samples

The PECaD survey link was sent to 130 partners. Of the 130 survey links sent via email, no email addresses bounced back. Eighty people consented to take the survey resulting in a 62% response rate. Of the 80 respondents, there were 69 complete surveys and 11 partial surveys. For questions regarding target area and role within PECaD, participants could select more than one response. For PI/JP, the survey link was sent to 35 partners, including researchers, core partners, site executive directors, site trainers/facilitators and recruiter coordinators. Of the 35 survey links sent via email, 23 surveys were completed resulting in a 66% response rate.

PECaD survey respondents were predominantly under the age of 65 years old (94%), female (81%), and African American (51%). The majority of respondents had been with the partnership for less than 5 years (72.5%). To maintain confidentiality among their study team, the PI/JP survey contained few demographic questions. Those who participated in the survey were predominantly female (82%) and Hispanic/ Latino (68%). In addition, 70% of respondents had been in collaboration with PI/JP for less than 3 years. These survey respondent characteristics demonstrate that the PECaD and PI/JP partners are representative of their respective target populations (African Americans and Hispanic/Latinos).

Survey Synthesis

Both CNP teams agreed that the surveys were too different to conduct valid statistical comparisons of key scale statistics and survey results. However, there were a number of valuable measurement considerations that permitted meaningful comparisons between the surveys. Using a conceptual framework for assessing group dynamics as an aspect of effective CBPR and community engaged partnerships,¹⁸ Table 2 shows the CBPR partnership characteristics assessed by each CNP, how each characteristic was operationalized for the survey, and the number of survey items for each characteristic.

Table 2.

Effective CBPR Partnership Characteristics Assessed^a

Characteristic	Operational Definition (No. of Items)
	Missouri CNP—PECaD	Minnesota CNP—PI/PJ
Environmental
Previous collaboration	Length of time with PECaD (1); Effectiveness on CBPR Principle 1 (1)	Previous experiences in research collaborations (2)
Community response to problem	—	Impact (2)
Geographic/cultural diversity	Race/ethnicity of collaborator (1); geographic target area(s) of work (1)	Ethnicity of collaborator (1); rural/urban location (1)
Organizational contextb	—	Project synergy with organizational goals/priorities (2); organizational commitment (4); organization and collaborator characteristics (11)
Structural
Membership	Role within PECaD	Role within PI/JP (1)
Complexity	—	—
Formalization	—	—
Group dynamics characteristics of effective partnerships
Shared leadership, including task and maintenance leadership behaviors	—	—
Two-way open communication	Comfort level for expressing opinions: communication (4); Perceived level of openness (3)	Open and honest communication (4)
Recognition of conflicts and constructive conflict resolution	—	—
Cooperative development of goals and shared vision	Effectiveness on CBPR Principle 6 (1)	Shared goals (1)
Participatory decision making processes that are flexible and use consensus for important decisions	Effectiveness on CBPR principles 3 and 4 (2)	Shared decision making (2)
Agreed upon problem-solving processes	—	—
Shared power, influence, and resources	Level of influence and power of self and others in group (2)	Shared resources and influence (4)
Development of mutual trust	Perceived level of trust (3)	Knowledge and understanding of others (1)
Collaborative evaluation of both task/goal and process objectives	Effectiveness in CBPR principle 7 (1)	Capacity to work together (1)
Well-organized meetings with collaboratively developed agendas and facilitation consistent with these characteristics (management)	Member Involvement Facilitation Scale (5)	—
Intermediate measures of partnership effectiveness
Perceived effectiveness of the group in achieving its goals.	How well partnership uses members’ time (1); accomplishments/impact of group (3); overall effectiveness on CBPR principles (9)	Collaboration effectiveness in reaching goals (1)
Perceived personal, organizational, and community benefits of participation	Community benefits of participation (1); effectiveness on CBPR principle 2 (1)	Perceived personal benefit (2) perceived community benefit (1)
Extent of member involvement	Member satisfaction with role (1)	Satisfaction with influence (1)
Shared ownership and cohesiveness/commitment to collaborative efforts	Membership satisfaction with influence (1); effectiveness on CBPR principle 8 (1)	Shared ownership (1)
Group and community empowerment; Future expectations of effectiveness	Community empowerment (3)	Future collaboration (1)
Output measures of partnership effectiveness
Achievement of program and policy objectives	—	Collective Impact (1)
Institutionalization of programs and/or partnerships	Effectiveness on CBPR principle 9 (1)	Sustainability (3), knowledge transfer (1), enhanced networks (2)

Notes. CBPR, community-based participatory research; CNP, Community Networks Program; PeCAD, Program for the Elimination of Cancer Disparities; PI/PJ, Padres Informados/Jovenes Preparados.

^aBased on conceptual framework from Shultz et al.¹⁸

^bPI/JP addition to framework.

Although the conceptual framework¹⁸ was not the only framework or theoretical model used during the independent development of the two surveys, the framework provided a theoretical common ground for synthesis of the survey review. For example, the CE and CBPR principles in Table 1 act as higher level constructs, with some overlap, within which the relevant conceptual dimensions found in the conceptual framework for assessing group dynamics as an aspect of effectiveness of CBPR partnerships.¹⁸ These dimensions include environmental characteristics, structural characteristics, group dynamics characteristics of effective partnerships, partnership programs and interventions, intermediate measures of partnership effectiveness, and output measures of partnership effectiveness. Both CNP surveys included questions adapted from the instrument developed from this framework. The PECaD CBPR survey and the PI/ JP CBPR survey both contained intermediate measures of partnership effectiveness, specifically with respect to perceived effectiveness of the group in achieving its goals; however, the questions selected for inclusion on our respective surveys differed. An example of this from the PECaD survey is the question “Related to the group(s) that YOU are involved with in PECaD, how important do you think the group(s) work is to the community as a whole?”, which is one of four group accomplishments/impact questions included from the seven questions on the Schulz et al. instrument.¹⁸ From PI/JP, “Over the past year, to what extent have you felt that our collaboration has been effective in achieving its goals” was one of two group accomplishments/impact questions included from the seven questions on the Schulz et al. instrument.¹⁸

The surveys contained a relatively similar numbers of items, but emphasized different conceptual areas based on the level of focus and rationale. For example, given the goal to assess adherence to CBPR principles at the program level, the PECaD survey focused on assessing group dynamics and effectiveness at achieving the principles of CBPR. In contrast, although the PI/JP survey addressed this topic, given the project goal of understanding variation in collaboration across sites, more emphasis was placed on environmental contexts that contribute to partnership development. Also, operational definitions employed by each CNP were different, particularly owing to the program-level focus of the PECaD evaluation and the project-level focus of the PI/JP evaluation. For example, under the group characteristic of “Shared power, influence, and resources,” this characteristic was defined as “level of influence and power of self and others in the group” by PECaD and “transparency with respect to the project budget and allocation of resources, equitable distribution of available resources, and input regarding allocation of resources” by PI/JP.

DISCUSSION

This case study focused on two distinct CBPR surveys from two CNPs to demonstrate CBPR assessment at the project and program levels. The purpose of comparing these two distinct but related survey instruments was to identify how CBPR and CE principles were incorporated into both surveys. Operational definitions of characteristics differed based on level of evaluation (program vs. project) and purpose of survey. PECaD focused on adherence to CBPR principles and effectiveness in implementing the CBPR approach. PI/JP was interested in assessing organizational contexts that contributed both to collaboration and implementation outcomes. Despite differences in content focus operational definitions, both CNPs acknowledged the importance of evaluating CBPR at the program and project levels, their processes, and outcomes to understand the factors that contribute to their success and to demonstrate the impact CBPR has on the outcomes of their work. Although each survey began at different stages in the CBPR process, both surveys focused on assessing group dynamics of effective partnerships and involvement in and satisfaction with the collaboration. Both surveys also attempted to quantify processes that should be inherent within an effective CBPR partnership.

There were several limitations to this theoretical application case study. First, we were unable to make meaningful statistical comparisons between the two surveys. These limitations were primarily owing to small sample sizes, level of evaluation, and differences in operational definitions. Each survey had different sample sizes because of the level of evaluation. However, response rates for the Missouri and Minnesota surveys were comparable (62% and 66%, respectively). Regarding limitations owing to level of evaluation, PECaD did not collect data from respondents on which components of PECaD they are specifically involved in on the 2011 survey because this was a program-level evaluation of the implementation and effectiveness of the CBPR process therefore, identifying project-level CBPR evaluation comparable to the PI/JP survey was not possible. The revised 2013 PECaD survey includes more questions regarding partnership involvement components and may provide an opportunity to get a better picture of the CE within different partnerships and roles. Finally, the differences in operational definitions of effective community partnership characteristics initially made it challenging to compare the surveys as a case study of applied theory. Commonalities in some survey questions demonstrated that there were similar domains being measured, such as trust, openness, and communication. The differences provided an opportunity to examine ways in which common measures of effective community partnership characteristics can be used across partnerships reflecting diverse priority populations and long-term health outcomes.

CNP centers are required to implement the CBPR approach in different geographic locations and their respective minority and medically underserved populations. Therefore, much can be learned from the successes and challenges of CBPR implementation at program and project levels across CNP centers. We recognize that a balance is recommended in comparing CNP centers in their CBPR approach. Not all communities are the same; unique contextual factors as well as collaborators’ priorities shape each partnership. Conversely, certain established principles of CBPR and CE, lead to quality CBPR, and we need to evaluate CBPR processes and outcomes in a more standardized way.

Recent work has moved the science of CBPR forward by establishing psychometric properties of a number of measures of CBPR processes²⁴ that could be applied broadly to community engaged projects; however, there is a need to understand the appropriateness of measures and utility of measures for projects with diverse goals and outcomes.²⁵ Future directions for evaluation of CBPR partnerships, processes, and outcomes should include systematically reviewing existing assessments, validating assessments in different partnerships with different goals and outputs, developing measures to assess program- and project-level processes and outcomes, and recommending which measures to use at various levels and specific activities and populations.