Measurement Approaches to Partnership Success: Theory and Methods for Measuring Success in Long-Standing CBPR Partnerships

Abstract

Background

Numerous conceptual frameworks have been developed to understand how community-based participatory research (CBPR) partnerships function, and multiple measurement approaches have been designed to evaluate them. However, most measures are not validated, and have focused on new partnerships. To define and assess the meaning of success in long-standing CBPR partnerships, we are conducting a CBPR study, Measurement Approaches to Partnership Success (MAPS). In this article we describe the theoretical underpinnings and methodological approaches used.

Objectives

The objectives of this study are to: 1) develop a questionnaire to evaluate success in long-standing CBPR partnerships; 2) test the psychometric qualities of the questionnaire; 3) assess the relationships between key variables and refine the questionnaire and theoretical model; and 4) develop mechanisms and a feedback tool to apply partnership evaluation findings.

Methods

Methodological approaches have included: engaged a community-academic national Expert Panel; conducted key informant interviews with Expert Panel; conducted a scoping literature review; conducted a Delphi process with the Expert Panel; and revised the measurement instrument. Additional methods include: conduct cognitive interviews and pilot testing; revise and test final version of the questionnaire with long-standing CBPR partnerships; examine the reliability and validity; analyze the relationship among variables in the framework; revise the framework; and develop a feedback mechanism for sharing partnership evaluation results.

Conclusions

Through the application of a theoretical model and multiple methodological approaches, the MAPS study will result in a validated measurement instrument and will develop procedures for effectively feeding back evaluation findings in order to strengthen authentic partnerships to achieve health equity.

Background

CBPR has received increasing recognition as a valid approach to examine and address social and health inequities.^1–5 CBPR involves partnerships between researchers and community entities that build upon community strengths; embrace principles of co-learning, equitable engagement, power sharing, and capacity building; and focus on enhancing understanding of a given phenomenon and translating findings into interventions and policy change to address health inequities.⁶ In the United States there has been a proliferation of CBPR partnerships with increased funding opportunities and dissemination of results.^7–9

Over the past 15 years, numerous conceptual frameworks have been developed to understand and evaluate how CBPR partnerships function^4,5,10–16 and multiple measurement approaches designed to assess key dimensions of these conceptual models.^{14,15,17–20} However, most measures are not adequately tested and validated;^17,21 with few exceptions, such as the work by Oetzel and colleagues.²⁰ Furthermore, measurement development has focused primarily on new CBPR partnerships with considerably less emphasis on defining and measuring the factors that contribute to the success of long-standing CBPR partnerships and their ability to achieve intermediate and long-term outcomes, such as perceived benefits and costs,^22,23 sustainability,^24–29 and policy advocacy.^6,30,31 To address this gap in the literature and to define and assess the meaning of CBPR partnership success and the factors that contribute to success in long-standing CBPR partnerships, we are conducting MAPS, a five-year, multi-phased CBPR study aimed at furthering the science and practice of CBPR, and providing a validated questionnaire for long-standing CBPR partnerships as well as newly forming CBPR partnerships to evaluate and sustain their efforts towards achieving health equity and success. The purpose of this article is to describe the theoretical underpinnings and methodological approaches in the MAPS study.

Objectives

In this study, we define long-standing CBPR partnerships as those in existence 6 years or longer, a time period that coincides with partnership continuation after a typical 5-year funding cycle for research. The objectives of our study are presented below, followed by a description of the CBPR partnership involved and a discussion of the theoretical and methodological approaches we use to reach these objectives.

• Objective 1: Define CBPR partnership success and develop a questionnaire to assess partnership success and its contributing factors in long-standing CBPR partnerships.

• Objective 2: Test the psychometric qualities of the questionnaire in a sample of long-standing CBPR partnerships existing six years and longer.

• Objective 3: Analyze survey data collected in Objective 2 to assess the relationships between key variables and to use the results to refine the questionnaire and the theoretical model.

• Objective 4: Develop mechanisms to feed back and apply partnership evaluation findings, and widely disseminate the questionnaire and feedback tool in a readily accessible and usable format.

Theory and Methods

CBPR Partnership: The Detroit Community-Academic Urban Research Center

The MAPS study is being conducted through the Detroit Community-Academic Urban Research Center (Detroit URC), a long-standing CBPR partnership established in 1995 with initial funding from the Centers for Disease Control and Prevention.^32,33 The overarching goal of the Detroit URC is to foster and support CBPR partnerships to examine and address the social and physical environmental determinants of health to reduce and ultimately eliminate health inequities in Detroit. The Detroit URC is guided by a Board composed of members of eight community-based organizations, two health and human service agencies, and an academic institution (see Acknowledgements).^32,33 As described below, in keeping with the principles of CBPR adopted by the Detroit URC Board^32,33 which guide our approach to this work, the Board was actively involved in the initial development of the theoretical model which informs this effort.¹⁴ The Board also contributed to the design of this study and continues to be involved in all aspects of the MAPS project through discussions at monthly meetings and regular email correspondence. In addition, several members of the Board are co-authors on this manuscript.

Conceptual Framework Guiding Proposed Design and Methodology

The MAPS study builds upon and extends the conceptual framework of Schulz, Israel, and Lantz for understanding and assessing the effectiveness of CBPR partnerships.^14,15,30 This framework was initially developed in the late 1990’s to guide the evaluation of the Detroit URC,^15,30 building on the work of Sofaer¹⁶ Lasker and Weiss,¹² and Johnson and Johnson.³⁴ This framework proposed six broad areas that contribute to partnership functioning: environmental characteristics, structural characteristics, group dynamics characteristics, partnership programs and interventions, intermediate measures/outcomes of partnership effectiveness, and output measures/long-term outcomes of partnership effectiveness, with a particular emphasis on the role of group dynamics.¹⁵

As briefly described below, since the initiation of this framework others have developed helpful frameworks for understanding and assessing CBPR partnership functioning. Wallerstein, Duran, and colleagues,^{4,5,13,17,20,35} acknowledge drawing upon the Schulz, Israel and Lantz framework³⁵ in creating a somewhat similar one with four major components: context, group dynamics and equitable partnerships, intervention/research, and outcomes. They further elaborate and expand upon the many dimensions of each of these four components and have developed and tested a comprehensive measurement instrument that has examined the pathways of how partnership process contributes to outcomes in community-engaged partnerships, over sampling Native American and Alaskan Native partnerships, and including partnerships at all stages of development and not only those that adhere to CBPR principles.^{4,5,13,17,20,35}

Kyodyakov and colleagues developed a conceptual model showing the impact of community engagement in research partnered projects on outcomes.³⁶ Also, building upon the work of Lasker and Weiss,¹² their model places synergy in the center, in which they suggest that “partnership characteristics impact partnership functioning, as well as synergy and perceived personal and community/policy-level outcomes; partnership functioning influences partnership synergy; and synergy influences both perceived personal- and community/policy-level outcomes”.^36(p197)

Jagosh and colleagues,³⁷ drawing upon an extensive review of the CBPR literature, used a realist evaluation methodology which included partnership synergy theory as their middle-range theory to inform their model. The results of their analysis supported the key dimension of trust in developing partnership synergy, which in turn leads to partnership sustainability, and subsequently population-level outcomes.³⁷

Although there are similarities in many of the dimensions of these different conceptual models (e.g., group process, trust, synergy) and a focus on examining pathways that contribute to partnership outcomes, the present study has several unique aspects that are intended to fill some gaps in the literature. For the purpose of the MAPS study, the earlier framework developed by Schulz, Israel and Lantz^14,15,30 is further developed and the dimension of success has been added based on an extensive review of the literature.^{4,5,11–17,20,21,25–29} Briefly described, and as shown below in the adapted model (Figure 1), the extent to which a CBPR partnership is able to achieve its long-term outcomes (e.g., sustainability, tangible community and health benefits, health equity), is influenced by intermediate outcomes (e.g., synergy, shared ownership, benefits and costs, partnership equity) of partnership effectiveness, which are shaped by the programs and interventions of the partnership. These are influenced by the group dynamics characteristics of the partnership (e.g., trust, communication, leadership, decision making) which are, in turn, shaped by a partnership’s structure (e.g., membership). All of the above-mentioned processes are influenced by the broader environmental factors or context within which a partnership operates (e.g., social, economic, cultural).

Figure 1: Conceptual Framework for Understanding and Assessing Success in Long-Standing Community-Based Participatory Research Partnerships.

Source: Adapted from original model by Lantz, et al⁽³⁰⁾ and Schulz, Israel, and Lantz⁽¹⁵⁾, and Israel et al^(14,71), drawing upon the work of Lasker & Weiss⁽¹²⁾; Sofaer⁽¹⁶⁾; and Wallerstein and colleagues⁽¹³⁾.

As shown in Figure 1, we include a new theoretical dimension in this adapted version of the conceptual framework, that posits that CBPR partnership success is a separate construct that is over and above and a function of intermediate and long-term partnership outcomes. This includes, for example, expanded relationships and influence that extends beyond the partnership, intangibles associated with the partnership (such as, genuine friendship, good will, high level collaboration), and personal enrichment. As described above, much of the previous work has largely focused on the conceptualization and measurement of the preceding dimensions of the framework (e.g., partnership structure, group dynamics), and their association with intermediate and long-term outcomes, especially as these relate to the early phases of partnership formation, and not solely focused on CBPR partnerships. In contrast, our study aims to provide a more in-depth understanding of the concept of success in long-standing, explicitly identified CBPR partnerships, and the intermediate and long-term outcomes that contribute to success (depicted in the three expanded boxes on the right-hand side of Figure 1). To achieve this aim, we are developing and validating a questionnaire to measure partnership success as well as the key intermediate and long-term outcomes posited in the framework that are essential for achieving success in long-standing CBPR partnerships. We also intend to further revise this conceptual framework as informed by the results of this process.

Objectives and Methods

As noted, this study is guided by the conceptual framework presented in Figure 1. The study objectives are derived from our review of the literature and experience which points to the identified gaps in the literature. To meet these objectives and further refine the theoretical model, our approach involves several methods, briefly described in accordance with the objectives of the study (Table 1). The University of Michigan Institutional Review Board reviewed the MAPS study and determined that it is exempt from ongoing institutional review board review under category two of federal exemption categories.

• Objective 1: Define CBPR partnership success and develop a questionnaire to assess partnership success and its contributing factors in long-standing CBPR partnerships.
• Objective 1.1: Establish and engage an Expert Panel of academic and community members actively involved in CBPR. As an initial step in meeting Objective 1, we established a national panel of six academic and six community experts in CBPR who are engaged in numerous activities throughout the project period. Experts were selected through reputational sampling by the academic research team and the Detroit URC Board members based on long-term experience in CBPR, contributions to the peer-reviewed literature, and diversity with respect to geography, race and ethnicity, and area of research. Two community and two academic partners involved in the Detroit URC and affiliated CBPR partnerships also serve on the Expert Panel for a total of 16 members (see Acknowledgements).
• Objective1.2: Reach conceptual clarity on “CBPR partnership success” and identify key dimensions and indicators. We used two approaches to meet this objective. First, we conducted key informant interviews with all 16 members of the Expert Panel to identify relevant dimensions and indicators of partnership success. In addition to asking a broad, open-ended question to identify outcomes, we specifically asked Panel members to define success in long-standing CBPR partnerships and discuss whether there is a distinction between success and outcomes; they also discussed follow-up questions on key dimensions of our theoretical model, as depicted in Figure 1 (e.g., sustainability, synergy, equity). Interviews were conducted in person or via video- or teleconferencing and lasted approximately 60 minutes. The interviews were recorded and transcribed for analysis, which involved the process of open coding and constant comparison.³⁸ Second, we conducted a literature search using the Joanna Briggs Institute framework for conducting scoping reviews³⁹ including the PRISMA.³⁹ In line with a scoping review approach,³⁹ our aim was to identify the multi-dimensionality of what was referred to as “outcomes” and “success” in CBPR partnerships over the past decade, what indicators and measures of outcomes and success in CBPR partnerships had been published, and what gaps remained in the existing research. We searched three databases (PubMed, CINAHL, Scopus) for literature meeting our inclusion and exclusion criteria and published between 2007–2017. Following a multi-step process to yield final articles to include, we ultimately reviewed 26 articles from which we identified key domains and indicators.⁴⁰
• Objective 1.3: Create a preliminary pool of items for review by Expert Panel. Themes identified through the data analysis of the key informant interviews, dimensions and indicators identified through the scoping review, and published measures of success related to our model²⁰ informed the generation of a preliminary item pool. For those constructs where measurement was either lacking or psychometrically unsound, we created new items. The resulting preliminary item pool was categorized along seven key dimensions with underlying items: equitable relationships (22 items), partnership synergy (7 items), reciprocity (9 items), competence enhancement (12 items), sustainability (18 items), realization of benefits over time (17 items), and achievement of intermediate and long-term goals/objectives (11 items), for a total of 96 items.
• Objective 1.4: Conduct a Delphi process with the Expert Panel to determine content validity of the item pool. We then conducted a Delphi process⁴¹ with the Expert Panel to determine content validity of the drafted items (i.e., ensuring the items measure outcomes and success in long-standing CBPR partnerships from the perspective of CBPR experts) and improve the clarity and comprehension of items with the goal of producing a set of items for the questionnaire that could be pilot tested.^42,43 As depicted in Figure 2, the Expert Panel participated in two rounds of the Delphi process through email correspondence using Qualtrics⁴⁴ (100% response rate). In the first round they assessed 96 questionnaire items on level of importance (5-point Likert scale from Very Important to Not at all Important), and during the second round they assessed 79 questionnaire items on the extent to which they were reflective of partnership success (3-point Likert scale from Reflective to Not Reflective). After each round the research team analyzed the quantitative data and qualitative results and revised the number and wording of the questionnaire items accordingly. The third round of the Delphi process was conducted in a face-to-face meeting at which the Expert Panel discussed remaining items where there was considerable variability in how Panel members assessed the extent to which they are reflective of outcomes and success, and items where Expert Panel members provided qualitative comments that suggested a lack of clarity (see Figure 2 for details). Based on the results of this Delphi process, we revised the item pool for construct validity testing.
• Objective 1.5: Conduct cognitive interviews with six individuals from two long-standing CBPR partnerships (three members each), involving equal numbers of community and academic members. The cognitive interviews are to help identify potential sources of response error and improve readability so that we can further revise the questionnaire as needed.^45,46 Cognitive interview protocols specifically address question comprehension, retrieval of relevant information from memory, and mapping of the response process (construct validity). The results from the cognitive interviews will be compiled and used to revise the questionnaire items.
• Objective1.6: Pilot test the questionnaire with two partnerships (two community and two academic partners from each) and revise/finalize the questionnaire based on results. The questionnaire will be pilot-tested with two CBPR partnerships that meet the recruitment criteria for the study (described elsewhere in this article). Pilot testing will assess the logistics of survey administration, length of the questionnaire, and recommendations for improving the clarity and flow of questions. We will inquire about respondent burden. The results will help us revise and finalize the questionnaire and inform the design of survey administration.
• Objective 2: Test the psychometric qualities of the questionnaire in a sample of long-standing CBPR partnerships existing six years and longer.
• Objective 2.1: Conduct purposive sampling to identify CBPR partnerships that meet the selection criteria. A purposive sampling method will be used to identify appropriate CBPR partnerships to include in the survey.^47–50 This sampling method is suited to our interest in capturing observant, reflective members of the universe of long-standing CBPR partnerships who are knowledgeable about the process and outcomes of community-based partnerships, and are both able and willing to share their knowledge. Although the use of a random sample would have the benefit of being able to generalize to the population of long-standing CBPR partnerships, no definitive and comprehensive list of such partnerships exists. Thus, we are using a systematic process to determine partnerships that meet the criteria for long-standing CBPR partnerships, as specified below. For example, we will use recommendations from the Expert Panel, colleagues in the field of CBPR, databases of funded CBPR initiatives (such as, the NIH RePorter), and the literature to identify partnerships.
• Objective 2.2: Recruit longstanding CBPR partnerships to participate in the survey. We expect to have 55 long-standing CBPR partnerships complete the survey. Our initial eligibility criteria include: (1) have been in existence at least 6 years and continue to operate; (2) show evidence of following CBPR principles and norms as noted by Israel and colleagues;^32,33 (3) conduct ongoing partnership evaluation; (4) show evidence of dissemination; and (5) consent to participate. Power analysis⁽⁵¹⁾ was done using a starting number of 100 contacted partnerships. Assuming 75% participation rate, we expect to send the survey to 75 partnerships. Of these partnerships, we expect 55 would successfully complete the survey, for a response rate of approximately 75%. Assuming that each partnership has on average 13 members, the final sample will include 712 survey respondents, which would allow us to achieve 80% statistical power^52–54 and be confident that the statistical psychometric methods will provide robust estimates of reliability and validity of our instrument. Based on our power calculations, in order to achieve a final sample of 55 partnerships, we will examine characteristics of all identified partnerships and select an initial set of 73 partnerships based on diversity in geographic location, size, health issues addressed, and other demographics of the communities involved. We will identify and contact an academic and community member of the leadership for the selected partnerships to explain the purpose of the study and what it would entail to participate. Agreement for participation from the partnership leaders will include a commitment to engage all individual partnership members to complete the survey. Each partnership will be compensated up to $2,000 for their participation in the study.
• Objective 2.3: Administer the questionnaire to members of recruited CBPR partnerships. We will use the pilot-test results (Objective 1.6) to refine our methods for survey administration and enlist the help of the identified partnership leaders to ensure a satisfactory response rate. If there is no particular administration preference identified, we will employ the following procedures: send a personalized email to invite members of the participant CBPR partnerships to fill out an online questionnaire and, if no response after three follow-ups, distribute and collect a paper-based questionnaire in a sealed envelope by leaders of selected CBPR partnerships.
• Objective 2.4: Perform psychometric analysis using multiple approaches to validate the MAPS questionnaire. We will examine reliability (internal consistency, one month test-retest reliability, and interrater reliability) and validity (criterion validity, content validity, construct validity, convergent and discriminant validity).⁵⁵ Recognizing that the hierarchical^56,57 structure inherent in the data collected for this study (e.g., multiple respondents describing the same partnership) can result in incorrect conclusion about the factor structure of the scales, we propose to use multilevel structural equation models^58–61 to assess construct validity (Multilevel Factor Analysis),^62,63 concurrent, predictive, convergent, and discriminant validity (intraclass correlation),^60,61 and internal consistency (multilevel Cronbach’s alpha).^64,65

Table 1.

MAPS Objectives and Methods Used

Objective 1: Define partnership success & develop & pilot test the survey questionnaire

1.1 Establish, engage, and collaborate with Expert Panel

1.2 Identify and clarify key dimensions and indicators (i.e., scoping literature review, key informant interviews)

1.3 Create preliminary questionnaire

1.4 Conduct Delphi process with Expert Panel (content validity)

1.5 Conduct cognitive interviews (construct validity)

1.6 Pilot test and revise questionnaire (face validity)

Objective 2: Test the psychometric reliability and validity of the questionnaire

2.1 Conduct purposive sampling

2.2 Recruit long-standing CBPR partnerships

2.3 Administer the questionnaire

2.4 Perform psychometric analysis to examine reliability and validity*

Objective 3: Analyze survey data to assess the relationships between key variables and revise and refine the questionnaire and theoretical model based on the results.

3.1 Revise and finalize questionnaire

3.2 Further refine theoretical model

Objective 4: Develop mechanisms to feed back and apply partnership evaluation

4.1 Develop feedback mechanism/tool

4.2 Disseminate knowledge gained, questionnaire and feedback mechanism/tool to CBPR partnerships

^*Psychometric analysis will examine internal consistency, test-retest reliability, interrater reliability, content validity, construct validity, and discriminant validity. It will also estimate latent partnership success and agreement among members.

Figure 2.

Delphi Process with Expert Panel

Furthermore, we will extend the multi-rater ordinal model^66,67 to our setting that involves multiple items and the nested structure of members (i.e., raters) under a partnership. This general model will enable us to estimate latent success for individual partnerships as well as another set of parameters for rater severity (i.e., every member has a personal tendency to give higher, middle, or lower ratings). Like the item response theory, this model involves a large number of parameters and thus will require the computational technique of Markov chain Monte Carlo⁶⁸ to sample the posterior distributions of these parameters.

We will use a generalized measure of agreement to calculate the agreement among members under each construct for each partnership.^69,70 Following the conventional approach, CBPR partnership success would be indicated by averaging the scale scores of questionnaire items from all the members in the same partnership.

• Objective 3: Analyze survey data collected in Objective 2 to assess the relationships between key variables and to use the results to refine the questionnaire and theoretical model.
• Objective 3.1: Revise and finalize the questionnaire. Based on the results of the psychometric testing and data analyses of the 55 partnerships that participate in the study, we will further revise and finalize the questionnaire.
• Objective 3.2: Further refine the theoretical model. Based on the results of the analyses of the key informant interviews, scoping review, field notes from the in-person meeting of the Expert Panel, and analysis of the survey data, we will further refine the conceptual model guiding this study.
• Objective 4: Develop mechanisms to feed back and apply partnership evaluation findings, and widely disseminate the feedback tool in a readily accessible and usable format.
• Objective 4.1: Develop, test, and refine a mechanism to feed back and apply evaluation findings to enhance CBPR partnership success. We will develop and make accessible a feedback tool that will be beneficial to CBPR partnerships interested in evaluating and improving their partnership process and outcomes. We will develop a mechanism for CBPR partnerships to share and interpret evaluation results so that they can apply findings to enhance partnership success. A practical template will be developed that provides guidelines and examples of how to present, discuss, and apply results. Using a participatory process in accordance with our CBPR principles, the Detroit URC Board, composed of community and academic partners, will be actively involved through monthly Board meetings and ongoing electronic communication in the design of this tool. The Board will build on its more than 25 years of experience evaluating their partnership, in which key survey results have been systematically fed back to the partnership and needed steps have been taken to improve the partnership.^{14,15,30,32,71} We will engage the community and academic members of the Expert Panel in the development and refinement of the feedback tool through a face-to-face convening and regular electronic communication. We will also review the literature and draw upon feedback tools developed by others, for example, Wallerstein and colleagues.³⁵ We will pilot test the tool with an affiliated partnership of the Detroit URC and make revisions accordingly.

Members of the MAPS research team will send survey results and offer assistance to each of the CBPR partnerships involved in the validation study, using the newly developed mechanism to feed back, interpret, and apply the findings. We will feed back estimated latent scores for success and the estimated member agreement results under each construct, to all participating partnerships. A manual on how to score these measures will be provided so that partnerships can easily score themselves in the future. This will allow partnerships to identify constructs where they are meeting what the MAPS study considers to be parameters of success and constructs where further attention is needed to improve success.

• Objective 4.2: Disseminate the knowledge gained, questionnaire and feedback mechanism/tool in a readily accessible and usable format. In keeping with the CBPR principle regarding dissemination of study findings, the knowledge gained, innovative measurement instrument, and feedback mechanism produced from this research will be disseminated widely and jointly by our community and academic partners to multiple audiences through multiple venues and media.

Conclusions

Although many long-standing CBPR partnerships have emanated from funding support over the past 25 years, there remains a lack of consensus on what defines success among these partnerships and what factors contribute to CBPR partnership success. Moreover, the processes and outcomes of CBPR partnerships have been measured in different ways with few validated instruments. Most of the measures that do exist emphasize early partnership formation rather than long-standing partnerships and do not always survey all members of the partnership. Through the application of a theoretical model, and multiple methodological strategies, as described here, the MAPS study addresses this gap in the literature. It is both timely and needed, particularly in light of the NIH’s institutional commitment to community-academic partnership approaches to research. One of the benefits of the MAPS project is that it will contribute to the field of CBPR in further defining what success means in long-standing CBPR partnerships, and in conceptualizing the factors that contribute to such success. It will also result in a validated measurement instrument of success in such partnerships, where not many validated instruments exist in the field. Furthermore, this instrument, which will be widely disseminated, will enable partnerships to assess their process and outcomes, and will also provide procedures for effectively feeding back evaluation findings in ways that strengthen engagement and authentic partnerships aimed at addressing health inequities.