A Logic Model for Community Engagement within the CTSA Consortium: Can We Measure What We Model?

Abstract

The Clinical Translation Science Award (CTSA) initiative calls upon academic health centers to engage communities around a clinical research relationship measured ultimately in terms of public health. Among a few initiatives involving university accountability for advancing public interests, a small CTSA workgroup devised a community engagement (CE) logic model that organizes common activities within a university-community infrastructure to facilitate community engagement in research. While the model focuses on the range of institutional CE inputs, it purposefully does not include an approach for assessing how community engagement influences research implementation and outcomes. Rather, with communities and individuals beginning to transition into new research roles, this article emphasizes studying community engagement through specific relationship types and assessing how expanded research teams contribute to the full spectrum of translational science.

The authors propose a typology consisting of three relationship types—engagement, collaboration and shared leadership—to provide a foundation for investigating community–academic contributions to the new CTSA research paradigm. The typology shifts attention from specific community–academic activities and, instead, encourages analyses focused on measuring the strength of relationships through variables like synergy and trust. The collaborative study of CE relationships will inform an understanding of CTSA infrastructure development in support of translational research and its goal, which is expressed in the logic model: better science, better answers, better population health.

The infrastructure within academic health centers (AHCs) to support community-engaged research has grown rapidly following the announcement of the Clinical Translational Science Awards (CTSAs) in 2006.¹ The director of the National Institutes of Health (NIH) at that time, Elias Zerhouni called the CTSAs “the first systematic change in our approach to clinical research in 50 years.”²

Research and Public Accountability

With an emphasis on realigning institutional relationships and organizing new community and clinical networks, the NIH launched the CTSA program under the auspices of the National Center for Research Resources (NCRR) and then transitioned it in 2012 to the recently formed National Center for Advancing Translational Sciences (NCATS). The CTSA program supports an array of assets to streamline science, transform training environments, and improve the conduct, quality, and dissemination of clinical and translational research. The CTSA Roadmap initiative is developing translational research as a means to more readily and systematically bridge stages within clinical research discovery processes.^3,4

Each CTSA institution is expected to participate actively in the CTSA national consortium. The consortium, comprising 60 AHCs, included among its strategic goals a commitment to developing and strengthening community–academic research partnerships to enhance the capacity of AHCs to improve the health of communities and the nation.⁵ The challenges faced by the consortium for fostering cross-institutional collaboration and community engagement as forms of team science are vast and include the voluntary nature of participation in consortium activities and the fact that the efforts of those individuals who work within consortium teams to advance science are not always or uniformly recognized by consortium, institutional, and community leaders. Despite the challenges, and amidst evolving NIH, consortium, and institutional aims, researchers and evaluators pursue and assess community engagement (CE) and community-engaged research (CEnR).

The CTSA adoption of CE aligns with other projects that address the public accountability of universities to their surrounding communities.^6-9 Academic–community research partnerships that use scientific methods to produce socially responsible and community-beneficial knowledge have encountered both facilitators of and challenges to these partnerships. One of these challenges is assessing and evaluating the contributions of these partnerships.^9-11 While efforts to develop institute evaluation strategies for translational science are emerging,^12,13 the CTSA program commitment to bi-directionality (co-independent variables) as a characteristic of translational science has complicated evaluation of community engagement. CTSA evaluators, initially focused on how internal institutional reorganization supported improvements in academic research processes, have only recently begun to examine if and how community–academic interactions contribute to research and knowledge production.¹⁴

Toward a Science of Community Engagement

The CTSA strategic goals include measuring the impact of community engagement on translational research through the rapidly evolving gold standard of health improvement at the population level.⁵ Consortium researchers have not consistently used terms and indicators necessary to advance a shared approach to assessing the contribution of CE and CEnR for clinical and translational research, improvement of population health outcomes, and alleviation of health inequities across diverse and often underserved populations.¹⁵ CTSA assessment of CE started with a small set of activities, including consortium use of CE consultants to explore institutional-identified areas for CE improvement;¹⁶ CE infrastructure development often involved constituting community advisory boards and reporting rudimentary counts of participants involved in CE activities (e.g., number of event attendees, pilot grant applications, researchers or community members educated). The expectation that translational research demonstrate population health benefits that are simultaneously cost effective requires coordinated inquiry into community input within the research process and its contribution to accelerating the realization of pragmatic, or “real world,” value.

To address some of the challenges outlined above, we present a logic model to guide community engagement metrics (see Figure 1),¹⁷ an outgrowth of team science and the collaborative structure of the Outcomes Workgroup of the CTSA Community Engagement Key Functions Committee. In this article, we link the logic model to a typology of community–academic relationships to better evaluate the effect of CE on clinical translational science. The categories—engagement, collaboration, and shared leadership—focus attention on understanding how processes for establishing and maintaining long-term relationships with community-based groups build trust and improve research.¹⁸ The proposed typology is a way to minimize the inconsistent use of terms about community-engaged and participatory research in the current literature. Adopting clearly defined relationship terms will create a common framework for correlating and evaluating community-academic relationship types with specific CE and CEnR activities.

Figure 1.

A logic model to guide community engagement metrics. The goal of this model is to foster consistent use of terms about community engaged and participatory research and thereby help create a common framework for correlating and evaluating community–academic relationship types with specific community engagement (CE) activities and community-engaged research (CEnR). The model also helps integrate assessment focused primarily on clinical outcomes with assessments of how community–academic relationships influence the conduct of research and contribute to those clinical outcomes.

CTSA indicates Clinical Translation Science Award; CEnR, community-engaged research; PBRN, Practice-based Research Network; CBRN, community-based research network; CBO, community-based organization; IRB, institutional review board; PI, principal investigator.

Model, Then Measure

Logic models have historically helped to articulate theories of change by describing how interventions are intended or assumed to produce outcomes; they are useful especially in new fields to facilitate hypothesis-driven research and evaluation. As a contributor to change theories, logic models can help research partners demonstrate and respond to epistemological challenges. Several recent logic models provide a structure to organize the study of community–academic research partnerships and their impacts.^14,19-22 In particular, one ongoing NIH-funded project on community-based participatory research (CBPR) has identified existing instruments and tools for evaluating collaborative research partnerships.²³ The project has put onto the Web a matrix linking variables to a CBPR conceptual model with questionnaire items that can be used to examine the correlations between partnering processes and specific practice, policy, and health outcomes within the model.^24,25 A “key informant” survey for principal investigators, and a “community engagement survey” that consolidates relevant partnership and outcome questions into a comprehensive instrument are also being tested.^25,26 That project's logic model and accompanying quantitative and qualitative instruments hold great potential for advancing an assessment across the consortium for CE and CEnR contributions to translational science.

Rather than a logic model, the British National Health Service commissioned a literature review of CE under the label public involvement in research. Conducted to increase the understanding of how public involvement influenced health sciences research, this meta-analysis concluded that public involvement added value throughout the research process by increasing recruitment for research of all types, improving clinical trial design, particularly in relation to the relevance for participants of outcome measures and enhancing satisfaction for both researchers and research participants.²⁷ This literature review also found limited evidence for public involvement in research due to the lack of structure and guidance from peer-reviewed journals for reporting on public involvement.²⁷ The review revealed limitations in data collection and reporting on CE processes, which also remain largely unaddressed and underfunded within the CTSA consortium.

Historically, both institutional variation in valuing consortium participation and also the academic reward structure (i.e., advancement for individual efforts rather than collaborative ones) have impeded CE and CEnR assessment and evaluation.^15,28 The challenges to cooperation across the consortium seem unacknowledged in the first request for applications (RFA) put forth under the auspices of NCATS, which encourages each CTSA program to “build on unique institutional strengths” while anticipating continuity in “areas that were highlighted in previous CTSA solicitations”; however, it is important to note one previous area emphasized developing “innovative models for patient and community engagement in research. ”²⁹

The development of the CE logic model described in this article demonstrates a team science approach with the potential for advancing a collective agenda. The logic model encompasses a full infrastructure that can advance the study of CE and CEnR as characteristics of translational science at CTSA institutions by facilitating instrument development and subsequent collection of comparable data. The CTSA CE logic model “inputs” column shows common CE activities that were developed in response to early CTSA RFAs. In 2010, common CE activities across the consortium, which still remain today, include educating academic researchers about engaging communities in research and educating community members about clinical research, funding pilot programs to facilitate community–academic interaction, examining CEnR research milestones and timelines for tenure and promotion decisions, and developing administrative structures to support increased academic engagement with community groups, practice-based networks, and clinical and consortium partners.

Based on these activities, the logic model characterizes expected short, intermediate, and long-term outcomes as part of an overall infrastructure that supports published studies of community–academic participatory research. As a model of common CE activities at CTSA institutions, the designated “intermediate results” column (see Figure 1) provides a structure for assessment among consortium partners through shared measurement strategies such as institutional review board (IRB) responsiveness to CEnR research,^13,30 measuring university capacity to support community-engaged research according to a measure of time from issuance of notice of award (NOA) to issuance of subcontracts to community partners, the time needed to recruit the first participant, or perhaps the extent of racial and ethnic diversity among individuals actually enrolled in studies. More specific population health outcomes would depend on the specific investigations undertaken within each CTSA program and the development of measures (in collaboration with community partners) shared across the CTSA consortium to assess and to compare community engaged health interventions.

Because the CE logic model names and focuses on CE activities and infrastructure outcomes, its usefulness extends beyond any single project with specific targeted outcomes and allocation of resources. This model provides an opportunity for understanding the contributions of CE and CEnR to translational science by integrating assessment focused primarily on clinical outcomes with assessments of how community–academic relationships influence the conduct of research and contribute to those clinical outcomes.

As a network of relationships, the CTSA consortium shares meaning-systems and organizes patterns of interaction, demonstrating the two primary characteristics of a community. Communities, like cultures, have been studied according to (1) cognitive processes and systems of meaning, and (2) structures and patterns of human social organization and interaction. These two methodological orientations to the study of community are incorporated into the recently revised Principles of Community Engagement,³¹ which explicates CE principles and how they function in contributing to improving public health. This second edition expands on methods for studying relationships that are inherently bi-directional, and advocates evaluation studies to contribute to a more complete understanding of CE. The logic model, therefore, provides the next step to supporting a collaborative evaluation structure of these co-dependent relationships and developing infrastructures.

The CE Logic Model: Structuring the Study of Community Engagement

In this article, we propose three categories of relationships, adapted from the CE continuum within the Principles of Community Engagement, to structure inquiry into and measurement of constructs within the logic model in order to learn how community participation may advance the translational science goal of linking clinical research to improved health in communities and the nation. The three relationship categories for structuring knowledge development—engagement, collaboration, and shared leadership—represent distinct, though not mutually exclusive, relationship types for the study of CTSA-sponsored CE activities.³¹

Engagement signifies an intention to exchange information and possibly resources (including money) through an individual event or a short-term series of events (including clinical trials). Outcome expectations are for all participants indeterminate due to the absence of ongoing information exchange, dialogue and shared goals.

Collaboration signifies an intention for the members of a partnership to cooperate over time for the purpose of achieving specified goals; these goals may or may not be shared. Partners determine the duration of information and resource exchange and establish measurable goals, typically from a particular perspective and potentially shaped by formal agreements and defined responsibilities.

Shared leadership signifies an intention for the partnership to achieve shared goals. Partners share resources and information to enhance capacity for bi-directional exchange and achieve expected outcomes. Partners develop a shared participatory evaluation plan,supported through information exchange and dialogue, with assessment focused on activity implementation, on specific health and broader capacity outcomes, and on sustainability.

By studying relationships and the institutional variations in these relationship categories, it becomes possible to study CE and CEnR activities within unique local contexts and to collect comparable data of how academic engagement in community life influences clinical research methods, community health outcomes, and strategies to inform health policy. The collection of comparable data depends upon consortium efforts to develop shared instruments for evaluating relationships developed through CE and CEnR.

The first step in the use of the logic model may be for each individual CTSA program to organize its vision of community engagement relationships using the above typology. No institution pursues every aspect of community engagement represented in the logic model, or specific CE activities, in precisely the same way. Each CTSA program could assess the variation in its multiple community engagement projects matched against the program's vision, and then use the logic model to identify the potential outcomes based on that assessment. The logic model's concise summary of input activities could inform evaluation and could help to shift the study of CE and CEnR from specific projects and goals toward the types of relationships that support project development, protocol implementation, and shared interpretation and dissemination of results.¹⁶ However, the challenge for the consortium is not only to build infrastructure that connects researchers to communities, but equally to build community infrastructure to support the implementation and dissemination of research findings in order to best foster health within each community context. As current experience and various Institute of Medicine reports make clear, knowledge and evidence are not in themselves sufficient to improve the quality of care.^32,33

With the CE logic model providing a point of departure for CTSA programs, an additional challenge exists in marshaling cooperation and collaboration among consortium members to identify and deploy metrics and measurement strategies to study how community–academic relationships produced through CE and CEnR activities advance translational science. Existing evaluation methods and metrics—which are emerging among CBPR and other participatory evaluation studies to link participation processes with practice, system, policy and health outcomes—can help advance this effort.^22-24

Trust as Core to Productive Relationships

While the logic model provides a set of inputs and short-term to longer-term outcomes, the question remains, How does change occur, and what processes correlate with or predict outcomes? The logic model organizes activities and expected outcomes while presuming that relationships, which may differ by vision and practice across the three types defined here, make a difference, especially those distinguished by synergy and trust. A recent realist evaluation review in the Milbank Quarterly applied partnership synergy as an intermediate theory of change across multiple partnerships, showing the value of relationships in contributing to outcomes.^10,34

The concept of trust has been well articulated in the literature as a key CE metric, with assessment typically focused on community attitudes toward researchers.³⁵ However, trust is a social phenomenon and therefore subject to analysis and interpretation from different perspectives.^35,36 Dynamic and context-bound, trust is a mediator of cooperation; both trust and distrust shape relationships and are direct attributes of bi-directionality.^16,34,37 Understanding trust may also enable researchers to avoid reifying or objectifying any particular community.³⁸ While the CE logic model situates bi-directional trust as a short-term result, all three relationship categories involve trust and are found throughout the logic model (e.g., between and within communities and research/science teams, organizations, and institutions). Trust as a measure of partnership strength, modification, and re-evaluation requires a multi-methods evaluation strategy.^12,39-42

Many stories have been told about community distrust of research. The Tuskegee syphilis study involving African Americans,^43-45 secret radiation experiments on mentally disabled children,^4,6 the recent Havasupai lawsuit against Arizona State University for consent form violations,⁴⁷ and “good will” studies that portray academic researchers as the sole possessors of research knowledge and skill, contribute to the history of community distrust of research.⁴⁸ “Mosquito,” “helicopter,” and “seagull” research are among the negative characterizations of researchers who minimally interact with communities or impose their own context-bound views while extracting value for their own benefit, subsequently abandoning communities once their purposes are met.^49-51 Stories of inequities in academic–community relationships produce feelings of disenfranchisement, marginalization, and disrespect for local experience and knowledge, particularly among vulnerable communities that traditionally lack a voice within the academy.^52,53 A study of researcher attitudes about the minority community around one academic health center found direct correlations between researchers' discomfort at being in community settings and the extent to which those researchers engaged the community through research.⁵⁴ Given the number of academic medical centers that are geographically proximate to minority neighborhoods and populations, other CTSA programs might benefit by replicating this study question.

Community–academic research partnerships that address public health problems and health disparities expand the national capacity to reduce or eliminate seemingly intractable health inequities.^10,55-58 However, some researchers encounter communities that seek to limit partnership scope (e.g., solely to advise on a project or to broker subject/participant recruitment); such researchers contend that assuming communities always prefer a more engaged community-based participatory approach is presumptuous.^59-61 Researchers also note that variability in engagement and partnership strength may result from the extent of trust and the approach to relationship consolidation or formalization,⁶² from strategies for disseminating findings and translating research into practice and policy,⁶³ and from improved community assets, research capacity, and public health outcomes.^64,65

The recognition of the importance of measuring trust within academic–community relationships has been growing, beginning with one of the earliest tools from the Prevention Research Centers.⁶⁶ In addition, the University of Rochester developed a survey to evaluate community trust of their academic health center as part of its CTSA evaluation. Finally, as part of its national CBPR cross-study, the University of New Mexico has developed trust measures, encompassing a spectrum from mistrust and proxy trust to ideal trust, for its internet survey tool of community–academic partnerships as well as for its interview questions.^22,35 The hypothesis is that partnerships with greater collaborative and shared leadership would have less mistrust and functional trust, and more evidence of ideal and sustainable trust.

Within the CTSA consortium, the construct of trust provides a collective opportunity for all sites to evaluate trust both as an independent and a dependent variable. Specific CEnR projects could use the measures for their own partnerships; yet on a broader scale, each CTSA program could conduct data collection within their community advisory boards and academic leadership committees to assess trust development over time and across types of engagement. Such a study of trust could also enhance the study of each CTSA program's capacity to function in multidisciplinary teams, the capability to adapt to complex situations, and a commitment to engagement and partnership that transcends specific projects.^20,67

In sum, the hypothesis for engaging a community in research is that intentionally structured and respectful academic and community relationships can create shared meanings and structures to advance translational research and result in improvements in the health of communities and the nation.⁶⁴ The CTSA consortium is moving toward developing pragmatic clinical and community trials that will inform this hypothesis only if the research outcomes data inform both the specific research question and the contextual relational (i.e., methodologic and implementation) components of the intervention. In essence, research cannot leave the controlled laboratory environment and then ignore its relationship to the social contexts.

Supporting the Full Spectrum of Translational Research

Translational research is often described as moving knowledge from bench to bedside or moving from an academic research focus on discovery into a clinical practice, primary care, and community setting focus on application.^68,69 Translational science has also been accorded the lofty objective of facilitating knowledge transfer throughout the entirety of the U.S. health care system.⁷⁰ A successful translational project requires both CE, with an emphasis on the bi-directionality of knowledge moving between the community and the academy, and relationships characterized by productive partnerships.

We have endeavored to show why CTSA programs, individually and collaboratively, should use the CE logic model to evaluate and assess CE activities according to three relationship categories: engagement, collaboration, and shared leadership. Further, we encourage consortium research teams to use trust and synergy as core metrics to organize assessment of their relationships. Without a coordinated evaluation strategy, the systematic evidence and insights necessary to assess how CE transforms the research process or how it advances the translational science goal of improving health will not be generated.

Recognizing individual variation between, and competing interests within, CTSA institutions, the NIH CTSA/NCATS Integration Working Group recommended that institutions participating in the consortium set aside and contribute internal funds and resources in predictable amounts to sustain CE and CEnR infrastructure. In response, translational researchers must commit to collaborating and to developing shared metrics to study CE and CEnR, determining whether viable and productive community–academic relationships support the full spectrum of translational research. ²⁸ Consortium participants can take the lead by committing to transparency and by collaboratively approaching CE and CEnR metrics and instrument development and assessment as an open source project; the CTSA Consortium Coordinating Center (C4) provides infrastructure functionality to support open source knowledge development.

With such cooperation within and between academic institutions and communities, we will be able to collectively and more effectively address inequities and redress disparities. Demonstrating engagement, collaboration, and shared leadership within the consortium will make it possible to realize the goal of translational research expressed in the long-term results part of the CE logic model: better science, better answers, better population health.