Community Engagement in the CTSA Program: Stakeholder Responses from a National Delphi Process

Abstract

In response to the Institute of Medicine (IOM) Committee's December 2012 public request for stakeholder input on the Clinical and Translational Science Award (CTSA) program, two nonprofit organizations, the Center for Community Health Education Research and Service, Inc. (CCHERS) and Community‐Campus Partnerships for Health (CCPH), solicited feedback from CTSA stakeholders using the Delphi method. Academic and community stakeholders were invited to participate in the Delphi, which is an exploratory method used for group consensus building. Six questions posed by the IOM Committee to an invited panel on community engagement were electronically sent to stakeholders. In Round 1 stakeholder responses were coded thematically and then tallied. Round 2 asked stakeholders to state their level of agreement with each of the themes using a Likert scale. Finally, in Round 3 the group was asked to rank the Round 2 based on potential impact for the CTSA program and implementation feasibility. The benefits of community engagement in clinical and translational research as well as the need to integrate community engagement across all components of the CTSA program were common themes. Respondents expressed skepticism as to the feasibility of strengthening CTSA community engagement.

The National Institutes of Health (NIH) launched the Clinical and Translational Science Award (CTSA) program in 2006 to transform the research enterprise at academic medical centers across the United States.1 The overall goal of the CTSA program is to facilitate research and its translation to real world settings,2 which requires broad‐based partnerships and “bi‐directional dialogue” between academic researchers and community and industry stakeholders.3 Specifically, the CTSA initiative aims to provide: a home for clinical and translational research, infrastructure to support clinical and translational research including that conducted with community and industry partners, training and preparation for the scientific workforce to engage in clinical and translational research, and mechanism for collaboration across and within sites.4 To date, 61 CTSAs have been funded with award totals ranging from 4 to 23 million dollars annually5; and for the 2012 fiscal year the program budget was 461 million dollars.6 Furthermore, 65% of the NIH extramural funding budget goes to institutions with CTSAs.6

Community engagement is a key function of the CTSA program and is critical to shortening the time between discovery and its application.2 With the exception of the most recent CTSA funding announcement, community engagement has been a required core component of CTSAs since their inception.3, 5

With the Fall 2012 move of the CTSA program from the National Center for Research Resources (NCRR) to the newly formed National Center for Advancing Translational Sciences (NCATS), Congress mandated an independent assessment of the CTSA program.6 The assessment conducted by an ad hoc expert committee convened by the Institute of Medicine (IOM) was designed to assess the efficacy of the overall program and to provide guidance with respect to program mission and goals moving forward.6 This paper describes feedback collected by two nonprofit organizations, the Center for Community Health Education Research and Service, Inc. (CCHERS) and Community‐Campus Partnerships for Health (CCPH), from a national group of CTSA stakeholders to inform the IOM study, with a particular focus on community engagement. Implications for the strategic direction of the CTSA program are then discussed.

The Delphi Method

The Delphi method (Delphi) is an exploratory method designed for group consensus building.7 The technique was developed in 1950 by the RAND corporation and was intended to be used as “judgment tool” to build consensus among experts on a topic.8 Delphi has been widely utilized over the last few decades by researchers and practitioners alike, and Delphi methods can be seen in scholarly publications spanning academic disciplines from nursing and social work, to policy, industry trades, and tourism. Delphi is ideal in the context of policy research and in circumstances characterized by complexity and limited knowledge, whereby experts in the field have the most up to date information.9

Delphi involves an initial survey or questionnaire that is presented to a diverse panel of experts. A facilitator then collects and analyzes the experts’ responses by identifying key themes and associated rationales, then compiles the data. The initial questions as well as responses are sent back to the group who again responds, giving consideration to the summary of themes and rationales provided by the facilitator. The process continues until the group nears consensus.7 A key element of the process is the anonymity, which is designed to limit bias and to center the discussion on the topic at hand.7 This method lends itself to exploring the value of community engagement in research, where power dynamics between the academy and community have been found to inhibit communication.10 Delphi assures every voice is heard and given equal weight.

Methods

In response to the IOM Committee's December 2012 public request for stakeholder input, the Delphi method was used to solicit feedback from a broad group of CTSA stakeholders. The Delphi was conducted in February 2013 through a partnership between CCHERS and CCPH.11 In employing the Delphi method, CCHERS and CCPH sought to gain a contextualized understanding of stakeholder perceptions of community engagement as a CTSA priority, how to both operationalize and improve community engagement, and to explore appropriate metrics for community engagement.

The partnership

CCHERS is a nonprofit community‐based organization in Boston that promotes the development of “academic community health centers,” that integrate education, research, and service to influence and change health professions education, improve healthcare delivery, and promote health systems change to eliminate racial and ethnic disparities in health. CCPH is a national nonprofit membership organization that promotes health equity and social justice through partnerships between communities and academic institutions. Since 2011, CCHERS and CCPH have been convening diverse stakeholders from across the nation to examine community engagement in CTSAs specifically and in publically funded research more broadly.12

Procedures

CTSA involved researchers and community stakeholders (n = 250) were invited to participate in the Delphi process. The invitees included members of the CCPH‐CTSA Member Interest Group as well as participants from two NIH‐funded National Community Partner Forums on community‐engaged health research. All participants were sent an electronic questionnaire, which included open‐ended items exploring participant perceptions of the CTSA program (a copy of the initial questionnaire and invitation are provided at the end of this paper). Follow‐up rounds (2) were also conducted electronically. Questionnaires were sent to the entire sample, despite participation in the previous round. Participants were given a 4‐day period to respond and were included in follow‐up rounds regardless of whether they responded in previous rounds. In each round, individuals were asked to identify their connection to the CTSA program: (1) community partner, (2) institution‐based researcher, (3) institution‐based staff, (3) stakeholder, or (5) other. Those who selected “other” were asked to describe their connection.

Round 1 included six open‐ended items exploring opinions of the CTSA program. In order to ensure alignment between the feedback provided and the charge of the IOM Committee, the questions used were identical to those posed by the IOM Committee to an invited panel on community engagement at its January 24th 2013 meeting13: (1) Why should community engagement be a priority, (2) How can the CTSA program be improved to more fully engage partners, (3) What are the benefits to community engagement, (4) What are the current gaps and barriers associated with engagement, (5) What metrics are being used to assess community engagement, and (6) What metrics should be used to assessment community engagement.

Participants were encouraged to provide thoughtful responses of up to 250 words assessing community engagement within the CTSA program based on their expertise and experience with the program. The project team analyzed group responses thematically by question. Common themes were then listed as choices for their respective Round 1 item. In Round 2 individuals were asked to rate their level of agreement with each of the choices based on the Round 1 data using a four‐point Likert scale. Finally, in Round 3, respondents were asked to rank the Round 2 items in terms of the potential impact on the CTSA program and implementation feasibility. The data for the second and third rounds were exported to Microsoft Excel 2007 and descriptive statistics were generated for each item.

Results

The response rates for Rounds 1, 2, and 3 were 32%, 46%, and 48%, respectively. Overall CTSA community partners represented the plurality of respondents, followed by institutional researchers and staff, as well as those who identified as other. The other category included former CTSA members, both community partners and research staff, as well as people affiliated with research institutions but not directly with CTSA. Table 1 illustrates the percent of responses by type of involvement with the CTSA program.

Table 1.

Respondent characteristics

Participants	Round 1	Round 2	Round 3
Number responding	n = 59	n = 83	n = 40
Community partner	45%	52%	50%
Institution‐based researcher	24%	19%	19%
Institution‐based staff	14%	14%	22%
Stakeholder	5%	8%	3%
Other	22%	18%	15%

With respect to why community engagement should be a priority four key themes emerged: it (1) makes the translation of results more likely, (2) ensures relevance and applicability of research to the concerned communities, (3) engenders trust in the community for research and researchers, and (4) improves participant recruiting and retention. An overarching theme was that health is local, the generation of knowledge to address community health is more robust if informed by communities (broadly defined), and the research translation efforts will be more successful if community knowledge is embraced. This point is illustrated by the response that follows.

The ultimate success of the entire health research enterprise depends on individuals and communities embracing and adopting the new scientific knowledge that it produces. The research process itself will ultimately be more productive if individuals and communities identify their priority problems and proposed solutions and, thus, frame the research questions themselves (35 respondents).

Participants were asked to reflect on how the CTSA program could be improved to more fully engage community organizations and patient advocacy organizations. Six key areas of improvement were identified, each illustrated below with a respondent quote. First was funding for community partners to participate. It was suggested by one respondent that CTSAs should develop and support mechanisms, which allow community and advocacy groups to be the prime awardees of CTSA supplement opportunities and other funding mechanisms for community‐academic partnerships that are affiliated with CTSAs (59 respondents). Beyond funding in and of itself, outlining community engagement expectations in the funding criteria was seen as a possible improvement. One respondent recorded that CTSAs should have financial and/or scoring incentives to demonstrate their level of community engagement (35 respondents). Further programmatic recommendations included, a clear delineation of expectations related to community engagement and the integration of community engagement across cores. One respondent recommended additional institutional resources devoted to foster and facilitate linkages between Community Engagement Cores with other Cores and with the community [are needed] (59 respondents). Finally, training and education were identified as ways to improve community engagement in the CTSAs. More specifically, education and training for institutional partners about the importance of community engagement and research literacy training for community partners.

Themes were then ranked on a scale of themes and average scores for impact and feasibility. 1–6, 1 being the lowest level of impact and least feasible and 6 being the highest. Themes are presented in Table 2. Column one lists the themes, while columns two and three outline participant perceptions of their impact and feasibility, respectively. Respondent rankings were averaged; a lower ranking represents a higher perceived level of impact/feasibility. Research literacy training for community partners and education for academic stakeholders on how to work with community were seen as having the highest impact on engagement and as being the most feasible with respect to implementation. Providing funding for community outreach and compensating partners was also seen as feasible, but ranked lower with respect to potential impact.

Table 2.

Recommendations for the CTSA program on how to fully engage partners

Themes from open‐ended responses	Impact*	Feasibility*
Provide sufficient funding for community outreach, including better compensation for community partners	4.52	3.18
Include specific expectations for what CE must involve	3.85	4.43
Include specific expectations for what CE must involve in funding criteria for CTSAs	3.42	4.02
Integrate CE into each core rather than keeping a separate core	3.53	3.71
Provide education and training to institutional partners concerning the importance of community engagement	3.25	2.79
Offer research literacy training to community partners	2.52	2.85

*Average based on 6‐point scale.

Respondents were asked to identify the benefits of engaging community organizations and patient advocacy groups in the CTSA. Common themes identified and their rankings with respect importance to the CTSA mission are presented in Table 3.

Table 3.

The benefits and successes associated with community engagement

Themes from open‐ended responses	Importance to the CTSA mission*
Health outcomes in the community are visibly improved and disparities are addressed	4.42
The research has improved relevance and efficacy	4.37
Trust is built between the community and the institutions	3.42
Recruitment and retention of participants is improved and more representative of populations of focus	3.29
Community partners offer fresh perspective to the research enterprise	2.97
Cultural consideration is addressed in the research	2.68

*Average based on 6‐point scale.

The benefits described by respondents were largely linked to translating research findings into practice or policy as illustrated by this response: Community organizations can also help tailor the research questions, research design, and use of the research findings to better serve the needs of the community (18 respondents). Respondents ranked the fresh perspective offered by community partners and the expertise they could provide with respect to cultural appropriateness as more important to the mission of the CTSA as less important to the CTSA mission.

Perceived barriers to engaging community in a meaningful way included: lack of respect for community engagement, community distrust of the academy, inequitable treatment of community partners, cultural disconnects between institutions and the community, little funding to support community partners.14 More importantly, however respondents were concerned there was no clear goal or mechanism for engaging community and that community engagement was not a CTSA requirement. As highlighted by one respondent:

There needs to be a clear goal around engaging these constituents in the research enterprise. They could be helpful in identifying the most important questions to ask to improve health and welfare. Each CTSA should have a community advisory group that supports this effort and also serves as a venue for dissemination and utilization of results (31 respondents).

Table 4 includes each of the themes identified as well as its potential impact on the CTSA program and its feasibility. Of the barriers identified the lack of funding for community infrastructure in the form of research literacy training and community distrust of academic institutions were seen as both having the greatest impact on the CTSA program and the most feasible to address.

Table 4.

The barriers to meaningful engagement

Themes from open‐ended responses	Impact*	Feasibility*
Institutional partners do not respect the value of community engagement in research	4.23	4.05
Inequitable treatment of community partners in distribution of funding and in governance/leadership roles	3.79	3.64
There is little or no funding for compensation of community partners	3.76	4
Established academic culture does not have proper protocol for engaging community partners	3.56	3.43
Members of the community distrust the institutions performing the research	3.45	2.9
There is little or no funding for research literacy training for community partners	2.18	2.95

*Average based on 6‐point scale.

The last two questions addressed by respondents were related to the measurement of community engagement, asking what current metrics were in place and their importance to CTSA leadership. Respondents identified ways community engagement in the CTSA program was being measured; however, some reported that community engagement was being measured inconsistently if at all. As noted by one respondent:

I don't think there are any established and unified metrics that are being used to measure CE across CTSA programs. There's been a lot of discussion but I don't think any things has been determined or adopted. This hurts CTSA CE programs ability to measure, document, communicate their value in a way that's understood by CTSA leadership—locally and nationally (56 respondents).

Known metrics are listed in Table 5. Column one describes the measure and column two includes each measures ranking with respect to perceived importance to CTSA leadership on a scale of one to three. Respondents felt CTSAs are using measures that may get at community engagement, but not so much at its impact. One respondent stated:

The CTSA program appears to depend on conventional metrics that are easily quantified: grant dollars awarded and manuscripts published. This will show an increase in the amount of research, but will not indicate the degree to which the research is translational (58 respondents).

Table 5.

Metrics being used to measure the progress of the CTSA program in community engagement

Themes from open‐ended responses	Importance to CTSA leadership*
Grant dollars awarded for community engagement research and related activities	2.28
Number of papers published by the CTSA	1.92
Number of community‐based organizations engaged or number of projects that incorporate community engagement	1.79

*Average based on 3‐point scale.

Respondents were then asked about what metrics should be in place to measure CTSA progress in community engagement. Those identified were ranked from one to six based on their potential meaningfulness to the CTSA program and the likelihood to seeing change or progress with respect to measurement in the next 5 years. The number of new projects initiated and the number of new or sustained partnerships were identified as the most meaningful measures for the CTSA program and the most likely areas in which to see change or progress anytime in the next 5 years. The full results are presented in Table 6.

Table 6.

Metrics that should be used to measure the progress of the CTSA program in community engagement

Themes from open‐ended responses	Meaningfulness as measure*	Likelihood of seeing change or progress in next 5 years*
Level of integration of community partners in the research team	3.59	3.36
Documented research outcomes (translation, community health outcomes, policy changes, etc.)	3.43	3.00
Allocation of funds to community partners and other nonacademic stakeholders	3.26	2.95
New and sustained community partnerships	2.79	2.90
Number of new projects initiated	1.92	2.79

*Average based on 5‐point scale.

Discussion

Delphi respondents overall articulated the benefits of community engagement in clinical and translational research and expressed the need to integrate community engagement across all components of the CTSA program. However, there was skepticism among respondents as to the feasibility of strengthening CTSA community engagement, perhaps reflecting the fact that community engagement in the last CTSA funding announcement was not explicitly required. Moreover, doubt may be associated with the lack of value placed on the real and potential contributions of community partners to research, inclusive of the lack of compensation for their time and expertise. These findings are supported by a recent study exploring the compensation and role of community representatives (CRs) in CTSAs that found that although most sites had CRs on their advisory boards and community engagement cores, few (11%) had CRs on CTSA‐wide leadership committees and their compensation was low.15

The Delphi study informed the IOM Committee, which in its report cited our preliminary findings5 recommending community engagement be integrated both across all CTSA components and across the continuum of clinical and translational research (T1‐T4). The director of NCATS has indicated his agreement with the IOM Committee recommendations and his intention to “implement them, beginning immediately.”16 Doing so, however, is likely to prove challenging given the barriers to community engagement in the CTSA program identified by Delphi respondents.

There are several limitations to our study that should be noted. We found a greater percentage of individuals participated in Delphi Rounds 2 and 3 than Round 1. The low response rate in Round 1 may be associated with the open‐ended nature of the questions in the round, as is typical of the Delphi method.17 Rounds 2 and 3 involved ranking, whereas Round 1 was more time consuming as it required a brief 250 word or less essay. This may introduce a degree of bias as the remaining rounds were shaped by the responses presented in Round 1. We collected data on whether respondents were based in community or academic settings, we did not ask respondents about other demographic characteristics. Lastly, our sampling strategy posed limitations. Although we drew upon existing networks of community and academic partners with expertise in community engagement and the CTSA program, we undoubtedly missed inviting the participation of key individuals involved in CTSA community engagement.

Despite its limitations, the Delphi process allowed us to engage a diverse group of stakeholders in gaining a more contextualized understanding of their perceptions of community engagement as a CTSA priority, how to operationalize and improve community engagement, and what might be appropriate metrics for community engagement.

Conclusions

The CCHERS‐CCPH Delphi process engaged key stakeholders in the CTSA program and served as a valuable tool for informing the recommendations of the IOM Committee given the number of times it was cited in the committee report. Our findings underscore the importance of engaging community stakeholders in the CTSA program while also uncovering critical concerns and challenges to moving community engagement in research from the margin to the mainstream. Although the NCATS Director has expressed support for the IOM recommendations, his ability to follow through may be limited by the direction of the National Institutes of Health more broadly, as buy‐in must be agency‐wide with the leadership of NIH extolling the critical importance of community engagement to shortening the time between discovery and application, and to improving the uptake of research findings by the public. There must also be buy‐in by the community and the easiest and most sustainable way of accomplishing this is by valuing, recognizing, and supporting genuine community engagement in the CTSA program and throughout the NIH.