Community-Researcher Partnerships at NIAID HIV/AIDS Clinical Trials Sites: Insights for Evaluation & Enhancement

Abstract

Background

Community engagement has been a cornerstone of NIAID's HIV/AIDS clinical trials programs since 1990. Stakeholders now consider this critical to success, hence the impetus to develop evaluation approaches.

Objectives

The purpose was to assess the extent to which community advisory boards (CABs) at HIV/AIDS trials sites are being integrated into research activities.

Methods

CABs and research staff (RS) at NIAID research sites were surveyed for how each viewed: a) the frequency of activities indicative of community involvement; b) the means for identifying, prioritizing and supporting CAB needs; c) mission and operational challenges.

Results

Overall, CABs and RS share similar views about the frequency of community involvement activities. Cluster analysis reveals three groups of sites based on activity frequency ratings, including a group notable for CAB-RS discordance.

Conclusions

Assessing differences between community and researcher perceptions about the frequency of, and challenges posed by specific engagement activities, may prove useful in developing evaluation tools for assessing community engagement in collaborative research settings.

Introduction

In 2006, the U.S. National Institute of Allergy and Infectious Diseases (NIAID) restructured its clinical trials networks for HIV/AIDS, and initiated development of a comprehensive utilization-focused evaluation system for this program. A collaboratively-authored evaluation framework identified factors critical to the success of this large research initiative (1), among which community involvement and the relevance of network research to community were ranked highly, across a broad array of stakeholders. This generated the impetus to develop measures to meaningfully assess the activities that constitute community engagement.

Historically, community participation has been a cornerstone of NIAID's research activities since early in the HIV/AIDS epidemic (2, 3). In 1990, addressing demands of AIDS activists, NIAID established Community Advisory Boards (CABs) to provide community input to network trial design and prioritization. Soon thereafter NIAID made community involvement a requirement for clinical research site funding (2). To support community representatives and enhance their meaningful contribution, NIAID developed training programs for CAB members throughout the networks. In 2006 Community Partners was formed with the mission to: 1) improve intra- and inter-network community input; 2) support training for local CABs; 3) increase representation of vulnerable populations and communities in resource-limited areas; 4) address challenges to trial participation, and; 5) harmonize best practices for community participation.

With more than 20 years experience fostering community partnerships, NIAID recognizes an opportunity to begin evaluating community-researcher engagement. Whereas early assessments were limited to determining whether or not a network or site had a CAB, we are now exploring the extent to which CABs are being effectively integrated into research activities. This report details results from a survey of CABs and research staff (RS) at NIAID sites. The survey was designed by Community Partners, in association with the Office of HIV/AIDS Network Coordination (HANC), and NIAID staff. Iterations of the survey were critiqued and modified by each group, individually and together, until a consensus was reached. The survey was intended to assess: 1) the frequency of activities indicative of: community involvement in protocol selection and implementation, efforts to assure research relevance, and communication/collaboration; 2) means by which community needs are identified, prioritized and supported; 3) mission-related and operational challenges faced in community-researcher collaborations.

Materials and Methods

Survey purpose and design

A self-administered survey questionnaire was utilized by HANC, to reach the globally-distributed clinical trials sites. Both CABs and RS were queried about the frequency of occurrence of specific activities, and the types of challenges faced in their engagement. The rationale for this approach was to learn about how similarly (or dissimilarly) these partners viewed the functioning of their engagement. Respondents were asked about: 1) the frequency of activities reflective of community involvement in protocol selection and implementation, assuring research relevance to community, and community-researcher communication and collaboration; 2) mission and operational challenges; 3) means for identifying, prioritizing and supporting CAB needs. The community involvement activities selected for study, and the mission and operational challenges selected for assessment in the survey, were derived from a guideline and best practices document authored by the Community Partners, “Recommendations for Community Involvement in NIAID HIV/AIDS Clinical Trials Research” (4). Given the importance of this document in guiding how sites should approach collaboration and interaction with the community, framing our assessment with these activities was viewed as a logical first step in a more evolved understanding of community engagement. RS were requested to consult site research colleagues and complete a single survey, per site. CABs were requested to meet, review the survey, and designate an individual to submit a single response for their CAB. Respondents were given six weeks to complete and enter their data directly online, using Survey Monkey (5).

Sampling

From 163 clinical research sites and 138 CABs, 105 RS (64%) and 92 CAB (66%) responses were received. Fifty-six sites submitted both a CAB and RS response, six of which were excluded for missing data, with the final set for analysis consisting of data from 50 paired responses. Within pairs, CAB and RS reported on the same community-researcher collaboration, providing a unique opportunity to learn how researchers and community members working together, view their collaboration. Whether the 50 sites selected for analysis are representative of the all sites is uncertain. The subset selected for the paired analysis appeared similar in several respects to the overall set of responding sites (n=105; 64%), but may have differed somewhat from the non-responding sites (n=58; 36%), among which a slightly higher proportion were affiliated with prevention research, and a slightly lower proportion were involved in both treatment and prevention research (Table 1).

Table 1. Site Demographics.

Site Descriptor	Sites Submitting a CAB1 Response (n = 92)	Sites Submitting a RS2 Response (n = 105)	Sites Submitting Complete CAB and RS Responses (n =50)	Sites Not Submitting a RS Response (n = 58)
US-based	54.9	52.0	66.0	65.5
Non US-based	45.1	48.0	34.0	34.5
Affiliation with treatment research	58.7	58.1	60.0	62.0
Affiliation with prevention research	16.3	14.3	12.0	24.1
Affiliation with both treatment and prevention research	22.8	26.7	28.0	13.9
> 50% former/current clinical trial participants on CAB	34.2	34.0	31.8	NA3

All results expressed as percentages of total (n) for the column

¹CAB: Community Advisory Board;

²RS: Research Staff;

³Not Available

Data analysis

Our analyses focused on three main questions: 1) what, if any, similarities exist in the frequency rating patterns across different best practice activities? 2) how are sites grouped in terms of how CABs and RS rate activity frequencies? and, 3) based on this grouping, are there operational or mission-related challenges that are associated with sites? First, to evaluate the similarity of CAB and RS responses on the activities for community involvement, research relevance, and communication and collaboration, we conducted a multivariate profile analysis for each activity set. This enabled us to determine if different patterns of ratings were present, and if so, on which activities there were differences. To identify variation of specific activities from the mean frequency within each set, we evaluated the profiles in terms of activities for which group averages fell outside of the confidence interval of the pooled profile. We compensated for multiple testing of activities within each set by establishing an alpha rate of .01 generating a 99% confidence interval. Second, a case-wise hierarchical cluster analysis using Ward's algorithm(6) was conducted to classify groups of sites, based on the CAB and RS frequency ratings across the different activity areas. Finally, we examined the mission-related and operational challenge assessments in relation to the clustered sites through a separate univariate analysis of variance of the overall number of challenges by cluster of sites. Subsequent to these overall mean difference tests, we examined which challenges, if any, were significantly different by cluster.

Results

Comparison of best practice frequency ratings

CAB and RS rated (on a seven point Likert-type scale, with: 1=never; 7=always) the perceived frequency of occurrence of 25 best practice activities, eleven of which were CAB-based and fourteen describing RS functions. In using this scale, we assumed frequency could be measured as an interval variable with the primary interest to determine if the two groups differ in their responses. Analysis of data from the 50 paired responses revealed a striking degree of similarity between the two groups. We detected no difference in the multivariate profiles of CAB and RS frequency ratings for the sets of activities related to community involvement, relevance of research, or communication and collaboration (Figures 1-3). Also, activity frequency ratings by both groups appear unrelated to whether an activity is specific to CABs or RS. We did observe significant variation in some of the activities reported to occur most frequently and least frequently within the community involvement [F(10, 89) = 13.30, p < .001], relevance of research [F(5, 94) = 5.36, p < .001], and communication and collaboration activities [F(7, 92) = 35.64, p < .001]. Given the variation (non-flatness) of these rating profiles, we assessed which of the rated activities significantly deviated from the pooled means for the respective set of activities. Applying a 99% confidence limit to account for multiple testing, we examined the marginal means for each activity in a set relative to the estimated parameters of the pooled means for each set.

Figure 1.

Community Involvement activities. Marginal mean frequency ratings, by CAB and RS. Dotted lines represent the pooled mean ratings for all activities in the set. Rating Scale: 1=Never; 7=Always. Abbreviations: CAB = Community Advisory Boards; RS = Research Staff. * For each pair, activity rating significantly different than the pooled mean at p < .01.

Figure 3.

Communication and Collaboration activities. Marginal mean frequency ratings, by CAB and RS. Dotted lines represent the pooled mean ratings for all activities in the set. Rating Scale: 1=Never; 7=Always. Abbreviations: CAB = Community Advisory Boards; RS = Research Staff. * For each pair, activity rating significantly different than the pooled mean at p < .01.

Figure 1 shows that CABs and RS rated the activity frequency for RS update the CAB about site studies significantly higher than the respective pooled means for the entire set of community involvement activities. In contrast, RS and CABs indicated that the activity: CABs assist the community in understanding informed consents occurred significantly less frequently than their respective pooled means for the set of community involvement activities. Figure 2 shows that CAB and RS rated the activity RS involve CAB in reviewing site research plan significantly lower than their respective pooled means for the set of relevance of research activities. Figure 3 shows that CAB and RS indicated that the activities RS support CAB activities and are actively engaged and CAB and RS work together to clarify the mission and the work occur significantly more frequently than the respective pooled mean for the entire set of communication and collaboration activities. In contrast, CAB and RS rated the activities RS performs formative research and stakeholder analyses, CABs identify self-evaluation criteria and discuss with research staff, and CABs work with research staff to share information with community as occurring significantly less frequently than their respective pooled mean frequencies for the entire set of communication and collaboration activities.

Figure 2.

Relevance of Research activities. Marginal mean frequency ratings, by CAB and RS. Dotted lines represent the pooled mean ratings for all activities in the set. Rating Scale: 1=Never; 7=Always. Abbreviations: CAB = Community Advisory Boards; RS = Research Staff. * For each pair, activity rating significantly different than the pooled mean at p < .01.

Although highly comparable patterns of responses were found between CAB and RS overall, the range of frequency ratings from CAB and RS (data not shown) suggested that there may be substantial variation among individual sites. As such, we sought to determine if there were subgroups of sites. Using individual sites' summed total frequency ratings for the community involvement, relevance of research, and communication and collaboration activities, from both CAB and RS responses, a hierarchical cluster analysis produced three groups. Figure 4 shows the plot of average CAB and RS ratings for the three groups of sites, across the three activity areas. The largest group of sites (n=19) was characterized by high average frequency ratings (mean range =5.62 to 6.41) across all three areas of activity, from both CABs and RS (High group). A second group of sites (n = 17), was characterized by substantially lower average frequency ratings across all three areas, from both CAB and RS (mean range = 3.51 to 4.14) (Low group). A third group (n = 14) was characterized by high frequency ratings for activities related to community involvement, relevance of research, and communication and collaboration as assessed by CABs (mean range = 6.34-6.42), but substantially lower frequency ratings for all three activity areas as assessed by the RS (mean range =4.25 to 4.75). This group, (Mixed group), representing 28% of the sites, appears quite distinct from the rest, with a level of apparent discordance between CAB and RS not seen in the other 72% of sites.

Figure 4.

Site clusters: Mean frequency ratings, by CAB and RS, by site cluster (High, Mixed, Low). Data represent the mean rating for all activities within each of three sets: Community Involvement; Relevance of Research; Communication and Collaboration. Rating Scale: 1=Never; 7=Always. Abbreviations: CAB = Community Advisory Boards; RS = Research Staff.

Mission-related and operational challenge assessment

CABs and RS also provided information indicating which activities, or aspects of community-researcher interactions, posed significant challenges. Challenges were categorized in the survey, as either mission-related or operational-related, with respondents selecting all that applied from a checklist. Among the three site groupings identified in the cluster analysis there were no differences in the overall mean number of mission-related challenges by group as reported by either CAB or RS. However, the presence of two particular challenges, when indicated by both CABs and RS as significant, was associated with a specific group. CAB member recruitment and participation and Effective communication between CAB and broader community were indicated as challenges more often by the Low group than by either the High or Mixed. There were no differences in the overall mean number of operational-related challenges by group, as reported by either CAB or RS, and none of the operational-related challenges was significantly associated with any of the groups.

CAB needs assessment, budget awareness, resource prioritization

We also gathered information regarding: a) the frequency and means by which CAB support needs are assessed; b) budgets for CAB activities, and; c) means and impact of prioritizing CAB resources. The strong overall agreement between CAB and RS persisted. Both reported that almost two-thirds (CAB: 63.8%; RS: 64%) of sites use more than one method, and nearly all CABs have their needs assessed at least yearly. Similarly, CABs and RS agreed on CAB budget awareness, with 32% of CABs, and 42% of RS (difference NS) reporting that their CAB was aware of the budget for CAB activities. Finally, regarding CAB expenditure prioritization, 44.9% of CABs and 46.0% of RS indicated a joint effort involving both partners. Sixteen percent of CAB and 20% of RS (difference NS) reported that the means by which CAB resources are prioritized pose obstacles to CAB functioning. None of these variables differed significantly among the three groups of sites.

Discussion

Since its initiation in 1990, community engagement has emerged to take its place among the factors now considered as critical to the success of NIAID's HIV/AIDS clinical trials networks (1). However, whereas other factors (e.g., scientific objectives, operations and management, resource utilization) have been subject to evaluation, little has been done to research the practices associated with effective community engagement or its meaningful integration into research activities. This study reports on our initial effort to explore this vital yet complex aspect of NIAID's HIV/AIDS clinical trials programs at the research site level, to better understand the extent to which community engagement is occurring.

Our most salient finding was that overall, CABs and research staff had a high degree of concurrence about the frequency with which a diverse array of activities took place at their sites (Figures 1-3). This agreement was robust, observed in all three activity categories, and across a broad range of frequency ratings, and was not related to whether an activity was CAB- or RS-specific. We presumed that observed congruence is indicative of a common view of the extent to which the recommended best-practices to community engagement manifest at research sites. We surmise that our results signify, for most CABs and RS, a good awareness of the engagement activities at their sites.

Despite strong overall CAB-RS agreement, cluster analysis revealed a subset of sites (28%) in which CAB and RS activity assessments were discordant. Within this group, CABs consistently gave high frequency ratings for all three activity sets, compared to RS. At this time, we are unable to identify the reason(s) for the discordance. None of the factors including: 1) US vs. non-US; 2) prevention vs. treatment research; 3) proportion of CAB members with experience as trial participants; 4) means or frequency of CAB needs assessment; 4) budget awareness among CAB members; or 5) means of CAB expenditure prioritization, correlated with any of the clusters. Future research will investigate other potential correlates: site longevity, CAB membership turnover, and funding for community activities (including paying CAB members) to determine if any of these distinguish discordant sites from the others. We observed one cluster-specific association, albeit for the “Low” frequency group, in their joint (both CAB and RS) designation of two significant mission-related challenges: CAB member recruitment and participation and Effective communication between CAB and broader community. These findings bear some resemblance to those of Cox et al. (2) who described a subset of AIDS research unit-affiliated CABs with lower participation.

We found variability in the perceived frequency of different activities, with several occurring at above average rates. Why these activities are reported to take place more frequently than others is unknown. Several explanations are plausible, including CAB and RS skills in addressing the activities, the general nature of the activities with multiple elements contributing to the frequency, and the importance of the activities to CABs and RS. By contrast, we also saw four low frequency activities potentially signifying their relatively lower importance, or an ineffectiveness in the CAB-research staff interaction. In the case of the evaluative activities by CAB and RS receiving the lowest frequency ratings, it may be that sites lack expertise or resources to support such work, that they happen at much longer intervals or they hold a lower priority. Collaborative, participatory data analysis of these findings, and future studies shall seek to extend these results and aim to identify activities that should be given priority for community engagement.

There are limitations to our approach and results. For instance, the use of a self-administered survey, the research partners' objectivity, the collection of a single, consensus-type response from each site, and the assessment of the perceived frequency with which activities ‘usually’ occur at sites, (7,8) constrained our ability to fully understand the complexity of community engagement. At this early juncture, it would be premature to generalize our findings to all sites. Though the subset of sites selected for analysis appears similar in several respects to the overall set of responding sites, slightly greater than a third of sites did not return a survey. If the non-responding sites were lower functioning (or differed in some other material way), their absence may have biased our findings. Also, our analytic treatment of the 7-point response scale as interval data (rather than ordinal) could be subject to debate. Our primary interest was in seeing if CABs and RS differ, rather than precisely estimating the central tendency of the frequencies. Thus, even if the data were ordinal, we believe that they would be similarly ordinal for both groups. Unless the nature of the “misinterpretation” of intervals differed systematically between the two groups, similar non-normal distributions in both groups would not bias a test of mean differences. As such, given the intent of our analysis, we believe that our treatment of the data as interval is justified in this context. Our survey may also have had an observable process use effect (9) as indicated by respondent comments reflecting the stimulation of evaluative thinking and learning from the survey questions. Further research is needed to determine the replicability of findings or their relevance to other contexts, given the many different models of community engagement in health research (10-15). We studied CAB-RS partnerships because this has been the primary form of engagement in our programs (2, 16-18) for many years. Though CABs certainly can vary in many ways (2, 16-21), the NIAID site CABs are joined in a well-established long-term partnership framework, operate in similar contexts, and follow a shared set of community-authored best practice guidelines (4). As such, we believe that their comparison is reasonable.

Many have pointed to the challenges of evaluating the effectiveness of community engagement in health research including the complexities of defining community and specifying what is meant by effectiveness (10, 13, 20, 22-26). The National Institutes of Health Clinical and Translational Science Awards (CTSAs) have outlined fundamental principles and guidelines (27). Our approach is distinct in that it used a collaborative, participatory process as the basis for conceptualizing not only the survey described here, but a larger, integrated evaluation framework that identified several categories of success factors, outputs and measures for evaluating the NIAID HIV/AIDS clinical trials networks (1). We also utilized a systems thinking perspective (28, 29) in the way that we integrated different success factors (e.g., scientific agenda setting, relevance of research to communities, communication and collaboration) with the evaluation of the community engagement. Rather than looking at community engagement in isolation as a ‘stand alone’ activity, we seek to study and understand it in the context of other key success factors so we can learn how they interrelate. To date, we are unaware of any comparable efforts to assess community engagement in the context of such a large, complex research initiative. In our context, community engagement is represented in the form of CAB-RS partnerships, with effectiveness seen primarily in terms of the beneficial impact on the research. Green and Mercer (15) have indicated that, in such partnerships, the benefits to research are usually in the form of improved applicability and usability in the settings in which the studies take place. Our study gathered quantifiable data about three sets of activities that relate closely to these benefits. However, since these activities are processes and not outcomes, they do not measure the effectiveness of CAB-RS partnerships on the research, but with further study, may be shown to be useful indicators.

We plan to explore other means of assessing the impact of community engagement in our research. Presuming that effective community-researcher partnerships result in studies with greater relevance and fit to local communities, we plan to investigate trial parameters (e.g., screening, enrollment, and retention rates, meeting demographic targets, etc.) which may be indicators of quality engagement (if other influences are accounted for). Systematic case studies of select protocols, especially if integrated with process modeling, could provide a detailed picture of how CABs and RS engage in protocol selection and implementation, and point towards better or additional measures. We also seek collaboration with experts in theory-based approaches (30), especially if these can be adapted to address ways in which partnerships influence research.

Because most research does not get translated into health improvements (31-35), researchers and funders are increasingly accountable for demonstrating impact (36, 37) and assessing “payback” (38). Community partnerships can play a vital role in knowledge dissemination for policy making and implementation (39, 40). These and subsequent efforts to improve evaluation will be a key component in the effectiveness of these partnerships.