Developing a framework for evaluating ethical outcomes of Good Participatory Practices in TB Clinical Drug Trials

Abstract

Good Participatory Practice Guidelines for TB Drug Trials (GPP-TB) were issued in 2012, based on similar guidelines for HIV prevention and reflecting growing acceptance of the importance of community engagement and participatory strategies in clinical research. Though the need for such strategies is clear, evaluation of the benefits and burdens are needed. Working with a diverse group of global TB stakeholders including advocates, scientists, and ethicists, we used a Theory of Change approach to develop an evaluation framework for GPP-TB that includes a clearly defined ethical goal, a set of powerful strategies derived from GPP-TB practices for achieving the goal, and outcomes connecting strategies to goal. The framework is a first step in systematically evaluating participatory research in clinical trials.

Introduction

Community engagement in health research is increasingly promoted as a means of enhancing the ethical foundation of research, safeguarding scientific outcomes, and broadening the social benefits accruing from the research enterprise (Ahmed et al. 2010; Emanuel et al. 2004; NBAC 2001; NIAID 2011; Ramsay et al. 2014). However, community engagement practices are wide-ranging, and the health research and community contexts where they are applied vary along a multitude of dimensions (Lavery, et al., 2010; Tindana, et al., 2007). This makes it challenging to evaluate what works, under what conditions, and why (King, et al., 2014; MacQueen, et al., 2015). As a critical step toward meeting this challenge we developed an evaluation framework for ethics-driven engagement strategies developed for the context of a globalized clinical research agenda.

In 2012 the Stakeholder and Community Engagement Workgroup (SCE-WG) of the Critical Path to TB Drug Regimens (CPTR) released Good Participatory Practice Guidelines for TB Drug Trials (GPP-TB) (Boulanger, et al., 2013; Critical Path to TB Drug Regimens, 2012). The GPP-TB was adapted from the Good Participatory Practice Guidelines for Biomedical HIV Prevention Trials (GPP-HIV) (UNAIDS, AVAC, 2011), which were developed by UNAIDS and AVAC (an advocacy group focused on accelerating the ethical development of biomedical HIV prevention) with broad stakeholder input in the aftermath of global controversies surrounding early HIV pre-exposure prophylaxis (PrEP) trials. The PrEP controversies had many layers but ultimately centered on concerns about exploitation of vulnerable populations in resource-constrained settings (Mack, et al. 2004; Singh & Mills, 2005). Trials of new TB regimens raise similar challenges, though controversies of the magnitude of those seen for HIV treatment and prevention trials have not emerged. Some exploratory work on the generalizability and effectiveness of GPP-style approaches for fostering ethical outcomes has been conducted (Mack, et al., 2013; Newman, et al., 2015) but no formal evaluation attempted.

The GPP documents explicitly consider the challenges of disease-specific clinical trials coordinated through global networks and coalitions that bring together public, private, non-profit and for-profit stakeholders. In this regard GPP are a response to the emergent challenges of 21st century global health research where local communities and global stakeholders are increasingly connected. GPP provide a framework for integrating local and global perspectives and concerns. They represent a solution to the tension between universal ethical principles and local values, between aspirational human rights goals and the practical means of achieving aspirations in widely diverse contexts, without falling into the trap of ethical relativism. Reflecting this complex global reality, community engagement in the GPP framework is a component of a broader stakeholder engagement strategy that includes national and international stakeholders in addition to the local community.

The GPP-TB were developed to provide trial funders, sponsors, and research team members involved in TB drug trials with a principle-based framework on how to effectively engage stakeholders in TB drug trials. Developed with the primary intent of serving the needs of the CPTR initiative, the authors nonetheless hoped the document would be of service to a wide range of TB research audiences and contexts. The document provides a detailed overview of how community and stakeholder engagement are conceived for GPP-TB purposes and the rationale for establishing GPP-TB. Six principles and three benchmarks form the underlying ethical framework (summarized in Table 1). As stated in the document, “principles are values that can be adhered to, while benchmarks are outcomes indicating whether or not the principles are being realised” (Critical Path to TB Drug Regimens, 2012, p. 16). From these principles and benchmarks a set of good participatory practices are derived and organized according to the general sequence they are likely to be implemented over the course of a typical TB drug trial (see Table 1).

Table 1.

An evaluation framework for Good Participatory Practices in TB Clinical Trials.

Framework element	Definition	Notes
GPP-TB Ethical Goal	TB clinical trials demonstrate social value, achieve increased access across stakeholders, and meet standards of acceptability.	The ethical goal represents the ultimate impact sought through implementation of GPP-TB. The GPP-TB document states that “the unique social, ethical, and medical context of TB drug trials in endemic countries call for more robust engagement with host communities and stakeholders more broadly” (p. 15). The document does not provide an explicit statement regarding ethical goals, but notes that “the principles and benchmarks described…serve as the ethical framework underlying the GPP-TB” (p. 16).
Social value	A trial is responsive to global and local needs and equitably alleviating the burden of TB disease, where “equitably” refers to the fair distribution of benefits from the research.
Increased access	A trial is responsive to all TB stakeholders (end users, communities, researchers, drug companies, policy makers, providers) with regard to access to the drug/product, to trial results regardless of the outcome, to communities where trials can be conducted, and to markets for drug/product.
Standards of acceptability	A trial includes consideration of whether the drug/product is one that end users, decision makers, policy makers, providers and funders will use and support.
Powerful strategies	Core GPP-TB programmatic strategies intended to address the ethical goal.	These overlap with the practices outlined in Section 5 of the GPP-TB document but generally cross-cut those practiceswhich is what makes them powerful. Each strategy is intended to reflect practices to be enacted by multiple groups of stakeholders, reflective of the overall participatory approach encompassed within GPP-TB.
Accountability mechanisms	This strategy is powerful for ensuring transparency and ownership of the research process so that stakeholders achieve outcomes of integrity and efficacy through shared information. Mechanisms for accountability can include use of narrative stories in the monitoring and engagement processes that describe what went wrong and point to recommendations for avoiding similar mistakes or problems in the future. Baseline assessment is an important component so everyone understands where they are starting and can gauge change.
Community mapping	This strategy is powerful for establishing a description of the local context (ethnographic mapping). Through mapping we can identify needs (cyclical) and develop an understanding of community so that we can ensure research is mutually beneficial (complementarity of GPP-TB). It is also important to describe the research context and the global public health context as they relate to TB, to understand the opportunities, needs, and constraints within which research agendas are developed, funded, and implemented. Ethnographic understanding of research and public health communities is a needed complement to that of local communities.	The GPP-TB practices are organized in the general sequence of steps in TB drug trials: Site selection Community mapping Trial planning Engagement planning Communication planning Issue management planning Site activation Development of a policy on research-related harms Trial conduct Informed consent process Development of a policy on access to treatment and ancillary care Site closure Trial closure and results dissemination Access to investigational products
Shared learning	This strategy is powerful for providing awareness-raising among all stakeholders. Shared learning about community context may be evident in activities such as CAB work, uptake of local community suggestions by researchers, and contributions to the informed consent process. Shared learning about the research context may be evident in research literacy programs that ensure local community stakeholders are clear about what can be accomplished in a given trial and are empowered to decide with researchers how clinical trials are implemented. Measures of success may include mitigation of misconceptions about research and community contributions to research protocols and the language/vocabulary used to describe studies. Other anticipated outcomes of a shared learning strategy include enhanced stakeholder ownership of trials and/or the research process, transparency and accountability/efficiency/complementarity. Shared learning encompasses communication and engagement strategies.
Responsible advocacy	This strategy is powerful for engaging all TB stakeholders, ensuring resources are available to conduct TB clinical trials, and ensuring access as an element of the ethical goal of GPP-TB. It includes consideration of the role of regulatory bodies, pharmaceutical companies, reduction of barriers, and improving access once research is concluded.
Deliberation	This strategy is powerful for ensuring options for mutual gain are pursued when trade-offs in GPP-TB principles or benchmarks are needed. Deliberation entails “formal discussions and negotiation between the various stakeholders who have a legitimate interest in the consequences that a trade-off between considerations might have” (GPP-TB, p. 17). GPP-TB suggests the use of deliberation when there is a need to actively balance principles.
Short-term outcomes	Outcomes that we would expect to be able to observe and measure within a short timeframe as a result of implementation of one or more of the powerful strategies, e.g., by the end of one year. Increased transparency and integrity in representing individual and stakeholder group interests & expectations, and in maintaining scientific and ethical standards. Increased identification of needs and understanding of context: local community, research community, global public health community. Increased shared stakeholder knowledge: literacy on community context, research context, and TB. Increased awareness, political will, and resources for TB clinical trials and community engagement. Increased use of formal discussion & negotiation with all stakeholders to ensure options for mutual gain are pursued when trade-offs in principles or benchmarks are needed.	It is assumed that use of a given powerful strategy will result in an observable positive change over time. This means it is also assumed that the reason for using the powerful strategies is to support positive change and not simply to maintain a status quo.
GPP-TB principles	Values to be adhered to and which are assumed to help achieve the ethical goal.	The principles have been placed in the logic model as values that are realized through implementation of the powerful strategies. Values are not strategies or outcomes, but rather considerations to be upheld and balanced against each other.
Respect	Implies a special relationship based on acknowledgment, attention, and value.
Fairness	Refers to the way stakeholders treat and negotiate with each other, emphasizing honest acknowledgment of one’s interests and motivations to ensure there is no active or passive deception when negotiating with other parties.
Integrity	Refers to choosing actions that are consistent with one’s value system and living up to commitments and promises inclusive of representing interests of constituency groups. Includes adherence to accepted scientific standards.
Transparency	Refers to bringing to the open one’s interests and motivations in a given TB trial. Requires clear, honest, open, and timely communication inclusive of roles and expectations of stakeholders, the constituents they represent, and the extent to which their input may influence trial-related decisions.
Accountability	A bidirectional commitment to thinking through the justification for one’s actions, based in the recognition that one’s actions have direct repercussions on the various stakeholders involved in the partnership.
Autonomy	Recognizes the entitlement of stakeholders to support, amend, or refuse proposals to conduct research in the particular geographic location where they are active. It is dependent on knowledge and therefore requires opportunity to understand local, national, and global issues specific to a trial.
GPP-TB benchmarks	Outcomes indicating whether or not the GPP-TB principles are being realized.	In the logic model these are conceptualized as cross-cutting outcomes we would expect to be able to observe and measure as a result of principled use of the powerful strategies.
Mutual understanding	Sociocultural competency: understanding of cultural norms, practices, beliefs; social circumstances; diverse stakeholder perspectives; appreciation of implications of power differences Research competency: understanding scientific process; developing appropriate trial designs; collecting, analyzing & disseminating data to ensure valid results
Complementarity	Funders, sponsors, research teams & local public health systems partner to maximize the impact that research and health care have on local populations
Efficiency	Eliminating wastes of time and effort as much as possible
Intermediate and long-term outcomes	Measureable outcomes or results that cascade from meeting the benchmarks to achieving the ethical goal. Investigational products demonstrated to be effective in trials are made available to trial participants and local community stakeholders. Stakeholders are clear about what a trial can accomplish in a particular community. Stakeholders are empowered to decide together if a given trial is appropriate in a given community.	These outcomes reflect the cumulative impact of TB clinical trials research.

Actual application is nonetheless presumed to vary according to specific details of a given trial in a given context at a given point in history. Each step in the trial process is defined, its relevance to GPP described and special considerations noted, and practice details specified with reference to responsible parties (trial funders, sponsors, and research teams).

The relatively recent introduction of GPP-TB in 2012 provides a unique opportunity to look at the process by which global guidance is perceived, interpreted and implemented within diverse contexts. Working with global TB clinical trials stakeholders, we developed an evaluation framework for GPP-TB. The framework forms a critical first step in designing an appropriate evaluation strategy. Here we describe the process used to develop the framework, the assumptions (or hypotheses) underlying the framework’s causal logic, and a research agenda for conducting an evaluation based on the framework.

Method

To develop an evaluation framework for GPP-TB, we established a project advisory board and then brought together board members with other global TB clinical trials stakeholders for a 2-day meeting in Decatur, GA, USA in October 2013. The timing and location were chosen to take advantage of the annual meeting of the Community Research Advisory Group (CRAG) of the Centers for Disease Control and Prevention (CDC)-sponsored TB Trials Consortium (TBTC). Participants in the framework development process included CRAG members and CDC staff as well as other international TB advocates, ethicists, community representatives, researchers, and TB trials funders/sponsors (see Acknowledgments). Following the meeting, the evaluation framework was refined through ongoing discussion with members of the project advisory board. All were familiar with, and many had been actively engaged in the development of the GPP-TB.

We used a Theory of Change (TOC) framework to develop the evaluation strategy (Anderson, 2005; Connell, et al., 1995). All TOC approaches emphasize techniques that are collaborative, participatory and practical or applied. TOC frameworks link practices to outcomes in order to be able to explain how and why outcomes are (or are not) achieved. This is done by making assumptions explicit and hypothesizing why particular practices are expected to generate specific outcomes. A TOC approach emphasizes stakeholder participation in defining the causal pathway from existing conditions to a desired future. In alignment with the Theory of Change approach, we (1) sought consensus in defining a clear ethical goal of GPP-TB, (2) worked backwards to identify appropriate and reasonable strategies to achieve the goal, and (3) used an iterative participatory process to refine the framework.

Identification of the ethical goal

Consensus on the ethical goal was an essential first step to ensure we were all in agreement about what was to be evaluated and why. The group first reviewed the principles outlined in GPP-TB and concluded that the principles were more relevant for understanding how to achieve an ethical outcome. Discussion therefore focused on identifying the implicit ethical goal of implementing participatory practice guidelines. Core themes in the discussion included providing better TB treatment options, empowering people to make informed choices, the potential influence of GPP-TB on funder timelines, understanding the constraints and challenges researchers faced in the research process, and maintaining efficiency in finding new treatments. Stakeholders at the meeting were keenly aware of the limited resources available for TB clinical research, and the discussion highlighted a felt need to focus on GPP-TB contributions to improving the process of TB trials research. With these considerations in mind, consensus was reached with the following statement:

GPP-TB ETHICAL GOAL: TB clinical trials demonstrate social value, achieve increased access across stakeholders, and meet standards of acceptability.

Each of the components of the ethical goal (social value, access and acceptability) were then further defined (see Table 1).

Identification of strategies

The next step was to identify a set of powerful strategies for reaching the goal. To qualify as powerful a convincing argument or causal hypothesis had to be made for how a proposed strategy would lead to intermediate outcomes that in turn would lead to achieving the ethical goal. Each proposed strategy had also to be aligned with the GPP-TB principles. Community mapping quickly emerged as a powerful strategy for researchers and local stakeholders to act as effective partners. Community mapping refers to a broad set of methods for describing a local context geographically, socially, politically, and economically (Dunn, 2007; Parker, 2006). Development of a communications strategy was viewed as critical for increasing shared stakeholder knowledge, including local community understanding of research and researcher understanding of local communities. The importance of advocacy as a strategy was noted with reference to access to the results of research, be it a new medical technology or information. A key aspect of access centered on ensuring equitable patient access to drugs including consideration of the role of regulatory bodies, pharmaceutical companies, and reduction of barriers and improving access once research is concluded. Advocacy was described as also needed to ensure resources were available for TB research. Access was viewed as a double-sided issue such that drug companies needed access to markets in tandem with patients needing access to drugs. The issue of ownership of the research process and the concept of “meaningful engagement” were also discussed in relationship to access and as closely aligned with the GPP-TB principle of accountability. Discussions about accountability resonated with concerns for responsible advocacy, in particular, who advocates are speaking for and their credibility as representatives. Accountability was further referenced in relationship to trust as an element of engagement.

From an evaluation perspective, participants noted the importance of conducting a baseline assessment together with on-going assessments to determine if use of the strategies led to the hypothesized enhancements in ethical outcomes. Several stakeholders emphasized the importance of developing narratives and stories to inform monitoring and evaluation and to ensure accountability. Hypothetical scenarios were discussed to explore how use of the powerful strategies could result in short-term outcomes that were aligned with the GPP-TB benchmarks and principles, and how, over time, those outcomes could in turn lead to achievement of the ethical goal.

Refinement of framework elements

Following the meeting, notes were consolidated into a draft framework including definitions. This initial draft was then reviewed with the project advisory board through a series of conference calls where several issues were highlighted. Two notable refinements with regard to the powerful strategies emerged. First, there was considerable discussion around what a communications strategy implied. If the key point of the strategy was to increase stakeholder shared knowledge (community and research literacy), the group favored the use of “shared learning” to describe this powerful strategy. Second, a comparison of the initial draft of the framework with the GPP-TB document revealed that the initial draft framework did not include deliberation, a strategy highlighted in GPP-TB for balancing principles and benchmarks against each other if dilemmas arose.

Deliberation refers to formal discussions and negotiation between the various stakeholders who have a legitimate interest in the consequences that a trade-off between considerations might have

(Critical Path to TB Drug Regimens, 2012, p. 17).

Deliberation was seen as having some commonality with shared learning because deliberation would be difficult without it, but also as a distinct strategy that warranted standing on its own.

The advisory board also discussed the implications for the use of the word “increased” to describe short-term outcomes (Table 1). There were concerns that this wording could negatively impact researchers who may in fact be engaged in activities reflective of the strategies (e.g., providing resources to support community engagement, holding community events to describe research activities). Would they interpret the wording to imply ever-escalating and unreasonable demands? By way of counterargument, it was noted that emphasizing increases placed a value on continued striving to improve. In this vein, reference was made to quality assurance and quality improvement as meaningful models. It was agreed that additional language would be incorporated into the definitions to reflect the intent of focusing on improvements rather than enumerating deficits.

The advisory board further stressed that the strategies and outcomes can and should reflect a two-way process to demonstrate improvements for both the community and the science. The two-way concept was viewed as central to all elements of the discussion of GPP-TB and needed to be clearly identified throughout the logic model.

Results

Based on the combined contributions of participants at the stakeholder meeting and discussions with the project advisory board members, a final evaluation framework was developed. Five powerful strategies were identified along with measurable short-term outcomes or indicators that were assumed to follow from those strategies. The GPP-TB principles were to be realized through these outcomes, and were cumulatively expressed as the GPP-TB benchmarks. The cumulative impact would be measured as intermediate and long-term outcomes that ultimately contributed to achieving the ethical goal. Full definitions for elements in the framework are listed in Table 1; a simplified flow diagram of the core elements of the framework is provided in Figure 1.

Figure 1.

Overview of an evaluation framework for Good Participatory Practices in TB Clinical Trials.

Best Practices

The Good Participatory Practices model is a multi-layered and nuanced approach to supporting and enhancing the ethical foundations of global health research. As evidenced by the supporting documentation and training materials available and in development for GPP, resources and staffing for implementation are not trivial concerns.¹ Evaluation is an appropriate and needed endeavor to understand how GPP contributes to ethical goals, to assess the potential for unintended negative consequences, to improve the practices as needed, and to maximize the impact of the resources allocated to GPP.

We feel strongly that there is no need to prove the general concept that community engagement, stakeholder engagement, and participatory practices add value to health research. There is ample evidence of unwanted outcomes that can result in the absence of such practices, including exploitation of vulnerable populations, stigmatization of communities, unintended negative consequences, barriers to research and ultimate access to life-saving health interventions, and the perpetuation of historical mistrust in the absence of transparency and dialog. What is needed is thoughtful evaluation focused on improving engaged, participatory practices in order to maximize the potential for research to improve the health of communities and minimize the potential for harm. By doing so, research becomes more efficient as well as more ethical.

As seen in the process used here to develop an evaluation framework for GPP-TB, the assumed causal chains from discrete practices to achieving a broad ethical goal are complex, prone to feedback loops (and, hence, non-independence of measures), and contingent on starting conditions (e.g., historical and cultural particularities). Translating such an evaluation framework to a research design requires a thoughtful match between the evaluation questions asked and the methods used to generate the answers. In the case of GPP-TB we argue that the appropriate questions center on understanding how combinations of practices in varying contexts contribute to (a) realizing or (b) hindering achievement of the three elements of the ethical goal.

Research Agenda

Our framework provides a means of generating hypotheses about the causal pathways from site- and study-specific use of a GPP-based approach to enhanced ethical outcomes of the global TB trials research enterprise. For example, data can be collected on the use of each of the powerful strategies, including the variety and intensity of activities related to each, and on associated short-term outcomes or indicators. This would allow evaluation of the assumptions about the relationships between specific strategies and these short-term outcomes. Over time, the relationship between short-term and intermediate or long-term outcomes could be assessed by looking at multiple trials in multiple sites. For example, do cumulative increases in transparency and integrity correspond with shared decision-making about site involvement, availability of effective products, or clarity on potential trial outcomes? Are all five powerful strategies necessary to achieve the intermediate and long-term outcomes? Are there combinations of fewer than five strategies that are sufficient? Is there a specific strategy that is inadequate on its own but is a necessary component of all successful combinations of strategies? Do successful combinations of strategies or the intensity of effort needed within a strategy vary with the local context?

Full use of the framework requires analysis at the level of the trial, the site, and TB clinical research overall. Strategies used at a given site for a given trial are assumed to have local short-term impacts with regard to achieving GPP-TB benchmarks and enacting GPP-TB principles. Over time, it is assumed that continued use of the strategies will result in an iterative strengthening of benchmarks and principles. Broad use of strategies across multiple sites and multiple trials is assumed to result in cumulative long-term outcomes and achievement of the elements embedded in the ethical goal of GPP-TB.

There are, however, a number of practical challenges that must be addressed in conducting such an evaluation. First, the nature of GPP makes it problematic to use experimental designs that require randomization. It is unlikely that TB trials stakeholders would accept being told whether or not they can draw upon GPP guidelines or which strategies they are allowed to use when negotiating relationships in the context of clinical research in vulnerable populations facing a deadly disease. Such an experiment would be difficult, if not impossible to control, even if all parties were to agree that the attempt was ethical. The strategies included in the theory of change we developed are not specific to GPP-TB; they reflect best practices from the broader field of community engagement and participatory research that were identified through our consensus process as most powerfully aligned with GPP-TB. It may be feasible to randomize exposure of research teams at specific sites to packaged training on particular engagement strategies and tools to support engagement. Research teams may be willing to participate in such a study if they knew the full training package would be made available to all randomized arms at the end of the study. However, it would still be difficult (if not impossible) to prevent teams from using strategies and tools they are already familiar with or that they learn about through other mechanisms. The research question that would be answered in a randomized training trial would be the impact of the training on the use of strategies---an important but more limited question than the one we set out to address in developing our theory of change evaluation framework. In fact, a randomized training design would be stronger if first we evaluate the long causal chain from the powerful strategies through short-term and long-term outcomes to achievement of the ethical goal.

Second, GPP-TB is at heart a transformative intervention. This means that GPP-TB cannot be implemented in such a way that local context and dynamics can be controlled or held constant. Rather, the whole point is to transform context and dynamics. We therefore need to understand the impact of the powerful strategies on the context and whether that impact generates long-term outcomes aligned with the ethical goal. There will likely be multiple pathways to achieving that goal, reflective of the global and local starting conditions. This makes the use of quasi-experimental evaluation designs as problematic as fully randomized designs.

All these factors point toward the need for evaluation approaches that are capable of modeling multiple pathways to a desirable outcome and illuminating the dynamic interaction between local conditions and a diversity of potential practices. We chose to use a theory-based framework as the starting point for an evaluation design in order to support our ability to describe the cumulative impact of contexts, practices, and processes on both short term and long term outcomes [Weiss, 1995]. From here we envision an evaluation design that combines process, implementation, and realist evaluation. Process and implementation evaluation are closely related and focus on documenting how interventions are implemented as well as fidelity of the process to the specified intervention design (Rossi, Lipsey & Freeman, 2004). Findings are informative for understanding observed impacts and outcomes from the intervention. Realist evaluation centers on understanding causality through a focus on identifying “what works in which circumstances and for whom” (Pawson, 2002). In contrast to typical experimental and quasi-experimental designs, realist evaluation does not presume or require independence of variables. Rather, it assumes dependent (contingent) relationships. Realist evaluation has been used to explore partnership synergy and trust building in community-based participatory research (Jagosh, et al, 2015). All three approaches rely on the use of mixed methods (qualitative and quantitative). Of particular note is the recent application of qualitative comparative analysis (QCA) to realist evaluation (Sager & Andereggen, 2012; Thomas, O’Mara-Eves & Brunton, 2014). QCA differs from typical statistical approaches to analyzing causality in its use of Boolean algebra, set theory and minimization logic to identify necessary and sufficient conditions that account for both observed outcomes and their absence (Schneider & Wagemann, 2012). As with process, implementation and realist evaluation, the QCA method is theory driven. It is also less constrained by sample size considerations than statistical approaches, since it is explicitly concerned with dependent relationships rather than measuring independent effects.

Educational Implications

The process of developing the GPP-TB evaluation framework brought home the limited attention and resources given to evaluation of the presumed ethical benefits of participatory research models. The evaluation challenges reflect the complexities of policy, advocacy, and community-based interventions, as distinct from the kinds of individual-level interventions that clinical researchers, funders and sponsors are more familiar with. Clinical research networks have a solid understanding of rigorous clinical trial research designs but generally have limited understanding of how to evaluate a context-dependent policy intervention such as GPP. If we are to improve our understanding of effective and ineffective participatory practices in clinical research networks, funders and sponsors need to improve their understanding---and acceptance---of alternative evaluation designs.

In addition to the theory-driven evaluation approach outlined in this paper, there is a need for practical monitoring and evaluation of participatory research models. To this end, a group of partners from across the TB and HIV research field have developed a new set of monitoring and evaluation tools that introduce a framework of indicators by which the impact of community and stakeholder engagement on certain phases and outcomes of clinical research may be measured (Hannah, et al., 2014). This toolkit, Engagement for Impact, is being piloted at the clinical research site-level by personnel responsible for implementing GPP and engagement programs. Analysis of the resulting data holds promise for building an evidence base for the mechanisms by which incorporation of GPP and engagement programs into clinical trials improves practice and outcomes.

Evaluating outcomes and impacts with the intention of improving participatory practices in diverse settings requires a set of skills and knowledge that are distinct from what is required to develop and support implementation of participatory guidance. There is much to be gained from including evaluators in the development of participatory guidance and policies so that evaluation becomes an explicit, integrated component of the guidance. This requires going beyond the inclusion of a generic overview of evaluation design to developing and refining appropriate tools for conducting evaluations that can answer well-defined questions of process, outcome, and impact. An evaluator perspective alone is not sufficient, however. Any evaluation that seeks to understand and enhance the use of participatory research methods needs to include community and stakeholder knowledge and experience in the evaluation design and implementation.

Biographies

Kathleen MacQueen is a senior social scientist at FHI 360. She has conducted extensive research on the social, behavioral and ethical aspects of biomedical HIV prevention trials, which in turn has led to work on participatory research more broadly. She is the principal investigator for this study, provided overall leadership for the manuscript, and led the writing on all sections.

Natalie Eley is a research associate at FHI 360. Currently, her research interests are participatory approaches, ethics, and social health research. She is the study coordinator for the research study reported and was responsible for documentation of the evaluation framework development process, which formed the basis for the sections on identifying and refining the framework elements.

Mike Frick is the TB/HIV project officer at Treatment Action Group in New York, NY. He coordinates the Community Research Advisors Group, the community advisory board to the U.S. Centers for Disease Control and Prevention’s TB trials network, and conducts advocacy to support community engagement in TB research. He assisted in the facilitation of the project advisory board and development and refinement of the theory of change framework described in this paper.

Laia Ruiz Mingote is a journalist by training who has devoted her professional career to the promotion of human rights; she is a strong advocate of the inclusion of affected communities in the public health decision-making process, in order to avoid what she calls the “Enlightened Absolutism of Science.” She is a member of the advisory board for this project and a former co-chair of the Community Research Advisors Group. She contributed to the development and refinement of the theory of change framework described in this paper.

Alicia Chou is a project manager at the Reagan-Udall Foundation (RUF) for the FDA working on the Critical Path to TB Drug Regimens (CPTR) project. She supports the Stakeholder & Community Engagement Workgroup, which focuses on developing methods and tools to facilitate patient and stakeholder engagement and good participatory practice in TB research. More broadly, she assists in developing engagement strategies to ensure broad stakeholder participation in all of RUF’s work areas. She contributed to the development and refinement of the theory of change framework described in this paper.

Stephanie Seidel is a Senior Manager of Community Engagement and Stakeholder Relations at the TB Alliance, and is focusing on the development and implementation of research site-level strategies for community involvement in clinical trials. She is a member of the advisory board for this project and a partner in the development of the Engagement for Impact monitoring and evaluation toolkit for the measurement of community engagement impact on clinical trials. She contributed to the development and refinement of the theory of change framework described in this paper.

Stacey Hannah is a Senior Program Manager at AVAC where she manages AVAC’s Good Participatory Practices program and oversees training related activities of AVAC and its partners. She is a member of the advisory board for this project and a partner in the development of Engagement for Impact, a monitoring and evaluation toolkit for impact of community engagement on clinical trials. She contributed to the development and refinement of the theory of change framework described in this paper.

Carol Dukes Hamilton is Director of Scientific Affairs at FHI 360 and Professor of Medicine at Duke University. She has led and participated in TB and TB-HIV clinical trials for over 20 years, including work focused on making the informed consent process more informative and less cumbersome, and streamlining IRB review in multi-site clinical trials. She is a co-investigator for the study, serving as liaison with TB Trials Consortium leadership and site PIs, contributing to protocol and the development and refinement of the theory of change framework described in this paper.