A World Health Organization tool for assessing research ethics oversight systems

Abstract

Although most countries have ethical oversight systems for health-related research involving human participants, mechanisms for assessing the quality of those systems are not regularly used, particularly in low-resource settings. To address this gap, the Regulatory System Strengthening, Regulation and Safety unit and Health Ethics and Governance unit of the World Health Organization (WHO) recently released a tool for benchmarking ethics oversight of health-related research involving human participants. The tool provides a simple, easy-to-measure set of indicators for assessing the quality of research ethics oversight systems without the need to invest a great deal of resources. The tool comprises 48 indicators divided across three areas: (i) the national context; (ii) research ethics committees; and (iii) institutions that conduct health-related research involving humans, such as academic medical centres. Indicators related to the national context are intended to be evaluated in a single assessment applicable to the country as a whole, whereas indicators related to research ethics committees and research institutions are evaluated on an entity-by-entity basis. Some countries may choose to assess a representative sample of research ethics committees and institutions; alternatively, national authorities might ask research ethics committees and institutions to undertake self-assessments and report the results. Research ethics committees or institutions could also use WHO’s tool on their own as part of a process of quality improvement. WHO is working with global partners to disseminate the tool and support global implementation. Widespread use of the tool is expected to enhance policy coherence in ethics oversight and facilitate multinational research.

Introduction

A core requirement of international guidelines on the ethics of health-related research involving human participants is that an effective system of ethical oversight exists at institutional and national levels. Such a system should ensure that: (i) the proposed research is reviewed in advance by a competent research ethics committee; (ii) the research is monitored to verify that ethical requirements are being followed; and (iii) mechanisms are in place to promote accountability, transparency and public engagement. By now, most countries have established some sort of research ethics system. A few have developed mechanisms for overseeing the quality of those systems, including national bodies like the National Health Research Ethics Council in South Africa and regional collaborations like the SIDCER-FERCAP Foundation, which was established to promote the development of human research ethics.^1,2 Yet, in most of the world, particularly in low-resource settings, mechanisms for assessing the quality of research ethics systems are not regularly used.

To address this gap, in 2023, the Regulatory System Strengthening, Regulation and Safety unit and the Health Ethics and Governance unit of the World Health Organization (WHO) released a tool for benchmarking ethics oversight of health-related research involving human participants.³ This tool was developed under the guidance of an international expert advisory group, which included representatives of global networks of research ethics committees as well as leading researchers, regulators and academics from across WHO regions. A draft of the tool was first published on WHO’s website for comments and was subsequently revised to reflect the many insightful replies received. Before the tool was released, it was piloted and refined at workshops in Egypt, India, Kenya, Nepal, Nigeria and Pakistan, which together included participants from more than 100 research ethics committees, national ethics committees, national regulatory agencies and research institutions.

Goals of the research ethics oversight tool

The primary purpose of the research ethics oversight tool is to provide a relatively simple, easy-to-measure set of indicators that can be used to provide a basic assessment of the quality of research ethics oversight systems in countries and institutions without the need to invest a great deal of resources. The tool is not a normative ethical document. Instead, it takes as a given the ethical commitments articulated in existing international guidance documents, including long-standing WHO guidelines for research ethics committees;⁴ the comprehensive set of ethical guidelines for health-related research involving humans jointly issued by the Council for International Organizations of Medical Sciences in collaboration with WHO;⁵ and two widely accepted declarations on research ethics issued by the World Medical Association.^6,7 In essence, the tool is intended to provide a means of determining whether these guidelines are being adhered to.

The tool offers several potential benefits for health research and ultimately public health. First, the tool provides a basis for identifying areas in which a research ethics oversight system does not fully conform to internationally accepted ethical standards. Second, the tool makes it possible to establish benchmarks against which future improvements in system performance can be measured and to evaluate the impact of future policy changes. Third, the tool could potentially be used by partners in multinational research as a basis for confidently deferring to another country’s ethics review process, thereby reducing the number of unnecessary and time-consuming duplicative reviews. Finally, the tool provides a means for funders and others engaged in efforts to strengthen countries’ research ethics capacities to identify where resources could be most usefully used and to monitor the impact of any interventions they support.

One of the most important characteristics of a good set of indicators is that “the gathering of the required information must be technologically feasible and affordable and must not overburden the system.”⁸ With that goal in mind, rather than attempting to define a comprehensive set of quality measures, the tool focuses on a few basic indicators of quality that any well-functioning research ethics oversight system can reasonably be expected to possess. In developing the tool, an effort was made to strike a balance between the three major types of indicator: (i) those that assess the organizational characteristics of the system (structural indicators); (ii) those that assess the manner in which the system carries out its activities ( procedural indicators); and (iii) those that assess the extent to which the system achieves its objectives (outcomes-based indicators).

The research ethics oversight tool was modelled on WHO’s Global Benchmarking Tool,⁹ the primary means by which WHO evaluates and supports the strengthening of regulatory systems for medical products. The Global Benchmarking Tool consists of a series of indicators and subindicators related to the overarching national regulatory system and eight core regulatory functions: (i) registration and marketing authorization; (ii) vigilance; (iii) market surveillance and control; (iv) licensing establishments; (v) regulatory inspection; (vi) laboratory testing; (vii) clinical trials oversight; and (viii) national regulatory authority lot release. Together, these indicators are designed to cover the full life cycle of medical products.

More generally, WHO has long relied on indicators to measure adherence to internationally accepted ethical and regulatory standards. For example, WHO has developed indicators to monitor countries’ compliance with their obligations under the International Health Regulations.¹⁰ These indicators measure, on a scale of 1 to 3, the level of implementation of mechanisms for identifying and responding to global public health threats.¹¹ Similarly, WHO has established indicators for assessing countries’ progress towards meeting the goals of the Framework Convention on Tobacco Control.¹² Examples include indicators of: (i) whether countries have established and are enforcing smoke-free policies; and (ii) the extent to which countries have reduced morbidity and mortality from exposure to second-hand smoke.¹³

Overview of the research ethics oversight tool

The tool comprises seven categories and is designed to evaluate three aspects of the research ethics oversight system: (i) the national context; (ii) research ethics committees; and (iii) research institutions, which are defined as institutions, such as academic medical centres, whose employees or agents conduct health-related research with humans (Box 1). The seven categories cover the essential components of an effective research ethics oversight system, without which the system cannot be considered adequate. The international expert advisory group identified potential quality measures for each category through a brainstorming process, and ultimately settled on 48 unique indicators. For each of the 48 indicators, the research ethics oversight tool provides guidance for assessors, which includes a description of each indicator, examples of relevant evidence to review and a rating scale (Fig. 1).

Box 1. Assessment categories in the WHO global benchmarking tool for strengthening research ethics oversight systems, 2023.

1. Legal provisions and regulatory framework

2. Structure and composition of the research ethics committee

3. Research ethics committee's resources

4. Research ethics committee's procedures

5. Mechanisms for promoting the transparency of the research ethics committee

6. Mechanisms for the research ethics committee to monitor its own performance

7. Research institutions

Fig. 1.

Structure of the WHO global benchmarking tool for strengthening research ethics oversight systems, 2023

WHO: World Health Organization.

Note: The research ethics oversight tool’s categories are described in Box 1.

Indicators related to the national context are intended to be evaluated in a single assessment that is applicable to the country as a whole. Countries may choose to entrust these assessments to national authorities, such as a health ministry, a research ministry or another regulatory agency; or they could rely on external experts, for example, by commissioning a report from an academic institution. By contrast, indicators related to research ethics committees and research institutions must be evaluated on an entity-by-entity basis. In countries with a relatively small number of research ethics committees and research institutions, a single entity could feasibly conduct all the necessary assessments. However, in countries with an extensive research infrastructure and many local research ethics committees, the number of assessments required could reach into the hundreds. In these settings, national oversight authorities might assess a representative sample of research ethics committees and research institutions. Alternatively, national authorities might ask research ethics committees to undertake their own self-assessments and report the results. Research ethics committees might also choose to undertake self-assessments on their own initiative as part of a process of quality improvement. In general, externally conducted assessments are preferable because they help ensure that standards are applied consistently. However, when resources are constrained, self-assessments can be a cost-effective means of generating a large number of assessments over a relatively short time period.

Assessing the national context

The indicators used to assess the effect of the national context on the quality of the research ethics oversight system are included in Category 1: Legal provisions and regulatory framework. Most of these indicators relate to the extent to which the basic elements of the research ethics oversight system have been codified into legally binding instruments, such as statutes, regulations, decrees or legally binding guidance documents. For example, one indicator relates to the existence of legal provisions that explicitly require health-related research involving humans to be reviewed and approved by a research ethics committee, with the possible exception of specified categories of low-risk studies. Reviewers are asked to examine a country’s laws to determine whether this requirement has been either: (i) fully implemented (that is, there exist legal provisions requiring a research ethics committee to review all health-related research involving humans before recruitment begins, with the possible exception of specified categories of low-risk research); (ii) partially implemented (that is, there exist legal provisions requiring a research ethics committee to review only some types of research, such as government-funded studies); or (iii) not implemented (that is, there are no legal provisions requiring research ethics committee review).

Some of the national indicators are designed to ensure that research ethics committees have adequate legal authority to conduct their mission. For example, one indicator relates to the existence of legal provisions that explicitly give research ethics committees the authority to suspend or terminate a study if a committee determines that the study no longer meets the criteria that justified its initial approval. Another relates to the existence of legal provisions which ensure that a research ethics committee’s decision not to approve a study cannot be overruled, except when a regulatory agency or court has determined that an abuse of authority has occurred.

The national indicators also ask whether a country has legal provisions ensuring that research participants have access to medical treatment for any injuries directly resulting from their participation in the research; and that participants and their dependents are protected from the financial consequences of any injury or death suffered by the participant as a direct result of their participation. This particular indicator is grounded in a broad international consensus, as reflected in influential documents such as the Declaration of Helsinki,⁶ and in the legal requirements of some jurisdictions (for example, the European Union).¹⁴ However, several countries have not yet fully implemented these provisions, including some that host many research sponsors.¹⁵

In addition to legal provisions, one indicator for the national context relates to the existence of national, subnational, multinational or local oversight authorities, or their combination, that support research ethics committees and ensure they adhere to applicable ethical and legal requirements. Depending on the country, the oversight authority might be a health ministry, another government agency or an independent agency created specifically to provide oversight of research ethics committees. For a country to be considered fully compliant with this indicator, not only must an oversight entity exist but there must also be evidence that the entity provides ongoing technical assistance, coordination and monitoring of research ethics committees.

Assessing research ethics committees

The indicators used to assess research ethics committees are divided into five categories: (i) structure and composition; (ii) resources; (iii) procedures; (iv) mechanisms to promote transparency; and (v) mechanisms to monitor committees' own performance. Indicators related to structure and composition are designed to reinforce basic standards for the membership of research ethics committees, such as the necessary inclusion of nonscientific members, the appointment of members for a fixed term rather than indefinitely, and arrangements to ensure that members are not removed before their term expires unless they have been found to have substantially breached their duties. The research ethics oversight tool directs assessors to focus not only on what is stated in research ethics committee policies but also on the extent to which those policies are followed. For example, assessors are asked to determine whether the research ethics committee has a policy to invite nonmembers to contribute to the review of research that raises issues beyond the scope of the members’ expertise or experience, and also to determine how frequently nonmembers are actually invited to participate.

The indicators related to resources focus on: (i) whether the research ethics committee has sufficient and competent staff; (ii) whether members of the committee and staff receive initial and continuing training; and (iii) whether the committee has adequate facilities, equipment, technological support and financial resources. The research ethics oversight tool suggests the kind of evidence that can help assessors make judgments about the adequacy of resources. For example, with regard to financial resources, relevant evidence includes: (i) the sources of funding for the research ethics committee; (ii) the committee’s budget for the current and preceding year, with information on how those budgets were determined; (iii) information about any budget proposals by the committee that were denied, the basis for the denials and the impact of the denials on the committee’s operations; (iv) the number of applications reviewed annually by the committee; (v) committee member and staff surveys on the adequacy of the committee’s budget, if available; (vi) the committee’s annual financial report; and (vii) information about any activities that were not performed because of budgetary constraints. Ultimately, the assessor’s goal is to determine whether the research ethics committee has “adequate financial resources and reliable source(e)s of funding” and that the funding mechanism “does not create financial incentives to approve or reject particular studies.”³

The indicators related to procedures look at basic aspects of how research ethics committees conduct their activities. For example, assessors are asked to determine: (i) whether the committee has appropriate guidelines for the submission and screening of applications; (ii) whether it has written procedures to ensure that its deliberations adhere to relevant ethical criteria; (iii) whether it has adequate time before and during committee meetings for the meaningful review of research procedures; and (iv) whether it has procedures to ensure that decisions are made in a timely manner and are promptly communicated to the principal investigators. Reflecting WHO’s increasing emphasis on the need to conduct research rapidly during public health emergencies,¹⁶ assessors are also asked to determine whether the research ethics committee has procedures for ensuring the fast-track review of proposals in emergency situations.

Indicators in the areas of transparency and monitoring relate to specific types of information that research ethics committees should make publicly available (for example, committee guidelines and details of procedures and funding sources) or available on request (for example, a list of committee members and their titles, a list of principal investigators, and the dates on which research proposals were approved by the committee). Two indicators in these categories require assessors to determine the committee’s receptivity to questions and feedback from research participants and investigators. Relevant evidence for these indicators includes: (i) the availability of contact information for the committee on the committee’s website and on committee-approved consent forms; (ii) the existence of a comprehensive frequently-asked-questions webpage; and (iii) examples of actions taken by the committee in response to the feedback it has received.

Finally, the research ethics oversight tool includes three indicators that assess the extent to which research ethics committees proactively engage in efforts to monitor their own performance. The first indicator relates to the existence of: (i) mechanisms for obtaining feedback from investigators and research participants about their experience of the research studies; and (ii) mechanisms for instituting appropriate corrective actions in cases where problems are revealed. The second indicator relates to the existence of a system for ensuring the consistent and coherent application of policies and procedures. Such a system might include mechanisms such as regular debriefings with committee members and staff, or retrospective reviews of selected applications. The third indicator is more open-ended than the others; it relates to whether the research ethics committee regularly conducts internal reviews of its performance.³ The tool includes suggestions for ways in which a research ethics committee might engage in this kind of internal self-assessment but it does not recommend any particular mechanism. The point is simply to encourage committees to develop a practice of engaging in some form of self-critical analysis.

Assessing research institutions

The institutional context in which research ethics committees do their work is crucial for their proper functioning. The category for research institutions is the shortest section of the research ethics oversight tool. The category is not designed to provide a basis for conducting a comprehensive assessment of institutional quality, but instead to identify a few markers of institutions’ commitment to protecting research participants. The Council for International Organizations of Medical Sciences has just released comprehensive guidelines on good governance practice for research institutions,¹⁷ which will be complementary to WHO’s tool.

Several of the indicators in this category seek to ensure that institutions provide adequate oversight of their affiliated researchers. For example, the tool assesses whether an institution verifies that all proposals for health-related research involving humans are submitted to a registered research ethics committee if any part of the research is to be conducted by a researcher affiliated with the institution. Other indicators relate to the existence of ethics training programmes for researchers, and of policies on researchers’ declaration and management of conflicts of interest. Institutions are also expected to have a process for investigating allegations of unethical conduct by researchers, and for imposing consequences when unethical conduct is determined to have occurred.

A second focus of the institutional indicators is to ensure that institutions with their own research ethics committees adequately support those committees. Thus, one indicator asks whether such institutions ensure that their committees have access to necessary resources, from either the institution itself or external funders. Assessors are also asked to determine whether committees are provided with legal support, especially if an institutional research ethics committee has been challenged in court. Finally, assessors are asked to determine whether the institution facilitates the lodging of complaints by research participants or prospective participants about studies conducted by researchers affiliated to the institution, either through the institution itself or at a national or regional level. If a complaint system already exists for research participants at the national or regional level, the institution simply needs to include information about how to submit complaints through that system on the institution’s own website. If no such system exists, however, the institution would need to establish its own internal process for reviewing and responding to complaints.

Implementation and next steps

The WHO research ethics oversight tool is not the first instrument designed to measure the quality of research ethics systems; a 2020 review found nine other such instruments in use by national regulatory agencies and private entities throughout the world.¹⁸ However, this tool is the first explicitly designed to be adaptable to all countries regardless of their existing level of research ethics capacity or available resources. Countries with limited resources and a low volume of research activity can use the tool to quickly establish baseline measurements through self-assessments that rely on simple data collection methods, such as reviews of paper records and discussions with research ethics committee staff. Countries with more developed systems might choose to implement externally conducted assessments involving more elaborate data collection methods, such as focus groups of research participants or regular retrospective reviews of research ethics committee meeting reports.

The WHO tool comes with a related user guide,¹⁹ which explains how the tool can be implemented through self-assessment, peer review or external evaluation. The tool and user guide have been translated into five languages, and plans are underway to develop an electronic data submission tool that will facilitate assessment and scoring. In addition, WHO has made extensive efforts to disseminate the tool and support its implementation, including sponsoring implementation workshops in the Lao People's Democratic Republic and Senegal in 2024; and in Lithuania, Rwanda and Tajikistan in 2025. In responding to demands for additional workshops, WHO is working with its partners, which include the Paul Ehrlich Institute; the Global Network of WHO Collaborating Centres for Bioethics; and regional networks such as the African Vaccine Regulatory Forum, the Forum for Ethical Review Committees in the Asian and Western Pacific Region, and the European research ethics committees' network. We hope that these workshops and other international collaborations will lead to the widespread adoption of the research ethics oversight tool in all WHO regions, and that this will stimulate lasting improvements in countries’ health research ecosystems.