An Evaluation of the Middle East Research Training Initiative (MERETI) Tool in Assessing Effective Functioning of Research Ethics Committees (RECs): Experience with a Kenyan REC

INTRODUCTION

Africa has recently witnessed a marked increase in the number of research ethics committees (RECs). This is in consort with the increasing number of studies involving human participants that are being conducted there through various initiatives, and the efforts made to build capacity for stronger systems for human research protection in the region (NEBRA, 2006; Kass et al., 2007; Abbot & Grady, 2010, Ndebele et al., 2014). RECs play a crucial role in providing ethical oversight for research involving human subjects and are a key requirement in all international ethics guidelines on research with humans (ICH, 1997; CIOMS, 2002; WHO, 2011; WMA, 2015). Research involving human participants in Africa requires efficiently run RECs that comply with international ethical standards and best practices. One of their key responsibilities is to ensure that risks to study participants are minimized and that they are reasonable in relation to anticipated benefits and to the importance of the knowledge that is to be generated (USA Department of Health & Human Services, 2009).

Assessing how well RECs are functioning is challenging. There is no consensus on how to measure the ethical quality of their reviews, nor their impact on research practice (Silverman et al., 2014). Furthermore, there may be no direct correlation between the functioning of a REC and the ethical standards of the studies it approves (Love & Fost, 2003). The standard way of ensuring that RECs comply with international ethical standards is by an accreditation process and continuous external assessments. However, this assessment often requires financial and human resources not always easily available in resource-constrained countries and there is no set of tools acceptable internationally for such accreditation.

Another way of measuring a REC's performance is by internal evaluation. The effective functioning of a REC could be considered to be a surrogate marker for how well it plays its role in protecting study participants. Effective functioning in this context is described in terms of the committee's structure, efficiencies, member experience, and the resources it can access in order to fulfil this role. This view assumes that high quality ethical review of studies will result in research that better complies with applicable ethical principles (Coleman & Bouësseau, 2008). Internal evaluation of a REC has the advantage of being relatively inexpensive, and therefore saving much needed resources. One approach to evaluation is to use a standard self-assessment tool with multiple indicators of effective functioning such as organizational structure, requirements for training members in research ethics, protocol review procedures, turnaround time for reviewing proposals, and the resources available to RECs, among others (Wolf et al., 2002; Fauriel et al., 2004; Burke, 2005; Silverman et al., 2014; Silaigwana & Wassenaar, 2015).

A tool specifically designed to evaluate the effective functioning of RECs in developing countries has been developed by bioethics experts in the Middle East under the auspices of the Middle East Research Ethics Training Initiative (MERETI). The tool, accessible in full within the article describing its creation, was developed with an aim of achieving a more accessible self-assessment tool for RECs, that would reflect pragmatic aspects of human subjects protection, be based on international standards, be straightforward in its completion, and be relevant to the administrative process that exist in many RECs during their early stage of development (Sleem et al, 2010). It provides quantitative data of how well RECs comply with international ethics standards, identify areas where RECs are performing well, and areas that require improvement (Silverman et al., 2014). High scores are indicative of effective RECs. Developers of the tool propose that it can be used to help RECs evaluate and enhance their operations, provide quality improvement for best practices and develop educational initiatives.

However, since only one REC member usually completes the MERETI tool, it is not clear whether the scores obtained reflect the overall committee's performance, as the scores may be influenced by personal viewpoints and bias. For this reason, this study examined the consistency of results obtained using the MERETI tool administered by several members of the same REC independently to assess the performance of their committee.

METHODS

This cross-sectional observational study involved members of an accredited institutional REC in Kenya. The REC meets once a month and reviews more than 400 research protocols annually. All 17 members of the REC were eligible to participate. They were approached individually, by phone or email, at which time the study's objectives and methodology were explained. A face-to-face meeting was then requested from those who indicated willingness to participate, during which the study was explained in more detail. Participants were then enrolled into the study upon giving their written informed consent. This was followed by a researcher administering a standard questionnaire to each participant to obtain demographic data including age, gender, duration of membership on the REC, prior ethics training (type and duration), position on the REC, and whether employed by the institutions affiliated to the REC. Each participant then self-administered the MERETI tool (Sleem et al, 2011), to evaluate the functioning of the REC. Briefly, the tool has ten major components: 1) organizational aspect of the committee; 2) membership and educational training of members; 3) submission arrangements and materials; 4) committee minutes; 5) review procedures; 6) review of specific items in the protocol; 7) communicating committee decisions; 8) continuing review of approved studies; 9) committee resources; and 10) workload of the committee. Answers to questions in the components were scored 1 to 5 points, as per the tool's instruction. No names of study participants were used. Instead, unique identifying numbers were used.

Data Analysis

Scores obtained were compared in relation to the duration of REC membership, the interval since their last research ethics training and in relation to their gender, using the Mann Whitney U-test, a non-parametric statistical test for comparing two groups with small sample sizes of data that cannot be assumed to be normally distributed (Nachar, 2008). A p-value <0.05 was considered statistically significant. Questions or comments raised by participants were examined for any descriptive context as to potential themes or trends.

Ethical Consideration

Ethical approval to conduct this study was granted by two institutional ethics review boards, one in Kenya and the other one in South Africa, prior to the recruitment of any study participant.

RESULTS

Sixteen of the seventeen REC members were contacted and requested to participate in this study. One member out of the country on study leave was excluded. Thirteen (76%) members consisting of 8(62%) males and 5(38%) females responded to the invitation and were enrolled in the study. A slight majority of 7(54%) had been REC members for 2 years or more. Six REC members (46%) had had their most recent research ethics training within the last 2 years. All trainings were either at a workshop or were web-based. None of the REC members had been trained in research ethics at a diploma or a degree level. All study participants were employees of the institutional affiliates of the REC. Furthermore, they were all researchers consisting of 2 pharmacists, 7 clinicians with multiple specialities, 1 dentist, 1 ethicist, and 2 social scientists. See Table 1. The only non-researcher in the REC did not participate in the study.

Table 1.

Demographics of study participants

Gender	Male	8(62%)	Female	5(38%)
Duration of REC membership	Less than 2 yrs	6(46%)	2 or more yrs	7(54%)
Last ethics training	Less than 2 yrs	6(46%)	2 or more yrs	7(54%)
Researcher/Non-researcher ratio	Researcher	13 (100%)	Non-researcher	0%
Employment	Institutional	13(100%)	Non-institutional	0%

First, the Chair's scores were compared with the median score of the rest of the members. Then, scores of all members including the Chair were compared in relation to duration of REC membership, last research ethics training, and gender.

REC Chair's score compared to other REC members

Table 2 shows the Chair's scores compared to that of the other members. The overall score of the Chair (180) was higher than the overall median score of the twelve REC members (160), a difference of 11%. The scores could not be compared statistical due to insufficient data in one comparator group. The majority of the scores contributing to this difference were in the subcomponents dealing with requirements for REC membership and educational training.

Table 2.

REC Chair's scores in relation to other members' scores

	REC Chair	Other REC Members (median)	Maximum possible score
Overall score	180	160	200
Organizational Aspects	43	44	54
Membership & Educational Training	28	17	30
Submission Arrangements & Materials	11	11	12
Committee Minutes	11	9	13
Review Procedures	11	8	11
Review of Specific Items in the Protocol	43	40	43
Communicating Committee Decisions	5	5	5
Continuing Review of Approved Studies	15	15	16
Committee Resources	15	14	16

Scores in relation to duration of REC membership

Members were grouped into two. Six of them had been REC members for up to 2 years while 7 had been members for 2 years or more. The overall median score of members who had served for up to 2 years was 160, compared to 162 for those who had served for 2 years or more. The difference was not statistically significant (p=0.72). Similarly, there was no statistically significant difference in the median scores of the two groups in the various subcomponents of the tool. See Table 3.

Table 3.

Median scores in relation to respondent's membership duration on the REC

	≤ 2 years	> 2 years	p-value
Overall score	160	162	0.72
Organizational Aspects	44.5	43	0.77
Membership & Educational Training	14.5	19	0.35
Submission Arrangements & Materials	11	11	0.94
Committee Minutes	8	10	0.62
Review Procedures	7.5	8	0.39
Review of Specific Items in the Protocol	41	39	0.15
Communicating Committee Decisions	4	5	0.28
Continuing Review of Approved Studies	15	14	0.28
Committee Resources	14	14	0.94

Scores in relation to ethics training

Participants were grouped into two with regard to their last research ethics training. Six members had had their last research ethics training within the last 2 years while 7 members had had their last training 2 or more years prior to the interview date. Members with their last research ethics training within 2 years had a median score of 156, whereas those with their last training 2 or more years earlier had a median score of 163. There was no statistically significant difference in these overall scores (p=0.94) and in the scores in the subcomponents. See Table 4.

Table 4.

Median scores in relation to respondent's last ethics training

	≤ 2 years	> 2 years	p-value
Overall score	163	156	0.94
Organizational Aspects	44	42.5	1.0
Membership & Educational Training	18	17	0.94
Submission Arrangements & Materials	11	11	0.48
Committee Minutes	8	8.5	0.83
Review Procedures	8	8	0.39
Review of Specific Items in the Protocol	41	38.5	0.30
Communicating Committee Decisions	4	5	0.52
Continuing Review of Approved Studies	15	14.5	0.77
Committee Resources	15	14	0.71

MERETI scores in relation to gender

The overall median score by male and female members did not significantly differ statistically at 155 versus 167, respectively (p=0.27). Scores in the subcomponents were also not significantly different. See Table 5.

Table 5.

Median scores in relation to respondent's gender

	Males	Females	p-value
Overall score	154.5	167	0.27
Organizational Aspects	42.5	45	0.51
Membership & Educational Training	17	18	0.51
Submission Arrangements & Materials	11	10	0.24
Committee Minutes	8	10	0.88
Review Procedures	8	9	0.27
Review of Specific Items in the Protocol	40	41	0.42
Communicating Committee Decisions	4	5	0.24
Continuing Review of Approved Studies	14.5	15	0.38
Committee Resources	14	14	0.94

MERETI scores in relation to employment relationship to the institution

Since all the study participants were employed by the two institutional affiliates of the REC, it was not possible to examine for differences in their scores obtained in relation to whether their employer was affiliated to the REC.

General observations on MERETI tool

Members’ comments indicated that the tool made them think critically about issues they had not previously paid attention to, such as having a policy for disclosure and management of conflict of interest by researchers, and taking detailed minutes indicating REC composition at meetings. Some members proposed using the tool as a checklist by RECs to improve their operations.

DISCUSSION

This study evaluated the utility of the MERETI tool by examining whether several members from the same REC would generate similar scores. It first compared scores provided by the REC Chair to the median score of the rest of the REC members. Thereafter, scores provided by all study participants were compared in relation to their duration of membership of the REC, gender, research ethics training, and whether or not they were employed by the institutions to which the REC is affiliated.

Although it was not possible to compare statistically the scores provided by the REC Chair with the median score of the rest of the REC members, these scores differed only at face value. Thus, the assessment of the REC's performance was rated consistently irrespective of whether it was assessed by the REC Chair alone, or by the entire REC. Since REC assessment using the MERETI tool is normally conducted by the Chair alone, this finding supports using the tool as currently used.

The bulk of the scores contributing to a higher REC Chair's score were in the subcomponent dealing with membership and research ethics training. This may be explained by the fact that this component had some questions that required special knowledge, which may not be available to REC members outside of leadership positions. Examples include the requirements for appointment of the REC Chair, requirements for appointment of other REC members, and documentation of training received by REC members. For instance, most members did not know whether there were criteria used for their appointment to the REC but indicated that their appointments must have been based on criteria known to the Chair. Also, some members did not know whether the REC kept documentation detailing members’ training in human subjects’ protection whereas the information was available to the REC Chair. It would appear that, although it is good for REC members to know this information, it is not crucial since lack of this knowledge does not hinder the members assessment of the performance of the REC. To the point that effective functioning is a surrogate for the ability to fulfil its supervisory ethical role, this knowledge would not necessarily be material to their review of human subject research.

Although it may be argued that gender differences may influence how one assesses a REC using a standard questionnaire tool, this study did not find any gender-based difference in the overall scores or in the scores in the components of the tool. Gender neutrality in scores obtained is important for assuring objectivity of the tool. Differences in the scores obtained between males and females would make the tool subjective raising doubts as to whether these scores truly reflect the REC's performance, thereby reducing its utility.

It is probable that the duration of service on a REC has an effect on how members evaluate its performance. Members who have served on the REC for longer may be more familiar with procedures and operation of the REC than newer members, resulting in higher ratings. In addition, experience gained in reviewing many research protocols and participating in REC discussions may make members more aware of the presence of certain documents without having to ask the REC administrators. Conversely, experience on the REC could also lead to a bias in assessment of how well or badly the REC is performing in comparison to the past. However, this study did not find any differences in the median scores provided by members who had served on the REC for 2 or more years compared to the scores of those who had been member for less than two years. This may be a reflection of the fact that new members are orientated to REC's procedures upon their appointment to the committee. This may serve to familiarize them with the procedures that older members would be more familiar with. However, the fact that several participants, including those who had been members for many years, indicated that they had learnt a great deal about operations from participating in the REC suggests that orientation alone does not completely equip new members for their task. The suggested use of the tool as a checklist may serve to highlight key areas that REC's need to address to improved their effective functioning and thereby their protection of human subjects. Furthermore, it is possible that our analysis in relation to duration of REC membership may have been affected by the wide range in the time on the REC among those members with 2 or more years of service. There may have been a difference in scores between REC members with 15 years of service and members who had served for only 3 years. However, such sub-analysis was not possible due to the small number of study participants.

It is also possible that recent refresher training in research ethics creates heightened awareness of the procedures and requirements of an effective REC and may thus have biased assessment. However, this was not observed since the median total score provided by REC members whose last research ethics training was 2 or more years prior did not differ from the median score of members whose last ethics training was within the last two years. However, the effect of these trainings on the members is not easy to assess given differences in their type and duration.

This study also intended to investigate whether being an employee of the institution with which the REC is affiliated may have influenced the way the members scored using the MERETI tool. However, this comparison could not be carried out because all of the REC members who participated in this study were employees of REC's institutional affiliates.

Conclusions

The MERETI tool was consistent irrespective of whether it was completed by the Chair or by the whole research ethics committee, irrespective of assessors gender, research ethics training and duration of membership of the REC. Qualitative findings suggest that REC members found completion of the tool to be a valuable educational experience or lead them to want to learn more about their REC and its formation, and as a useful checklist for RECs interested in improving their operations. The study has also highlighted some challenges of scoring questions and suggests a refinement of the tool to make it even more robust.

Limitations of the study

The sample size in this study was small. A larger study with members from multiple RECs that includes members not affiliated with an institution is needed to confirm or refute these findings.

Best practice

There is a rapid increase in REC in developing countries in consort with the increase in the number of research involving humans in these countries. It is crucial that these RECs operate at a level that meets international standards. Self-assessment tools for RECs are important for their quality improvement and for ensuring best practices in the performance of their crucial role of protecting research participants. For such tools to be relied on, they should be easy to use and be able to provide consistent results when used to assess the performance of a REC.

Educational implication

A reliable and consistent self-assessment tool can be a useful educational source for a REC for measuring its performance and planning training programmes for improving its functioning. Furthermore, information obtained from self-assessment can be of great benefit in rationally allocating the meagre resources available to the REC in areas that can optimize its functioning.