Those Responsible for Approving Research Studies Have Poor Knowledge of Research Study Design: a Knowledge Assessment of Institutional Review Board Members

Abstract

Background:

Institutional Review Board (IRB) members have a duty to protect the integrity of the research process, but little is known about their basic knowledge of clinical research study designs

Methods:

A nationwide sample of IRB members from major US research universities completed a web-based questionnaire consisting of 11 questions focusing on basic knowledge about clinical research study designs. It included questions about randomized controlled trials (RCTs) and other observational research study designs. Potential predictors (age, gender, educational attainment, type of IRB, current IRB membership, years of IRB service, clinical research experience, and self-identification as a scientist) of incorrect answers were evaluated using multivariate logistic regression models.

Results:

148 individuals from 36 universities participated. The majority of participants, 68.9% (102/148), were holding a medical or doctoral degree. Overall, only 26.5% (39/148) of participants achieved a perfect score of 11. On the six-question subset addressing RCTs, 46.6% (69/148) had a perfect score. Most individual questions, and the summary model of overall quiz score (perfect vs. not perfect), revealed no significant predictors – indicating that knowledge deficits were not limited to specific subgroups of IRB members. For the RCT knowledge score there was one significant predictor: compared with MDs, IRB members without a doctoral degree were three times as likely to answer at least one RCT question incorrectly (Odds Ratio: 3.00, 95% CI 1.10-8.20). However, even among MD IRB members, 34.1% (14/41) did not achieve a perfect score on the six RCT questions.

Conclusions:

This first nationwide study of IRB member knowledge about clinical research study designs found significant knowledge deficits. Knowledge deficits were not limited to laypersons or community advocate members of IRBs, as previously suggested. Akin to widespread ethical training requirements for clinical researchers, IRB members should undergo systematic training on clinical research designs.

1. INTRODUCTION

University institutional review boards (IRBs) are the main gatekeepers for approval of clinical research studies in the US. IRBs are charged with protecting human participants’ rights and welfare, ensuring that approved studies are ethically and scientifically sound, and that they adhere to federal regulations. This role is fulfilled by detailed examination of study protocols, which often result in requests for change in the research protocols rather than simply accepting or denying them (1-4). Medical IRBs are usually composed of scientists, physicians, ethicists, and patient and community representatives. Academic IRB service is typically voluntary and uncompensated, although recognized as a service activity for members with academic rank. While mandatory and recurring training in research ethics is ubiquitous for IRB members, formal training in principles of clinical study design and methodology is rare. Aside from the lay members of an IRB, the scientific and methodological competence of IRB members is assumed to be present because of their credentials (e.g. MD, PhD, or other relevant advanced degree, experience, or specialized knowledge).

Whether IRB members are truly competent to evaluate the scientific soundness of proposed clinical research studies is unknown. We found no published empirical research on IRB members’ knowledge about clinical research study designs. Instead, much research on IRBs has focused on the attitudes of IRB chairs (5-7) and who speaks at IRB meetings (8). A recent systematic review on IRB evaluation considered studies which provided empirical data about IRB structure, process, variation and outcome but did not report any studies which evaluated the knowledge of IRB members on topics relevant to their duties (9). However, several recent findings suggest that important deficits in knowledge may be prevalent. A study of expert instructors (i.e. MDs) for continuing medical education (CME) programs found deficits in basic knowledge about randomized controlled trials (RCTs), and consequently bias in the manner in which instructors presented RCT results to the CME audiences (10). Another study showed that when IRB members at multiple sites are presented with the same research proposal, their reactions vary (11). Variations have been noted in the acceptable methods for recruitment of subjects (12, 13), designation of risk level (14, 15), type of concerns expressed or changes required (1-4, 12), and final approval vs. disapproval (4, 16). Finally, empirical evidence from a systematic review of 43 studies showed that IRBs in the United States differ in their approval decisions significantly (11). Given this observed inconsistency in IRB judgment of and reactions to the same proposal, it is imperative to investigate potential contributory factors. One of these factors might be IRB member knowledge regarding clinical research study designs and methods. In this paper, we report the results of a brief knowledge quiz on clinical research study designs that was administered to a nationwide sample of current and recent IRB members at 36 leading research universities in 2012-2013.

2. METHODS

As part of a larger study on factors that influence decision-making for clinical research study approvals by IRB members, we investigated knowledge about clinical research study design using a clinical research design knowledge quiz. Our study was approved by the IRB at the University of South Florida (IRB#: 107911). Our target population consisted of university IRB members in the United States. Sample identification, participant enrollment and informed consent, and administration of our web-based study questionnaire were completed from June 2012 to February 2013.

The target sample for this study was identified based on multiple strategies. Member lists from the Association of American Medical Colleges (AAMC) (n=122 universities) and the Public Responsibility in Medicine and Research (PRIM&R) (n=170 individuals) were used. Websites of AAMC members were searched and IRB administrators contacted to obtain the names and contact information for individual IRB members (current and recent). The final sample list comprised 1,398 individuals from 128 institutions. Multiple contact attempts were made using e-mail, letters, and postcards over the course of 7 months during late 2012-early 2013. Calculation of the response rate for the study was done by dividing the number of completed and partial responses by the sum of cases estimated to be eligible. That is, after adjustment for ineligible cases (n=238) using a widely-accepted method the final response rate was 20% (232/1160) (17). Of the 232, 148 participants completed all the sections of the study including the knowledge quiz and were included in the analysis for this manuscript.

The clinical research design knowledge quiz was developed to assess basic knowledge about RCTs, other clinical research designs, and the purpose and importance of randomization. The quiz questions were sequenced near the end of the main survey questionnaire. Each quiz question consisted of a short statement which respondents had to rate as “true” “false” or “don’t know.” We calculated knowledge quiz scores by summing up the total number of correct answers for the 11 questions; therefore the range of possible scores was 0 to 11 with a perfect score being 11. We also created an RCT knowledge score by summing the total number of correct answers to the 6 questions that focused specifically on RCTs (range of possible scores 0 to 6 with a perfect score being 6). We included respondents who completed the questionnaire but did not record an answer for every question (i.e. they chose to skip at least 1 question). The questions were purposefully selected to be easy and straightforward ones which no knowledgeable IRB member should have difficulties answering.

We examined several potential predictors of incorrect quiz responses in our statistical analyses. These predictors were age, gender, educational attainment, IRB type (medical versus socio-behavioral), length of IRB service, current membership status, self-reported clinical research experience, and self-designation as a scientist. To assess the independent effects of these potential predictors, we ran multivariate logistic regression models. The dependent variables in these models were: (a) the summary score for all 11 quiz questions, dichotomized as not perfect score vs. perfect score; (b) the summary score for the 6 RCT questions (not perfect vs. perfect); (c) each of the quiz questions individually (i.e. 11 separate models), modeling the likelihood of an incorrect vs. correct response.

3. RESULTS

Total amount of 148 participants completed the knowledge quiz (Table 1) and were included in the analysis. These 148 individuals represent 36 universities. Characteristics of the respondents and their universities, in total and stratified by quiz score, are shown in Table 2. These IRB members were predominantly middle-aged and 56.8% (84/148) were female. A majority (68.9%; 102/148) had either an MD or other terminal doctoral degree (e.g. PhD, PharmD, EdD). Almost all of these IRB members (83.1%; 123/148) were associated with Carnegie Foundation designated Research-Very High universities representing the leading US Universities (the top tier of the ranking). A majority of participants were current Medical IRB members, had clinical research experience, and considered themselves to be scientists. Finally, almost all (93.9%; 139/148) had 2 or more years of experience (range: 1-32 years) on an IRB. IRB members who did not achieve a perfect score on the knowledge quiz had somewhat lower educational attainments and shorter lengths of IRB service, and were less likely to report clinical research experience or that they considered themselves to be scientists.

Table 1.

Brief Knowledge Quiza about clinical study designs. a Questions were administered in random order

Table 2.

Characteristics of the Respondents and Their Universities, by Knowledge Quiz Score. a Includes one member with an Associate’s degree.. b Includes one member with a DDS.. c Missing for 3 respondents.. d Based on the 2010 Carnegie Foundation Classification. e Note: Freestanding research institutes and medical schools which do not grant PhDs are not listed in the Carnegie Foundation rankings. This category also includes 2 federal employees.

Overall, only 26.5% (39/148) of respondents answered all 11 quiz questions correctly. However, incorrect answers were not clustered in a single question, as illustrated by Figure 1, which depicts the percent of respondents with an incorrect answer to each question. Each question was correctly answered by a majority of respondents, with the percent incorrect ranging from 4.1% to 33.3%.

Figure 1.

Percent incorrect by question on the clinical research knowledge quiz

However, after multivariate adjustment (Table 3), there were no significant predictors of a not-perfect score on the knowledge quiz, indicating that knowledge deficits were not clustered in specific subgroups of IRB members. For 7 out of 11 individual questions, there were also no significant predictors of a wrong answer. For 2 questions that focused on Phase I trials, members of Behavioral IRBs were 5-7 times more likely to answer incorrectly compared with Medical IRB members. Moreover, for one of the phase I trial questions IRB members with shorter duration of IRB service were more likely to answer incorrectly compared with IRB members with longer duration of service.

Table 3.

Predictors of Wrong Answers on the Knowledge Quiz for 148 IRB Members: Multivariate Logistic Regression Resultsa, a All models included the following predictors: age, gender, educational attainment, type of IRB, years of IRB service, clinical research experience, and self-identification as a scientist., b The model was not properly specified because of zero cells for some groups.

For the RCT questions sub-score, only educational attainment was significantly associated with a not-perfect score. Compared with IRB members with a medical doctorate (MD), IRB members without a doctoral degree were 3 times as likely to answer at least 1 RCT question incorrectly (OR=3.00, 95% CI 1.10-8.20, p=0.03). The bivariate association between RCT score and educational attainment is shown in Figure 2. The majority of physicians answered all questions correctly (65.9%; 27/41), compared with fewer than half of those with doctorates (44.3%; 27/61) or a bachelors or master’s degree (32.6%; 15/46).

Figure 2.

RCT knowledge by IRB member educational attainment

4. DISCUSSION

In this first nationwide study of IRB member knowledge about clinical research study designs, we found significant knowledge deficits, despite the high educational attainment and considerable years of IRB experience of the respondents. These results are somewhat alarming because they indicate that those who are charged with oversight in the protection of research integrity may not be adequately equipped to fulfill their duties.

We investigated age, gender, educational attainment, IRB type, length of IRB service, current membership status, self-reported clinical research experience, and self-designation as a scientist without finding any significant predictors of a not-perfect score. That is, there were no significant predictors of a not-perfect score on the knowledge quiz reveals that these knowledge deficits were not limited to laypersons or community advocate members of IRBs, as had been previously suggested in the literature (6, 19).

IRBs are generally concerned only with the protection of human subjects in research, a task that seemingly requires only ethical knowledge and judgment. However, these two kinds of knowledge, ethical and methodological, are actually interrelated, and both are necessary for the work of IRBs. This principle is expressed in the Federal Common Rule, the guideline to which all IRBs report. Criterion One of CFR 46.111 states, “Risks to subjects are minimized: (i) by using procedures which are consistent with sound research design and which do not unnecessarily expose subjects to risk” (emphasis added) (20). This principle makes clear the presupposition that underlies an IRB’s task- that both knowledge of proper scientific methods and ethical knowledge and judgment are required for successful protection of human subjects. For example, it is axiomatic that a research study cannot be considered ethical if it has no potential for generating valid scientific knowledge, even if there are no particular ethical concerns raised by the details of the study protocol, because of concerns about resource waste, pointless inconvenience to subjects, and opportunity cost (21).

Deficits in IRB member methodological knowledge may impact very specific goals of the medical research community. In our contemporary milieu there is increased emphasis on replacing traditional placebo-controlled trials with comparative effectiveness trials. In our study, 21% (31/148) of respondents believed that an RCT must have a placebo control group, and 26% (38/148) did not understand a fundamental justification for and strength of randomization – namely to create study groups with similar characteristics and thus minimize confounding by both measured and unmeasured participant characteristics. These findings are consistent with a study of CME speakers (physician specialists) which found RCT methods knowledge deficits – specifically that given a simple example, speakers made incorrect calculations for relative risk reduction (32%), absolute risk reduction (26%), and number needed to treat (21%) (10).

Moreover, it is important to note that IRB members do not just approve or disapprove research studies; they often elicit and require specific changes to study protocols prior to final approval being granted. Therefore another important empirical question is whether IRB-required modifications to study protocols are ever in conflict with principles of good study design. Without knowledge of these methodological choices in study design and presentation, IRB members cannot properly guard against unethical research designs.

Given our findings, it is perhaps not surprising that investigators continue to question or dispute IRB decisions (4, 22-26). Given that competence is necessary to maintain trust, the lack of knowledge of some IRB members about research methodology has the potential to seriously undermine relationships between regulators and those who are regulated.

5. STUDY LIMITATIONS

The most important limitation of our study is that it is not known how the knowledge deficits observed in our study relate to the decisions that IRB members reach in real-world settings. We believe this is an important area for further empirical research. Nevertheless, the dialectical relationship between theoretical knowledge and real-world performance is not unique to the issue at hand; yet societies have been determining competence based on formal knowledge assessments since the dawn of organized education. Another limitation of our study is that our target population did not include the members of commercial IRBs, and there is no published research on the methodological competence of these individuals. Finally, response rate of 20% seems low. However, recent developments in survey research methodology indicate that response rates might not be necessarily associated with quality or representativeness of a survey (27). That is, instead of response rate the focus should be on the representativeness of the sample. Indeed despite the low response rate, our respondents are representative of IRB members at major research universities; most had doctoral degrees and ample experience. Given that the participants in our survey were almost uniformly recruited from the leading US universities, our results probably indicate upper bound of knowledge on research methodology of IRB members. Therefore, we believe it is extremely unlikely that our findings are a pure artifact of a low response rate.

6. CONCLUSIONS

In conclusion, our study provides preliminary evidence that would initiate discussion regarding policy changes in how IRBs are regulated. Though our study found knowledge deficits in methodology and did not investigate possible deficits in ethical judgment, the inter-related nature of ethical judgment and methodological knowledge suggests that this methodological knowledge deficit will negatively impact IRB success. If IRBs members are to succeed in their mandated mission of protecting human subjects, required and periodic systematic training on clinical research designs may be called for.

Authors’ contributions

The study was conceived and designed by BD and RM. The knowledge quiz was developed by RM, BD and AK. RM oversaw the programming, instrument testing and data collection work by the UVa Center for Survey Research, carried out under the direction of TG. EBP and RM analyzed the data. All authors contributed to the interpretation of results and writing and preparation of the final manuscript.