Assessing the Quality and Performance of Institutional Review Boards: Levels of Initial Reviews

Abstract

How well institutional review boards (IRBs) follow Common Rule criteria for levels of initial protocol review has not been systematically evaluated. We compared levels of review as determined using the Office for Human Research Protections (OHRP) human subject regulations decision charts of 313 protocols that had been approved by IRBs. There was a 97.8% agreement between 140 protocols that were reviewed by full board and the levels of review according to OHRP criteria. Likewise, there was a 93.8% agreement between 113 protocols that were reviewed using an expedited review procedure and OHRP criteria. However, there was only 75% agreement for exempt protocols. Specifically, 10 (16.7%) of the 60 exempt protocols were found to require IRB review, i.e., 6 protocols requiring expedited review and 4 protocols requiring full board review. Conducting non-exempt research without prior IRB approval constitutes serious noncompliance. Our data suggest that exempt protocols need more scrutiny.

INTRODUCTION

Institutional review boards (IRBs) and their equivalents, Research Ethics Committees or Ethics Review Boards, play an important role in protecting human subjects participating in research. In the U.S., the Federal Policy for the Protection of Human Subjects, also known as the Common Rule, requires that no research involving human subjects, unless it is deemed to be exempt from IRB review, can be initiated until it has been reviewed and approved by the IRB (U.S. Department of Health and Human Services, 2018). Research activities that present no more than minimal risk to human subjects and involve only procedures listed in one or more of the expedited review categories may be reviewed by the IRB using the expedited review procedure (U.S. Department of Health and Human Services, 1998). Thus, there are three levels of IRB reviews according to the Common Rule:

• Exempt from IRB review for exempt research,

• Expedited review for research that is eligible for expedited review procedure, and

• Full board review at a convened IRB meeting for the rest of the non-exempt research including those protocols that are qualified for expedited review, but where the IRB decides not to use the expedited review procedure.

It should be pointed out that technically the Common Rule only applies to federally funded research, even though in practice most institutions adopt it as an overarching regulatory/ethical framework. In some other countries (e.g. South Africa), a single national guidance document is legally binding on all research and researchers irrespective of funding source (Cleaton-Jones & Wassenaar, 2010).

Inconsistency or variation in IRB practice has been well documented by multisite studies when multiple IRBs were used for the review of protocols (Friesen, Yusof & Sheehan, 2019). These inconsistencies include, but are not limited, to levels of initial protocol review, required modifications, and IRB determinations (Driscoll, Currey, Worrall-Carter, et al., 2008; Goldman & Katz, 1982; Nahmias, Grigorian, Brakenridge, et al., 2018; Petersen, Simpson, Sorelle, et al., 2012). While the recent implementation of the revised Common Rule, which requires single IRB review of multi-site studies (Menikoff, Kaneshiro, & Pritchard, 2017: U.S. Department of Health and Human Services, 2018), eliminates most of the problems associated with the use of multiple IRBs, it does not address the underlying inconsistency of IRB practices.

A previous study evaluating how well IRBs follow Common Rule protocol approval criteria revealed that in their review of protocols, IRBs mostly discussed the informed consent documents (98% of the time), but failed to address risk minimization (21% of the time), the risk to benefit ratio (57%), equitable subject selection (60%), data monitoring (54%), privacy and confidentiality (25%), and protection of vulnerable populations (13%) (Lidz, Appelbaum, Arnold, et al., 2012). Similar observations were reported by Wassenaar et al., for Research Ethics Committees in South Africa (Tsoka-Gwegweni & Wassenaar, 2014; Silaigwana & Wassenaar, 2019). However, so far no study has evaluated how well IRBs follow Common Rule criteria for levels of initial protocol review.

The Office for Human Research Protections (OHRP) maintains regulatory oversight of human subjects research conducted or supported by the U.S. Department of Health and Human Services (HHS). OHRP has developed decision charts to guide IRBs and investigators in determining whether an activity is human subjects research that must be reviewed by the IRB in accordance with the Common Rule and under what level of review (U.S. Department of Health and Human Services, 2016). However, it is not known how well or consistently these charts are applied in practice. Therefore, as part of our assessments of the quality and performance of IRBs at the Department of Veterans Affairs (VA) (Tsan, 2019), we used the OHRP decision charts to determine levels of review for a sample of protocols that had been approved by VA IRBs. A comparison of levels of review as determined using OHRP criteria and those that were actually used by VA IRBs for the review and approval of human research protocols, allows us to determine the performance of VA IRBs in following the Common Rule criteria for levels of initial protocol review.

METHODS

Selection of protocols

Ten IRBs, including 9 VA facility IRBs and the VA Central IRB, were recruited to participate in this study. All these 9 VA facilities utilized their own VA IRBs as their IRBs of record, and each had more than 170 active human research protocols in 2010. The goal was to select up to 15 protocols in each level of IRB review between January 1, 2010, and December 31, 2011, from each site. The IRBs at Sites 7 and 9 did not conduct expedited review of protocols as a matter of local institutional policy. Therefore, only full board-reviewed protocols and exempt protocols from these 2 sites were included. A total of 313 protocols, including 140 full board-reviewed protocols, 113 expedited reviewed protocols, and 60 exempt protocols, were available for analysis.

Determination of levels of review according to OHRP criteria

Two research team members were trained as qualitative coders by an IRB chair, the principal investigator who was a senior IRB member, and a qualitative methodologist, to use OHRP human subject regulations decision charts to determine whether a research protocol was human subject research, and if it was human subject research, whether it qualified as an exempt protocol or qualified for IRB review using an expedited review procedure.

A level of review codebook was constructed through the development of an algorithm of questions designed to triage protocols into the appropriate level of review according to OHRP human subject regulations decision charts. Coding proceeded site by site so that all documents at one site were coded before coding the next site. Coding at each site began with a period of orientation to the documents contained in the records of the site to be coded. The coders made an initial determination regarding the level of review based solely on the protocol and without knowledge of the level of review actually approved by the site’s IRB. However, after making an initial determination, the coders examined the entire IRB record to determine the level of review actually used by the IRB. If discrepancies were noted, the coders reviewed relevant documents to ensure that there were no errors and searched for other evidence that might justify the different level of review. If additional evidence was found supporting the IRB’s action, the coders changed the level of review to correspond with the IRB’s action. If no additional evidence was found, the original coding stood despite its discrepancy with the IRB’s action.

The two coders then proceeded to independently code a random sample of 20% of the site’s protocols. After the initial coding, the coders met face-to-face to discuss coding discrepancies and adjudicate each discrepancy. If the pooled Kappa was greater than 0.80, the remaining protocols were coded by the primary coder. If the pooled Kappa was less than 0.80, both coders independently co-coded another 20% of randomly selected protocols and calculated the inter-rater agreement for the second sample. This process was repeated until the pooled Kappa was 0.80 or greater. In the final sample, across all sites, the inter-rater agreement on adjudicated OHRP levels of review was excellent on a sub-sample of 67 protocols (kappa =0.97) for the levels of review.

Protection of human subjects

This study did not involve human subjects and did not record any individually identifiable information. Therefore, no IRB review and approval was required.

Data analysis

The actual levels of review by site IRBs were cross-tabulated with the expected levels of review based on the objective OHRP criteria for all protocols. Agreement on the classification of protocols as full board, expedited, or exempt was quantified in terms of percentage agreement between the IRB of record and the coders trained to apply the OHRP algorithms.

RESULTS

Full board-reviewed protocols

Table 1 shows levels of review as determined by OHRP algorithms for the 140 protocols granted full board reviewed. Seventy-one protocols (50.7%) were determined to be neither qualified for exemption nor eligible for expedited review according to OHRP criteria. These 71 protocols could only be reviewed by IRB full board at a convened meeting. Sixty-six protocols (47.1%) were qualified for expedited review; 1 (0.7%) was qualified as an exempt protocol, and 2 (1.4%) were determined to be not human subject research. All but 3 of the 71 protocols adjudicated to require full-board review contained details that precluded exemption or expedition; however, the remaining 3 lacked sufficient detail for the coders to determine the appropriate level of review. Although some IRBs might handle similar situations by requiring further details from the investigators before making a determination on the level of review, for the purposes of this study we considered these 3 as properly referred to the fully convened IRB because the OHRP decision charts require full-board review for any study that does not satisfy criteria for exemption or expedition.

Table 1:

Levels of review determined using OHRP algorithms for protocols granted full-board review

Site #	Full-boarda,b	Expedited	Exempt	Not human research	Total
1	14	0	0	0	14
2	10	5	0	0	15
3	4	11	0	0	15
4	10	1	0	0	11
5	5	6	0	0	11
6	6	9	0	0	15
7	2	13	0	0	15
8	13	0	0	1	14
9	1	12	1	1	15
10	6	9	0	0	15
Total	71 (50.7%)	66 (47.1%)	1 (0.7%)	2 (1.4%)	140 (100%)

^aColumns from left to right were in decreasing intensity of levels of review.

^bBold prints were used to denote columns that were concordant with the granted level of IRB review.

Since protocols that are eligible for expedited review can be reviewed either using an expedited review procedure or by IRB full board at a convened meeting, only 3 (2.1%) protocols, i.e., 1 exempt protocol and 2 non-human subject research protocols, should not have been reviewed by IRB full board at a convened meeting. For the purpose of calculating percent agreement, the first 2 columns, (i.e., full board and expedited reviews) were combined demonstrating a very high degree of agreement (i.e., 97.8%) between actual VA IRB full board-reviewed protocols and levels of review, i.e., full board and expedited, as determined using OHRP criteria.

Thus, of the 140 IRB full-board reviewed protocols, 3 (2.1%) protocols i.e., 1 exempt and 2 non-human subject research protocols, received more intense review than needed according to OHRP criteria, no (0%) protocol received less intense review, and 137 (97.8%) protocols were concordant.

Alternatively, in view of the fact that 66 (47.1%) protocols were eligible for expedited review and could have been reviewed using expedited review procedure, one could consider these protocols received more intense review than needed according to OHRP criteria, and only 71 (50.7%) protocols were concordant for full board review. However, this interpretation not only is inconsistent with the Common Rule, but also is unfair to the participating IRBs. These IRBs decided to use full board review for some or all protocols that were eligible for expedited review with the understanding that this was permissible under the Common Rule. In fact, two IRBs did not use expedited review procedure at all as a matter of institutional policy.

Protocols reviewed using an expedited review procedure

Table 2 shows levels of review as determined by OHRP algorithms for the 113 protocols granted expedited review: 106 (93.8%) were determined to be eligible for expedited review according to OHRP criteria, 5 (4.4%) were qualified as exempt protocols, 1 (0.9%) was determined to be non-exempt and not eligible for expedited review, and 1 was not human subject research. Thus, there was a high degree of agreement (i.e., 93.8%) between actual VA IRB expedited reviewed protocols and levels of review as determined using OHRP criteria.

Table 2.

Levels of review determined using OHRP algorithms for protocols granted expedited review

Site #	Full-boarda	Expeditedb	Exempt	Not human research	Total
1	0	13	2	0	15
2	0	13	2	0	15
3	0	15	0	0	15
4	0	9	1	0	10
5	1	12	0	1	14
6	0	15	0	0	15
7	0	0	0	0	0
8	0	15	0	0	15
9	0	0	0	0	0
10	0	14	0	0	14
Total	1 (0.9%)	106 (93.8%)	5 (4.4%)	1 (0.9%)	113 (100%)

^aColumns from left to right were in decreasing intensity of levels of review.

^bBold prints were used to denote the column that was concordant with the granted level of IRB review.

In summary, of the 113 protocols reviewed by IRBs using the expedited review procedure, 6 (5.3%) protocols, i.e., 5 exempt and 1 non-human research protocols, received more intense review than needed according to OHRP criteria, 1 (0.9%) full-board review protocol received less intense review, and 103 (93.8%) protocols were concordant.

Exempt protocols

Table 3 shows levels of review as determined by OHRP algorithms for the 60 protocols exempted from IRB review. Due to local policies and differing research portfolios, only 6 out of 10 sites contributed protocols that had been exempted from IRB review. Of these 60 exempted protocols, 45 (75%) were qualified as exempt protocols according to OHRP criteria, 6 (10%) were determined to be eligible for expedited review, 5 (8.3%) were not human subject research, and 4 (6.7%) lacked detail sufficient to qualify for expedition or exemption and as described above these were adjudicated as requiring full-board review. Therefore, there was only 75.0% agreement between actual VA exempted protocols and levels of review as determined using OHRP criteria. In addition, a total of 10 protocols that required IRB review according OHRP criteria, were implemented as exempt protocols without IRB reviews, (i.e., 4 requiring full board review and 6 requiring IRB expedited review, all of which were from 2 sites, i.e., sites #3 and #8).

Table 3.

Levels of review determined using OHRP algorithms for protocols granted exempt from IRB review.

Site #	Full-boarda	Expedited	Exemptb	Not human research	Total
1	0	0	12	0	12
2	0	0	5	0	5
3	2	4	7	2	15
4	0	0	7	0	7
5	0	0	0	0	0
6	0	0	0	0	0
7	0	0	0	0	0
8	2	2	11	0	15
9	0	0	3	3	6
10	0	0	0	0	0
Total	4 (6.7%)	6 (10.0%)	45 (75.0%)	5 (8.3%)	60 (100%)

^aColumns from left to right were in decreasing intensity of levels of review.

^bBold prints were used to denote the column that was concordant with the granted level of IRB review.

In summary, of the 60 protocols exempted from IRB review, 5 (8.3%) non-human research protocols received more intense review than needed according to ORHP criteria, 10 (16.7%) protocols, i.e., 6 eligible for expedited review and 4 requiring full board review, received less intense review, and 45 (75%) protocols were concordant.

Site level differences

Disagreements between IRBs and coders using OHRP criteria varied among sites. For example, disagreement on full-board review protocols was limited to sites 8 and 9; for expedited review protocols, sites 1, 2, 4, and 5; and for exempt protocols, sites 3, 8, and 9. Discrepancies were most variable regarding the protocols granted full-board review where 8 of 10 sites granted full board reviews to at least 1 protocol eligible for expedited review procedures, and the site-specific proportion of protocols that could have been expedited ranged from 9% (1/11, Site 4) to 87% (13/15, Site 7).

DISCUSSION

The results presented in this study demonstrated that there was a very high degree of agreement between protocols that were either reviewed by VA IRB full board at a convened meeting (97.8%) or reviewed using an expedited review procedure (93.8%) and the levels of review as determined using OHRP criteria. There was only 1 (0.9%) out of 113 protocols reviewed by VA IRBs using expedited review procedure that should have been reviewed by a higher level of review, i.e., full board review at a convened meeting, according to OHRP criteria. On the other hand, among the 140 VA IRB full board-reviewed protocols, 66 (47.1%) could have been reviewed using a lower level of review, i.e., expedited review, according to OHRP criteria. The expedited review procedure is not only less labor intensive, but also on average 44 days faster than full board review (Varley, Feske, Gao, et al., 2016). The costs associated with assembling fully convened IRB meetings are significant given advanced training of members and the time involved. Therefore, it would be much more efficient and beneficial to investigators if IRBs utilized expedited review procedures to the greatest extent permissible by the regulations, thus minimizing the delays and costs associated with full board review. An additional incentive for using expedited review procedure is that under the revised Common Rule, the annual IRB continuing review requirement of ongoing research for studies that undergo expedited review is no longer required (Menikoff, Kaneshiro, & Pritchard. 2017; U.S. Department of Health and Human Services, 2018).

In contrast, the agreement between VA IRB exempted protocols and levels of review as determined using OHRP criteria, was poor. Specifically, 10 (16.7%) of the 60 exempted protocols were determined to require IRB review, i.e., 6 protocols requiring expedited review and 4 protocols requiring full board review, according to OHRP criteria. Conducting non-exempt research without prior IRB approval constitutes serious noncompliance that should be reported to OHRP along with a remedial action plan to prevent its recurrence in the future as required by federal regulations (U.S. Department of Health and Human Services, 2011). This finding was surprising, because VA policy required that for exempt protocols, the exempt status should be determined by the IRB chair or an experienced IRB member designated by the chair and that the applicable exempt category/categories be documented (U.S. Department of Veteran Affairs, 2019).

One could argue that the coders in this study may have made an inaccurate determination of levels of review in these 10 protocols using OHRP criteria. However, these two coders were quite consistent in using OHRP criteria to determine levels of review for expedited review and full board review protocols. In addition, all 10 protocols in question were from 2 IRBs (sites 3 and 8), suggesting that the problem(s) might have arisen specifically from these two specific IRBs rather than the 2 coders. Site level differences were also observed in protocols granted full board review that might otherwise qualify for expedited procedures, suggesting that site-specific interventions might be designed to improve either the quality of review (Sites 3 & 8) or the overall efficiency (all sites except Sites 1 and 8).

There are ample examples in the literature of research that was conducted under the pretense of exempt protocols, but were later found to require IRB approval (Cooper & McNair, 2019). Exactly how exempt status of a protocol is determined in non-VA research institutions is not clear. The Common Rule does not define who should make such a determination. However, because of the potential for conflict of interest, OHRP recommends that investigators not be given the authority to make an independent determination that human subjects research is exempt and that institutions should implement exemption policies that most effectively address the local setting and programs of research (U.S. Department of Health and Human Services, 2020a). The recently implemented revisions to the Common Rule markedly expanded the categories for exemption, some of which require limited IRB review (Menikoff, Kaneshiro, & Pritchard. 2017; U.S. Department of Health and Human Services, 2018). The impact of revised Common Rule on the compliance of exempt research regulations remains to be determined.

Our study is the first to look at this issue systematically. While the total number of exempt protocols examined, i.e., 60, in this study was small, in view of the above findings, we suggest that further studies should be carried out to determine how much non-exempt human subjects research is conducted without prior IRB approval. Once non-exempt research is carried out without prior IRB approval, there are few options available to remedy the noncompliance. Prevention remains the best strategy to ensure that no non-exempt human subjects research is initiated prior to IRB approval (Tsan, 2020).

Limitations of this study:

Our study is limited to only 10 VA IRBs. The results may not generalize to other VA IRBs or to IRBs in the private sector. In addition, our data and procedures predate the revised Common Rule. However, we believe that our findings and conclusions remain applicable in the post, revised Common Rule era. The revised Common Rule expanded exempt categories from the original six categories to eight categories. These changes included:

• Adding some restrictions on Category 1. Educational practices,

• Expanding Category 2. Educational tests, surveys, interviews, and observation of public behavior,

• Replacing Category 3. Research on public officials with Research involving benign behavioral interventions,

• Expanding Category 4. Research on existing data to Secondary research that does not require consent,

• Expanding Category 5. Public benefit or service programs,

• No change on Category 6. Taste and food evaluation,

• Adding new Category 7, Storage of identifiable private information or identifiable biospecimens for secondary research for which broad consent is required, and

• Adding new Category 8, Secondary research using identifiable private information or identifiable biospecimens for which broad consent is required (Anderson, 2018).

Thus, most of the original six exempt categories remain effective. In addition, the revised Common Rule did not change any expedited review categories nor the eligibility criteria for expedited review (Menikoff, Kaneshiro, & Pritchard. 2017; U.S. Department of Health and Human Services, 2018). Nonetheless, It would be important to carry out similar studies to determine the impact of the revised Common Rule on levels of initial protocol review by IRBs using our current results as a baseline for comparison.

Conclusions:

In this study we demonstrated that 47.1% of protocols reviewed by full-board IRBs at a convened meeting could have been reviewed using an expedited review procedure. While this is permissible under federal regulations, we suggest that IRBs should utilize expedited review procedure to the greatest extent possible to improve efficiency. In addition, 16.7% of IRB exempted protocols required either expedited review or full board review. Conducting non-exempt research without prior IRB review and approval is a serious noncompliance. Thus, exempt protocols need more scrutiny to ensure that no non-exempt human subjects research is initiated prior to IRB approval.

Best Practices

Based on the results of this study, we recommend that IRBs should 1) optimize efficiency by using expedited review procedures as much as possible to review those protocols eligible for expedited review, and 2) augment and monitor their procedures for exemption to ensure that no non-exempt human subjects research is initiated prior to IRB approval.

Research Agenda

Future research should be conducted to determine the extent, if any, of non-exempt human subjects research that is carried out intentionally or unintentionally as exempt protocols without prior IRB approval, especially in the post revised Common Rule era. Specifically, we suggest the following studies from a representative sample of IRBs:

• Determine how exempt status of a protocol is determined and by whom using a questionnaire survey.

• From a sample of exempt protocols (both before and after the implementation of revised Common Rule), determine how many of these protocols are exempt (under which categories), qualified for expedited review, or requiring full board review, according to OHRP criteria, using OHRP Human Subject Regulations Decision Charts (Pre-2018 and 2018 Requirements) (U.S. Department of Health and Human Services. 2016 & 2020b).

• Compare pre- and post-revised Common Rule results to determine the impact of revised Common Rule.

Educational Implications

Prevention is the best strategy to ensure that no non-exempt human subjects research is initiated prior to IRB approval. Investigators and designated individual(s) who is authorized to make determinations on exempt status should be educated and trained on the various exempt categories and how to determine whether a study is qualified to be exempt using OHRP Human Subject Regulations Decision Charts (U.S. Department of Health and Human Services. 2020b). When in doubt, OHRP should be consulted.