How Closely Do Institutional Review Boards Follow the Common Rule?

Abstract

Purpose

To determine how closely institutional review board (IRB) discussions reflect the ethical criteria specified in the Common Rule federal regulations

Method

Between November 2006 and July 2009, the authors observed, audio-recorded, transcribed, and coded protocol reviews from 20 IRB meetings at 10 leading academic medical centers. They also reviewed each of the applications discussed to identify the Common Rule criteria--(1) risk minimization, (2) risk/benefit comparison, (3) equitable subject selection, (4) informed consent, (5) data monitoring to ensure safety, (6) privacy protection and confidentiality, and (7) protection of vulnerable subjects--that were both relevant to the study and not adequately addressed in the application. They then determined if the IRB addressed each of the relevant and not discussed Common Rule criteria in their discussions.

Results

IRBs made no mention of many of the Common Rule criteria that required their discussion--in 17/82 (21%) reviews, they failed to address risk minimization; in 52/91 (57%), risk/benefit comparison; in 31/52 (60%), equitable subject selection; in 32/59 (54%), data monitoring; in 13/52 (25%), privacy and confidentiality; and in 7/55 (13%), protection of vulnerable populations. However, they discussed informed consent in 102/104 (98%) reviews and raised questions about, or requested changes about, informed consent for 92/104 (88%) protocols.

Conclusions

These findings suggest that essential elements of human subjects protection are not implemented uniformly across IRBs. While not directly addressing this issue, the current proposed changes to the Common Rule offer an opportunity to improve, in general, the effectiveness of IRBs to protect human subjects.

Institutional review boards (IRBs) play a critical role in protecting human research subjects in the United States. Federal regulations mandate that every institution receiving federal research funds establish a committee to review certain ethical issues in research involving human subjects. Review is only required for federally-supported research involving human subjects, however most medical and academic institutions require IRB review for all research involving human subjects, with a narrow list of exceptions, regardless of the funding source.^1–2

Despite the importance then of IRBs in human subjects research, we know relatively little about how they function. Several studies, dating to the 1970s, included surveys of the composition, staffing, and workload of IRBs, and the types of protocols that they reviewed.^2–7 These studies, however, primarily assessed the outputs of the review process, e.g., the number and type of changes requested in protocols, and found that IRBs focused principally on consent forms.⁸ Other studies looked at the variation in decisions when IRBs at different sites reviewed the same protocol and found substantial differences between IRB decisions, calling into question the reliability of such determinations.^9–11

However, the existing research has not examined what IRBs discuss when they review a protocol. What is known on this subject stems largely from surveys of IRB members.^{7, 12–17} In the 1970s, Gray and Cooke examined the performance of a single IRB by reviewing its meeting minutes.¹⁸ More recently, there have been several ethnographic descriptions of one or two IRB panels.^{6, 19} One of the most important questions that remains unanswered is the degree to which IRB deliberations address the primary elements of human subjects protection regulations.

We studied the factors that contribute to IRB decision-making at ten major academic medical centers (AMCs) by observing and audio-recording IRB meetings. Our analyses, presented here, focus on the extent to which these committee discussions addressed the ethical criteria in the federal regulations for the protection of human subjects, known as the Common Rule.²⁰

Method

Data collection

We observed in person and audio-recorded 20 IRB discussions at ten AMCs (CWL, SM, and other research staff). We recruited sites from among the 25 AMCs that received the most funding from the National Institutes of Health (NIH) in 2004. We chose this criterion because these AMCs do much of the clinical research in the United States and because, as larger institutions, they have the ethical and administrative resources for comprehensive reviews of research applications. At each site, we studied two panels that reviewed general applications (i.e., we excluded specialized panels such as those that reviewed only pediatric or transplant protocols). Between November 2006 and July 2009, we observed and audio-recorded one meeting of each panel and, to ensure that we examined the complete evaluation of the protocols, we only recorded and analyzed discussions of new or resubmitted proposals. The median number of such proposals discussed at each meeting was 5 (range 3 to 10).

This study received IRB approval at each of the ten study sites, as well as at the home institutions of the principal investigator (CWL) and co-investigators (PSA, RA). We informed all potential participants of the goals and procedures of our study before we observed the IRB meetings (and all participants could review our grant application). We then asked participants for their written consent to be audio-recorded and interviewed; 263 of 295 potential IRB members and staff consented. In keeping with our agreement with the participating IRBs, we did not transcribe the statements made by those IRB members who did not give their consent, although we noted the topic of what they said. Subsequently, we excluded eight of the 114 protocols from our data because a refusing participant was the primary reviewer (statements by non-consenting members who were not reviewers constituted 0.06% of all speaking turns). We excluded 2 additional reviews because we did not have the original applications for those protocols.

Coding

First, we transcribed the audio recordings from the 20 meetings, then we redacted the information from the transcripts and the original applications that would identify the site, the principal investigator (PI), or the protocol (SM and other research staff). Next, we screened each application to determine whether the IRB should have explicitly considered if the protocol met each Common Rule criterion. Two coders independently assessed whether each Common Rule criterion was relevant to the protocol and, if so, whether it had been addressed adequately in the application. Thus, for example, the equity of subject selection was relevant only if some otherwise eligible group had been excluded from the study (e.g., non-English speakers), and the application had not provided a convincing case for the exclusion (e.g., pregnant women should be excluded from studies of drugs not yet tested for their impact on fetuses). See Table 1 for a description of our inclusion coding rules for each of the seven Common Rule criteria.

Table 1.

Institutional Review Board Applications: Coding Rules for Determining Relevance and Need for Explicit Discussion of Each of the Seven Common Rule Criterion

Common Rule criterion	Relevance screen: Criterion not relevant if:	Discussion screen: No discussion needed if:
Risk minimization	Study poses no or minimal risks (i.e., no risks greater than would be encountered in everyday life)	Risks are actually minimized in a way that would be recognized by a reasonable person
Risk/Benefit comparison	Study poses no risks other than those related to confidentiality	If relevant, must always be discussed
Equitable subject selection	Study excluded no otherwise eligible group (i.e., everyone with the condition being studied is eligible to enter)	A clear and convincing justification is provided for the exclusion of an otherwise eligible group, demonstrating that there are no reasonable means of including them
Informed consent	Always relevant except if study is not enrolling any subjects	Must always be discussed
Data monitoring to ensure safety	Either (1) study poses no or minimal risks, or (2) study is not an intervention (i.e., study utilizes observational, survey, or interview methods, or is only based on analysis of existing data)	Data Safety Monitoring Board will monitor the study
Privacy protection and confidentiality	Study collected no identifiable information	Standard confidentiality precautions have been taken (e.g., locked files, password-protected computer files, etc.). Standard privacy precautions include the protection of sensitive data, such as criminal activity, illicit drug use, mental illness, AIDS, etc.
Protection of vulnerable populations	Study included no vulnerable populations	If relevant, must always be discussed

Agreement between the coders in our study was 72.4%. We addressed any disagreements by a group consensus process, and most were easily resolved. We developed this relevance screen to account for the possibility that reviewers determined in their pre-review that a criterion was appropriately dealt with in the application and therefore did not need to be discussed in the meeting.

Then, for each relevant criterion that was not sufficiently addressed in the application, two members of the research group (CWL, PC, SM, and other research staff) independently coded the transcripts of the discussion of that protocol at the IRB meeting. We used the Common Rule list of required determinations as our standard for adequate ethical review²⁰; these criteria were: (1) risk is minimized and (2) is reasonable in relation to benefits; (3) the subject selection process is equitable; (4) informed consent is sought and documented; (5) data are monitored to ensure subject safety; (6) subject privacy and confidentiality are protected; and (7) vulnerable subjects receive appropriate protection. The coders classified the IRB determinations of each relevant and unaddressed criterion into one of four mutually exclusive categories: (1) no mention of the criterion; (2) the criterion was mentioned, but there was no decision or action taken related to the criterion; (3), the committee raised questions for the investigator or requested changes to the protocol with respect to the criterion; or (4) the criterion was judged to be addressed acceptably. We did not attempt to judge the correctness of the IRBs’ evaluations of the protocols with respect to these criteria, nor did we evaluate the reasoning that led to these judgments. Instead, we focused on the more basic aspects of the IRBs’ decision-making--was the criterion discussed, and, if so, did the IRB decide whether the protocol satisfied the criterion?

Almost all disagreements on the coding of the IRBs’ discussions reflected a coder missing a code and were easily resolved by group review and agreement of the coding team. We assessed reliability for the independent coding process by computing Kappas (range = .42 to .72). We (LS, WG) analyzed the data using the SAS 9.2 freq procedure (The SAS Institute, Cary, NC).

Results

In all, we observed and audio-recorded the IRB discussions of 104 protocols (87 new and 17 resubmitted protocols) discussed by 201 assigned reviewers, with some assigned to review more than one protocol. See Table 2 for data on the characteristics of these 104 protocols. See Table 3 for a summary of the determinations of the IRBs with respect to each of the required Common Rule criterion.

Table 2.

Characteristics of the 104 Observed Protocol Reviews Discussed by the Institutional Review Boards at Ten Academic Medical Centers

Characteristic	No.	% of total
Field of medicine
Infectious diseases	11	11
Oncology	23	22
Neurology/Psychiatry	10	10
Circulatory system	12	12
Other	48	46
Total	104	100
Review status
New	87	84
Deferred	17	16
Total	104	100
No. of sites
Single-site	53	51
Multi-site	51	49
Total	104	100
Therapeutic/Non-therapeutic
Therapeutic	60	58
Non-therapeutic	44	42
Total	104	100
Study type
Observational	24	23
Intervention	80	77
Total	104	100
Study design
Phase I	14	13
Phase II	32	31
Phase III	27	26
Feasibility	2	2
Laboratory	7	7
Survey/Interview	2	2
Other	20	19
Total	104	100

Table 3.

Overview of Institutional Review Boards’ (IRBs’) Actions Regarding Each Relevant and Unaddressed Common Rule Criterion

Common Rule criterion	Authors’ determinations		IRBs’ actions
	Discussion not needed, no. (%)	Discussion needed, no. (%)	Criterion acceptably addressed, no. (%)	Questions raised or changes requested, no. (%)	Criterion mentioned but no decision made or action taken, no. (%)	Criterion not mentioned, no. (%)
Risk minimization	22/104* (21%)	82/104 (79%)	32/82 (39%)	31/82 (38%)	2/82 (2%)	17/82 (21%)
Risk/Benefit comparison	13/104† (13%)	91/104 (87%)	25/91 (28%)	11/91 (12%)	3/91 (3%)	52/91 (57%)
Equitable subject selection	52/104‡ (50%)	52/104 (50%)	7/52 (13%)	12/52 (23%)	2/52 (4%)	31/52 (60%)
Informed consent§	0/104 (0%)	104/104 (100%)	5/104 (5%)	92/104 (88%)	5/104 (5%)	2/104 (2%)
Data monitoring to ensure safety	45/104¶ (43%)	59/104 (57%)	13/59 (22%)	11/59 (19%)	3/59 (5%)	32/59 (54%)
Privacy protection and confidentiality	52/104**(50%)	52/104 (50%)	11/52 (21%)	26/52 (50%)	2/52 (4%)	13/52 (25%)
Protection of vulnerable populations	49/104†† (47%)	55/104 (53%)	21/55 (38%)	23/55 (42%)	4/55 (7%)	7/55 (13%)

^*These 22 studies included those of minimal risk.

^†These 13 studies included those with no risks other than to confidentiality.

^‡These 52 studies included those for which no otherwise eligible group was excluded or those for which a clear justification was provided for those who had been excluded.

^§The Common Rule contains two separate criteria related to consent, i.e., that consent will be sought (absent a waiver) and that it will be documented; for the purposes of our analysis, we combined these two criteria.

^¶These 45 studies included those with minimal risks, no intervention, or a Data Safety Monitoring Board review in the application.

^**These 52 studies included those for which no identifiable information was collected or those for which standard confidentiality precautions were included in application.

^††These 49 studies included only those that did not include vulnerable populations.

Risk minimization

We determined that 22 of the 104 (21%) applications met our inclusion criteria (as described in Table 1) for not needing explicit determinations regarding risk minimization. Only 2 of the remaining 82 (2%) IRB reviews, for which risk minimization had to be addressed, made no mention of the risks of the study. However, this group, in which risks were not mentioned, included a phase 1 oncology study of a new combination of approved chemotherapy agents and a placebo-controlled phase 3 trial of a new treatment for a previously untreatable disorder, both of which posed significant risks.

The Common Rule also specifically requires that the IRB determine whether the risks have been minimized. In 32 of the 82 (39%) cases, which we assessed as needing an explicit determination, the IRB indicated that the risks were minimized or otherwise acceptable. In another 31 (38%) cases, the IRB asked for more information or proposed changes to reduce the risks. In 2 (2%) cases, the IRB discussions included some mention of whether the risks were minimized or appropriate, but no follow-up and no explicit decision on whether the risks were appropriate. In the remaining 17 (21%) cases, risk minimization was not discussed at all.

Risk/Benefit comparison

We excluded 13 of the 104 (13%) protocols by determining that a comparison of the risks and benefits either was not relevant or had been adequately addressed and therefore did not require explicit discussion. In 25 of the remaining 91 (27%) protocols, the IRB explicitly approved the risk/benefit ratio, and, in 11 (12%), they asked for more information or requested changes. In another 3 (3%) protocol reviews, the IRB mentioned both risks and benefits, but there was no follow-up. In the remaining 52 (57%) instances, either risks or benefits were not mentioned in the protocol review.

Equitable subject selection

All 104 IRB protocols included a description of the subjects to be recruited, but the IRB discussions usually focused on the scientific appropriateness of the sample, not the equity of the recruitment. We determined that 52 of the 104 (50%) protocols either did not exclude any categories of subjects or provided a clear justification for those who they had excluded (e.g., pregnant women were excluded from drug trials). We eliminated these 52 protocols from our subsequent analysis. The IRB explicitly approved the equity of subject selection for 7 of the remaining 52 protocols (13%), and returned 12 (23%) to the PI with questions or suggested changes. However, in reviewing 31 (60%) of the protocols that excluded some category of subjects without giving an adequate explanation, the IRB did not mention the equity of the subject recruitment. The IRB discussions of the final 2 (4%) protocols included some criticism or questions about equity but no follow-up or decision on the criterion.

Informed consent

In contrast to the other criteria, which sometimes were not discussed, IRBs routinely discussed informed consent. They approved unchanged consent forms for 5 of the 104 (5%) protocols, while recommending or requesting changes to 92 (88%) others. In 5 (5%) reviews, an IRB committee member criticized or suggested changes to the consent form but no further action was taken. In the remaining 2 (2%) reviews, the IRB made no mention of informed consent. These two instances included a study that was not recruiting subjects at the review site and another for which the IRB reviewers were so critical of the design that they never discussed informed consent.

Data monitoring to ensure safety

The Common Rule states that data must be monitored to ensure subject safety when appropriate.²⁰ The IRB application forms at all ten sites asked for a data monitoring plan when a protocol posed more than minimal risk and included an intervention. We determined that data monitoring was not appropriate in 45 of the 104 (43%) protocols, either because the study posed minimal risk, lacked an intervention, or already had Data Safety Monitoring Board (DSMB) review. Thirteen of the remaining 59 (22%) protocols were approved, and 11 (19%) included a request for a change or for further information. The IRB mentioned the criterion but did not make a decision for another 3 (5%) protocols and did not discuss it at all in the remaining 32 (54%) reviews.

Privacy protection and confidentiality

The Common Rule mandates that adequate provisions be taken to protect the privacy of subjects and to maintain the confidentiality of data, when appropriate.²⁰ We determined that 52 of the 104 (50%) protocols either did not include the collection of identifiable information or had plans to maintain subjects’ confidentiality. In the remaining 52 (50%) protocols, this criterion was relevant. It was discussed and approved in 11 (21%) protocols. The IRB requested modifications or more information in 26 (50%) reviews. In another 2 (4%) reviews, the IRB mentioned but did not make a decision on the criterion. They did not mention confidentiality and privacy issues in the remaining 13 (25%) reviews.

Protection of vulnerable populations

The Common Rule states that IRBs “should be particularly cognizant of the special problems of research involving vulnerable populations, such as children, prisoners, pregnant women, mentally disabled persons, or economically or educationally disadvantaged persons.”²⁰ This criterion was not relevant in 49 of the 104 (47%) protocols, which did not involve any identified vulnerable populations. In 21 of the remaining 55 (38%) protocol reviews, the IRB explicitly approved the inclusion or exclusion of vulnerable subjects from the study. In another 23 (42%) reviews, the committee asked questions or requested changes concerning the protection of vulnerable subjects. However, in 4 (7%) reviews, the criterion was mentioned, but there was no follow-up. It did not come up at all in the remaining 7 (13%) reviews.

Summary

For each protocol in our study, we noted if the IRB made a clear determination on each of the Common Rule criterion that was relevant to the protocol. A clear determination meant that the IRB either agreed that the criterion was acceptably addressed in the protocol, raised a question for the PI, or requested a modification. We only looked at the criteria that we determined in our earlier coding to be relevant and not adequately addressed in the study application. The IRB made a mean of 3.1 (standard deviation =1.3) clear determinations per protocol and did not make a determination on a mean of 1.7 (standard deviation=1.2) criteria. The IRB made clear determinations on all relevant criteria for 20 of the 104 (19%) protocols.

Discussion

We found that not all IRBs discussed all of the criteria mandated by the Common Rule. First, in over 20% of the protocol reviews of studies that posed greater than minimal risk, IRBs did not consider whether the studies’ risks were appropriately addressed in the application or whether those risks could be minimized. Second, IRBs did not compare the risks to benefits in 57% of the protocol reviews that had risks other than those to confidentiality. Third, that IRBs did not consider equity in subject selection in 60% of the applications that excluded categories of subjects suggests that they are not routinely determining whether the benefits and burdens of research are distributed evenly among populations. Finally, IRBs in only about 40% of reviews of relevant protocols either approved or proposed revisions to plans to monitor data for potential safety issues, in 70% made determinations with regard to confidentiality, and in 20% did not address the protection of vulnerable populations. In summary, although we could not listen to the audio recordings of the meetings without appreciating the seriousness with which the reviewers took their responsibilities, we found that the IRBs frequently failed to discuss many of the human subjects protection criteria mandated by the Common Rule.

Of note, the Common Rule applies only to research supported by a federal department or agency.²⁰ However, through the Office for Human Research Protections, all ten sites that participated in our study had formally agreed to apply the Common Rule to non-federally supported research as well. Yet, the failure of these IRBs to address consistently all required parts of the Common Rule does not necessarily mean that they violated a law. AMC staff or IRB committee members might have considered these issues outside the formal IRB meetings then not discussed them in the meetings. However, that they did not discuss these criteria raises questions about the effectiveness of the meetings to implement the ethical mandates of the Common Rule.

In contrast, the IRB committees paid careful attention to consent forms and processes. They almost always discussed the consent forms and nearly 9 in 10 protocol reviews resulted in questions about, or proposals for revisions of, consent forms. This finding is consistent with other research that has found that IRBs regularly and often exclusively recommend changes to consent forms.⁸

Limitations

Our study has four limitations. First, we only judged the adequacy of IRB decision-making to a limited degree. For example, if an IRB determined that a protocol had addressed a criterion appropriately, we did not judge whether the protocol actually met the standard of the Common Rule. In this sense, our findings regarding the adequacy of IRBs discussions of these ethical issues are conservative. A second limitation concerns the analysis of resubmitted applications. Although reviewers usually discussed the prior review, and we found few differences between the discussions of resubmitted and new protocols, some of these topics may have been discussed in a prior meeting and not in the one that we observed. This limitation, however, does not threaten our overall conclusions, because we found that there were no statistically significant differences between resubmitted and new protocols on any of the Common Rule criteria. Third, the generalizability of our findings may be limited because our IRBs were exclusively from the largest AMCs in the United States. Finally, the IRBs that we observed may have altered their behavior under the scrutiny of our investigators. However, if our presence had an effect, it likely led to a greater focus on the Common Rule criteria.

In Summary

We found that essential elements of human subjects protection were not implemented uniformly across IRBs. A mechanism for ensuring that consideration is given to each of the seven key criteria of the Common Rule might help to improve the reliability of IRB determinations and to ensure that IRBs review all criteria.²⁰

The Department of Health and Human Services is currently proposing changes to the Common Rule.²¹ These proposed changes will address many problems, but not the deficiencies that we found in our study--IRBs’ failures to assess all required Common Rule criteria, i.e., to perform a comprehensive protocol review. Although any reform must avoid bogging down IRBs in bureaucratic consideration of criteria that are minimally relevant to a given protocol, we recommend clarifying the meaning of “IRBs’ shall determine” to improve the human subjects’ protection process. The current impetus toward regulatory reform offers an opportunity to address this issue and to improve the effectiveness of IRBs to protect human subjects. IRBs should revise their procedures to improve their coverage of the basic ethical issues covered by the Common Rule, and future research should assess the success of these revisions.