Protecting human subjects participating in research

Abstract

Objectives: Institutions conducting research involving human subjects establish institutional review boards (IRBs) and/or human research protection programs to protect human research subjects. Our objectives were to develop performance metrics to measure human research subject protections and to assess how well IRBs and human research protection programs are protecting human research subjects. Methods: A set of five performance metrics for measuring human research subject protections was developed and data were collected through annual audits of informed consent documents and human research protocols at 107 Department of Veterans Affairs research facilities from 2010 through 2021. Results: The proposed performance metrics were: local adverse events that were serious, unanticipated, and related or probably related to research, including those that resulted in hospitalization or death; where required informed consent was not obtained; required Heath Insurance Portability and Accountability Act authorization was not obtained; non-exempt research was conducted without IRB approval; and research activities were continued during a lapse in IRB continuing reviews. Analysis of these performance metric data from 2010 through 2021 revealed that incident rates of all five performance metrics were very low; three showed a statistically significant trend of improvement ranging from 70% to 100%; and none of these five performance metrics deteriorated. Conclusions: Department of Veterans Affairs human research protection programs appeared to be effective in protecting human research subjects and showed improvement from 2010 through 2021. These proposed performance metrics will be useful in monitoring the effectiveness of human research protection programs in protecting human research subjects.

Introduction

Institutional review boards (IRBs) and comparable entities, such as research ethics committees and ethics review boards, have been established for the primary purpose of protecting human subjects participating in research [1]. Since the establishment of the IRB system in the 1970s, research institutions have delegated the authorities and responsibilities of protecting human research participants to IRBs [2].

However, a number of events occurring at the turn of this century suggested that IRB oversight, as practiced at the time, was insufficient in protecting human research subjects. Two young individuals, Jesse Gelsinger and Ellen Roche, who out of altruism volunteered in phase one clinical trials at the University of Pennsylvania and Johns Hopkins University, died on September 17, 1999, and June 2, 2001, respectively, as a result of egregious noncompliance by the investigators, IRBs, and institutions involved [3-5]. In addition, a number of major academic institutions’ federally funded research programs were temporally suspended due to persistent and serious noncompliance with federal regulations [3,6]. It became clear that, in addition to IRBs, investigators, institutions, sponsors of research, research volunteers, and the federal government all share responsibility for protecting human research subjects [7,8].

Substantial efforts were made in the early 2000s to improve our systems for protecting human research subjects, including, but not limited to, stronger federal oversight of research, implementation of voluntary external accreditation of human research protection programs, improved training for investigators and IRB members, improved monitoring and reporting of adverse events, and greater involvement of research participants and the public in these reform efforts [6,7]. While the investment was substantial, it is not clear whether these improvement efforts have resulted in improved protections for research participants.

Despite repeated calls for measuring the effectiveness of IRBs and the enhanced measures listed above in protecting human research subjects, there has been little or no empirical evidence in the literature demonstrating that IRB review and other measures do, in fact, protect human research subjects effectively [9,10]. This has led some investigators to question whether IRBs actually protect human research subjects [11,12]. Critics who are frustrated with the perceived burdens imposed by IRBs, in the absence of demonstrated effectiveness, have suggested that IRBs delay or prevent important research from taking place, while increasing the cost of research and impeding investigators’ opportunities for academic promotion [13,14].

Measuring the incidence of actual harms to participants in research undergoing IRB review versus research not undergoing IRB review would provide a direct comparison of the effectiveness of IRB review. In order to demonstrate that IRBs protect human research subjects, it would be ideal to carry out a prospective, randomized trial, in which comparable research projects are randomly assigned to two groups, one receiving IRB review (the intervention group) versus one not receiving IRB review (the control group). If research subjects in projects receiving IRB review experienced less harms than subjects in projects not receiving IRB review, then one could reasonably conclude that IRBs are effective in protecting human research participants [2,15].

Unfortunately, this ideal study cannot be carried out for at least three reasons. First, the Federal Policy (Common Rule) for the Protection of Human Research Subjects stipulates that no research can be initiated prior to IRB approval, unless it is deemed to be exempt from the Rule’s requirements [16]. Second, there are no widely accepted strategies for directly measuring the effectiveness of human research subject protections [2,17,18]. Finally, given the well-documented harms experienced by human subjects prior to any requirements for IRB review, it could be considered unethical to conduct human research without some form of objective oversight.

Although it would not be possible to conduct the prospective, randomized trial described above, it might be possible to conduct a large retrospective comparison of research receiving versus not receiving IRB review. However, such a comparison would require development of objective criteria sufficient to ensure the equivalence of the research studies compared and a sufficiently large sample of studies from which to draw. This kind of large retrospective comparison would be a major undertaking and would probably have to be conducted on a national scale and at great expense.

Nevertheless, a practical approach focusing on actual harms to human subjects is needed to provide a meaningful measure of the effectiveness of IRB review (and potentially other research protections) at the institutional level.

In this report, we propose a set of performance metrics for assessing the effectiveness of human research subject protections and use data collected from 107 Department of Veterans Affairs (VA) facilities conducting research involving human subjects from 2010 through 2021 to demonstrate the feasibility and utility of implementing these proposed metrics.

Measuring harms to research participants

Human research participants may experience two types of harms: concrete harms, such as physical or psychological injury, and dignitary harms, such as violations of autonomy or privacy rights. Measuring the incidence of these two types of harms actually experienced by research subjects would constitute a direct measure that institutions could use to demonstrate the effectiveness of their IRB reviews and human research protection programs.

Concrete harms

Concrete harms to subjects are reflected by the adverse events (i.e., physical or psychological harms) actually experienced by subjects participating in research. Examples might include death, disease progression, untoward drug effects, medical device malfunctions, failed surgical techniques, or interventions resulting in psychological harms.

Adverse events can be anticipated or unanticipated, and related or unrelated to the research. Anticipated adverse events related to research are the foreseeable risks of the research interventions (e.g., the administration of investigational drugs or the use of medical devices). These foreseeable risks are typically described in the research protocol, investigator’s brochure, and informed consent document that the IRB reviews. They must be disclosed to potential participants considering research participation [19].

Serious adverse events that are unanticipated and related to research are especially important because they constitute actual harms to subjects associated with previously unknown risks that were not disclosed to them. These events typically require substantive changes in the research protocol and informed consent document, and/or corrective actions to protect the safety and welfare of future participants [19]. As demonstrated in the case of Jesse Gelsinger and Ellen Roche, unanticipated, serious, research-related adverse events often occur due to egregious noncompliance by investigators, IRBs, and/or institutions [3-5].

The Common Rule requires that unanticipated, serious adverse events be promptly reported to the IRB, institutional official, agency head, and the Office for Human Research Protections (OHRP) [16,19]. The OHRP defines serious adverse event as any adverse event that (1) results in death; (2) is life-threatening (places the subject at immediate risk of death from the event as it occurred); (3) results in inpatient hospitalization or prolongation of existing hospitalization; (4) results in a persistent or significant disability/incapacity; (5) results in a congenital anomaly or birth defect; or (6) based upon appropriate medical judgment, may jeopardize the subject’s health and may require medical or surgical intervention to prevent one of the other outcomes listed in this definition [19].

We propose that the incidence of adverse effects determined by the IRB to be unanticipated, serious, and related (or possibly related) to the research would constitute a useful metric for capturing concrete harms actually experienced by research subjects.

Institutions could monitor trends in this metric to access the relative effectiveness of its IRB reviews (and other human research protections) over time.

Dignitary harms

We propose that dignitary harms associated with violations of human autonomy rights occur when research subjects are not given the opportunity to exercise meaningful informed consent. Enrolling subjects in research without obtaining informed consent is contrary to the ethical principle of respect for persons of the Belmont Report, and is a violation of the subject’s right to be respected as an autonomous person [20]. Thus, the incidence of failure to obtain informed consent from subjects (or their legally authorized representatives) constitutes a metric directly reflecting dignitary harm to human autonomy rights.

We propose that dignitary harms associated with violations of human privacy rights occur when research subjects are not afforded all the protections required under the Heath Insurance Portability and Accountability Act (HIPAA) [21]. Thus, the incidence of failure to obtain HIPAA authorization from subjects (or their legally authorized representatives) constitutes a metric directly reflecting dignitary harms to human privacy rights.

Other metrics that may indirectly reflect dignitary harms to subjects include the incidence of (non-exempt) research conducted without IRB approval and the incidence of research conducted without continuing IRB review because meaningful informed consent and full HIPAA protections are uncertain under those circumstances.

In addition, the Common Rule requires IRBs to ensure that research has met eight approval criteria that satisfy all three ethical principles of the Belmont Report [16] and reflect basic human dignitary rights. Thus, when human research is conducted without prospective IRB review, approval, and oversight, or when investigators continue research activities during lapses in required IRB continuing review, research participants are not afforded Common Rule protections, including assurance of their basic dignitary rights. They are also exposed to situations that could result in serious harm.

Methods

We sought to demonstrate the utility of the performance metrics proposed above by examining quality assurance data available from the 107 Department of Veterans Affairs (VA) medical facilities with human research programs.

Data collection

The VA Office of Research Oversight (ORO) has collected quality assurance performance metric data on VA human research protection programs each year starting in 2010. VA facility research compliance officers were required to conduct audits of all informed consent documents annually and regulatory audits of all human research protocols once every three years using auditing tools that ORO had developed (available at https://www.va.gov/ORO/orochecklists.asp). Approximately one third of all active human research protocols were audited each year. For protocols that had been active for more than three years, protocol regulatory audits were limited to the most recent three years of research. Results of these audits conducted between June 1 and May 31 of each year were collected from all VA research facilities through a web-based system [22,23].

Ethic statement (human subject protections)

This was a quality assurance project. It did not involve human subjects and no individually identifiable information was collected. Therefore, no IRB review and approval were necessary [24].

Data analysis

Data on human research subject protection performance metrics from 2010 through 2021 were used in this study. We used analysis of ordinal categorical data as described by Agresti [25] to determine the trend of change from 2010 through 2021. This was performed using JavaStat ordinal contingency table analysis available at www.statpages.info. A value of P < 0.05 was considered to be statistically significant. For those performance metrics with statistically significant changes, we also calculated percent changes from 2010 through 2021 using the following formula: Percent change = [(rate in 2021 - rate in 2010) ÷ rate in 2010] × 100 [23].

Results

Assessing human research subject protections

Unanticipated, serious adverse events related to research

Metrics reflecting unanticipated, serious adverse events constitute direct measures of concrete harms experienced by human research subjects.

Table 1 shows data from 2010 through 2021 on local adverse events that were determined by IRBs to be unanticipated, serious, and related (or probably related) to research, as well as those adverse events resulting in hospitalization or death. The numbers of protocols audited each year ranged from 2,102 in 2010, to 4,249 in 2012.

Table 1.

Local adverse events determined to be serious, unanticipated, and related or probably related to research

	2010	2011	2012	2013	2014	2015	2016	2017	2018	2019	2020	2021	p value1	% Change2
Total active protocols	14,944	16,421	16,602	16,568	16,244	15,769	15,699	15,279	15,258	15,061	14,637	15,015
Protocols audited	2,102	3,558	4,249	3,834	4,183	3,980	3,801	3,573	3,564	3,569	3,348	3,540
Local adverse events serious, unanticipated, and research related	25 (1.19%)3	43 (1.21%)	17 (0.40%)	29 (0.76%)	13 (0.31%)	45 (1.13%)	15 (0.39%)	13 (0.36%)	39 (1.09%)	15 (0.42%)	18 (0.54%)	13 (0.37%)	0.0005	-70%
Resulted in hospitalization	11 (0.52%)	10 (0.28%)	5 (0.12%)	2 (0.05%)	0 (0.00%)	4 (0.10%)	2 (0.05%)	3 (0.08%)	9 (0.25%)	0 (0.00%)	3 (0.09%)	-4	0.0013	-83%
Result in death	0 (0.00%)	0 (0.00%)	0 (0.00%)	2 (0.05%)	0 (0.00%)	0 (0.00%)	0 (0.00%)	1 (0.03%)	0 (0.00%)	1 (0.03%)	0 (0.00%)	-	0.7844	N/A5

¹Determined using analysis of ordered categories for the trend of changes from 2010 through 2020 or 2021.

²Percent change from 2010 through 2020 or 2021.

³The numbers in parentheses were the percentages of the total number of protocols audited.

⁴Data not collected.

⁵N/A denotes not applicable.

The rates of local adverse events that were determined to be unanticipated, serious, and related (or probably related) to research were low, ranging from 0.31% (i.e., 0.31 events per 100 protocols) in 2014 to 1.19% in 2010. They showed a statistically significant trend of change, decreasing from 1.19% in 2010 to 0.37% in 2021, an improvement of 70%.

The rates of these adverse events resulting in hospitalization ranged from 0.00% in 2014 and 2019, to 0.52% in 2010. They showed a statistically significant trend of change, decreasing from 0.52% in 2010 to 0.09% in 2020, an improvement of 83%.

The rates of death resulting from these adverse events were extremely low. In only 3 out of 12 years were any deaths reported (i.e., 1 or 2 deaths representing 0.03% and 0.05%), and there was no statistically significant trend of change from 2010 through 2020.

Informed consent and HIPAA authorization

Metrics reflecting failure to obtain informed consent and failure to obtain HIPAA authorization constitute direct measures of dignitary harms experienced by human research subjects.

Table 2 shows data from 2010 through 2021 on informed consent documents audited each year; informed consent documents that were not obtained; HIPAA authorizations required; and HIPAA authorizations that were not obtained.

Table 2.

Informed consent document and Health Insurance Portability and Accountability Act authorization

	2010	2011	2012	2013	2014	2015	2016	2017	2018	2019	2020	2021	p value1	% Change2
Total active protocols	14,944	16,421	16,602	16,568	16,244	15,769	15,699	15,279	15,258	15,061	14,637	15,015
Protocols audited	-3	15,978	16,546	16,522	15,730	15,765	15,629	15,264	15,233	14,892	13,985	12,066
ICDs4 audited	89,216	100,832	99,013	102,085	93,206	86,389	89,024	90.153	82,849	73,331	57,828	35,323
Informed consent not obtained	-	-	358 (0.36%)5	110 (0.11%)	89 (0.10%)	95 (0.11%)	29 (0.03%)	34 (0.04%)	85 (0.11%)	74 (0.10%)	38 (0.07%)	138 (0.39%)	0.0000	+8%
HIPAA authorization Required	-	95,916	96,290	97,297	87,528	82,577	86,109	87,045	78,372	69,970	52,756	33,356
Authorization not obtained	-	1,383 (1.44%)	827 (0.86%)	1,164 (1.20%)	783 (0.89%)	698 (0.85%)	486 (0.56%)	572 (0.66%)	518 (0.66%)	529 (0.76%)	535 (1.01%)	477 (1.43%)	0.0000	-7%

¹Determined using analysis of ordered categories for the trend of changes from 2011 or 2012 through 2021.

²Percent changes from 2011 or 2012 through 2021.

³Data not collected.

⁴Abbreviations used: ICD, informed consent documents; HIPAA, Health Insurance Portability and Accountability Act.

⁵The numbers in parentheses were the percentages of the total number of informed consent documents or HIPAA authorizations audited.

The numbers of informed consent documents audited ranged from 35,323 in 2021 to 102,085 in 2013. The number of informed consent documents not obtained, which included missing informed consent documents as well as informed consent documents not signed by the subjects (or legally authorized representatives), was small, ranging from 0.36% in 2012 to 0.03% in 2016. There was a statistically significant trend of change, decreasing from 2012 to 2016 and then increasing from 2016 to 2021. As a result, the percentage difference between 2012 (0.36%) and 2021 (0.39%) was only 8%.

The number of required HIPAA authorizations audited ranged from 33,356 in 2021 to 97,297 in 2013. The number of required HIPAA authorizations not obtained was small, ranging from 0.56% in 2016 to 1.44% in 2011. There was a statistically significant trend of change, decreasing from 2011 to 2016 and then increasing from 2016 to 2021. As a result, the percentage difference between 2011 (1.44%) and 2021 (1.43%) was only 7%.

Initial and continuing IRB reviews

Metrics reflecting failure to obtain required initial review and failure to obtain required continuing IRB review while continuing research activities, constitute indirect measures of dignitary harms experienced by human research subjects.

Table 3 shows data from 2010 through 2021 on protocols conducted and completed without IRB review and approval; protocols requiring IRB continuing reviews; and protocols in which investigators continued research activities during lapses in required IRB continuing review.

Table 3.

Institutional review board initial and continuing reviews

	2010	2011	2012	2013	2014	2015	2016	2017	2018	2019	2020	2021	p value1	% Change2
Total active protocols	14,944	16,421	16,602	16,568	16,244	15,769	15,699	15,279	15,258	15,061	14,637	15,015
Protocols audited	2,102	3,558	4,249	3,834	4,183	3,980	3,801	3,573	3,564	3,569	3,348	3,540
Conducted without required IRB3 approval	1 (0.05%)4	2 (0.06%)	1 (0.02%)	0 (0.00%)	0 (0.00%)	0 (0.00%)	0 (0.00%)	0 (0.00%)	0 (0.00%)	0 (0.00%)	0 (0.00%)	0 (0.00%)	0.0049	-100%
Protocols requiring IRB continuing review	1,606	2,942	3,411	3,112	3,593	-5	3,162	3,094	3,035	2,861	2,547	2,147
Continued research during lapse	2 (0.12%)	6 (0.20%)	4 (0.12%)	3 (0.10%)	11 (0.31%)	-	0 (0.00%)	0 (0.00%)	3 (0.10%)	0 (0.00%)	2 (0.08%)	0 (0.00%)	0.0030	-100%

¹Determined using analysis of ordered categories for the trend of changes from 2010 through 2021.

²Percent change from 2010 through 2021.

³Abbreviation used: IRB, institutional review board.

⁴The numbers in parentheses were the percentages of the total number of protocols audited or the number of protocols requiring IRB continuing reviews.

⁵Data not collected.

The numbers and rates of protocols conducted without IRB approval were very small, and they happened only in the first 3 years from 2010 through 2012, ranging from 0.02% in 2012 to 0.06% in 2011. After 2012, there were no research protocols conducted without IRB prior approval. Thus, from 2010 to 2021, there was a statistically significant trend of change, decreasing from 0.05% in 2010 to 0.00% in 2021, a decrease of 100%.

The numbers and rates of investigators continuing research activities during lapses in IRB continuing reviews were very small, ranging from 0.00% to 0.31% in 2014. There was a statistically significant trend of change, decreasing from 0.12% in 2010 to 0.00% in 2021, an improvement of 100%.

Discussion

In this report, we proposed a set of five performance metrics to measure the effectiveness of protections for human research subjects. The first metric (unanticipated, serious adverse events related to the research) captures concrete harms actually experienced by subjects participating in research. The second and third metrics (failure to obtain informed consent and failure to obtain HIPAA authorization) capture dignitary harms actually experienced by human research subjects.

The final two metrics (failure to obtain required initial IRB review and failure to obtain required continuing IRB review while continuing research activities) constitute indirect measures of dignitary harms that may or may not have been associated with harms actually experienced by subjects. However, these two metrics are important to measure because these situations place subjects at risk of harm in the absence of objective oversight. For example, in 2016, an investigator at Southern Illinois University injected himself and at least 17 other “volunteers” with a live attenuated herpes simplex virus vaccine that he had developed without approval from the IRB, submission of an investigational new drug application (IND) to the Food and Drug Administration (FDA), or obtaining informed consent from participants [26].

Ideally, all of these five human research subject protection performance metrics should have zero incidence rates, and in any case, their incidence rates should always be kept as low as possible (i.e., the lower the incidence rates, the better the human research subject protections).

The proposed performance metrics allow us to monitor trends in the effectiveness of human research protection programs in protecting human research subjects, as well as to identify areas of vulnerability for quality improvement purposes. For example, the VA data presented in this report revealed that:

● Incidence rates of all five human research subject protections performance metrics were very low (i.e., mostly less than 1%), and some were or approached 0%.

● Three of the five performance metrics showed statistically significant trends of improvement from 2010 through 2021, i.e., local adverse events that were unanticipated, serious, and related (or probably related) to research improved by 70%, those resulting in hospitalization improved by 83%, and failure to obtain required initial IRB review and failure to obtain required continuing review both improved by 100%.

● None of the five proposed human research subject protections performance metrics deteriorated from 2010 through 2021.

Thus, based on the proposed human research subject protections performance metrics, VA human research protection programs appeared effective in protecting human subjects participating in research, and that they showed improvement from 2010 through 2021.

The Common Rule was extensively revised in 2018 with the stated goal of enhancing human research subject protections and reducing burdens to investigators and IRBs [27]. The revised Common Rule was implemented on January 21, 2019. The proposed metrics for measuring human research subject protections could be used to assess whether the revised Common Rule in fact improved human research subject protections by comparing data collected before and after implementation of the revised Rule.

Limitations

One could argue that when human research is carried out without IRB review and approval, there would be no relevant records to review. However, with the general awareness of the requirement for IRB approval prior to the initiation of any nonexempt human research, it is hard for any investigators to hide human research activities that are not approved by IRBs. Unauthorized research like the above mentioned live attenuated herpes simplex virus vaccine study [26] will eventually come to light. Another potential source of research conducted without IRB approval is non-exempt research carried out (intentionally or un-intentionally) as exempt protocols without IRB approval. Institutions should periodically monitor exempt protocols to ensure that they are in fact exempt from IRB review and approval [28].

It is not clear whether high-risk research, such as cancer research protocols or significant-risk device studies, has a higher incidence of unanticipated, serious adverse events related to the research as compared to minimal risk research. If that were the case, institutions conducting more high-risk research would likely be found to be less effective in protecting human research subjects than institutions conducting mostly minimal risk research using the proposed performance metrics. Further studies are necessary to clarify this issue.

While we believe that the proposed metrics capture most concrete and dignitary harms actually experienced by research participants, there may be other metrics that better capture the effectiveness of human research subject protections. Better metrics can only be developed if more research institutions begin using the proposed metrics, or other metrics of their choice, to assess the effectiveness of their human research protection programs and publish the results. The experience gained and lessons learned through this process will eventually guide us to a set of mutually agreed upon performance metrics for measuring human research subject protections.

Summary

We proposed a set of five performance metrics for measuring human research subject protections: Unanticipated serious adverse events related to the research, failure to obtain informed consent, failure to obtain HIPAA authorization, failure to obtain required initial IRB review, and failure to obtain required continuing IRB review while continuing research activities. We used VA quality performance data collected from 2010 through 2021 to demonstrate the feasibility and utility of implementing these proposed performance metrics.

Disclosure of conflict of interest

None.