Data Integrity in the Pharmaceutical Industry: Analysis of Inspections and Warning Letters Issued by the Bioresearch Monitoring Program Between Fiscal Years 2007–2018

Abstract

Background

Warning Letters issued by the Food and Drug Administration’s Bioresearch Monitoring (BIMO) Program provide insight into data integrity issues and other research misconduct in the premarket side of the pharmaceutical industry. The objectives of this study were to understand the common compliance issues for clinical investigators, institutional review boards, sponsors of clinical studies, good laboratory practice laboratories, and bioequivalence studies and to see how compliance has changed over time.

Methods

Warning letters and closeout letters issued by the BIMO program between US fiscal years 2007–2018 were analyzed by categorizing regulatory violations into violation themes. Inspections during the same time period were analyzed based on the assigned inspection classification.

Results

A combined total of 300 warning letters were analyzed as a part of this study. The most common violations found in all warning letter categories included failing to follow and maintain procedures and poor documentation practices. Inspection results show that overall the percentage of Official Action Indicated results has decreased over time, while the percentage of No Action Indicated results has increased.

Conclusion

Although the number of warning letters has decreased over the past decade and inspection results have been improving, there are still significant data integrity and other regulatory compliance issues found in the premarket side of the pharmaceutical industry. It is unclear if the reduction of warning letters is due to improved compliance in the industry or other factors unrelated to compliance.

Background

Data integrity has become a hot topic in the pharmaceutical industry because of how easily data can be manipulated and compromised in the modern computer-based era. The US Food and Drug Administration (FDA) is especially aware of this, noting in their data integrity guidance that an increasing number of current good manufacturing practices (cGMP) violations involving data integrity have been observed in recent years [1]. Data integrity is typically defined in the pharmaceutical industry using the acronym ALCOA: data must be Attributable, Legible, Contemporaneously recorded, Original, and Accurate. In more recent years, this acronym has been extended to become ALCOA+, with the “+” referring to complete, consistent, enduring, and available, although the “+” does not technically add any new requirements to the original ALCOA and only provides extra emphasis [2].

According to the FDA, data integrity is important in the pharmaceutical industry for ensuring the safety, efficacy, and quality of drug products [1]. Data integrity issues can lead to poor quality products released on the market, which can have a negative impact on public health. Some examples of regulations in cGMP that are directly related to data integrity involve maintaining original data records (21 CFR 211.180), securing data from alteration or inadvertent erasure (21 CFR 211.68), and reviewing records for accuracy (21 CFR 211.182, 211.186(a), 211.188(b)(11), 211.194(a)(8)). However, data integrity violations are not limited to manipulated or mishandled data. The validity of data can also be challenged when standard procedures are followed incorrectly, leading to data that is inaccurate in the first place.

While data integrity issues can be hard to quantify and study in the real world, FDA-issued inspection citations and warning letters provide a tangible record of data integrity violations in the pharmaceutical industry.

Overview of FDA Inspections

Inspections are the FDA’s way of ensuring that firms are complying with cGMP and other applicable regulations. The FDA inspects all drug and device manufacturers that market FDA-regulated products, including foreign manufacturing facilities [3]. At the end of an inspection, the FDA classifies the inspection under one of three categories [4]: A No Action Indicated (NAI) classification indicates that no compliance issues were found that require further action. A Voluntary Action Indicated (VAI) classification indicates that some regulatory violations were found but the FDA does not intend to pursue further regulatory action. An Official Action Indicated (OAI) classification indicates that significant violations were observed in the inspection and the FDA intends to take further regulatory action. The inspector will issue an FDA Form 483 at the end of an inspection that lists the most significant regulatory violations found [5]. If the violations in a 483 are serious enough and not corrected quickly, the FDA may decide to issue a warning letter. Electronic 483s are posted in a database on the FDA website, though these do not represent all 483s issued [6].

Warning letters are letters issued by the FDA to firms or individuals who have been caught violating regulations found within the Code of Federal Regulations (CFR), generally as the result of an inspection [7]. These letters contain detailed summaries on major violations found and they identify what must be done to correct the violations, typically with a response deadline of 15 business days. All of these letters are available to the public on the FDA’s website [8]. A combined total of 3538 warning letters were issued within the pharmaceutical industry (1418 for medical devices, 1263 for drugs, 108 for biologics, and 749 for veterinary medicines) from fiscal year (FY) 2009–2018 [9]. If the violations of the warning letter are not addressed properly within the timeframe, the FDA may consider further regulatory penalties, such as injunction or seizure of products.

The Bioresearch Monitoring Program

In 1977, the FDA developed a task force specifically for investigating clinical studies that would later become known as the Bioresearch Monitoring (BIMO) Program [10]. This task force was created because of congressional hearings between 1975 and 1976 that determined a need for improved monitoring of investigational research studies. The BIMO Compliance Program Guidance Manuals (CPGM) state that the objectives of the BIMO program are to protect the rights, safety, and welfare of subjects involved in FDA-regulated clinical trials, to verify the accuracy and reliability of clinical trial data submitted to the FDA in support of research or marketing applications, and to assess compliance with the FDA’s regulations governing the conduct of clinical trials [11]. Today, the BIMO program inspects clinical investigators, institutional review boards (IRB), sponsors, monitors, and contract research organizations (CROs) involved in clinical trials, in vivo bioavailability/bioequivalence (BA/BE) studies, and nonclinical labs subject to Good Laboratory Practice (GLP) regulations. Warning letters issued by the BIMO program made up about 6% of all warning letters issued to the pharmaceutical industry between FY 2009–2018 (205 BIMO program warning letters, 3538 total warning letters).

Closeout Letters

Closeout letters are follow-ups to warning letters that the FDA releases when all of the issues cited in the warning letter have been corrected by the entity that received the warning letter [7]. The FDA closeout letter program was created to keep the public informed about the regulatory compliance following warning letters. This program is relatively recent and only warning letters issued on or after September 1, 2009 are eligible to receive closeout letters. Currently, closeout letters are the only publicly available notice on the FDA website that indicate that a firm or an individual has corrected the violations mentioned in the warning letter.

Literature Review

Previous studies have been published on the topic of BIMO program warning letters, though most have focused on letters issued to clinical investigators and IRBs [12–16]. Studies by Bramstedt [12], Gogtay et al. [13], Shetty and Saiyed [14], and Garmendia et al. [15] have shown that clinical investigators commonly receive citations for deviating from investigational protocols, inadequately reporting adverse events, and flawed informed consent processes. These researchers also found common issues with IRBs to be deviating from standard procedures, inadequate documentation, and timely review of research studies [13, 14, 16]. Shetty and Saiyed [14] found that sponsors commonly had issues with monitoring investigations, submitting annual reports, and obtaining investigator agreements.

The current study analyzes all of the warning letters issued by the FDA’s BIMO program from FY 2007–2018, including clinical investigator, IRB, sponsor, sponsor-investigator, and GLP warning letters. To the best of our knowledge, this is the first study that has analyzed warning letters issued to GLP laboratories and is the most extensive study on BIMO program warning letters to date. This study also analyzes closeout letters associated with these warning letters, which has not been studied before. An analysis of the BIMO program’s inspection metrics from FY 2007–2017 is included and is related to the warning letters issued.

Methods

This article does not contain any studies with human or animal subjects performed by any of the authors.

This study involved the analysis of 300 warning letters issued by the BIMO Program over the course of 12 FYs (October 1, 2006 through September 30, 2018). Warning letters were obtained from the publicly available warning letter database located on the FDA website [8]. Standardized violation themes (VTs) were created for each category of warning letters in order to tally the frequency of violations found in warning letters, except for clinical investigator warning letters in which the VTs designed by Garmendia et al. were used [15]. VTs were also grouped by which FDA center issued the warning letter (CDER, CDRH, CBER, or CVM).

The categories that BIMO program warning letters are listed as in the warning letter database are clinical investigator, IRB, sponsor/monitor/CRO, sponsor-investigator, and GLP. For the purpose of this study, sponsor/monitor/CRO and sponsor-investigator warning letters were analyzed together due to their similarity. The 25 closeout letters were also obtained from the FDA website. These letters can be found linked to the original warning letters that they referenced in the FDA warning letter database.

The BIMO program publicly releases inspection metrics for every FY on the FDA website [17]. These reports include the total number of inspections conducted for each category for the FY and what classification they received (No Action Indicated [NAI], Voluntary Action Indicated [VAI], or Official Action Indicated [OAI]). Data on domestic inspections for clinical investigators, IRBs, sponsors/monitors/CROs, GLP labs, and BA/BE studies from these yearly reports was compiled for FY 2007–2017 in order to compare inspection trends over time.

Statistical analysis was done to determine if there was a difference among the FDA centers in the frequency of specific VTs issued. Statistical analysis was performed using Rstudio software version 1.1.463 (Rstudio, Inc.) to calculate significance using Fisher’s exact test at a significance level of α/m, where α = 0.05 and m was the number of hypotheses (VTs) tested in a set of warning letters. This significance level is based on the Bonferroni correction for multiple comparisons [18].

Results

A combined total of 300 warning letters were analyzed as a part of this study (Table 1). The number of warning letters issued each FY has varied greatly, with more recent FYs seeing much fewer warning letters than 10 years ago. For comparison, 48 total warning letters were issued in FY 2007, while only 1 warning letter was issued in FY 2018. Clinical investigators received the most warning letters during this time period, while GLP laboratories received the least amount of warning letters. CDER issued 149 warning letters, CDRH issued 124 warning letters, CBER issued 26 warning letters, and CVM issued only 1 warning letter.

Table 1.

Warning Letters Issued Between FY 2007–2018 by Category.

	FY07	FY08	FY09	FY10	FY11	FY12	FY13	FY14	FY15	FY16	FY17	FY18	Total
CI	26	28	25	23	13	9	9	13	6	6	3	0	161
IRB	6	8	7	14	8	8	5	4	1	4	0	0	65
S + SI	13	10	2	7	6	5	1	7	4	3	4	0	62
GLP	3	1	4	1	1	0	0	1	0	0	0	1	12
Total	48	47	38	45	28	22	15	25	11	13	7	1	300

CI, clinical investigator, S + SI sponsor, and sponsor-investigator combined

IRB Warning Letters

A total of 65 IRB warning letters were analyzed in this study (Table 2). Of these warning letters, 25 (39%) were issued by CDER, 32 (49%) were issued by CDRH, and 8 (12%) were issued by CBER. The most common VT cited involved written procedures (77% of all warning letters), while the second most common VT involved documentation of meeting minutes (75%). Other common violations involved meeting attendance (51%) and membership issues within the IRB (42%). Using Fisher’s exact test for significance at a significance level of $\frac{0.05}{12}=0.0042$, there were no statistically significant differences in violations between warning letters issued by different FDA centers for any VT.

Table 2.

Frequency of VTs in IRB Warning Letters.

VT #	Violation Theme	CDER n = 25	CDRH n = 32	CBER n = 8	Total n = 65	P Value
1	Documentation of meeting minutes	17 (68%)	25 (78%)	7 (88%)	49 (75%)	0.5617
2	Timely review of research	5 (20%)	12 (38%)	1 (13%)	18 (28%)	0.2663
3	Not following procedures/no written procedures	18 (72%)	25 (78%)	7 (88%)	50 (77%)	0.7722
4	Informed consent	6 (24%)	8 (25%)	1 (13%)	15 (23%)	0.8441
5	Less than majority in meeting/no nonscientific member present	11 (44%)	18 (56%)	4 (50%)	33 (51%)	0.6521
6	Membership issues/no accurate roster	9 (36%)	14 (44%)	4 (50%)	27 (42%)	0.7770
7	Research documentation	6 (24%)	7 (22%)	2 (25%)	15 (23%)	1.0000
8	Expedited review	5 (20%)	6 (19%)	1 (13%)	12 (18%)	1.0000
9	Safety/risk	5 (20%)	1 (3%)	0	6 (9%)	0.0961
10	Conflict of interest	5 (20%)	6 (19%)	2 (25%)	13 (20%)	0.9129
11	Children/vulnerable populations	6 (24%)	3 (9%)	2 (25%)	11 (17%)	0.2127
12	Other	8 (32%)	4 (13%)	2 (25%)	14 (22%)	0.1861

Sponsor Warning Letters

A total of 62 sponsor and sponsor-investigator warning letters were analyzed, with 46 of these letters categorized as “Sponsor” and 16 of them categorized as “Sponsor-Investigator” (Table 3). Even though the BIMO program lumps sponsor, monitor, and CRO inspections into a single category, all warning letters under this category were addressed to sponsors. CDER issued 20 (32%) of these letters, CDRH issued 39 (63%), and CBER issued 3 (5%). The most common VT cited involved ensuring proper monitoring of the investigation (56%), with the next most common issues involving record keeping (50%), lack of investigational application review or IRB review (50%), and problems with investigators of the trials (48%).

Table 3.

Frequency of VTs in Sponsor Warning Letters.

VT #	Violation Theme	CDER n = 20	CDRH n = 39	CBER n = 3	Total n = 62	P value
1	Monitoring of investigation	12 (60%)	21 (54%)	2 (67%)	35 (56%)	0.9081
2	Record keeping	6 (30%)	25 (64%)	0	31 (50%)	0.0079
3	Submission of regular reports	0	11 (28%)	1 (33%)	12 (19%)	0.0145
4	Investigator issues	6 (30%)	22 (56%)	2 (67%)	30 (48%)	0.1093
5	FDA/IRB approval	9 (45%)	19 (49%)	3 (100%)	31 (50%)	0.3165
6	Informed consent/patient rights	3 (15%)	13 (33%)	0	16 (26%)	0.2606
7	Labeling/advertisement	0	7 (18%)	0	7 (11%)	0.1351
8	Adverse event reporting/evaluation	0	7 (18%)	1 (33%)	8 (13%)	0.0623
9	Provide FDA information/inspection on request	2 (10%)	1 (3%)	1 (33%)	4 (6%)	0.0640
10	Financial disclosure/conflict of interest	2 (10%)	5 (13%)	1 (33%)	8 (13%)	0.5047
11	Other	2 (10%)	0	1 (33%)	3 (5%)	0.0198

Based on Fisher’s exact test for significance at a significance level of $\frac{0.05}{11}=0.0045$, there were no statistically significant differences in violations between warning letters issued by different FDA centers for any VT. Medical device sponsors were the only ones to receive labeling and advertising violations. Sponsors of drug products received no violations relating to regular progress reports and adverse event reporting.

GLP Warning Letters

Warning letters issued to GLP labs are the least common type of BIMO warning letter. A total of 12 warning letters were analyzed, with 5 (42%) issued by CDER, 1 (8%) issued by CDRH, 5 (42%) issued by CBER, and 1 (8%) issued by CVM (Table 4). These letters on average cite more violations than any other warning letter type. Every GLP warning letter except for one specifically mentioned deficiencies in the quality assurance unit. Other frequent violations included failure to follow and maintain SOPs (75%) and failure to record or store data properly (75%). Of the 10 VTs devised for GLP warning letters, 6 of them were present in at least half of the warning letters.

Table 4.

Frequency of VTs in GLP Warning Letters.

VT #	Violation Theme	CDER n = 5	CDRH n = 1	CBER n = 5	CVM n = 1	Total n = 12
1	QA unit responsibilities	5 (100%)	1	4 (80%)	1	11 (92%)
2	Study protocols	2 (40%)	0	4 (80%)	1	7 (58%)
3	Inadequate reports	4 (80%)	0	3 (60%)	0	7 (58%)
4	SOPs	3 (60%)	1	4 (80%)	1	9 (75%)
5	Characterization of test article	3 (60%)	0	2 (40%)	0	5 (42%)
6	Calibration of equipment	3 (60%)	0	3 (60%)	0	6 (50%)
7	Recording/storage of data	5 (100%)	1	2 (40%)	1	9 (75%)
8	Personnel	1 (20%)	1	2 (40%)	0	4 (33%)
9	Handling unforeseen circumstances	0	0	2 (40%)	1	3 (25%)
10	Other	0	1	2 (40%)	1	4 (33%)

Clinical Investigator Warning Letters

Clinical investigator warning letters are the most common type of BIMO warning letters. A total of 161 warning letters were issued to clinical investigators during this time period, with 99 issued by CDER, 52 issued by CDRH, and 10 issued by CBER.

Only nine new warning letters have been issued to clinical investigators since Garmendia et al.’s analysis of clinical investigator warning letters, all of which were issued by CDER. In these warning letters, only three different VTs were cited: deviation from investigational plan (9 times), failure to maintain source documentation (4 times), and violations related to the investigational product (2 times). The data from the warning letters issued from FY16 to FY18 was combined with the data from Garmendia et al. to create a combined dataset from FY07 to FY18 (Table 5). The addition of 9 new warning letters did not have a significant effect on the overall dataset. The most common violations for clinical investigators were deviation from investigational plan (94%), failure to maintain source documentation (66%), and violations related to informed consent (46%).

Table 5.

Frequency of VTs in Clinical Investigator Warning Letters.

VT#	Violation Theme	CDER n = 98	CDRH n = 56	CBER n = 10	Total n = 164	P value
1	Deviation from investigational plan	93 (95%)	52 (93%)	9 (90%)	154 (94%)	0.5493
2	Failure to maintain adequate/accurate source documentation	61 (62%)	39 (70%)	8 (80%)	108 (66%)	0.4161
3	Informed consent	36 (37%)	33 (59%)	6 (60%)	75 (46%)	0.0184
4	Violations related to investigational product	32 (33%)	13 (23%)	4 (40%)	49 (30%)	0.3497
5	Failure to personally supervise the study	22 (22%)	4 (7%)	0	26 (16%)	0.0191
6	Failure to protect subject safety/report adverse events to IRBs/communicate with the IRB	31 (32%)	19 (34%)	3 (30%)	53 (32%)	0.9616
7	Submission of false information to the FDA and sponsor	1 (1%)	0	0	1 (1%)	1.0000
8	Failure to communicate with the sponsor	3 (3%)	1 (2%)	0	4 (2%)	1.0000
9	Financial disclosure	0	1 (2%)	0	1 (1%)	0.4024

Garmendia et al.’s analysis used Fisher’s exact test to determine significance at a significance level of α = 0.05 and found a significant difference in frequency for VT 5 (failure to personally supervise the study) among the different FDA centers. After applying the Bonferroni correction for multiple comparisons and adjusting the significance level to $\frac{0.05}{9}=0.0056$, none of the differences met the level of statistical significance.

(NOTE: In reviewing the FDA warning letter database, only 151 clinical investigator warning letters were found. One letter that was not available in the FDA database was found on a third-party website [19]. This does not match the 155 warning letters that were analyzed by Garmendia et al. [15]).

Inspection Results

Overall, inspection results have improved since FY 2007 (Table 6). The percentage of NAIs went from 51% in FY 2007 to 71% in FY 2017, showing a gradual increase since FY 2010. The percentages of VAIs and OAIs have also been lower in recent years. There are more OAIs issued than warning letters, showing that not every OAI leads to a warning letter.

Table 6.

Total Domestic BIMO Program Inspections with Classification.

Classification	FY07	FY08	FY09	FY10	FY11	FY12	FY13	FY14	FY15	FY16	FY17	Total
NAI	460 (51%)	557 (49%)	627 (47%)	621 (45%)	577 (50%)	520 (53%)	606 (55%)	746 (56%)	876 (63%)	943 (67%)	937 (71%)	7470 (56%)
VAI	367 (41%)	476 (42%)	630 (47%)	663 (48%)	504 (44%)	423 (43%)	470 (42%)	499 (38%)	444 (32%)	423 (30%)	347 (26%)	5246 (39%)
OAI	70 (8%)	98 (9%)	81 (6%)	107 (7%)	63 (6%)	35 (4%)	33 (3%)	81 (6%)	68 (5%)	49 (3%)	34 (3%)	719 (5%)
Total	897	1131	1338	1391	1144	978	1109	1326	1388	1415	1318	13,435
Warning Letters	48	47	38	45	28	22	15	25	11	13	7	299

Inspection results broken down by category can be found in Tables 7, 8, 9, 10, and 11. Clinical investigators received the most inspections, followed by bioequivalence studies, IRBs, sponsors/monitors/CROs, and GLP labs. Sponsors/monitors/CROs typically receive the highest percentage of OAIs out of all of the categories followed by GLP labs. Although bioequivalence studies received 103 OAIs during this time period, no warning letters were issued in this category. Some years will have more warning letters issued than OAIs issued in a particular category, which is due to warning letters being issued based on inspections from previous years.

Table 7.

IRB Inspections.

Classification	FY07	FY08	FY09	FY10	FY11	FY12	FY13	FY14	FY15	FY16	FY17	Total
NAI	51% (93)	46% (87)	43% (77)	44% (114)	52% (93)	47% (73)	45% (77)	50% (76)	59% (81)	66% (82)	77% (95)	948 (51%)
VAI	47% (86)	47% (88)	50% (89)	50% (129)	44% (79)	46% (72)	49% (84)	45% (68)	37% (51)	27% (33)	21% (26)	805 (44%)
OAI	2% (4)	7% (13)	7% (13)	6% (15)	4% (7)	7% (11)	6% (10)	5% (8)	4% (6)	6% (7)	2% (3)	97 (5%)
Total	183	188	179	258	179	156	171	152	138	124	124	1850
Warning Letters	6	8	7	14	8	8	5	4	1	4	0	65

Table 8.

Sponsor/Monitor/CRO Inspections.

Classification	FY07	FY08	FY09	FY10	FY11	FY12	FY13	FY14	FY15	FY16	FY17	Total
NAI	51% (36)	61% (55)	64% (74)	50% (62)	55% (70)	64% (51)	54% (53)	57% (79)	61% (72)	65% (73)	64% (67)	692 (59%)
VAI	24% (16)	20% (18)	28% (33)	38% (47)	36% (46)	31% (25)	44% (43)	35% (48)	31% (36)	26% (29)	30% (31)	372 (32%)
OAI	22% (15)	19% (17)	8% (9)	12% (15)	9% (11)	5% (4)	2% (2)	8% (11)	8% (9)	9% (10)	6% (6)	109 (9%)
Total	67	90	116	124	127	80	98	138	117	112	104	1173
Warning Letters	13	10	2	7	6	5	1	7	4	3	4	62

Table 9.

GLP Inspections.

Classification	FY07	FY08	FY09	FY10	FY11	FY12	FY13	FY14	FY15	FY16	FY17	Total
NAI	42% (23)	51% (22)	49% (24)	43% (35)	46% (19)	30% (12)	63% (29)	40% (17)	36% (13)	43% (19)	65% (22)	235 (46%)
VAI	51% (28)	44% (19)	39% (19)	53% (43)	44% (18)	68% (27)	35% (16)	53% (23)	56% (20)	50% (22)	35% (12)	247 (48%)
OAI	7% (4)	4% (2)	12% (6)	4% (3)	10% (4)	2% (1)	2% (1)	7% (3)	8% (3)	7% (3)	0%	30 (6%)
Total	55	43	49	81	41	40	46	43	36	44	34	512
Warning Letters	3	1	4	1	1	0	0	1	0	0	0	11

Table 10.

Clinical Investigator Inspections.

Classification	FY07	FY08	FY09	FY10	FY11	FY12	FY13	FY14	FY15	FY16	FY17	Total
NAI	52% (308)	50% (353)	48% (416)	46% (340)	53% (324)	56% (320)	56% (372)	58% (466)	64% (526)	66% (512)	73% (512)	4449 (57%)
VAI	40% (237)	41% (289)	46% (399)	45% (333)	41% (251)	41% (235)	42% (279)	37% (297)	33% (271)	32% (248)	26% (182)	3021 (38%)
OAI	8% (47)	9% (63)	6% (52)	9% (66)	6% (36)	3% (17)	2% (13)	5% (40)	3% (25)	2% (15)	1% (7)	381 (5%)
Total	592	705	867	739	611	572	664	803	822	775	701	7851
Warning Letters	26	28	25	23	13	9	9	13	6	6	3	161

Table 11.

Bioavailability/Bioequivalence Study Inspections.

Classification	FY08	FY09	FY10	FY11	FY12	FY13	FY14	FY15	FY16	FY17	Total
NAI	38% (40)	28% (36)	37% (70)	38% (71)	49% (64)	58% (75)	57% (108)	67% (184)	71% (257)	68% (241)	1146 (56%)
VAI	59% (62)	71% (90)	59% (111)	59% (110)	49% (64)	37% (48)	33% (63)	24% (66)	25% (91)	27% (96)	800 (39%)
OAI	3% (3)	1% (1)	4% (8)	3% (5)	2% (2)	5% (7)	10% (19)	9% (25)	4% (14)	5% (18)	103 (5%)
Total	105	127	189	186	130	130	190	275	362	355	2049

Closeout Letter Results

Out of 169 BIMO warning letters issued after September 1, 2009, only 25 (15%) letters received closeout letters as of December 2018 (Table 12). This indicates that only 25 letters have been publicly confirmed to be completely resolved. Using Fisher’s exact test, there was no statistically significant difference between the percentage of closeout letters issued for warning letters of each type. The average time to closeout was 518 days, ranging from 26 to 1729 days.

Table 12.

Closeout Letters and Warning Letters.

	Clinical Investigator	IRB	Sponsor	Sponsor- Investigator	GLP	Total
Closeout Letters	11 (13%)	5 (11%)	6 (22%)	1 (10%)	2 (50%)	25 (15%)
Warning Letters (After September 1, 2009)	84	44	27	10	4	169

Discussion

Warning Letters issued by the FDA’s BIMO Program provide insight into data integrity issues and other research misconduct in the premarket side of the pharmaceutical industry. This study presents data on the major compliance issues found during inspections of clinical investigators, IRBs, sponsors of clinical studies, and GLP laboratories. This data can provide an understanding of the impact the BIMO program has on ensuring integrity in clinical trials and can help those conducting clinical studies understand what the FDA looks for in inspections based on empirical evidence.

Violations cited in warning letters typically come straight from the BIMO program CPGMs [11]. The CPGMs go into detail on how the inspection process works and all of the inspection criteria. In preparation for inspections, a checklist could be prepared using the CPGM for guidance. Such a process would help the establishment to remain in compliance.

Looking at all of the warning letter datasets, documentation was a common issue. This resulted in the second most cited category of violations and included IRB meeting minute documentation. Failure to follow and maintain procedures was another common violation for clinical investigators, IRBs, and GLP labs; it is the most common violation for both IRBs and clinical investigators and second most common for GLP labs. This violation was fairly common for sponsors in the context of failure to maintain and follow procedures for monitoring clinical studies and investigators.

While the number of warning letters issued in the past few years was much lower than the number issued in the latter half of the 2000s, there were still a number of OAIs issued. This decrease in warning letters could be due to a number of reasons. It is possible that compliance has overall improved where fewer warning letters were needed, though this does not seem likely because the number of OAIs did not decrease nearly as much as the number of warning letters. It is also possible that recipients were better at responding to 483s issued to them and returning to a state of compliance before a warning letter was issued. Perhaps the BIMO program changed their practices for when to issue warning letters. Another possibility is that the recent restructuring by the FDA’s Office of Regulatory Affairs (ORA) and the BIMO program has caused this change. Since the goal of this restructuring is to create more time and resources for providing warning letters sooner, there may be an improvement in this moving forward [20, 21]. Regardless, without insider information from the FDA, it would be hard to determine the exact cause of the decrease in warning letters.

Fisher’s exact test was used to determine if there were statistically significant differences between each FDA center in the frequency that specific VTs were issued. However, when applying Bonferroni’s correction for multiple comparisons, it was found that there were no statistically significant differences among the centers in each data set. It is likely that a larger sample size of warning letters would be required in order to achieve the statistical power necessary to determine if there are actually any significant differences in these regards.

Conclusion

The findings of this study suggest that there are significant data integrity and other regulatory compliance issues in all aspects of the premarket side of the pharmaceutical industry. Inspections conducted by the BIMO program show that many clinical investigators, IRBs, sponsors, and GLP laboratories have issues with documentation and following defined procedures. However, inspection results have seen overall improvement during the past 12 years, potentially indicating improvement in compliance. While this study assessed some of the activities of the BIMO program, more research is required to understand the overall impact of this program for improving compliance in the industry.