Quality of Reporting of Serious Adverse Drug Events to an Institutional Review Board

Abstract

Purpose

Serious adverse drug event (sADE) reporting to Institutional Review Boards (IRB) is essential to ensure pharmaceutical safety. However, the quality of these reports has not been studied. Safety reports are especially important for cancer drugs that receive accelerated Food and Drug Administration approval, like imatinib, as preapproval experience with these drugs is limited. We evaluated the quality, accuracy, and completeness of sADE reports submitted to an IRB.

Experimental Design

sADE reports submitted to an IRB from 14 clinical trials with imatinib were reviewed. Structured case report forms, containing detailed clinical data fields and a validated causality assessment instrument, were developed. Two forms were generated for each ADE, the first populated with data abstracted from the IRB reports, and the second populated with data from the corresponding clinical record. Completeness and causality assessments were evaluated for each of the two sources, and then compared. Accuracy (concordance between sources) was also assessed.

Results

Of 115 sADEs reported for 177 cancer patients to the IRB, overall completeness of adverse event descriptions was 2.4-fold greater for structured case report forms populated with information from the clinical record versus the corresponding forms from IRB reports (95.0% versus 40.3%, P < 0.05). Information supporting causality assessments was recorded 3.5-fold more often in primary data sources versus IRB adverse event descriptions (93% versus 26%, P < 0.05). Some key clinical information was discrepant between the two sources.

Conclusions

The use of structured syndrome-specific case report forms could enhance the quality of reporting to IRBs, thereby improving the safety of pharmaceuticals administered to cancer patients.

Serious adverse drug events (sADE) are a significant cause of morbidity and mortality. When associated with chemotherapeutic agents, the detection and management of adverse events are particularly problematic, as these agents are designed to be toxic. For some cancer drugs, the Food and Drug Administration (FDA) has granted accelerated approval for marketing based on preliminary findings from phase II or short phase III studies; many of these drugs’ serious toxicities are identified several years after FDA approval (1). The main sources of safety signals for these drugs are reports of ADEs that occur during clinical trials. These case descriptions are closely reviewed by Institutional Review Boards (IRB), the FDA, and the drug manufacturer. It is possible that delays in identifying safety signals for accelerated approved cancer drugs result from incomplete reporting of these sADEs.

Two recent studies validate this concern. Adverse event descriptions associated with 15 drugs and reported from the clinical practice setting as structured event descriptions were 2-fold to 10-fold more completely described than the descriptions of the same events that are included in FDA safety databases (2). In a pilot study comparison of adverse event descriptions derived from unstructured versus structured case report forms administered to patients enrolled in one phase III clinical trial, 11 versus 238 ADEs were identified, respectively, and 14% versus 73% of trial participants, respectively, reported at least one ADE (3).

Translational Relevance.

The serious toxicities of rapidly approved cancer drugs are often identified several years after Food and Drug Administration approval. A critical component of drug safety monitoring is executed by Institutional Review Boards (IRB). We investigated the quality of reports submitted to an IRB during clinical trials of a rapidly approved agent, imatinib mesylate, and compared the IRB records with the information available in the corresponding clinical charts. We found that although submitted reports contained 95% of the information required by the IRB, causality data was frequently missing and available adverse event information was 2.4 to 3.5 times higher in the clinical record. We showed that with a specific, structured case report form, critical clinical data could be extracted more completely from patient records. Increasing the quality of reporting through the methods we have highlighted in this work could help ensure that the most timely and accurate safety information is available to patients and clinicians.

We reviewed a large number of ADE reports from the clinical trial setting for imatinib, a breakthrough anticancer agent that received accelerated FDA approval in 2001 for the treatment of chronic myelogenous leukemia (4). The time from the initial submission to the FDA of the New Drug Application to marketing approval was 73 days, by far the fastest development time of any new pharmaceutical to date (4). The benefit of the drug continues to be impressive: a 5-year follow-up study showed survival rates of 90% in chronic myelogenous leukemia with a very favorable safety profile (5). The associated ADEs are manageable: at the time of FDA approval, most imatinib-related ADEs identified during clinical trials were categorized as mild (grade 1) or moderate (grade 2) grade based on the Common Terminology Criteria for Adverse Events (CTCAE version 3) descriptive terminology (5, 6). In 2006, the FDA reported two possible imatinib-associated sADEs—peripheral/pulmonary edema and congestive heart failure (7). Moreover, two other serious clinical events, infectious complications and bone fractures, represent serious clinical events that have occurred among clinical trial patients who received imatinib, although there is no consensus that these events were caused by the drug (8-13).

The delay between approval and identification of sADEs associated with accelerated approved cancer drugs may be, in part, due to incomplete or inaccurate reporting to or processing of signals by safety boards, such as IRBs. To understand the process with imatinib, we investigated the completeness and accuracy of adverse drug reaction reports sent to the IRB. For comparison, data concerning the same events were independently abstracted from the available clinical record. To our knowledge, this is the first study to evaluate the quality and completeness of adverse event reports contained in IRB files.

Materials and Methods

Population, setting, and data

Relevant clinical trials were identified by reviewing a cancer trials research office electronic database that monitors trials for compounds under investigation as well as associated ADE reports submitted to both the IRB and the sponsor. For this study, an IRB that monitored the execution of 14 clinical trials with imatinib mesylate was chosen. The trials included one phase I, four phase I/II, seven phase II, and two phase III clinical trials assessing safety, efficacy, and various antileukemic effects of imatinib alone or in combination with another agent. The IRB was located at a National Cancer Institute (NCI)-designated comprehensive cancer center that was one of three sites where FDA licensing trials for imatinib were conducted. The study period was from 2001 to 2004. During the study period, any serious or unanticipated adverse events were required to be reported by each of the trials to the IRB. The Office for Human Research Protections had not yet revised this guidance to report only both serious and unanticipated adverse events to the local IRB.

All ADE reports submitted to the IRB from the 14 clinical trials were reviewed. The research charts and electronic patient medical records corresponding to each adverse event were also obtained and reviewed.

Development of structured case report form

A syndrome-specific modular adverse event case report form was developed for this study using defined data elements for each of four clinical events; infectious processes, fracture, congestive heart failure/pulmonary effusion, and peripheral and/or pulmonary edema. The clinical data elements selected were those required to complete an adverse event form using the CTCAE version 3.0 system. The CTCAE system is a standardization tool that defines adverse events and categorizes their severity through detailed description of symptoms (14). These data elements were grouped into categories representing signs and symptoms, laboratory tests, treatment given, and outcome. For example, for infection, the following data elements were recorded: history of fever, chills, or night sweats (signs and symptoms); complete blood counts, cultures (laboratory tests); and intravenous antibiotics, hospitalization, or other treatment (treatment). Additionally, the developed form included a causality assessment using the Naranjo instrument, a previously validated causality assessment tool (15). These forms were used in the abstraction of data from the IRB reports and the clinical record (see Supplementary Material).

Data collection and population of structured case report form

For each adverse event reported, a matched pair of structured case report forms were produced—the first was populated using data from the submitted IRB report and the second was populated by data from the corresponding clinical record. To accomplish this, the research nurse abstracted clinical information from each imatinib-related ADE report submitted to the IRB onto a structured case report form (“IRB Forms”). Continuation reports and additional supporting documentation submitted to the IRB (e.g., sponsor report forms, lab tests, or correspondence) were included as part of the IRB forms. Data were then abstracted from each matched research chart and electronic patient medical record (“primary source data”) onto a second, identical, structured case report form. Primary source data were reviewed for the time frame beginning with the diagnosis and ending with treatment of the suspected ADE.

Incidence, prevalence, and classification of reported sADEs

The submitted IRB sADE forms were qualified as possibly of the four types if the provided description met the CTCAE criteria for that type; some of these were missing complete information, and were classified as definite sADEs of the types listed if complete descriptions could be found in the source. Definite sADEs were used to calculate incidence (new sADEs per person year) and prevalence (sADEs per patient).

Completeness and accuracy (concordance)

Detailed definitions of “completeness” and “accuracy” were outlined prior to data collection and analysis (Table 1). The completeness for each report submitted to the IRB was first evaluated with respect to the clinical information required by the IRB (Table 1). Mean completeness scores were compared between adverse event reports derived from the IRB versus the original source data using one-way t test for dependent groups (Table 1). Accuracy, measured as concordance between the two sources (Table 1), for sADE’s descriptions in report forms was compared via McNemar’s test for correlated proportions.

Table 1.

Definitions of completeness and accuracy

Concept	Completeness	Accuracy and concordance*	Data elements
Event description and severity	Could one define the CTCAE category and grade?	Did the IRB description match the complete description? Did the IRB grade match the complete grade?	Signs/symptoms, laboratory tests, treatment, other required by CTCAE
Causality	Could one make an independent assessment of causality?	Did the investigator-reported causality match the Naranjo causality? Did the investigator-reported causality match the complete causality?	Investigator-reported relatedness and Naranjo questions (temporal relation to drug, alternative causes, recurrence of symptoms with drug)†
Dates of the event	Were there clear dates corresponding to AE start, stop, and reporting?	Did the AE start dates match (within 1 mo)?	AE Date Start, Stop, Report to IRB, Report to Sponsor
Action taken	Was it clear that action was or was not taken?	N/A	Consent form change, protocol change, drug discontinued
Summary	Contains key elements of description, severity, causality, and action taken	Were both sources concordant on key variables?

^*To be accurate, the variables must also be complete unless neither source had the information.

^†Recurrence was not required for the brief version of Naranjo.

Causality

Data for all IRB reports were abstracted onto a previously validated causality assessment instrument that was included in the structured report (15). This was repeated for the corresponding primary source data. Weighted kappa scores were used to assess agreement between investigator “relatedness” assessments of causality and assessments based on the Naranjo instrument. For kappa scoring of causality assessment, total agreement was defined as an exact match between the standard for each of four possible causality relatedness categories (no relation, possible, probable, and definitely related) and partial agreement was defined as any pair of answers that agreed that the event was at least possibly related to the drug.

Results

Final data set

In total, reports of 122 imatinib-related sADEs submitted to an IRB were reviewed. These reports involved 84 of the 177 cancer patients enrolled in the trials included in our study. Mean patient follow-up in these trials was 2 years. Three sADE reports were duplicate descriptions of the same clinical event and four sADE reports did not ade quately report a description and were excluded. A total of 115 unique sADE reports were included in our final study data set. Most adverse event descriptions included additional clinical details: 65% were also described in MedWatch adverse event forms (safety reports submitted to the FDA) or adverse event report forms submitted directly to the sponsor, 23% were accompanied by medical records documentation, and 23% of the IRB adverse event reports included no additional supporting documentation.

Incidence and prevalence of the four syndromes: IRB reports versus primary source data

After review of the 115 unique sADE reports submitted to the IRB, 58 (50.4%) of the reports were suspected sADEs and fully reviewed and 21 (18.4%) were classified as definite sADEs associated with four clinical toxicities (edema, heart failure, fracture, and infection; Table 2). Thus, combined suspected incidence was 16.4% and combined definite incidence was 5.9%. Incidence and prevalence rates for the four sADEs ranged from 0.3% (fracture) to 3.7% (infection; Table 2). Almost 12% of patients in the clinical trials developed at least one suspected sADE associated with one or more of the four prespecified toxicities.

Table 2.

Incidence and prevalence of ADEs from IRB reports and original source review

	All events
	N	Prevalence (per patient)
Total patients at risk	177	177 (100%)
Local ADEs reported to IRB	115	65.0%
Suspected sADE (edema, infection, fracture, pulmonary edema) from IRB reports	58	32.8%
Definite sADE from IRB reports	21	11.9%
Edema	4	2.3%
Fracture	1	0.6%
Pulmonary edema	3	1.7%
Infection	13	7.3%

Completeness: submitted IRB reports versus IRB requirements

Clinical information required by the IRB to assess the seriousness of an ADE was recorded for 95% of the suspected sADEs. The most commonly omitted data element was change in the consent form by the investigator as a result of IRB review of the sADE report.

Comparative completeness: IRB reports versus primary source data

We compared the completeness of ADE report descriptions derived from IRB reports with those independently obtained from primary source data (medical records and clinical trial case report forms; Table 3). Because some sADEs covered the same patient chart and topic, 84 charts were partially reviewed for the 115 sADEs and 58 charts were completely reviewed as being a possible sADE of the four types. On average, IRB forms were 40.3% complete versus 95.0% complete for ADE descriptions abstracted from primary source data onto syndrome-specific case report forms.

Table 3.

Completeness and accuracy of IRB ADE reports vs. original source

Completeness	IRB, N (%)	Original sources, N (%)	Difference (IRB-Available)	Accuracy (concordance) IRB vs. original source, N (%)*
Total	28/115 (24%)	77/84 (92%)	-68%†	11/58 (19%)
of event description	70 (61%)	83 (99%)	-38%†	49 (84%)†
of full causality‡	20 (17%)	66 (79%)	-62%†	17 (29%)†
basic causality	33 (28%)	78 (93%)	-65%†	24 (41%)†
investigator relatedness	109 (96%)‡	N/A	N/A	36 (62%)†
of dates	37 (32%)	77 (93%)	-61%†	3 (5%)‡
Average	40.3%	95%	-55%†	62.3%

^*Accuracy was defined as concordance between the two corresponding structured case report forms for each event.

^†McNemar’s P < 0.001 or one-sided t test P < 0.001.

^‡Not included in average or total.

For an individual ADE, summary reports from 24% of the IRB reports versus 92% of reports from primary source data contained a minimum of information required to fully characterize clinical events associated with one of the four prespecified clinical toxicities. Description of history-related findings that facilitated assessment of the severity of a suspected adverse event was less complete for adverse event descriptions derived from IRB reports versus those derived from primary source data (61% versus 99%, P < 0.05). Examples of clinical items not recorded in IRB adverse event reports included information on the treatment of edema and infections, formal diagnosis of the cause and site of infection, and appropriate diagnostic tests to evaluate the severity of new onset of edema. Information related to causality and dates of occurrence of the sADE had the greatest difference (-65% and -61% difference) for adverse event reports derived from IRB forms versus primary source data. Information required to assess causality (using the Naranjo instrument) was available for 28% of summary reports derived from IRB source data.

Accuracy (concordance): IRB reports versus primary source data

Only 19% of descriptions of 58 ADEs thought to be both related to imatinib and of the four syndromes were completely concordant between the IRB reports and the original source (medical chart) data. Examples of lack of concordance include differences in event description between the progress notes in the chart and the brief description in the IRB form (84% concordant) and relatedness of the event to imatinib (62% concordant, although this was not used in the total).

Incidence and prevalence: IRB reports versus primary source data

Overall, detailed information was available for 43 suspected sADEs; 16.9% of the reports were suspected sADEs and 5.9% were definite sADEs associated with four clinical toxicities (edema, heart failure, fracture, and infection; Table 2). Incidence and prevalence rates for the four sADEs ranged from 0.3% (fracture) to 3.7% (infection; Table 2). Incidence estimates of the individual sADEs were 2% based on information derived from IRB report versus 5% based on information derived from primary source data. Almost 12% of patients in the clinical trials developed at least one suspected sADE associated with the four toxicities. Medical record reviews identified 22 potential sADEs that had not been reported to the IRB. Inclusion of these suspected ADEs increased the incidence and prevalence estimates of the studied sADEs. For example, inclusion of reports derived from review of source data increased the estimated incidence of grade 3 or grade 4 edema 5-fold (from 1.1% to 5.9%); the increase was primarily peripheral edema. Primary data sources for clinical information for 58 adverse event reports categorized as “suspect” sADEs; 21 events were classified as definite sADEs (both based on the Naranjo causality assessment instrument). Review of primary data sources led to the identification of 23 additional suspected sADEs associated with the four clinical syndromes.

Assessment of causality comparison: IRB reports versus primary source data

Investigator-reported assessment of causality was compared with causality as assessed by information the abstractor obtained from the primary data source using the validated Naranjo instrument to guide the comparison (Table 4). Six forms were not included because they were missing causality estimates. For the remaining 109 forms, total agreement on these causality assessments was noted for 54 ADEs (50%) and partial agreement for 63 ADEs (58%). Overall weighted kappa was 0.12 (95% confidence interval, -0.04 to 0.28, P > 0.05). Thus, investigator assessment of causality was no better than chance compared with a more formal assessment of causality.

Table 4.

Causality from report with investigator’s causality (IRB-reported information only; N = 109)

Investigator relatedness	Naranjo relatedness
	Doubtful	Possible	Probable/definite
Doubtful	27	24	3
Possible	21	26	2
Probable/definite	0	5	1
Total agreement*	27/48 (56%)	26/55 (48%)	1/6 (17%)
Partial agreement	27/48 (56%)	33/55 (57%)	3/6 (50%)

^*Kappa = 0.12 (-0.04 to 0.28); P > 0.05.

Time cost

Mean estimated time to abstract adverse event information was 18.0 minutes to abstract clinical information onto syndrome-specific case report forms (SD of 7.3 minutes) for each of the 122 sADEs that had been reported to the IRB. Total time cost for abstracting additional clinical information for 23 events identified during the independent data collection process was 478 minutes.

Discussion

Compared with adverse event reporting guided by structured case report forms, the reports of imatinib-associated ADEs that were actually submitted to the IRB at this NCI-designated comprehensive cancer center were less complete as well as less accurate with respect to causality assessment information. These differences were unlikely to result from poor training of clinical trial investigators, poor quality of primary source data, or deficiencies in follow-up actions taken by IRB personnel because the forms were filled out per the basic regulations, but rather reflect the limitations of using unstructured adverse event case report forms. In interpreting our findings, several factors should be considered.

First, peripheral edema and pulmonary edema, two imatinib-associated sADEs reviewed in this study, were first identified during preapproval clinical trials and were included in safety information in the initial package insert for imatinib from 2001. Preapproval clinical trials found that two thirds of imatinib-treated patients developed edema, 5% developed grade 3 or 4 edema, and 5% developed grade 3 or 4 pulmonary dysfunction. In contrast, in our study, edema was recorded in only 2% of ADE reports submitted to the IRB and 4-fold as many unreported cases of peripheral edema were identified upon review of primary source data. Individual event descriptions for peripheral edema and pulmonary edema were incompletely described. Without additional chart review, the ability of the IRB to accurately assess the magnitude of the risk in these patients was limited, largely due to the lack of structured reporting information available on the IRB forms. Awareness of high rates of peripheral edema and severe dyspnea ultimately led clinicians in 2006 to suspect imatinib as a cause of unexplained congestive heart failure when evaluating 10 individuals who developed severe congestive heart failure after receiving an average of 7 months of the drug (8). All of these individuals had had normal left ventricular ejection fractions (usually >60%) prior to initiating imatinib; following imatinib treatment, the mean ejection fraction decreased to 25%. Basic science studies also support a causal relationship.

Second, incomplete clinical descriptions and data required for causality assessment were noted in IRB adverse event reports for the four selected clinical syndromes, whereas this clinical information was almost uniformly recorded in the primary source data. The incidence and prevalence of these sADEs based on primary data source review were 2-fold to 3-fold greater than estimates derived from the IRB database. Conversely, despite a 5% rate of sADEs associated with infections in the 14 clinical trials, only specific infections, such as varicella-zoster virus, have been associated with imatinib (16). Details about clinical infections are required for accurate assessment of potential sADEs. As well, investigators reviewing the 84 patient charts found an additional 22 sADEs of the types in question that were not reported to the IRB upon review of the 84 charts in question. This indicates that the incidence rates found in the study were likely much lower than incidence rates that would be found if the primary source data were used exclusively to look for suspected sADEs.

Third, adverse event reporting could be improved if pharmacovigilance efforts are guided by epidemiologic and pharmacologic considerations (17). Targeted evaluations could prospectively identify safety signals based on class effect or drug-drug interaction considerations, and safety assessments could be guided by structured syndrome-specific case report forms—assuring that information submitted to IRBs are complete and appropriate. A targeted approach to safety evaluation led investigators at another NCI-designated comprehensive cancer center to prospectively evaluate cardiovascular function for 75 patients with imatinib-resistant gastrointestinal stromal tumors who had participated in phase I/II clinical trials with sunitinib, a recently approved multitargeted tyrosine kinase inhibitor (18). These investigators identified frequent cases of severe cardiovascular events occurring at a median of 34 weeks of sunitinib treatment, including cardiac events (11%), congestive heart failure (8%), and 10% or greater reduction in left ventricular ejection fraction (28%). Cardiac dysfunction generally resolved upon withholding sunitinib and instituting medical management. Basic science studies indicate that sunitinib causes mitochondrial apoptosis in mice and in cultured rat cardiomyocytes. It should be noted that in the final reports of phase III licensing trials for sunitinib as a treatment for gastrointestinal stromal tumor or renal cell cancers, rates of grade 3 reductions in left ventricular ejection fraction of 0% and 2%, respectively, were observed. However, these trials were for short treatment periods and excluded patients who had previously received imatinib.

Fourth, important considerations for pharmacovigilance efforts relate to time and expense. For the adverse event reports reviewed in this study, time required for clinical trials office personnel to complete unstructured case reports averaged 14 minutes per individual ADE reported versus 18 minutes per individual ADE for a trained research assistant to complete structured case report forms.⁶ This 4-minute increase per adverse event report must be balanced against the benefits of identifying increased numbers of sADEs and improved completeness in describing these events. Going forward, structured syndrome-specific case report forms such as those used in this study could be incorporated into modified clinical trial reporting efforts being developed by the National Cancer Institute Biomedical Informatics Grid (caBIG) project.

Some limitations of our study should be considered. The IRB process may include informal acquisition of information that would facilitate determinations of severity, causality, and risk of the medication to patient safety. This information is often described in e-mail communications or faxed documents, but not recorded on IRB case report forms. Also, this study was conducted at a single NCI-designated comprehensive cancer center, similar assessment at other sites are needed. However, this site was one of only three sites where the FDA licensing trials for imatinib were conducted and safety information for a large number of patients and clinical trials was reviewed. Moreover, IRB case report forms used at this center are similar to those used at all NCI-designated comprehensive cancer centers (19).

In conclusion, we found that the accuracy and completeness of IRB sADE reports for imatinib-associated ADEs reported to the IRB at this NCI-designated comprehensive cancer center were poor. Developing structured syndrome-specific reporting efforts may ultimately improve the safety of pharmaceuticals administered to cancer patients.