Evaluation of the Value of Attribution in the Interpretation of Adverse Event Data: A North Central Cancer Treatment Group and American College of Surgeons Oncology Group Investigation

Abstract

Purpose

In March 1998, Common Toxicity Criteria (CTC) version 2.0 introduced the collection of attribution of adverse events (AEs) to study drug. We investigate whether attribution adds value to the interpretation of AE data.

Patients and Methods

Patients in the placebo arm of two phase III trials—North Central Cancer Treatment Group Trial 97-24-51 (carboxyamino-triazole v placebo in advanced non–small-cell lung cancer) and American College of Surgeons Oncology Group Trial Z9001 (imatinib mesylate v placebo after resection of primary gastrointestinal stromal tumors)—were studied. Attribution was categorized as unrelated (not related or unlikely) and related (possible, probable, or definite).

Results

In total, 398 patients (84 from Trial 97-24-51 and 314 from Trial Z9001) and 7,736 AEs were included; 47% and 50% of the placebo-arm AEs, respectively, were reported as related. When the same AE was reported in the same patient on multiple visits, the attribution category changed at least once 36% and 31% of the time. AE type and sex (Trial Z9001) and AE type and performance status (Trial 97-24-51) were associated with a higher likelihood of AEs being deemed related.

Conclusion

Nearly 50% of AEs were reported as attributed to study drug on the placebo arm of two randomized clinical trials. These data provide strong evidence that AE attribution is difficult to determine, unreliable, and of questionable value in interpreting AE data in randomized clinical trials.

INTRODUCTION

In March 1998, Common Toxicity Criteria (CTC) version 2.0 was implemented, introducing the collection and reporting of attribution of adverse events (AEs) to study treatment. Attribution was categorized as not related, unlikely related, possibly related, probably related, or definitely related to study treatment. Despite standardization, AEs are collected only if the treating physician actively asks the patient about a particular AE, or the patient spontaneously volunteers information to state that an AE has occurred. In general, there is no consensus with regard to reporting AEs that are deemed not related or unlikely related to study treatment. Moreover, attribution categories to be included in the relatedness to study treatment are debatable: Should attributable AEs include those deemed possibly, probably, and definitely related to study treatment? Should they be limited to the probably and definitely related to study treatment categories?

Collection and reporting of attribution adds time and cost to the clinical trial process, and its use while reporting AEs in the literature has varied. In general, assignment of attribution of an AE involves multiple interactions between the treating physician, the nurse/clinical research assistant and, in some instances, the pharmacist. In a case-control study assessing the factors that were related to reporting AEs related to study drug, it was shown that patient load, volume of prescriptions, and physician attitudes played a significant role.¹ A moderate to (sometimes) poor inter-rater reliability was demonstrated when multiple physicians reviewed medical records to identify AEs attributed to medical care rather than the disease process itself.² In the setting of a clinical trial, a central review of grade 5 AEs reported on a phase III intergroup trial in advanced colorectal cancer deemed 20 of the 23 grade 5 AEs reported as treatment related, whereas only 10 of the 23 grade 5 AEs were considered as treatment related by the local treating physician who initially assigned the attribution.³

The nocebo phenomenon, which refers to the AEs reported on a placebo arm, provides an excellent opportunity to assess the nonspecific AEs of a study treatment attributable to factors other than the active treatment component.⁴ Rosenzweig et al⁵ reported a 19% incidence of AEs in healthy volunteers during placebo administration using data from 1,228 volunteers from 109 double-blind, placebo-controlled pharmacology trials. On the placebo arm, subjects ≥ 65 years of age and subjects who received repeated dosing reported a higher percentage of AEs compared with their respective counterparts.⁵ Other factors reported to influence AE reporting include a patient's expectation/perception before treatment initiation and psychological, situational, and contextual influences.⁴ These previous studies report factors that are associated with a patient reporting an AE as related to study treatment while receiving placebo but do not assess factors that are likely to be associated with a treating physician's attribution of an AE.

In this article, we investigated whether clinician-judged attribution adds value to the interpretation of AE data in oncology trials using AEs and their reported attribution on the placebo arm of two randomized phase III trials.

PATIENTS AND METHODS

We performed a retrospective analysis of the North Central Cancer Treatment Group (NCCTG) Trial 97-24-51, which was a phase III, randomized, double-blind study of carboxyamido-triazole (CAI) and placebo in patients with advanced non–small-cell lung cancer.⁶ To investigate the consistency of these results and further validate our findings, we subsequently analyzed data from a second trial—American College of Surgeons Oncology Group (ACOSOG) Trial Z9001—which was a phase III, randomized, double-blind study of adjuvant imatinib versus placebo in patients following the resection of primary non–small-cell lung cancer. These trials were selected because they included a placebo arm, which provides the rare opportunity in oncology to evaluate AEs where the true relationship between the study treatment and the AE is known to be unrelated.

The NCCTG conducts primarily phase II and III multidisciplinary trials. It has more than 30 years of experience sponsoring more than 400 trials that enrolled more than 50,000 patients. NCCTG membership consists of more than 1,000 predominately community-based physicians. In contrast, the ACOSOG conducts mostly phase II and III surgical management trials and consists largely of general and specialty surgeons in both academic and community-based practices.

NCCTG Trial 97-24-51 was a double-blind trial enrolling patients who were stable or responding after one prior chemotherapy regimen for metastatic or locally advanced non–small-cell lung cancer. Therapy with either oral CAI or placebo continued until disease progression or severe toxicity occurred. No differences in overall survival between the treatment arms were observed (median overall survival, 11.4 v 10.5 months; log-rank P = .54). The AE profile of the CAI regimen was generally mild with 38% of patients reporting grade ≥ 3 AEs regardless of attribution.⁶

ACOSOG Trial Z9001 was a double-blind trial enrolling patients who had resection of primary gastrointestinal stromal tumors. Treatment with imatinib or placebo continued for 1 year or until disease progression occurred. There was a difference observed in recurrence-free survival between treatment arms in favor of the imatinib arm (1-year recurrence-free survival, 98% v 83%; log-rank one-sided P < .001). The toxicity profile was also mild, with 31% in the imatinib group and 18% in the placebo group reporting at least one grade ≥ 3 AE.⁷

All AEs reported during the active treatment and observation phases (period of the trial where patients are no longer being treated with study therapy but are being followed according to the study test schedule) of both trials were included for analysis. All AEs reported across all cycles of treatment and including all grades of severity were included. AE attribution for the two trials was collected per CTC version 2.0 (Trial 97-24-51) and version 3.0 (Trial Z9001). Severity of AEs (ie, grade) was categorized as 1 = mild, 2 = moderate, 3 = severe, 4 = life-threatening, and 5 = death. Attribution was categorized as not related, unlikely, possible, probable, or definitely related. We further grouped them as related (possible, probable, and definite) and unrelated (not related and unlikely related). This further categorization was used because grouping is frequently done and is cited in the literature when evaluating AE data.

To assess whether either patient or AE characteristics influenced physicians' preconceptions of AE attribution, we constructed generalized estimating equation (GEE) models to assess the association between an AE being deemed related to study treatment and a specific patient or AE characteristic. In these models, we adjusted for repeated AEs within patients and allowed an exchangeable correlation structure for observations within each patient. The characteristics of sex, age, performance status (PS), severity of AE, cycle in which AE was first reported, AE type, and an indicator of whether or not the AE was listed in the protocol as previously known to be related to the study treatment were evaluated as potential predictors of attribution.

To further evaluate the effect of physician-reported attribution on the AE rates reported in the trial, we compared the rate of an AE reported on the treatment arm to that reported on the placebo arm. The difference in these rates provides an unbiased measure of the true rate of AEs attributable to treatment for that specific AE, since the rate reported in the placebo arm should represent the rate expected due to the patient's underlying disease or comorbidities alone. We also calculated what the AE rate would have been using the common method of including only AEs that the treating physician deemed at least possibly related to treatment. These rates were then used to calculate the rate at which physicians over- or under-reported attribution. All AE types were evaluated for this analysis if they were found to be statistically significantly different between the treatment and placebo arms of the trial. All analyses were conducted using SAS version 9 (SAS Institute, Cary, NC).

RESULTS

NCCTG Trial 97-24-51

A total of 186 patients (94 CAI, 92 placebo) were accrued between April 1999 and January 2004, including 84 patients enrolled on the placebo arm who experienced at least one AE during the course of the trial. All patients were included for this analysis, regardless of eligibility status, because they were all evaluable for AEs. Fifty-eight percent of patients enrolled on the placebo arm of this trial were male, median age was 64 years, 54% had a PS of 1, and 79% had stage IV disease (Table 1). Time on study treatment was relatively short with a median of 2.6 months.

Table 1.

Baseline Characteristics for Patients in the Placebo Arm by Study

Characteristic	Trial 97-24-51 (n = 84)		Trial Z9001 (n = 314)
	No.	%	No.	%
Male		58		54
Age, years
Median	64		58
Range	35-87		18-91
PS
0		35		75
1		54		23
2-3		12		3
Stage			—
IIIA		1
IIIB		20
IV		79
Time on treatment, months
Median	2.6		11.5
Range	0.6-47.1		0.1-13.8

Abbreviation: PS, performance score.

A total of 1,009 AEs were reported in the 84 patients in the placebo arm throughout the course of treatment. The majority of these AEs were grade 1 to 2 (95%), 47% were reported on the first or second cycle of treatment, 61% were known toxicities of CAI, and 29% were neurologic (Table 2). Of the 1,009 AEs reported on the placebo arm of this trial, 47.3% were reported as at least possibly related to study treatment (Fig 1). Eighty-six percent of patients on the placebo arm had one or more AEs that were deemed at least possibly related to treatment. When the same AE was reported on the same patient multiple times over the course of study treatment (n = 198), 36% (72 of 198) of the time the attribution changed at least once, with 25% (50 of 198) changing from not related to related or vice versa.

Table 2.

Adverse Event Characteristics by Study

Characteristic	Trial 97-24-51 (%; N = 1,009)	Trial Z9001 (%; N = 6,727)
Severity
1-2	95	98
3-5	5	2
Cycle/visit
1-2	47	20
3-5	27	29
6+	25	51
Known drug-related event	61	35
Adverse event type
Neurologic	29	6
Hematologic	19	9
Pain	7	16
Pulmonary	8	3
Constitutional symptoms	6	10
Gastrointestinal	13	19
Other	19	36

Fig 1.

Distribution of attribution by trial.

Based on the GEE model, physicians were more likely to deem AEs related to study treatment if the patient's PS was 0 compared with 1 or 2 to 3 (54% v 47% v 34%; P = .05). There was also an association with AE type with constitutional symptoms being more likely to be deemed as related (72%) and pulmonary AEs less likely to be deemed related (15% overall; P = .01). The factors of sex, age, severity of AE, cycle of AE, and whether the AE was listed in the protocol as a known drug-related AE were not associated with attribution (Table 3). On the basis of the difference in AE rates between the placebo and active treatment arms, physicians overestimated the true rate of AE attribution on the treatment arm by a range of 57% for ataxia to as much as 247% for neurosensory AEs (Table 4).

Table 3.

Characteristics Tested for Association With Adverse Event Relatedness

Characteristic	Trial 97-24-51 (N = 1,009)		Trial Z9001 (N = 6,727)
	% Related	P	% Related	P
Sex		.69		.02
Male	46		52
Female	49		47
Age, years		.88		.07
< 50	53		60
50-59	45		50
60-69	46		41
70-79	43		47
80+	59		50
PS*		.05		.11
0	54		48
1	47		52
2-3	34		60
Severity (grade)		.22		.08
1-2	48		50
3-5	42		35
Cycle/visit		.69		.11
1-2	48		51
3-5	42		53
6+	52		47
Known drug-related event		.42		< .001
Yes	51		63
No	41		42
Adverse event type		.01		< .001
Neurologic	47		40
Hematologic	57		61
Pain	32		39
Pulmonary	15		18
Constitutional symptoms	72		63
Gastrointestinal	57		56
Other	41		48

^*Performance score (PS) was collected prior to each cycle for Trial 97-24-51 and at baseline for Trial Z9001.

Table 4.

Adverse Event Rates by Treatment Arm and Estimated Over- or Under-Reporting for Trial 97-24-51

Adverse Event	% CAI Patients (A)	% Placebo Patients (B)	% Patients With Event Due to CAI Based on Data (C = A − B)	% Patients With Event Due to CAI Based on Treating Physician (D)	% Physician Over- or Under-Reporting Attribution [E = (D − C)/C]
Anorexia	30	13	17	29	72
Ataxia	32	16	16	24	57
Fatigue	52	29	23	50	119
Nausea	60	30	29	55	90
Neurosensory	57	45	13	45	247
Vomiting	31	14	17	30	78

Abbreviation: CAI, carboxyamido-triazole.

ACOSOG Trial Z9001

A total of 713 patients (359 imatinib, 354 placebo) were randomly assigned between June 2002 and April 2007, including 314 patients enrolled on the placebo arm who experienced at least one AE during the course of the trial. All patients were included for this analysis, regardless of eligibility status, because they were all evaluable for AEs. Fifty-five percent of patients enrolled on the placebo arm of this trial were male, median age was 58 years, and 75% had a PS of 0; time on study treatment was relatively long with a median of 11.5 months (Table 1).

From these 314 patients, 6,727 AEs were reported throughout the course of treatment. The majority (98%) of these AEs were grade 1 to 2, 20% were reported on the first or second cycle of treatment, 35% were known toxicities of imatinib mesylate, and 19% were gastrointestinal (Table 2). Of the 6,727 AEs reported on the placebo arm of this trial, 49.5% were reported as at least possibly related to study treatment (Fig 1). Eighty-seven percent of patients on the placebo arm had one or more AEs that were deemed at least possibly related to treatment. When the same AE was reported on the same patient multiple times over the course of study treatment (n = 1,309), 31% (409 of 1,309) of the time, the attribution changed at least once with 18% (237 of 1,309) changing from not related to related or vice versa.

Based on the GEE model, physicians were more likely to deem AEs related to study treatment if the patient was male (52% v 47%; P = .02). There was also an association with AE type, with hematologic and constitutional symptoms being more likely to be deemed as related (61% and 63%, respectively) and pulmonary AEs less likely to be deemed as related (18% overall; P < .001). AEs that were listed in the protocol as known to be related to imatinib were also more likely to be deemed related by the treating physician (63% v 42%; P < .001). Factors that were not found to be associated with attribution included age, PS, severity of AE, and timing (ie, visit) of when the AE was reported (Table 3). On the basis of a comparison of the reported rates on the treatment and placebo arms, physicians overestimated the number of AEs attributable to treatment by as much as 252% for fatigue and underestimated rates by as much as 87% for edema of the limb (Table 5).

Table 5.

Adverse Event Rates by Treatment Arm and Estimated Over- or Under-Reporting for Trial Z9001

Adverse Event	% Imatinib Patients (A)	% Placebo Patients (B)	% Patients With Event Due to Imatinib Based on Data (C = A − B)	% Patients With Event Due to Imatinib Based on Treating Physician (D)	% Physician Over- or Under-Reporting Attribution [E = (D − C)/C]
Edema, head/neck	45	14	30	43	42
Nausea	50	27	22	45	100
Leukopenia	18	6	12	17	44
Anemia	45	29	17	36	117
Neutropenia	17	6	10	15	46
Rash/desquamation	33	19	13	29	119
Edema	10	3	7	8	17
Edema, limb	24	13	11	1	−87
Vomiting	24	14	10	17	66
Fatigue	53	40	13	47	252
Tearing	10	4	6	9	47
Pain, extremity	12	6	7	6	−3
Anorexia	16	9	7	13	80
Creatinine	12	6	5	7	32
Weight loss	10	5	5	5	7
Hypokalemia	9	4	5	3	−29
Constipation	12	18	—	6	—

DISCUSSION

In this analysis, we evaluated the value added by collecting AE attribution in more than 7,000 AEs from two phase III placebo-controlled clinical trials from two different oncology cooperative groups. Despite the fact that the two trials used in this analysis were in different disease settings (lung v GI), stage of disease (79% stage IV v 100% resectable), and general health of patients (35% PS = 0 v 75% PS = 0), the main findings were remarkably consistent. Both trials reported a large percentage of AEs on the placebo arm as at least possibly related to study treatment (47% and 50%), a large percentage of patients on the placebo arm with at least one AE reported as at least possibly related to treatment (86% and 87%) and of those, AEs reported more than once for the same patient (36% and 31%) changed in attribution over time. These data demonstrate that the blind was well maintained in both of these trials. Both trials also identified AE type to be predictive of attribution of AEs to treatment on the placebo arm, and both trials demonstrated a potential for significant over-reporting of AE rates when using the attribution of possibly, probably, and definitely related on the treatment arm. Moreover, sex in Trial Z9001 and PS in Trial 97-24-51 also predicted for attribution relationship on the placebo arm. These data strongly support that AE attribution is difficult to determine, unreliable, and influenced by physicians' preconceptions.

There are limitations to this analysis that are worth noting. A relatively small number of patients had severe (grade ≥ 3) AEs reported (n = 50, Trial 97-24-51; n = 135, Trial Z9001). It is interesting to note that the attribution was not found to be associated with the severity of the AE, but this analysis is limited by the relatively small sample of grade 3 to 5 AEs in this population. One could hypothesize that physicians may put more time and effort into determining attribution for severe AEs. If physicians did not need to determine attribution for the large percentage of AEs routinely reported on clinical trials with a grade of 1 to 2, they would have significantly more time to spend determining attribution of the AEs that many would consider the most important. Mahoney et al⁸ noted that in 26 NCCTG trials conducted between January 1999 and November 2001, 97% of all routine AEs were grade 1 and 2. Roche et al⁹ in their review of factors affecting workload of cancer clinical trials also found that the collection and monitoring of AEs are major components contributing to the overall time and resources required for the conduct of clinical trials.

In our analysis, we included AEs with an attribution of possible in our related category, which is consistent with standard practice. Unfortunately, such a large portion of the related events (42% in Trial Z9001; 36% in Trial 97-24-51) were indeed classified as possible that it precluded a meaningful analysis considering only AEs classified as probable or definite as being related. It may be worthwhile to conduct cognitive interviews to understand how clinicians determine attribution.

It is interesting to note, on the basis of the between-arm comparisons within each trial, that the majority of the time, misattribution resulted in overestimating the rate of AEs on the treatment arm. However, we did observe on Trial Z9001, for the AE edema of the limb, physician assessment of relationship appeared to under-report (11% v 1%), which might represent an even larger area of concern. Critically, the requirement to provide the assessment of an AE to treatment in a randomized trial replaces the fundamental benefit provided by randomization with a subjective assessment. If all events are collected, the direct comparison of reported AEs between arms provides an unbiased and objective assessment of the true AE rate attributable to the element of the treatment that differs between study arms.

Given the substantial over-reporting of AE rates, under-reporting of some AEs, and variability in attribution on the same patient/event reported over time, we have demonstrated that attribution is very difficult to determine. One might hypothesize that treatment attribution would be especially difficult in the setting of advanced disease where the patients have significant comorbidities. Interestingly, however, attribution appeared equally difficult in the setting of Trial Z9001, which was in a limited disease setting.

Because of the strength and consistency of the data on these two trials, we recommend at a minimum improving the process for AE attribution collection through standardization and training. Realizing that large-scale training and standardization can be resource intensive, consideration should be given for possible discontinuation of collection of attribution on grade 1 to 3 AEs in a randomized phase III setting. For nonrandomized trials, we recommend consideration of collection of attribution for grade 3 AEs on a study-by-study basis, with consideration of no collection of attribution for grade 1 and 2 AEs. Because of the limited numbers of grade 4 and 5 AEs and the medical relevance of such AEs, we recommend the collection of attribution for grade 4 and 5 AEs in all trials, with the caution that reliability of such attribution data must be viewed critically.

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Although all authors completed the disclosure declaration, the following author(s) indicated a financial or other interest that is relevant to the subject matter under consideration in this article. Certain relationships marked with a “U” are those for which no compensation was received; those relationships marked with a “C” were compensated. For a detailed description of the disclosure categories, or for more information about ASCO's conflict of interest policy, please refer to the Author Disclosure Declaration and the Disclosures of Potential Conflicts of Interest section in Information for Contributors.

Employment or Leadership Position: None Consultant or Advisory Role: Ronald P. DeMatteo, Novartis (C) Stock Ownership: None Honoraria: Ronald P. DeMatteo, Novartis Research Funding: None Expert Testimony: None Other Remuneration: None

AUTHOR CONTRIBUTIONS

Conception and design: Shauna L. Hillman

Collection and assembly of data: Shauna L. Hillman, Sumithra J. Mandrekar, Brian Bot, Ronald P. DeMatteo, Edith A. Perez, Karla V. Ballman, Heidi Nelson, Jan C. Buckner, Daniel J. Sargent

Data analysis and interpretation: Shauna L. Hillman, Sumithra J. Mandrekar, Brian Bot, Ronald P. DeMatteo, Edith A. Perez, Karla V. Ballman, Heidi Nelson, Jan C. Buckner, Daniel J. Sargent

Manuscript writing: Shauna L. Hillman, Sumithra J. Mandrekar, Brian Bot, Ronald P. DeMatteo, Edith A. Perez, Karla V. Ballman, Heidi Nelson, Jan C. Buckner, Daniel J. Sargent

Final approval of manuscript: Shauna L. Hillman, Sumithra J. Mandrekar, Brian Bot, Ronald P. DeMatteo, Edith A. Perez, Karla V. Ballman, Heidi Nelson, Jan C. Buckner, Daniel J. Sargent