Unintended consequences of evolution of the Common Terminology Criteria for Adverse Events

Abstract

Background:

Adverse events (AEs) on Children’s Oncology Group (COG) trials are reported manually by clinical research assistants (CRAs). The Common Terminology Criteria for Adverse Events (CTCAE) was developed to provide standardized definitions for identifying and grading AEs. The CTCAE has expanded significantly over its five versions, but the impact of CTCAE definitional changes has not been examined.

Procedure:

This study compared AE number and ascertainment among the first four CTCAE versions using a case vignette. Each CTCAE version was used to create a list of AEs and grades by two separate CRAs.

Results:

The CTCAE expanded from 9 categories and 49 AEs in v1.0 to 26 categories and 790 AEs in v4.0. CRAs independently selected different approaches to AE ascertainment—comprehensive and parsimonious. The number of AEs identified in the parsimonious approach was stable with 10–14 in each CTC version. The comprehensive approach identified 9, 20, 29, and 37 AEs in CTC versions 1.0, 2.0, 3.0, and 4.0, respectively. Only approximately 65% of AEs were conclusively graded in versions 2.0 to 4.0 using the comprehensive approach.

Conclusions:

CTCAE has increased in complexity. Although this increased complexity allows for more granular AE reporting, these data demonstrate potential unintended negative consequences of increasing CTC AE complexity, including the risk of varying approaches to AE capture. A comprehensive evaluation of CTC AE definitions and CRA reporting practices across COG institutions and AEs are needed to improve the accuracy and efficiency of AE reporting.

1 |. INTRODUCTION

Cancer treatment is associated with significant treatment-related adverse events (AEs). In addition to the fact that AE reporting is legally mandated in the context of clinical trials, these reports are important because they provide estimates of toxicities that inform physicians, patients, and families in medical decision-making.¹ Furthermore, AE reporting determines safety endpoints after administration of investigational therapies.^2,3 These two functions are essential for the safe and ethical conduct of oncology clinical trials and interpretation of their results.

The Common Terminology Criteria for Adverse Events (CTCAE) was developed by the National Cancer Institute (NCI) Cancer Therapy Evaluation Program to be a worldwide standard for defining and grading AEs experienced by patients enrolled in oncology clinical trials.^4,5 CTCAE grades are assigned based on the potential impact the AE has on clinical management, activities of daily living (ADLs), dose modifications, or medication discontinuation. CTCAE definitions are the product of multidisciplinary, collaborative efforts.⁶

Version 1 of the Common Toxicity Criteria (CTC v1.0), created in 1983, was a standardized list of AEs commonly identified in oncology trials. Each AE included a grading scale to quantify severity.⁷ Although primarily intended for phase I clinical trials, CTC v1.0 became accepted for use in other trials.⁸ In 1999, the CTC Review Committee released a revised and expanded version, CTC v2.0.⁹ CTC v2.0 added systematic criteria for grading the acute effects of radiotherapy and added impact upon ADLs as a metric of AE severity.¹⁰ Although CTC v2.0 incorporated treatment-related toxicities inclusive of chemotherapy, radiation, and surgery, it was limited to acute AEs; late effects of treatment were missing, and criteria applicable to surgery-associated toxicities were limited.^1,10

CTCAE v3.0, published in 2006, focused on including a comprehensive list of surgical and late toxicities, and established pediatric-specific criteria for assigning toxicities.¹¹ CTCAE v4.0, published in 2009,¹² aimed to map AEs to the terms found in the Medical Dictionary for Regulatory Activities (MedDRA) System Organ Class (SOC).¹³ MedDRA is an extensive, clinically validated dictionary of medical terminology organized by SOC and used internationally to support public health monitoring and communication, as well as data analysis and management.¹⁴ The most recent version, CTCAE v5.0, was just released in November 2017 and includes additional AEs that were not included in prior versions.¹⁵

AE reporting on most oncology clinical trials is reliant on manual chart abstraction by local clinical research assistants (CRAs). In the Children’s Oncology Group (COG), CRAs are provided with modules with guidance as to how to ascertain AEs. They are advised to read the clinical trial protocol for guidance on AE reporting requirements, and are instructed to search the CTCAE quick reference manual or use the NCI Safety Profiler online CTCAE tool.¹⁶ CRAs are also trained to review the medical record to identify AEs and to confirm AE determination with the study principal investigator (PI) or treating physician.

As the CTCAE versions have evolved, the process of capturing an AE has become more labor intensive,⁷ with an increasing requirement for significant medical knowledge.¹⁷ Despite the evolving complexity of the CTCAE system, the infrastructure and processes for AE ascertainment have remained relatively the same. This situation has resulted in a vulnerability to both underreporting of AEs^18–20 and misspecification of AEs and grades.^21–23

Although researchers have proposed process improvement for AE collection and reporting,^6,8,17,24,25 there has been only very limited discussion of the CTCAE definitions themselves. Only two small studies have directly compared the differences in CTCAE v3.0 and CTCAE v4.0 definitions by contrasting definitions for oral mucositis.^5,26 This study aimed to gather preliminary data related to the impact of CTCAE version changes on the current system for clinical trial AE capture and reporting using a case vignette that can inform future studies regarding AE ascertainment. Based on prior experience that there is a great variability in AE reporting,²⁷ we sought to evaluate this potential variability by having two CRAs ascertain and grade AEs using each CTC version for the same clinical vignette. We hypothesized that each successive CTCAE version would necessitate capture of more distinct AEs for a specific clinical scenario resulting in increased variability in actual AE reporting and that there would be variability in AE capture between CRAs. The results of this study will lay the groundwork for the development of future studies regarding AE ascertainment strategies.

2 |. METHODS

2.1 |. Study design

We compared the four CTCAE versions published prior to 2017 using a case vignette of a child with acute myeloid leukemia (AML), as detailed below. The vignette was based on a patient’s medical chart and included a level of detail for symptoms comparable with that described in the medical record. As a secondary analysis, varying CRA approaches to AE ascertainment were compared.

2.2 |. Patient’s clinical summary

Patient X was a 15-year-old male who presented with fatigue, weight loss, and leukocytosis. He was diagnosed with AML, and induction therapy with cytarabine, daunorubicin, and etoposide was started. He experienced profound neutropenia and required blood transfusions for anemia and thrombocytopenia. Ten days after completion of his first cycle of chemotherapy, he developed diarrhea, intermittent nausea and emesis, poor oral intake, and weight loss of 9.5% of his body weight over a four-day period. He complained of abdominal cramping and pain due to frequent bowel movements. He then became febrile to 38.6°C, and blood cultures were drawn. He was begun on broad-spectrum antibiotics, and 12 hours after the blood culture was drawn, it was reported as positive for gram-positive cocci from both lumens. The causative agent was identified as Streptococcus Mitis (viridans group streptococcus). Two days after his first fever, he developed grunting, increased work of breathing, and cough with hemoptysis. His blood pressure dropped to 82/34, and he became tachycardic to 140 beats per minute. He was started on two liters of oxygen by nasal cannula without improvement in his respiratory status. At this time, a chest X-ray revealed right middle lobe and bibasilar atelectasis. He received 40 mL/kg normal saline without a significant change in his blood pressure and was transfused due to a platelet count of 12,000/μL. He continued to have grunting and reported new, diffuse abdominal pain with a soft, nondistended abdomen. Despite administration of a third 20 mL/kg normal saline bolus, he remained hypotensive. He was transferred to the pediatric intensive care unit (PICU) due to septic shock in the setting of streptococcus viridans central line-associated bloodstream infection. Bedside echocardiogram in the PICU showed decreased left ventricular function (ejection fraction of 22%). He had persistent hypoxemia despite oxygen administration by nasal cannula. Due to continued respiratory distress despite a trial of bilevel positive airway pressure (BiPAP), he was electively intubated. He was started on dopamine for hypotension, and his blood pressure normalized. His dopamine was discontinued within 24 hours of initiation. Oral intake was stopped, and total parenteral nutrition (TPN) was initiated due to significant mucositis, poor oral intake, and diarrhea. A CT scan performed due to continued diarrhea and abdominal pain showed colitis. His mechanical ventilation was weaned after two days and he was extubated to BiPAP and transitioned to nasal cannula. He remained on TPN and had no oral intake for six days, at which time his abdominal pain and diarrhea improved, and his diet was advanced to clears. At that point, he was transferred back to the oncology floor.

2.3 |. Identification of AEs for each CTC version

Each CTC version was manually applied to the clinical vignette by two CRAs each with one to two years of similar experience in using CTCAE for clinical trials. Each CRA received the same training on AE capture using CTCAE. Each CRA independently identified her own AE ascertainment approach based on her prior AE reporting experience and typical AE reporting style. The first CRA selected a parsimonious approach that identified the single most appropriate AE option for each sign or symptom in the narrative. The second CRA utilized a comprehensive approach that identified all potential AEs for each clinical sign or symptom contained in the vignette. For example, the parsimonious approach would report “adult respiratory distress syndrome (ARDS),” while the comprehensive approach would report “hypoxia,” “dyspnea,” and “ARDS.” Each CRA self-reported whether an AE could be conclusively graded. AEs that CRAs were unable to conclusively grade were reviewed by the physician PI for this study. Laboratory AEs were excluded as this investigation sought to assess AEs that require interpretation beyond a laboratory value.

2.4 |. Comparison of AE capture and consistency across CTCAE versions

The total number of AEs and grades identified were compared between CTC versions in an effort to qualitatively assess the changes between versions. Additionally, AE capture across each CTC version was evaluated by comparing the number of AEs conclusively graded using each approach. AE names and grades identified using the comprehensive approach for each CTC version were compared to evaluate grading consistency between CTC versions.

3 |. RESULTS

3.1 |. Evolution of the CTC

Between CTC v1.0 and 4.0, the number of AE categories increased from 9 to 26, and the number of individual AEs increased from 49 to 790 (Table 1 and Figure 1). Excluding v1.0, which was intended only for phase I trials, the number of categories only increased from 24 to 26, but the individual AEs increased by nearly 500. CTC v1.0 and v2.0 included grade 0 (none, normal, or no change) to grade 4 (severe). CTCAE v3.0 and 4.0 included grade 1 (mild) to 5 (death related to AE).

TABLE 1.

Number of categories and unique AEs in each CTC version

CTC version	Publication year	Categories	Number of AEs
CTC v1.0	1983	9	49
CTC v2.0	1999	24	295a
CTCAE v3.0	2006	28	373b
CTCAE v4.0	2009	26	790
CTCAE v5.0	2017	26	837

^aThere are an additional 39 AEs if “RTOG/EORTC Late Radiation Morbidity Scoring Scheme: Use for adverse event occurring greater than 90 days after radiation therapy” is included (total # of AEs: 334).

^bThere are an additional 537 AEs if all of the “Selects” drop-down options (e.g., surgery/intraoperative injury, pain, infection) are included (total # of AEs: 910).

FIGURE 1.

Exponential increase in the number of CTC AEs over each version

3.2 |. Comparison of AE capture across CTC versions

Table 2 displays the number of AEs identified by each CRA by CTC version. Not surprisingly, the comprehensive reporting approach identified more AEs than did the parsimonious approach except in v1.0, and the number of AEs identified by the comprehensive approach increased with each CTC version. For versions 2.0 through 4.0, only approximately 65% of AEs identified by the comprehensive approach and less than 43% of AEs identified by the parsimonious approach could be conclusively graded.

TABLE 2.

Number of conclusively graded AEs in the clinical scenario by CTC version

CTC version	Approach	Number of AEs reported	Number of AEs that CRA conclusively graded (%)	Number of AEs that MD conclusively confirmed (%)
CTC v1.0	Comprehensive	9	8 (88.9)	9 (100)
	Parsimonious	10	2 (20)	5 (50)
CTC v2.0	Comprehensive	20	13 (65)	15 (75)
	Parsimonious	14	5 (35.7)	12 (85.7)
CTCAE v3.0	Comprehensive	29	19 (65.5)	22 (75.9)
	Parsimonious	12	4 (33.3)	10 (83.3)
CTCAE v4.0	Comprehensive	37	24 (64.9)	28 (75.7)
	Parsimonious	14	6 (42.9)	14 (100)

CRA, clinical research associate; MD, medical doctor.

CRAs reported that the inability to conclusively grade an AE was due to insufficient clinical information, especially lack of granular information on the number of times an event (e.g., emesis) occurred or about the effect that the AE had on the patient’s ADLs. Other AEs were not graded due to inability to interpret the CTC definition. Of the AEs that the CRAs were unable to conclusively grade, 50%–100%, depending on CTCAE version and AE reporting approach, were able to be conclusively graded upon review by the study PI based upon physician clinical expertise.

3.3 |. Consistency between CTC versions

Table 3 shows considerable variation in AE grades between CTC versions using the comprehensive approach. Only 12 (32%) of the 37 AEs identified using CTCAE v4.0 had consistent grade assignments across versions, when the version included a related AE. The grade of another 12 AEs (32%) changed between versions due to different definitions in each version. Thirteen AEs (35%) only appeared in one version or could not be graded in at least one version, and therefore change over time could not be assessed. Two AEs only existed in a version that preceded CTCAE v4.0.

TABLE 3.

Variation in AE grading between CTC versions for the case vignette

Symptom in clinical summary	AE identified by comprehensive AE assessment approach (per CTCAEv4.0)	Grade
		CTCv1.0	CTCv2.0	CTCAEv3.0	CTCAEv4.0
Pain due to frequent bowel movements	Buttock pain	-	-	NG	NG
	Rectal pain	-	NG	NG	NG
	Anal pain	-	-	NG	NG
Diarrhea	Diarrheaa	3	3	3	3
	Fecal incontinence		NG	NG	3
	Proctitis	-	3	3	3
Mucositis	Anal mucositis	-	-	3b	NG
	Rectal mucositis	-	-	3c	NG
Colitis	Colitisa	-	3	3	3
	Enterocolitis	-	-	3d	3
Typhlitis	Typhlitis	-	3	3	3
Abdominal pain	Abdominal pain	-	NG	NGe	NG
	Stomach pain	-	-	-	NG
Loss of appetite/TPN requirement	Weight lossa	2f	1	3	3
	Anorexiaa	3	4	3	3
	Malabsorption	-	-	3	3
	Nauseaa	3	2	3	3
	Vomitinga	4	4	3	3
Cardiac symptoms	Ejection fraction decreaseda	-	-	-	3
	Heart failure	-	-	-	4
	Left ventricular systolic dysfunctiona	3g	NGh	3	4
	Left ventricular diastolic dysfunction	-	-	3	-
Respiratory symptoms	Dyspneaa	4i	4	4	4
	Cougha	-	NG	NG	2
	Hemoptysis	-	3	-	-
	Productive cough	-	-	-	NG
	Sore throat	-	-	-	NG
	Laryngeal hemorrhage	-	3j	3k	3
	Tracheal mucositisa	-	-	-	1
	Postnasaldrip	-	-	-	2
	Wheezing	-	-	NG	2
	Atelectasis	-	-	4	4
	Respiratory failure	-	-	-	4
	Adult respiratory distress syndrome		4	4	4
	Sinus tachycardiaa	4l	3	4	3
Low blood pressure	Hypotensiona	4	3	NG	4
Infection	Catheter-related infectiona	4m	4	4	4
Sepsis	Sepsisa	-	-	-	4
Elevated temperature	Fever	-	-	1	1
	Total number of AEs unable to be graded	0	5	8	9

Notes: Green, consistent grading of AE across CTC versions; red, inconsistent grading of AE across CTC versions; NG, AE unable to be graded definitively; -, AE not in CTC version.

Approach 2 also identified stomatitis in CTC v1.0, stomatitis, infection, and hypoxia in CTC v2.0, infection in CTCAE v3.0 and hypoxia and febrile neutropenia in CTCAE v4.0, which were not identified by approach 1 in those versions.

^aAE also identified by approach 2 (in at least one of the CTC versions).

^bIn CTCAEv3.0, anal mucositis is called “mucositis/stomatitis (functional/symptomatic)–Anus.”

^cIn CTCAEv3.0, rectal mucositis is called “mucositis/stomatitis (functional/symptomatic)–Rectum.”

^dIn CTCAEv3.0, enterocolitis is called “enteritis.”

^eIn CTCAEv3.0, abdominal pain is called “pain abdomen NOS.”

^fIn CTCAEv1.0, weight loss is called “weight gain/loss.”

^gIn CTCAEv1.0, left ventricular systolic dysfunction is called “cardiac function.”

^hIn CTCAEv2.0, left ventricular systolic dysfunction is called “cardiac left ventricular function.”

ⁱIn CTCAEv1.0, dyspnea is called “pulmonary.”

^jIn CTCAEv3.0, laryngeal hemorrhage is called “hemorrhage, pulmonary/upper respiratory—bronchopulmonary NOS.”

^kIn CTCAEv2.0, laryngeal hemorrhage is called “hemorrhage/bleeding with grade 3 or 4 thrombocytopenia.”

^lIn CTCAEv1.0, sinus tachycardia is called “dysrhythmias.”

^mIn CTCAEv1.0, catheter-related infection is called “infection.”

4 |. DISCUSSION

Over time, the number of CTC AEs has increased exponentially. As hypothesized, this study shows that the increase in total potential AEs in each version has unintended consequences. Despite a greater total number of AE reports with each successive version, at most 65% of these AEs could be graded by a CRA without guidance from an attending physician in versions 2.0 to 4.0. This preliminary work demonstrates that CTC expansion has allowed for more detailed AE reporting and dramatically different AE ascertainment approaches, both of which create AE reporting complexities. These initial data provide strong support for the necessity of further investigation into various methods for improving the current AE reporting system.

The comprehensive and parsimonious approaches to AE reporting highlight the impact of a nonstandardized approach to the use of CTCAE criteria. The comprehensive approach may lead to reporting redundancy. For example, using CTCAE v3.0 and v4.0, this approach reports both “colitis” and “enterocolitis” (Table 3). Moreover, capturing all possible AEs from a list of 790 options for a complex clinical scenario is labor intensive. The parsimonious approach, although more efficient, leads to a fraction of the potential AEs being identified and may underestimate rates of clinically relevant AEs. For example, the parsimonious approach may identify grade IV dyspnea rather than grade IV respiratory failure as the patient’s primary respiratory AE. Our data do not allow a definitive conclusion about which approach is optimal. COG recently released guidance advising investigators to take a parsimonious approach for AEs that are syndromes; only the overall syndrome rather than the symptoms should be reported.²⁸ Although supportive of this, we want to highlight its dependence on the definitional interpretability and clinical relevance of the CTCAE definitions.

Furthermore, the multiple ascertainment approaches potentially lead to bias in AE estimates. If approaches are balanced between study arms, the relative effect measures (relative risk, odds ratio) would be unbiased, but the absolute AE rates would underestimate true AE rates by treatment arm. Alternatively, if either approach were by chance associated with a specific arm, then both the absolute AE rates and relative effect measures could be biased. This potential bias may be of particular relevance if CRAs search for AEs more intensively on the experimental arm, as is mandated in current Adverse Event Expedited Reporting System (AdEERS) reporting.

The differences observed in AE ascertainment with these two approaches are consistent with those observed by Zhang et al, who reported that CTC definitions are often inappropriately applied in published clinical trial reports.²² The cause of the misuse of CTC definitions is not well understood and needs further investigation to improve the quality of AE reporting. CTC definitions are developed by a consensus panel that seeks to include a wide range of clinical stakeholders in AE classification. However, based on the available literature, CTCAE definitions do not appear to undergo testing prior to release or performance improvement once released.

In order to address potential gaps in CTCAE, some clinical trial groups have made modifications to enhance AE reporting. For example, in CTC v4, infection-related AEs are graded based on requirements for antibiotics, imaging, or surgery,¹² and do not distinguish sterile and nonsterile site infections nor capture the causative organism. To address these limitations, current COG AML trials require additional data reporting regarding infectious AEs,²⁹ and prior work has shown that this additional data reporting guidance has led to improvements in AE reporting.¹⁸ Incorporating this modification into CTCAE would help capture clinically relevant data that are not currently included.

The use of ADL criteria for grading is another important definitional issue that can be addressed. ADLs were introduced in CTC v2.0 as a metric for grading AEs, and in CTCAE v4.0, approximately 250 AE definitions include ADLs.¹² ADL ascertainments are complicated by their age- and patient-specific nature. Instrumental ADL refers to “preparing meals, shopping for groceries or clothes, using the telephone, managing money, etc.”³⁰ Self-care ADL refers to “bathing, dressing and undressing, feeding self, using the toilet, taking medications, and not bedridden.”³⁰ These references are differently applicable to all patients (e.g., adults, children, intellectually disabled, physically disabled), which renders grading the severity of an AE based on ADLs difficult across the spectrum of clinical trial participants.

The pediatric oncology clinical trial community recognizes these issues. In addition to clarifying CTCAE definitions, multiple pediatric protocols have sought to minimize the burden of AE reporting and decrease the heterogeneity in AE reporting approaches by identifying trial-specific targeted toxicities. Such targeted toxicity reporting may decrease the burden of reporting, but frequently results in the preferential reporting of the targeted toxicities.¹⁸ The comprehensive approach of capturing targeted toxicities should be recommended, but it must be understood that with this approach, estimates of non-targeted AE rates will likely be biased downward. Individual protocols need to state the comprehensiveness of key AE capture based on the aims of the study in order to produce consistent, accurate results.

In addition, we and other investigators have proposed potential alternative AE definitions based on resource utilization to improve AE reporting consistently.^18,23 Although such approaches may be helpful, a comprehensive and rigorous study to understand the causes of AE underreporting and misreporting is planned to identify the reporting issues identified in this and other reports. Such work would need to consider the operational impact and workflow realities of CRAs and physicians collecting AE data on oncology clinical trials. Improving CRA training on how to capture AEs and interpretation of current definitions could also be beneficial, and work to address the training is underway within COG. Although the often-cited underfunding of cooperative groups should not be used as an excuse for inadequate AE reporting, performing AE capture via manual chart review in its current form is likely not sustainable. We propose consideration of automated ascertainment to establish efficient and consistent capture especially if CTC definitions cannot be simplified. Automated capture would ensure a standardized approach to AE capture for each protocol, and the comprehensiveness of capture could be adjusted as needed for different protocols.

Although this study identifies important concerns regarding the current CTC reporting system and hypotheses for future research, several limitations of this study should be acknowledged. An important limitation is that CTC versions were compared using a clinical narrative rather than medical chart review and only evaluated by three reviewers (two CRAs, one attending physician). The clinical vignette was written to be as comprehensive as what would be found in a medical chart, and the lack of detail regarding the number of times an event occurs or ADLs parallels the unavailable level of detail in the medical record. However, conclusive AE capture may have been affected for some AEs by data that reviewers felt were missing from the vignette. In addition, the current version of CTCAE, v5.0, was not evaluated by the study team. However, given the modestly increased number of AEs in CTC v5.0, it seems likely that our results are applicable to CTC v5.0. Lastly, this study did not evaluate variability in physician interpretation of AEs and clinical data. Together, these limitations preclude definitive conclusions regarding the variability in AE reporting on COG clinical trials. Still, even as initial data with recognized limitations, the data provide a compelling rationale for further evaluation of the impact of CTC definitions on reporting completeness and accuracy.

In summary, although CTCAE definitions guide AE determination, CTCAE is complex due to its length, and the varying approaches to using CTACE may lead to inconsistent results. Despite the limitations of this preliminary study, the observed variability in AE reporting is substantial and serves as the basis for a future rigorous and comprehensive evaluation of CTCAE definitions and reporting processes. These and other data our and other research groups^18,23,31 have led to the development of an adverse event task force within COG that is working to survey CRAs as a first step to gather data about AE reporting practices and challenges in AE interpretation. We hope that those data will inform the development of improved CTCAE definitions and training, and standardize reporting processes within COG.

Abbreviations:

AdEERs

Adverse Event Expedited Reporting System

ADL

activities of daily living

AE

adverse event

AML

acute myeloid leukemia

BiPAP

bilevel positive airway pressure

COG

Children’s Oncology Group

CRA

Clinical Research Assistant

CTC

common toxicity criteria

CTCAE

Common Terminology Criteria for Adverse Events

MedDRA

Medical Dictionary for Regulatory Activities

PI

principal investigator

PICU

pediatric intensive care unit

NCI

National Cancer Institute

SOC

system organ class

TPN

total parenteral nutrition