Coverage Evaluation of CTCAE for Capturing the Immune-related Adverse Events Leveraging Text Mining Technologies

Abstract

Background: Immunotherapy is seen as a promising new treatment for cancer but it may also cause immune-related adverse events (irAEs). Post-market surveillance of immunotherapy drugs highly depends on the ability to capture and standardize irAE data. The Common Terminology Criteria for Adverse Events (CTCAE) is a potential terminology that can be leveraged for irAEs standardization. However, the capability of the CTCAE in irAEs standardization needs to be evaluated. Methods: We investigated the irAEs of six FDA approved cancer immunotherapy monoclonal antibodies (mAbs) and evaluated the coverage of the CTCAE for capturing irAEs. We manually identified irAEs from drug labels of the 6 mAbs as the gold standard. We assessed the performance of two text mining pipelines using the dictionary lookup of the CTCAE terms and identified irAEs. In the coverage evaluation, the CTCAE was compared with MedDRA, a standard terminology for regulatory science, for irAE standardization. Results: We manually identified 510 unique irAEs from the drug labels. When using the CTCAE as a dictionary to run the text mining pipeline, the precision, recall and F-measure value was 100%, 10.78% and 19.47%. After adding manually identified irAE terms into the dictionary, the recall and F-measure value significantly improved, increased to 95.69% and 97.31%, respectively. In the coverage evaluation, compared with MedDRA, the coverage rate of the CTCAE is only 13.50% when taking all the mining results together into consideration. Conclusion: With some limitations in our study, we clearly demonstrated that the CTCAE needs an extension to meet the irAE standardization task.

Introduction

By improving a patient’s immune system for therapeutic benefit in cancer, immunotherapy is seen as a promising treatment for cancer recently [1]. The United States Food and Drug Administration (FDA) has approved six immune checkpoint-blocking antibodies for the treatment of cancer since 2011. However, those immunotherapies could cause immune-related adverse events (irAE) due to the increasing activity of the immune system. IrAEs may affect many organ systems such as the gastrointestinal tract, endocrine glands, skin and liver [2]. Most of those irAEs are mild to moderate severity, but sometimes it can be serious, irreversible, or even fatal. In addition, due to the fact that these agents are new to the market, it’s important to conduct studies using real-world data to investigate their safety profiles. However, the irAEs in clinical data is usually heterogeneous and error-prone. The success of post-market surveillance of immunotherapy drugs also highly depends on the ability to capture and standardize irAE data. From the point view of data standardization, harmonized terminology is required to report and describe irAEs in order to interpret the safety data.

The Common Terminology Criteria for Adverse Events (CTCAE) [3] is a potential candidate terminology that may meet our requirement for irAEs standardization. The CTCAE is a descriptive terminology used for adverse events (AE) with standard grading scale and it has also been widely accepted throughout the oncology research community [4]. Although the CTCAE has been utilized to standardize and grade the irAEs in some immunotherapy related studies [5-8], few studies have been focused on investigating the coverage of the CTCAE in standardizing irAEs.

The objective of our study is to evaluate the feasibility of utilizing the CTCAE to standardize irAEs. In order to investigate the performance of the CTCAE in mining different irAE data sources, we choose drug labels, irAE-related publications and FDA Adverse Event Reporting System (FAERS) data as materials for a comprehensive evaluation. We use a customized text-mining pipeline based on a CTCAE dictionary and conduct an irAEs detection by FAERS data. We assess the coverage of the CTCAE for capturing the irAEs in both structured data (i.e., the FAERS data) and unstructured data including drug labels and literature.

Materials and Methods

Materials

CTCAE. The current released version is CTCAE 5.0 [3]. This version contains 837 AE terms with the Medical Dictionary for Regulatory Activities (MedDRA) codes. Most AE terms are associated with a 5-point severity scale. The AE terms are grouped by the MedDRA System Organ Class (SOC). In the CTCAE, “Grade” refers to the severity of the adverse event (AE). The CTCAE displays Grades 1 (mild) through 5 (extremely severe) with unique clinical descriptions of severity for each AE based on a general guideline.

MedDRA is a rich and highly specific standardized medical terminology [9], aiming to facilitate sharing of regulatory activities information for human medical products. MedDRA is organized by hierarchy structure. There are 5 hierarchy levels in MedDRA; System Organ Classes (SOCs), High Level Group Terms (HLGTs), High Level Terms (HLTs), Preferred Terms (PTs) and Lowest Level Terms (LLTs). In the latest version of MedDRA, there are more than 70,000 terms recorded. MedDRA has been widely used in the drug safety surveillance projects for the adverse events standardization.

cTAKES – Clinical Text Analysis and Knowledge Extraction System [10]. Built on an operable interface Unstructured Information Management Architecture (UIMA), cTAKES provides a pipeline for selecting which descriptors are used together and for determining the order of the descriptors (see detail in cTAKES 4.0 Component Use Guide). Dictionaries such as UMLS, SNOMED CT, and RxNorm are integrated into the cTAKES clinical pipeline. cTAKES discovers clinical named entities and clinical events using a dictionary lookup algorithm and a subset of the UMLS. The fast dictionary lookup annotator supports the definition of a custom dictionary that performs the same basic functions as the original dictionary lookup annotator to identify terms in text and normalize them to codes in a dictionary. The module comes with multiple possible pre-packed configurations and is also customizable and extensible. In this study, we developed a text-mining pipeline specific for the irAE detection using the fast dictionary lookup feature of the cTAKES.

Drug Labels from the Structured Product Labeling (SPL). The Structured Product Labeling (SPL) is a document markup standard approved by the Health Level Seven (HL7) and adopted by the FDA as a mechanism for exchanging product information [11]. The National Library of Medicine (NLM) DailyMed web site provides high quality information about marketed drugs derived from FDA SPLs [12]. The search functionality of the web site only supports a single drug name or NDC code input so a user cannot query against multiple medications simultaneously. Furthermore, the adverse events are described in free text (i.e. non machine-readable) under a number of section headings (e.g. the Adverse Reaction Section). In this study, we choose 6 monoclonal antibodies (mAbs) that are those approved by FDA for cancer immunotherapy and download their drug labels from DailyMed. Table 1 shows the basic information of these 6 mAbs.

Table 1.

Basic information of 6 cancer immunotherapy mAbs

Drug name	Active Ingredients	FDA approval year
YERVOY	ipilimumab	2011
KEYTRUDA	pembrolizumab	2014
OPDIVO	nivolumab	2014
TECENTRIQ	atezolizumab	2016
IMFINZI	durvalumab	2017
BAVENCIO	avelumab	2017

IrAEs related PubMed publication. In order to evaluate the performance of CTCAE in irAE-related publication, we retrieved publication from PubMed and built an irAE-related publication text-mining data set. The query “immune-related [All Fields] AND adverse [All Fields] AND events [All Fields]” (retrieve date: Jan. 2018) was used to retrieve publications from PubMed. A total of 679 irAEs related publications were obtained. Then we downloaded all the abstract text and full text of 20 review papers as the irAE-related publication text-mining data set.

FAERS is a database that contains adverse event reports, medication error reports, and product quality complaints resulting in adverse events that were submitted to FDA [13]. FAERS provides rich information on voluntary reports of suspected adverse events and has been widely used for drug safety signal detection and pharmacovigilance applications. And all the adverse events in FAERS have already standardized by MedDRA. Since September 2012, the format of the FAERS database has been changed. In this study, we used the new format of the FAERS data from September 2012 to March 2017. Among over 4 million patients’ adverse reports in FAERS during September 2012 to March 2017, a total of 24883 mAbs related AE reports were collected. And the number of patients for the 6 mAbs are 8556 (Ipilimumab), 5099 (Pembrolizumab), 12569 (Nivolumab), 893 (Atezolizumab), 27 (Durvalumab) and 5 (Avelumab).

Methods

In this study, we first conduct a manual review to create the gold standard from the drug labels of 6 mAbs and then use it to evaluate the performance of two cTAKES pipelines in extracting irAEs terms. In the coverage evaluation, irAEs terms from drug label are combined with irAE terms extracted from the publication and irAE signals detected from FAERS data. We also compared with MedDRA-based approach for the coverage evaluation of the CTCAE.

Drug label text mining and manual evaluation. In this study, we manually reviewed all the 6 mAbs drug labels to identify irAEs as the gold standard to calculate the baseline performance of CTCAE. Two authors (KR, GJ) manually reviewed the drug label text under the section WARNINGS AND PRECAUTIONS and the section ADVERSE REACTIONS of six mAb drugs (as shown in Table 1), and identified the irAE terms out of the drug label text, coming to consensus via discussion. Both KR and GJ have medical backgrounds, and KR is a medical oncologist with both clinical and research expertise in treatment toxicities. We also developed a customized text mining pipeline using the Fast Dictionary Lookup feature provided by the cTAKES v4.0 and created a custom dictionary based on the CTCAE terms using a bar-separated value (BSV) (a.k.a. pipe-separated) flat file. In order to make a comparison and provide a performance evaluation of cTAKES in our irAE extraction task, we also ran another enhanced pipeline which used both CTCAE and manual review results as dictionary. By implementing the pipeline, irAE terms could be extracted from the drug label text under the section WARNINGS AND PRECAUTIONS and the section ADVERSE REACTIONS. Then we evaluated the baseline performance using manually identified irAE terms as the gold standard, and standard measures of two pipelines (precision, recall and f-measure) were calculated respectively.

IrAE related publication text mining. In order to evaluate the coverage of CTCAE for mining irAEs in publications, we also performed the enhanced text mining pipeline as described above in our PubMed publication data set. Then we compared the number of CTCAE terms and all the irAEs terms which were extracted from those publication text to investigate the performance of CTCAE.

FAERS based irAE signal detection. In order to assess the feasibility of CTCAE on immunotherapy drug safety surveillance, we conducted an irAEs detection using FAERS data. Adverse event reports of 6 mAbs were extracted from FAERS and reporting odds ratio (ROR) algorithm was used to detect the irAEs (ROR algorithm is shown in Table 2 and Equation 1). When the case number of one irAE is more than 2 and the lower 95% confidence interval (95% CI) of ROR is more than 1, the specific adverse event would be seen as a positive irAE signal. For the purpose of evaluation, all the positive signals were integrated to two data sets respectively, the signals in the first set was annotated by CTCAE and in the other one was annotated by MedDRA.

Table 2.

The contingency table for the calculation of ROR algorithm

	Reports with target event	Reports without target event
Reports with mAbs	a	b
Reports without mAbs	c	d

$$\mathbf{R}\mathbf{O}\mathbf{R}=\frac{a/b}{c/d}$$ (Equation 1)

Comparison with MedDRA-based text mining pipeline. We implemented the third text mining pipeline using all the MedDRA terms (version 20.0) for the cTAKES dictionary lookup. We processed the drug labels and publication text using the MedDRA-based text mining pipeline and compared the results with those using enhanced CTCAE pipeline.

Results

Baseline performance of text mining pipelines. Table 3 shows manual performance of irAEs identification results. By the manual review, we totally collected 510 unique irAEs from all the 6 drug labels. As previously mentioned, in order to evaluate the feasibility of CTCAE in describing irAEs, we implemented two pipelines with different dictionaries respectively to make a comparison. In the first pipeline, we only used CTCAE as the identification dictionary and 55 irAEs terms were extracted from 6 drug labels. In the enhanced pipeline with both original CTCAE and additional manual identified irAE terms dictionary, we extracted 493 irAEs terms. Then, the precision, recall and F-measure value was calculated for those pipelines using the manually identified irAE terms as the gold standard. For the CTCAE only pipeline, the precision, recall and F-measure value is 100%, 10.78% and 19.47%, compared with 98.99%, 95.69% and 97.31% in enhanced pipeline. In addition, for the recall and F-measure value of each specific drug label, there is also an improving by the enhanced pipeline. This indicates that CTCAE may not cover many irAEs and add more adverse terms in it could improve the performance of CTCAE.

Table 3.

Performance of two text mining pipelines for the irAEs identification from drug labels of 6 mAbs.

Drug Name (mAb)	Manual identified irAEs terms	Dictionary	cTAKES Identified irAEs terms	TP - true positive	FP - false positive	FN - false negative	Precision (TP/(TP+FP))	Recall (TP/(TP+FN))	F-measure (2PR/(P+R))
YERVOY- ipilimumab	130	CTCAE Only	35	35	0	95	100%	26.92%	42.42%
		Enhanced	132	124	8	6	93.94%	95.38%	94.66%
KEYTRUDA- pembrolizumab	207	CTCAE Only	23	23	0	184	100%	11.11%	20.00%
		Enhanced	222	203	19	4	91.44%	98.07%	94.64%
OPDIVO- nivolumab	233	CTCAE Only	18	18	0	215	100%	7.73%	14.34%
		Enhanced	253	229	24	4	90.51%	98.28%	94.24%
TECENTRIQ- atezolizumab	154	CTCAE Only	14	14	0	140	100%	9.09%	16.67%
		Enhanced	156	143	13	11	91.67%	92.86%	92.26%
IMFINZI- durvalumab	186	CTCAE Only	20	20	0	166	100%	10.75%	19.42%
		Enhanced	188	173	15	13	92.02%	93.01%	92.51%
BAVENCIO- avelumab	154	CTCAE Only	29	29	0	125	100%	18.83%	31.69%
		Enhanced	153	146	7	8	95.42%	94.81%	95.11%
Total	510	CTCAE Only	55	55	0	455	100%	10.78%	19.47%
		Enhanced	493	488	5	17	98.99%	95.69%	97.31%

Out of 510 terms identified from drug labels, 156 terms (30.59%) are covered by the CTCAE whereas 354 terms (69.41%) are not included in the CTCAE. We also mapped those non-CTCAE terms into the SOC classes. Table 4 shows the SOC distribution of 354 non-CTCAE terms.

Table 4.

SOC distribution of 354 non-CTCAE terms. (Some terms may belong to more than one SOCs)

SOCs	No. of terms	SOCs	No. of terms	SOCs	No. of terms
Blood and lymphatic system disorders	13	Infections and infestations	36	Renal and urinary disorders	12
Cardiac disorders	7	Injury, poisoning and procedural	11	Reproductive system and breast	4
		complications		disorders
Ear and labyrinth disorders	2	Investigations	32	Respiratory, thoracic and mediastinal disorders	26
Endocrine disorders	17	Metabolism and nutrition disorders	22	Skin and subcutaneous tissue disorders	57
Eye disorders	12	Musculoskeletal and connective tissue disorders	17	Surgical and medical procedures	1
Gastrointestinal disorders	20	Neoplasms benign, malignant and unspecified (incl cysts and polyps)	6	Vascular disorders	20
General disorders and administration site conditions	29	Nervous system disorders	32
Hepatobiliary disorders	8	Pregnancy, puerperium and perinatal conditions	3
Immune system disorders	30	Psychiatric disorders	8	Couldn't classify in SOCs	44

irAEs terms identified from publication text. By the retrieval query we raised before, 679 abstracts and 20 reviews were retrieved as of January 24, 2018. Table 5 shows the irAEs publication text mining result by enhanced CTCAE pipeline. A total of 316 unique MedDRA PTs was identified. Among them, 246 unique terms are identified from abstract text, and 228 PTs are from 20 review full paper text.

Table 5.

Text mining result for irAEs related publications.

	PubMed Retrieval Result (Num. of Abstracts/Papers)	Enhanced CTCAE Text Mining Result (Unique Terms/Non CTCAE Terms)
Abstract	679	246/129
Review full paper	20	228/124
Total	679	316/166

irAEs signals identified based on FAERS data mining. A total of 94 irAEs signals were identified for the 6 mAbs from the FAERS data. All of the 94 irAE signals are covered by the CTCAE. Table 6 shows those 94 signal terms and their System Organ Classes (SOCs).

Table 6.

irAEs detection result based on FAERS data.

No.	irAEs Signal Term	MedDRA Code	SOCs	No.	irAEs Signal Term	MedDRA Code	SOCs
1	Eosinophilia	10014950	Blood and lymphatic system disorders	48	Stevens-Johnson syndrome	10042033	Injury, poisoning and procedural complications
2	Febrile neutropenia	10016288		49	Infusion related reaction	10051792
3	Leukocytosis	10024378		50	Infusion site extravasation	10064774
4	Ascites	10003445	Cardiac disorders	51	Alanine aminotransferase increased	10001551	Investigations Metabolism and nutrition disorders
5	Atrial fibrillation	10003658		52	Aspartate aminotransferase increased	10003481
6	Atrial flutter	10003662		53	Blood bilirubin increased	10005364
7	Atrioventricular block complete	10003673		54	Blood corticotrophin decreased	10005452
8	Conduction disorder	10010276		55	Blood lactate dehydrogenase increased	10005630
9	Myocarditis	10028606		56	Lipase increased	10024574
10	Pericardial effusion	10034474		57	Lymphocyte count decreased	10025256
11	Pericarditis	10034484		58	Neutrophil count decreased	10029366
12	Supraventricular tachycardia	10042604		59	Platelet count decreased	10035528
13	Ventricular arrhythmia	10047281		60	Dehydration	10012174	Metabolism and nutrition disorders
14	Adrenal insufficiency	10001367	Endocrine disorders	61	Chills	10008531	Musculoskeletal and connective tissue disorders
15	Hyperthyroidism	10020850		62	Myositis	10028653
16	Hypopituitarism	10021067		63	Rhabdomyolysis	10039020
17	Hypothyroidism	10021114		64	Encephalopathy	10014625	Nervous system disorders
18	Hypophysitis	10062767		65	Guillain-Barre syndrome	10018767
19	Conjunctivitis	10010741	Eye disorders	66	Leukoencephalopathy	10024382
20	Keratitis	10023332		67	Meningitis	10027199
21	Uveitis	10046851		68	Myelitis	10028524
22	Eyelid function disorder	10061145		69	Peripheral motor neuropathy	10034580
23	Abdominal pain	10000081	Gastrointestinal disorders	70	Peripheral sensory neuropathy	10034620
24	Colitis	10009887		71	Spinal cord compression	10041549
25	Colonic fistula	10009995		72	Facial nerve disorder	10061457
26	Duodenal perforation	10013832		73	Nephrotic syndrome	10029164	Renal and
27	Enterocolitis	10014893		74	Cystitis noninfective	10063057	urinary
28	Ileal perforation	10021305		75	Acute kidney injury	10069339	disorders
29	Ileus	10021328		76	Aspiration	10003504
30	Pancreatitis	10033645		77	Bronchial fistula	10006437
31	Proctitis	10036774		78	Bronchial obstruction	10006440
32	Small intestinal obstruction	10041101		79	Hypoxia	10021143
33	Small intestinal perforation	10041103		80	Pleural effusion	10035598	Respiratory, thoracic and mediastinal disorders
34	Enterocolitis infectious	10058838		81	Pleuritic pain	10035623
35	Capillary leak syndrome	10007196	General	82	Pneumonitis	10035742
36	Fatigue	10016256	disorders and	83	Pneumothorax	10035759
37	Disease progression	10061818	administration	84	Respiratory failure	10038695
38	Mucosal infection	10065764	site conditions	85	Stridor	10042241
39	Cholecystitis	10008612	Hepatobiliary disorders	86	Tracheal obstruction	10044291
40	Hepatic failure	10019663		87	Bronchopleural fistula	10053481
41	Bile duct stenosis	10051341		88	Non-cardiac chest pain	10062501
42	Myasthenia gravis	10028417	Immune system disorders	89	Pruritus	10037087	Skin and
43	Toxic epidermal necrolysis	10044223		90	Rash maculo-papular	10037868	subcutaneous
44	Cytokine release syndrome	10052015		91	Skin hypopigmentation	10040868	tissue disorders
45	Autoimmune disorder	10061664		92	Disseminated intravascular coagulation	10013442	Vascular disorders
46	Sepsis	10040047	Infections and	93	Portal vein thrombosis	10036206
47	Lung infection	10061229	infestations	94	Superior vena cava syndrome	10042569

Comparison with MedDRA-based text mining pipeline. We compared the CTCAE-based text mining results with the results from MedDA-based text mining. Figure 1 shows the coverage rate of the CTCAE in the different situations. Compared with MedDRA, the coverage rate of the CTCAE in identifying irAEs from the drug labels and from the publication is 9.98% and 12.42% respectively (Figure 1a, Figure 1b). For the irAEs signal detection study (Figure 1c), all of the 94 irAEs signals annotated by MedDRA terms could also be annotated by the CTCAE. We also merge all the results from different tasks into one set to make a coverage evaluation. A total of 200 unique CTCAE terms and 1482 MedDRA terms were found in all of three irAE mining tasks. The coverage evaluation shows that MedDRA could capture more irAEs terms that are not included in the CTCAE from unstructured irAE text. Rigorous evaluation of MedDRA text mining pipeline is beyond the scope of the paper and will be conducted in the future.

Figure 1.

CTCAE coverage rate for different text mining tasks (compared with MedDRA). 2a is the text mining result for drug labels; 2b is the result for publications; 2c is the result for FAERS-based signal detection; and 2d is the result for systematic evaluation.

Discussion

IrAEs have increasingly got a concern in clinical immunotherapy administration, especially for specifying and standardizing the severity scale. The CTCAE is a widely used terminology to assess and grade the irAEs in oncology research and evaluation. Although the CTCAE is a subset of MedDRA and only has 837 adverse event related terms, it provides detailed severity scale definition for every term which meets the needs of clinical application. However, due to the fact that irAEs are caused by some new agents in the market, CTCAE may not cover all the irAEs, and this will lead to the insufficient coverage of some irAEs and cause inaccurate clinical observational results. Therefore, it is important to evaluate the coverage of the CTCAE in capturing irAEs for the standardization purpose.

In this study, we respectively implement a manual evaluation and a systematic evaluation to assess the performance of the CTCAE based irAEs standardization. In manual evaluation, we found that many of irAEs are not included in the CTCAE by drug label manual review. The performance of text mining pipeline has been significantly improved when we added the manually identified terms into the dictionary, which also indicate that the CTCAE has to be extended with some more specific terms to satisfy the demand of irAEs standardization. In addition, the same irAE signal detection results reveal that the CTCAE could describe most of common irAEs. However, the text mining result in comparison with MedDRA-based pipeline shows that MedDRA may cover more irAEs. Note that MedDRA don’t provide the severity scale standard, so we think it may be necessary to add more terms and relevant severity level information to the CTCAE, for a better standardized description of irAEs.

We leveraged advanced text-mining technology for processing unstructured drug safety data. We noted that although the recall of our text-mining pipeline was improved significantly after enhancing the CTCAE dictionary with the irAE terms manually extracted from drug labels, some of the irAE terms could not be identified by the dictionary lookup method used in our pipeline. We have checked underlying reasons behind the unrecognized terms and found out that most of these unrecognized terms are related to the laboratory tests, including Aspartate aminotransferase increased, Alanine aminotransferase increased, Blood bilirubin increased, Blood thyroid stimulating hormone increased, Blood thyroid stimulating hormone decreased, Amylase increased, and Lipase increased. We believe that with this pattern identified, the issue can be fixed using a rule-based approach in the future update.

For the MedDRA-based text mining pipeline, we did not evaluate the baseline performance as the main purpose is to demonstrate that the coverage of the CTCAE for capturing irAEs is limited and additional efforts should be taken to produce a CTCAE extension with a relatively comprehensive list of irAEs coded in MedDRA. Therefore, despite the fact that we had limited evaluation on the text mining tool, the comparison of evaluation still demonstrated that the CTCAE needs an extension to meet the irAEs standardization task.

Conclusion

In this study, we implemented a number of text mining pipelines to assess the coverage of CTCAE for capturing irAEs for the purpose of standardization. We demonstrated that the CTCAE may satisfy the basic requirement for representing irAEs as illustrated in the FAERS signal detection, while CTCAE needs to be extended to cover more irAE terms to achieve the specific demand in advanced clinical research and application on irAEs.