To Adjudicate or Not Adjudicate

Corresponding Author

Immune checkpoint inhibitors (ICIs) are antibodies that target the programmed death receptor-1 (PD-1), programmed death-ligand 1 (PD-L1), and cytotoxic T-lymphocyte antigen-4 (CTLA-4) to treat a variety of tumor types by activating the patient’s immune response. Clinically significant immune-mediated adverse reactions have been observed in trials evaluating ICI therapies indicated for the treatment of hematological and solid tumor malignancies. The Warnings and Precautions section of the US Food and Drug Administration (FDA) Prescribing Information for ICIs describes the risks of severe and fatal immune-mediated adverse reactions and other clinically significant immune-mediated adverse reactions occurring at an incidence of <1%, including cardiac/vascular reactions such as myocarditis, pericarditis, and vasculitis.1, 2, 3, 4, 5, 6, 7, 8 Although immune-mediated cardiac and vascular toxicities are observed infrequently in clinical trials, events can be severe or fatal.1, 2, 3, 4, 5, 6, 7, 8, 9 The FDA Prescribing Information recommends that patients who develop grade 2, 3, or 4 myocarditis based on National Cancer Institute Common Terminology Criteria for Adverse events permanently discontinue ICI therapy. Mortality has been reported in up to 50% of patients receiving ICI therapy who develop myocarditis.^9,10 Although there has been increased reporting of myocarditis with the FDA’s voluntary adverse event reporting system in the postmarketing setting,⁹ the true rate of this adverse event in the real-world setting is unknown. Some data suggest that the frequency of myocarditis ranges from 0.25% to 2.48%.9, 10, 11, 12 The risk of other cardiovascular (CV) events in ICI-treated patients with and without underlying CV disease, including major adverse cardiovascular events, such as death, nonfatal myocardial infarction (MI), and nonfatal stroke, is unknown.

In this issue of JACC: CardioOncology, Kondapalli et al¹³ performed a retrospective observational study using electronic medical record, provider billing, and state and public data sources for all patients 18 years of age or older who received ICI therapy within the University of Colorado Health System from January 2011 to April 2019. The objective of this study was “to determine CV event occurrence in ICI-treated patients and to assess diagnostic accuracy by ICD code compared with adjudication utilizing established definitions and full source documentation review.” Using International Classification of Diseases (ICD)-9 and -10 codes and Medication Epic Identifier, Kondapalli et al¹³ captured medical history and ICI use, respectively. CV events of interest were prespecified and included MI, hospitalization for unstable angina (UA), hospitalization for heart failure (HF) or HF exacerbation requiring treatment, transient ischemic attack (TIA), stroke, hypertensive emergency, noncoronary (peripheral) vascular events, and venous thromboembolism (ie, either deep vein thrombosis and/or pulmonary embolism). Two cardiologists independently reviewed potential CV events identified by ICD code and related electronic medical record data using standardized definitions. Discrepancies in event adjudication were resolved by consensus. Events were categorized as pre-ICI, if they occurred before the first dose of ICI and as post-ICI, if they occurred on or after the day of the first dose of ICI.

The analysis cohort included 1,813 patients with a mean age of 62.5 ± 13.5 years. At baseline, approximately 48% of patients had hypertension, 16% had diabetes mellitus, 17% had prior coronary revascularization, and 11% were current smokers. ICI therapy had been administered to treat several hematological and solid tumor malignancies.

The investigators reported crude rates of adverse events because there were no statistical adjustments for competing risks or differential follow-up. Venous thromboembolic events (VTEs) were the most common CV event in this study. The 2 reviewers adjudicated VTE in 206 patients (11.4%) pre-ICI and 205 patients (11.3%) post-ICI therapy, including pulmonary embolism in 89 patients (4.9%) pre-ICI and 97 patients (5.4%) post-ICI therapy, and deep vein thrombosis in 146 patients (8.1%) pre-ICI and 48 patients (8.2%) post-ICI therapy. The reviewers also adjudicated MI in 33 patients (1.8%) pre-ICI and 54 patients (3.0%) post-ICI therapy, HF in 40 patients (2.2%) pre-ICI and 50 patients (2.8%) post-ICI therapy, stroke in 33 patients (1.8%) pre-ICI and 29 patients (1.6%) post-ICI therapy, and myocarditis in 1 patient (0.06%) pre-ICI and 6 patients (0.3%) post-ICI therapy. In addition, the reviewers adjudicated hypertensive emergency in 1 patient (0.06%) pre-ICI and 3 patients (0.2%) post-ICI initiation, and noncoronary vascular events in 5 patients (0.3%) pre-ICI and 2 patients (0.1%) post-ICI therapy. Post-ICI therapy, the reviewers adjudicated 1 event of TIA and 0 events of UA. A total of 954 patients (53%) died. See the Supplemental Appendix¹³ for crude events rates of adjudicated CV events pre-ICI initiation and for the number of adjudicated events of arrhythmias and pericardial diseases.

With respect to myocarditis, ICD codes and adjudication identified 10 (n = 1 pre-ICI and n = 9 post-ICI therapy) and 7 (n = 1 pre-ICI and n = 6 post-ICI therapy) events, respectfully. One of the myocarditis events that had been identified by ICD code as occurring post-ICI was adjudicated as a pre-ICI event because the patient had experienced viral myocarditis 5 years before being diagnosed with cancer.

Table 1 summarizes kappa coefficients and 95% CIs for the comparison of CV events identified by ICD code and those adjudicated through chart review.

Table 1.

Comparison of CV Events Identified by ICD Code and Adjudication

Event	Kappa Coefficient (95% CI)
Venous thromboembolic event	0.82 (0.79 to 0.85)
Myocardial infarction	0.74 (0.66 to 0.82)
Myocarditis	0.50 (0.20 to 0.80)
Heart failure	0.47 (0.40 to 0.54)
Hypertensive emergency	0.23 (0.01 to 0.44)
Transient ischemic attack	0.12 (0.00 to 0.24)
Unstable angina	0.08 (−0.06 to 0.22)

Byrt¹⁴ described interpretive categories of kappa as follows: 0.93-1.00: excellent agreement; 0.81-0.92: very good agreement; 0.61-0.80: good agreement; 0.41-0.60: fair agreement; 0.21-0.40: slight agreement; 0.01-0.20: poor agreement; ≤0.00: no agreement.

CV = cardiovascular; ICD = International Classification of Diseases.

Based on the kappa statistics, the investigators concluded that ICD codes correlated well with adjudicated events for VTE and MI, but not for HF and other events such as myocarditis where codes are less specific and adjudication may be more useful.

This study has some important limitations. In addition to the limitations acknowledged by the investigators, ICD coding may not have identified all potential CV events. Based on the design of the study, the adjudicators would be able to identify only CV events that were misclassified, but not necessarily new cases that were missed. The result is an underestimation of the background rate of events. Case #3 in Table 3¹³ underscores this point because it appears that this patient may not have received timely and appropriate diagnosis and treatment. This patient experienced chest pain and underwent a thoracentesis with resolution of symptoms, but inpatient serial troponin monitoring was consistent with a non–ST-segment elevation MI. Appropriate evaluation, diagnosis, and management of patients with cardiac symptoms is critical. The number of adjudicated myocarditis, hypertensive emergency, TIA, and UA events is also small. Therefore, the point estimates for correlation lack precision, and given the single health system analysis, these findings have limited generalizability. In addition, with 1 exception, this paper presumes that the myocarditis events that were adjudicated were related to ICI therapy,¹³ but sufficient information is not provided to make this determination. The fact that the analysis population consists only of patients who had received ICI therapy also means that the reviewers were not blinded to treatment allocation, which may introduce bias into the review process. In addition, although adjudication can be extremely useful in clinical trials and has resulted in clinically meaningful differences in event rates in some trials,¹⁵ there is no “truth” or gold standard. Adjudication has its own limitations and is largely dependent on the information shared with the clinical events committee. Finally, to determine whether adjudication would lead to a difference in outcomes compared with investigator-reported events, a randomized prospective trial in which the adjudication plan, trigger terms to identify potential events, and event definitions are prespecified, would be the best approach to avoid post hoc adjudication of selected events identified by ICD code. Comparing investigator-reported and adjudicated events to ICD coding diagnoses in such trials could also be useful.

In summary, although ICD codes can be used to identify potential CV events, and post hoc adjudication may be helpful in evaluating potential HF events and immune-mediated events such as myocarditis, adjudication may be most useful in the clinical trial setting in which a plan is prospectively specified with well-defined criteria and where clinical events committee queries can be addressed with additional information from the study site when needed.

Funding Support and Author Disclosures

This editorial reflects the views of the authors and should not be construed to represent the views or policies of the U.S. Food and Drug Administration. The authors have reported that they have no relationships relevant to the contents of this paper to disclose.