A Role for Artificial Intelligence in the Detection of Immune-Related Adverse Events

Summary:

In the article that accompanies this editorial, Sun and colleagues utilized large language models (LLMs) to detect immune-related adverse events (irAEs) from electronic health records and demonstrated that LLMs had higher sensitivity than ICD codes alone (94.7% vs 68.7% respectively) and similar specificity, requiring a fraction of the time compared to manual adjudication. This research represents a significant step in enhancing the efficiency of our efforts to better understand irAEs by leveraging data in the electronic medical records from patients treated with immune checkpoint inhibitors over the last 15 years to better predict these toxicities, with the goal of minimizing or mitigating them entirely.

Despite recent advancements and uptake in the use of immune checkpoint inhibitors (ICIs) for patients with multiple cancer types and stages, ICIs can trigger immune syndromes called immune-related adverse events (irAEs). While these immune responses can impact any organ system, common irAEs include gastrointestinal, rheumatic, endocrine, and dermatologic.^1–3 In addition to adversely impacting patients’ quality of life, irAEs may result in treatment interruptions or necessitate prolonged courses of high doses of systemic corticosteroids or other immune-suppression therapy, which may have long-lasting effects. Some irAEs can cause chronic effects, such as pulmonary toxicities, with patients requiring life-long oxygen, and some irAEs can be fatal.^4,5 Research on potential risk factors for irAEs and strategies to mitigate or minimize their risk have been hampered by a lack of valid methods to accurately detect and characterize irAEs from medical records in real-world settings despite the significant risk they pose to patients. In addition, this detection gap prevents researchers from leveraging large data sets, which have been instrumental in leading to breakthroughs in other research fields, such as genomics.⁶

The complex nature of irAEs underscores the urgent need for a detailed understanding of their frequency and severity. This is especially crucial given the current limitations in detection and reporting tools. IrAEs can range from mild to moderate, but in severe cases, they may cause permanent organ dysfunction and exhibit acute, life-threatening manifestations.⁷ In a meta-analysis involving 45,855 patients, the risk of all-grade irAEs was 39.8%, with 14.9% of patients experiencing grade ≥3 irAEs.⁸ The rapidly expanding use of ICIs in combination with other cancer therapies – including other immune therapies, radiation, chemotherapy, targeted therapies, cellular therapies, and antibody-drug conjugates - introduces additional layers of complexity in reporting and managing irAEs. Due to potential drug-drug interactions, patients receiving combination therapies may experience overlapping toxicities from different treatment modalities, and parsing out attributions is even more challenging. This underscores the need for detection and classification tools to identify and distinguish between reversible and irreversible toxicities and those caused by overlapping toxicities. Integrating large language models (LLMs) offers a promising solution to these challenges by providing advanced tools for accurately and efficiently identifying and characterizing irAEs in electronic medical records (EHR) data.

In this JCO issue, Sun et al. conducted a comprehensive retrospective study to investigate the precision of LLMs with retrieval augmented generation (RAG) in identifying and characterizing the most prevalent irAEs among hospitalized patients undergoing ICIs therapy. As such, clinical records were processed through an LLMs pipeline augmented with RAG. The use of RAG enables the LLMs to search a user-provided database of unstructured text, such as clinical notes, for relevant information. This approach enhances the LLM’s ability to generate accurate responses by incorporating additional context from the retrieved data. This study was motivated by the time and resource-intensive requirements of manual adjudication and the limitations of existing methodologies, such as International Classification of Disease (ICD) codes in detecting irAEs in large EHR data. The authors conducted a comprehensive manual review and adjudication of hospital admissions for 3,542 patients receiving ICIs treatment between 2011 and 2023. By leveraging both ICD codes and LLMs the study team compared the performance of these methods against adjudication (i.e., by a dedicated team of physicians and medical students) as the gold standard.

Results from Sun et al. are important as they show that the LLMs had a significantly higher sensitivity in identifying irAEs compared to ICD codes, with an overall sensitivity of 94.7% versus 68.7% for ICD codes. In irAEs-specific analyses, LLMs showed statistically significant improvements in sensitivity estimates for ICI-hepatitis (p<0.001), myocarditis (p<0.001), and pneumonitis (p=0.003), compared with ICD codes. Notably, the significantly higher sensitivity results of the LLMs relative to ICD codes did not come at the cost of specificity rates, which were comparable between the two methods. External validation results using a second institution’s dataset further supported the study’s main findings, with the LLMs outperforming the ICD codes in sensitivity (98.1% vs. 78.6%) and specificity (95.7% vs. 92.9%). The authors also observed an additional advantage of the LLMs approach ─ efficiency. In this study, the LLMs method required an average of 9.53 seconds per chart review, which was significantly less than the estimated average of 15 minutes needed for manual adjudication. The observed efficiency combined with high sensitivity and specificity underscores the potential of LLMs as a valuable tool in detecting irAEs.

This work is highly impactful because a reliable collection of irAEs data by manual review is onerous, time-consuming, and subjective even when reviewed by trained specialists. We and others have shown that relying on ICD codes and billing documentation is fraught with error^9–14; therefore, having access to extensive data with a systemic reliable method for irAEs identification would be of great utility to advance research. While the study by Sun et al. provides promising findings, some limitations warrant mentioning. The study employed a retrospective design and was conducted at institutions that both participated extensively in early clinical trials in immunotherapy and, therefore, may have more familiarity with irAEs – including a dedicated immune toxicity consultation service - which might affect the generalizability of the results. In addition, the validation analysis was performed using data from an external institution that is part of the same health system where the primary analysis was conducted. The LLMs demonstrated low positive predictive values for irAEs, which indicates high false positive rates. High false-positive rates will limit the reliability of using LLMs as results must be confirmed by supplementary review. The LLMs were only limited to detecting the presence of irAEs and lacked any assessment of severity, which limits the model’s ability to provide a comprehensive toxicity assessment. Moreover, the data utilized were only from inpatient records. Ideally, we would be able to identify irAE across the care continuum, especially in the outpatient setting, where early identification and treatment might prevent hospitalization entirely. Finally, the study focused on four irAE (colitis, hepatitis, pneumonitis, myocarditis), so it is not clear how the LLMs would perform in detecting other irAEs, such as neurotoxicities, which are rare but often life-threatening, or rheumatic irAEs, which are common but fraught with misdiagnosis.¹⁵

irAEs are often inadequately and inconsistently reported^10,11 due to complex data structures, the need for manual adjudication, and the use of ICD codes. In addition, the complexity and variability of irAEs and the lack of standardized irAEs-specific codes ICD contribute to their identification and reporting challenges.^7,9 Currently, ICD codes for autoimmune disease and drug-induced toxicities are often used to report irAEs in EHR data.¹⁶ The problem is further compounded by the general underrepresentation of true irAEs and overrepresentation of false irAEs when relying solely on ICD codes, as we have previously shown.¹⁷ As a result, current irAEs reporting practices may impede precise attribution and identification of irAEs and can lead to gaps in both patient care and future oncology research efforts. Notably, due to important advocacy work by the irAEs Consortium, ASPIRE, new ICD-10 modifier codes are expected to be released in October 2024, which is a significant and critical step forward (Table 1).¹⁶ However, concerns remain about the future uptake and utilization of the new codes. Additionally, these new codes will not help identify irAEs during the first 15 years of routine ICIs use, so tools such as the one developed by Sun et al. will remain important for learning from patients treated during this period. Additional systemic infrastructure building and investment are certainly warranted to ensure the ongoing safety of our patients being treated with ICIs, especially as these therapies enter the neoadjuvant and adjuvant settings where patients have undergone curative therapy.

Table 1.

New ICD-10 Codes Immune-Related Adverse Events

ICD-10	Description
Z92.26	Personal history of immune checkpoint inhibitor therapy
T45.AX5	Adverse effect of immune checkpoint inhibitors and immunostimulant drugs
T45.AX6A	Adverse effect of immune checkpoint inhibitors and immunostimulant drugs, initial encounter
T45.AX7D	Adverse effect of immune checkpoint inhibitors and immunostimulant drugs, subsequent encounter
T45.AX8S	Adverse effect of immune checkpoint inhibitors and immunostimulant drugs, sequela

^*To become Effective October 1^st, 2024

ICD-10 = International Classification of Disease 10^th Edition

Integrating large LLMs-based advanced detection tools into clinical practice presents a transformative potential for both irAEs management and oncology research. In clinical trials, LLMs-based tools can help precisely identify individuals with previous irAEs and ensure appropriate participant selection. LLMs can also be employed for real-time continuous surveillance and early detection of irAEs during follow-up. This proactive approach could facilitate intervention and tailored treatment adjustments based on patient responses, reducing the burden on patients by minimizing the need to collect patient-reported outcomes and for researchers and investigators in terms of data capture and documentation. Additionally, this approach can improve data quality and consistency through standardized assessments and reporting. Automating data review processes has the potential to reduce expenses, accelerate trial timelines, enhance efficiency, and boost cost-effectiveness.

The creation of infrastructure to implement LLMs identification and classification tools in outpatient clinics is crucial. Continuous monitoring in outpatient settings can aid in the early detection of irAEs and timely intervention. To ensure the practical applications of these models outside academic centers, addressing challenges such as infrastructure, safeguarding data privacy, integrating with existing systems, and gaining support from healthcare providers will be crucial.

In conclusion, the study by Sun et al. highlights the transformative potential of LLMs in detecting irAEs among hospitalized patients. Their findings demonstrate that LLMs significantly outperform traditional ICD codes’ sensitivity and efficiency, offering a promising solution for accurate irAEs identification. Despite limitations such as high false positive rates and a lack of severity assessment, the integration of LLMs into clinical practice can enhance patient selection, real-time monitoring, and data standardization in clinical trials. Addressing current detection and reporting challenges, LLMs provide a robust framework for improving patient care and advancing oncology research in managing irAEs.