Navigating the complexities of AI-enabled real-world data collection for oncology pharmacovigilance

The ubiquitous uptake of electronic health records (EHRs) in the United States, combined with advances in artificial intelligence (AI), presents new opportunities to leverage real-world data (RWD) - collected as a part of routine clinical care - to complement clinical trials for oncology pharmacovigilance. Clinical trials are an essential component of drug safety evaluation but have inherent limitations that make them a necessary but insufficient data source for cancer drug safety. ^1–3 Trials generally include selective populations that are not representative of many patients seen in cancer clinics who may be more susceptible to adverse events (AEs), and do not capture the full range of clinical practice.^{4, 5} Further, trials in the United States predominantly include white patients and individuals with a higher socioeconomic status, which may affect the generalizability of results.^6–8 Clinical trials may also underestimate the actual long-term burden of AEs due to incomplete reporting³ and limited or delayed reporting timelines. RWD in EHRs can provide more comprehensive evidence for drug safety and support a learning health system with real-time AE monitoring, thereby improving cancer treatment outcomes.⁹ Here, we discuss how AI, especially natural language processing (NLP), can help realize this potential and discuss the complexities of appropriate development of these AI-enabled methods. We propose the “Three Ps” framework - Processing, Pipelines, and Patient Outcomes - to inform effective pharmacovigilance drug safety measures incorporating AE information from EHR text (Table 1).

Table 1.

Overview of the Three Ps Framework for Language Model-Based Pharmacovigilance

Category	Element	Description
Processing	Definitions	Clear definition of AE information being extracted, and whether and how attribution to a treatment is defined.
	Accuracy	Precision and recall (sensitivity) of the model in identifying and extracting target AE information.
	Adaptability	Capacity of the model to incorporate new information and adjust its analysis accordingly.
	Annotation Guidelines	Development and release of clear annotation guidelines describing how AE information is labeled for model development and evaluation; inter-annotator agreement to report data labeling quality and d reproducibility.
	Code and Data Release	Sharing of code and data for verifiability and interrogation of findings.
Pipelines	Data Coverage	Comprehensive coverage of diverse data types and formats in EHRs to avoid gaps in data representation.
	Data Quality	Mechanisms and metrics to identify, quantify, and correct errors, inconsistencies, and biases in data to enable accurate, equitable causal inferencing.
	Data Standardization	Consistent harmonization and standardization strategies for comparing and merging unstructured and structured data.
	Systematic Reporting	Clear reporting of results at each stage of the pipeline and for the overall pipeline.
	Training and Validation Datasets	Reporting the number of notes for each split and breakdown according to AE type.
	Data Labeling	Documentation of labeling methods, inter-annotator agreement (for human labeling), and validation of automated labeling methods.
	Dataset Characteristics	Details about the demographics and clinical characteristics of the training and validation datasets.
Patient Impact	Early Detection	Capacity to enable early AE identification for timely intervention and management.
	Accurate Diagnosis	Capacity to enable correct AE diagnosis for correct specialist referral and management.
	Treatment Optimization	Potential of insights from methods to inform personalized treatment plans to reduce AE risk and enhance effectiveness.
	Quality of Life	Potential of insights from methods to minimize the impact of AEs on patients’ quality of life.
	Health Equity	Potential of insights to provide an accurate understanding of AE risk across patient groups, especially vulnerable populations.

Abbreviations: AE, adverse events.

The systematic and timely identification of real-world AE rates in EHRs are limited by their documentation in unstructured EHR text, requiring labor- and expertise-intensive manual review and abstraction of events. Relying only on structured data that can be directly abstracted, such as billing codes and laboratory data, often underestimates AEs, especially mild-moderate AEs, that can impact patients’ quality-of-life and ability to stay on an otherwise effective cancer treatment.^10–13 Furthermore, structured data are often under-specified and cannot provide the level of detail about severity, causality, and temporal trends necessary to inform clinical decision-making processes and carry out quality RWD studies.

To overcome the bottleneck of manual EHR curation for AE pharmacovigilance, Barman et al. report the results of NLP methods to automatically extract immune-related AE (irAE) occurrences from unstructured text. NLP is a field of AI that enables computational processing of human language for various downstream tasks, including automated information extraction. In recent years, a class of deep learning models called Transformer language models ^{14, 15} has gained traction across the field and now forms the backbone for many NLP methods. Barman et al. fine-tune a Transformer-based language model for the automated curation of clinical notes for irAE detection and compare the rates of irAEs detected by the language model versus a set of pre-defined billing codes. There was low agreement between myocarditis, encephalitis, pneumonitis, and severe cutaneous AEs identified by language model and structured data methods.

This study falls within a broader literature showing the promise of NLP for detecting AEs, including irAEs, in EHRs ^{12, 13, 16–24} and highlights the challenges and complexities of AI-augmented EHR pharmacovigilance. In this study, irAE rates were much lower than reported elsewhere; for example, their overall irAE rate of 20.8% is lower than the aforementioned rates in RWD studies and clinical trials, where they have been reported to impact up to 80% of patients.^24–26 Similarly, most studies report immune-related pneumonitis rates of 10–20%,^27–30 compared to 2.9% in this study. These discrepancies underscore the challenges of shifting from easy-to-measure and available structured data that likely suffer in sensitivity and specificity, to incorporating information in unstructured text that requires manually labeled ground truths to guide learning strategies. One potential explanation for the lower rates of irAE is the data labeling method for fine-tuning the NLP model. Here, automated methods were used to identify irAE-containing text, which was subsequently used for model development. The irAE rates highlight the potential ramifications of using non-expert verified labels for model development. Training and evaluating NLP-based methods on unreliable ground truths is an ongoing challenge in the field, and best practices still rely on significant manual annotation.

This study also touches on the challenge of attributing an AE to its inciting agent, i.e., causality extraction. This is essential in oncology, where most patients are exposed to varying combinations of therapies, and determining the causative agent is often a diagnostic challenge. While some AEs can be definitively attributed via biopsy, this is not frequently done in practice, and many others have no pathologic gold standard to determine causality. This leads to a reliance on supporting clinical data and judgment, both of which will be documented to varying degrees. Consequently, the accuracy and utility of NLP models in this context are inherently limited by the quality of the underlying documentation for direct attribution. As an alternative to relying on clinician judgment, statistical causal inference can be used to establish attribution from RWD,^{31, 32} which requires detailed, granular extraction of AEs, treatments, relative timings, outcomes, and other potentially contributing clinical and demographic factors – and an accounting for biases and inequities in healthcare delivery, as described below.

The Three Ps Framework for NLP-Enhanced Pharmacovigilance

Leveraging NLP to mine unstructured EHR text for AE information holds potential to improve and potentially automate pharmacovigilance for cancer care. The “Three Ps” framework may guide considerations when developing methods that incorporate information extracted from text (Table 1):

• Processing: Development and evaluation of NLP methods for EHR text processing that ensures consistency, reproducibility, and verifiability of AE findings.

• Pipelines: Considerations for combining language model-extracted AE information with other EHR sources for comprehensive data coverage, quality, standardization, interoperability, and systematic reporting.

• Patient Outcomes: Considerations for developing methods that align with the ultimate goal of pharmacovigilance: improving patient care.

Future directions for AI-enabled pharmacovigilance

Recent advancements in language models have led to the most current generation of large language models (LLMs), which may be able to make predictions without task-specific training examples or only a few examples to guide the model - diverging from traditional methods that require larger labeled datasets. If successful, these models may reduce time and resource constraints associated with data annotation. At present, specialized fine-tuned language models still outperform generalist LLMs for most specialized tasks, including causality extraction,^33–35 although, with continued advances, this new paradigm might catalyze advances in EHR-based pharmacovigilance if evaluated and implemented appropriately.

In the future, AI models that incorporate multi-modal EHR data, such as text, labs, vitals, imaging, and pathology, may improve AE diagnosis and attribution. This may strengthen clinical evidence, drive translational research, and provide real-time diagnostic decision support. Similarly, by taking full advantage of all data within a patient’s EHR, such models could provide more consistent and standardized severity grading, a challenge with manual abstraction³⁶.

Once EHR extraction processes and pipelines are developed, new methods and data-sharing approaches are needed to take full advantage of RWD while addressing its biases and limitations for inference. Information about AEs and other clinical factors that may contribute to them, including demographics, cancer diagnosis, comorbidities, social determinants of health, treatment details (e.g., dosage, timing, and route), and AE-directed treatment, is often documented over long time periods, by multiple providers across different institutions. Data silos, closed EHR systems, and variability in what healthcare providers choose to document can lead to incomplete documentation within a single healthcare system.³⁷ Efforts to generate findable, accessible, interoperable, and reproducible (FAIR) data,³⁸ including the adoption of consensus data standards,^39–42 will also be imperative to overcoming these obstacles. Some data models are beginning to include AE elements,⁴⁰ but more work is needed to expand them to comprehensively capture AE information about severity, causality, and timing. In parallel, validated measures of RWD quality, including uncertainty, chart completeness, and documentation bias, are urgently needed for successful and safe implementation.

Finally, fairness and equity will need to be considered in the design of any pharmacovigilance system, especially those that will be AI-assisted. AEs may not be distributed equally, and the only way to understand the risks and benefits of treatments in diverse populations is to prioritize the representativeness of the collected data. This includes evaluation and monitoring of performance across different patient groups and the design of systems that can be widely implemented at institutions with varying resource capacities. Relatedly, any pharmacovigilance system design must have a causal framework backbone that considers left and right censoring. For example, patients with worse outcomes due to social determinants of health or social determinants of care are given less opportunity to report AEs.⁴³ Further, they are more at risk for competing events such as death,^44–46 and patients who are not offered treatment are not included in the denominator population for AE reporting.

In conclusion, integrating AI, particularly NLP methods, with EHRs is a significant opportunity to advance oncology pharmacovigilance, which could improve treatment outcomes and cancer patients’ quality of life. Appropriate development, evaluation, and reporting will be essential to ensure that automated methods accurately identify and estimate AE rates so that we realize the full benefit and avoid the harms of AI-augmented pharmacovigilance.