Improving Pharmacovigilance Signal Detection from Clinical Notes with Locality Sensitive Neural Concept Embeddings

Abstract

Although pharmaceutical products undergo clinical trials to profile efficacy and safety, some adverse drug reactions (ADRs) are only discovered after release to market. Post-market drug safety surveillance – pharmacovigilance - leverages information from various sources to proactively identify such ADRs. Clinical notes are one source of observational data that could assist this process, but their inherent complexity can obfuscate possible ADR signals. In previous research, embeddings trained on observational reports have improved detection of such signals over commonly used statistical measures. Moreover, neural embedding methods which further encode juxtapositional information have shown promise on analogical retrieval tasks, suggesting proximity-based alternatives to document-level modeling for signal detection. This work uses natural language processing and locality sensitive neural embeddings to increase ADR signal recovery from clinical notes, with AUCs of ~0.63-0.71. Constituting a ~50% increase over baselines, our method sets the state-of-the-art for these reference standards when solely leveraging clinical notes.

Introduction

While the aim of pharmaceutical intervention is to ameliorate or remedy medical issues in patients, such interventions come with risk of side-effects, or adverse drug reactions (ADRs). With nearly half of the US population - and nearly ninety percent of those over the age of 65 - utilizing at least one prescription drug every year¹, significant economic and health burdens are associated with unintended effects of medication therapy, even when these are relatively rare. Limiting these adverse effects could both save billions in healthcare expenditure² and reduce unnecessary patient morbidity and mortality^3,4.

Pharmacovigilance systems that surveil on-market drugs or therapeutic biologic products for new or emerging safety concerns, seek to reduce ADRs by actively monitoring sources of observational data, such as the FDA Adverse Event Reporting System (FAERS)^5–8. Recently, multi-modal approaches that integrate numerous data sources, such as the literature or claims data, have seen increased investigation^9–15. Clinical notes in electronic health records are one source of potentially voluminous and rich ADR signal, containing documentation of patient conditions and treatments, including ADRs and medications captured in real time^14,16 and without the reporting biases that have been documented with adverse event report data^17,18. Despite this richness, use of clinical notes for pharmacovigilance faces many challenges. Their ever-growing volume necessitates scalable computational models while their inherent complexity as natural language makes computational modeling challenging¹⁶. Clinical notes contain narratives generated by healthcare professionals primarily in support of patient care by humans and not primarily for secondary informatics research. In contrast to FAERS reports, which are intended to facilitate secondary research in pharmacovigilance and report drugs based on their suspicion of causing an ADR, drugs and problems may occur in the same clinical note for many reasons, including direct therapeutic relationships or therapeutic relationships with comorbidities. Consequently, drug-ADR signals can be obfuscated by noisy signals unrelated to drug safety. Clinical natural language processing (NLP) is a burgeoning field aimed at addressing these challenges and enabling downstream secondary research^19,20.

One way to limit the complexity of unstructured clinical text data is to extract clinical concepts of interest using NLP methods (e.g., named entity recognition and normalization to concepts in a vocabulary)²¹. This does not provide estimates of association per se but does restrict the complexity and vocabulary involved, facilitating downstream analysis. While this alone could enhance signal detection for ADRs compared to unprocessed clinical text, the choice of signal detection methodology can be critically important. In the realm of pharmacovigilance, signal detection is done by calculating discrete statistical measures, such as the Proportional Reporting Ratio (PRR)²² or Reporting Odds Ratio (ROR)²³, which are known as disproportionality metrics. These measures reveal signals by calculating the relative abundance of report-level co-occurrence for concepts or terms of interest, such as a drug and ADR. However, such measures do not model the similarity between drugs or side-effects - each drug and ADR are considered independently and discretely based purely on co-occurrence between queried terms. Consequently, these methods cannot draw associations between drugs with similar actions (such as members of the statin family), or side-effects with related pathophysiological mechanisms (such as myocardial infarction and stroke). One family of methods which do infer and encode similarity based on shared contexts is neural embedding approaches, as popularized by the skipgram-with-negative-sampling (SGNS) architecture provided in the widely-used word2vec software package^24,25.

In pharmacovigilance, a similar neural embedding technique, Portanova et al’s aer2vec²⁶, was found to outperform discrete statistical measures on recovery of curated drug-ADR associations when trained on FAERS reports^11,26. A distinguishing feature of the aer2vec approach vis a vis typical word2vec style models is the recapitulation of the training objective at test or query time. In typical word2vec approaches using SGNS, input weights and output weights of a neural network are updated during training to optimize predicting context terms given an observed term as in the optimization objective:

$$argmax(O,C){\displaystyle \sum _{(o,c)\in D}\log \sigma (\stackrel{\rightharpoonup }{o}\cdot \stackrel{\rightharpoonup }{o})}+{\displaystyle \sum _{(o,\neg c)\in D\text{'}}-\log \sigma (\stackrel{\rightharpoonup }{o}\cdot \neg \stackrel{\rightharpoonup }{o})}$$

where O and C represent observed (input) term and context (output) term weight matrices, where $\stackrel{\rightharpoonup }{o}$ and $\stackrel{\rightharpoonup }{c}$ represent single vectors of O and C respectively, drawn from the set D of observed term-pairs o,c; σ represents the sigmoid function; and $\neg \stackrel{\rightharpoonup }{c}$ represents a random negative sample vector of C drawn from other contexts not likely occurring in the set D (that is, the unobserved context set D’)¹. Subsequent analysis is performed using the trained input weights only, with the output weights c discarded. In contrast, with aer2vec models the output weights are also retained, and at query time the sigmoid of the dot product between the input weight for a given observed term and the output weight for the suspected context term is calculated. This not only recapitulates the training objective but also provides a probabilistic interpretation of the association between the query terms. Such a formulation satisfies via probabilistic approximation queries of the form: what is the probability given a query term, that I would expect to observe a given second query term in its context? Such queries lend naturally to the field of pharmacovigilance; given some set of observational data, what is the probability of observing a drug in the context of a specified ADR? More importantly, such embedding techniques produce similar probabilities for these queries for analogous drugs and analogous ADRs. This is encoded during the training process, with entities sharing similar contexts having similar representations – a distinguishing feature from discrete statistical approaches.

One important modeling decision in embedding models is how to define context. In the case of aer2vec, context is defined at the document level. If an ADR occurs in a FAERS report, it will be used to predict all drugs in that report². However, FAERS reports do not contain natural language of the form seen in clinical narratives, where the order of the entities as they appear in context could be informative and important. That is, it may not be advantageous in a clinical narrative for a given drug to inform the prediction of every other entity within that narrative. Instead, it may be advantageous to examine defined, locally constrained context windows, as is the case with standard SGNS models that operate on a sliding window basis²⁴. In such a case, context may be defined by a narrow window of terms on either side of an observed term (e.g., a window radius of 5 around the observed term, for a total window size of 10 plus the observed term). We hypothesized that this narrower definition of context would constrain the associations learned by embedding models to emphasize those of greater utility for pharmacovigilance. Moreover, the relative position of juxtaposed terms within a context window could add additional information, as was observed on general domain tasks in Cohen and Widdows’ recent work²⁷. The derived model, Embeddings Augmented by Random Permutations (EARP), has several variants, each exploiting some element or combination of relative direction and distance from an observed term. That is to say, EARP variants encode not just if a term is likely to appear in a given context window with an observed term, but also what direction (to the left or right) and/or distance away from that term might a query term be most likely to appear within that context window. In Cohen and Widdows’ experiments, this improved performance on analogical reasoning tasks; clinical narratives may contain ordered information (e.g., it may be more likely for a drug to precede the utterance of an associated ADR) which could likewise see improved signal detection of drug-ADR associations.

In this paper, we propose an innovative computational method to recover ADR signal from NLP processed clinical notes. We hypothesized that extracting relevant concepts using an established clinical NLP tool would allow for recovery of drug-ADR signals with discrete statistical methods, more so with aer2vec models on account of their encoding of distributional similarity, and even more so when including proximity and direction-based features by generating locality sensitive neural concept embeddings with SGNS and EARP variants. Moreover, we deploy these variants in a novel way, reprising the training objective at query time as with previous aer2vec evaluations. In doing so we provide answers to fundamental questions about how best to model context when representing clinical notes for pharmacovigilance signal detection.

Methods

The data used in this analysis was a subset derived from The University of Texas Health Science Center at Houston (UTHSC-H) clinical data warehouse, with clinical notes spanning 2004 to 2015 collected at UT Physicians clinics and comprising a total of 4.67 million notes^28,29. This study has been approved by the Committee for the Protection of Human Subjects (the UTHSC-H IRB) under protocol HSC-SBMI-13-0549²⁹. The text from these clinical notes was processed with the Clinical Language Annotation, Modeling and Processing (CLAMP) NLP tool, a tool which has won numerous clinical NLP challenge tasks^30,31. We used the “run_comprehensive_pipeline” script included with the software package. Among other output, this produces structured data with semantic annotations for named entities and concepts. For our analysis, we retained only those records which contained at least one concept tagged as “problem” and at least concept tagged as “drug,” resulting in a final collection of 2,545,152 clinical notes. Additionally, for each of these tagged entities extracted by the CLAMP tool, we retained their Unified Medical Language System (UMLS)³² concept unique identifiers (CUIs) for problems or their RxNorm RXCUIs^33,34 for drugs. CLAMP additionally extracts these concepts in order of appearance in the clinical narrative, allowing for an ordered list of problem (e.g., ADR) CUIs and drug RXCUIs. For statistical (ROR/PRR) and note-level embedding (aer2vec) procedures, we took the additional step of ensuring that each record had a unique list of CUIs of both types by removing duplicates to mirror the adverse event reports these methods are intended to model (which generally do not contain repeated mentions of the same drug or ADR within a single report).

For comparison to typically deployed statistical methods in pharmacovigilance we utilized the PRR and ROR, methods which have seen use by the European Medicines Agency (EMA), Lareb, the FDA, and others^35,36. Additionally, we generated note-level embeddings using aer2vec. Note-level embeddings were trained by treating each note as a bag-of-words and training a model to predict every drug in a note given each observed problem. This model, but with reports rather than notes as the unit of analysis, was deployed in previous work leveraging FAERS reports and performed better on the reference standards (subsequently described) than PRR and ROR methods using that data source²⁶. Both the PRR and ROR statistics as well as the aer2vec embeddings were generated with the Semantic Vectors (SV) Java package^37,38. Aer2vec embeddings were trained similarly to previous work, with 5 negative samples, 5 training epochs and real valued vectors of 200 dimensions.

To process the ordered list of extracted concepts with locality sensitive embeddings, SGNS and EARP embeddings were utilized via the SV package. While SGNS uses a sliding window to restrict training context to a given radius, it does not encode the relative direction or distance from an observed concept within a given window. In contrast, EARP variants seek to encode that directional and proximity specific information. Directional (EARP_dir) and proximity based (EARP_prox) variations were used. In brief, EARP_dir and EARP_prox both consider a window of several terms around a given observed concept in context as in SGNS, but additionally seek to encode information about the direction (EARP_dir) or the direction and proximity (EARP_prox) of the context terms relative to an observed term. Encoding of this positional or locality specific information was accomplished via the use of permutations (operators that shuffle vector coordinates) in both EARP variants used in this analysis. Separate sets of vectors (or neural weights) are kept for observed terms and context terms, which the SV software denotes as embedding and elemental vectors, respectively. Permutation vectors are also kept which distinguish left of an observed term or right of an observed term for EARP_dir, and direction and distance to left or right from an observed term for EARP_prox. During training, EARP_dir optimizes the sigmoid of the scalar dot product between the embedding vector for the observed term permuted by the left or right permutation vector (whichever is applicable for the given context term) and the context vectors occurring within its window radius, as in the optimization objective:

$$argmax(O,C){\displaystyle \sum _{(o,c:p)\in D}\log \sigma (\stackrel{\rightharpoonup }{o}\cdot {\Pi }_p\stackrel{\rightharpoonup }{o})}+{\displaystyle \sum _{(o,\neg c:p)\in D\text{'}}-\log \sigma (\stackrel{\rightharpoonup }{o}\cdot {\Pi }_p\neg \stackrel{\rightharpoonup }{o})}$$

where O represents the weight matrix for the observed terms (the input weights) with $\stackrel{\rightharpoonup }{o}$ a vector of O, C represents the weight matrix for context terms (the output weights) with $\stackrel{\rightharpoonup }{c}$ a vector of C, and p represents the position of c relative to o, in the set D of observed positional term-pairs o,c; σ represents the sigmoid function; Π represents the permutation associated with position p; and $\neg \stackrel{\rightharpoonup }{c}$ represents a random negative vector of C drawn from other contexts not likely occurring in the set D (that is, D’). EARP_prox is the same, only the permutation encoding the direction and distance from the observed term is utilized instead of one just encoding relative direction. These permutations are deliberately constructed such that similarity as measured by the scalar product will degrade gracefully as the sliding window is traversed. For example, with x as a random vector and Π_p the permutation for position p, Π₁(x) • Π₂(x) > Π₁(x) • Π₃(x). Figure 1 depicts the difference for an example text window and specific observed term within that window. Additional details on EARP and its variants can be found in Cohen and Widdows²⁷ for the interested reader. In every case, models were trained with 5 negative samples, a window radius of 5, 5 training epochs and real valued vectors of 200 dimensions.

Figure 1.

Example schematic depicting the process for optimizing the representation of a given observed concept based on the embedding methods deployed: aer2vec, SGNS, EARP_dir, and EARP_prox. In all cases, the prediction of nearby terms is done using the sigmoid of the dot product between the observed concept (permuted if applicable) and the applicable context terms. In this study, concepts would be replaced with UMLS CUIs for ADRs and RxCUIs for drugs as extracted by the CLAMP tool from clinical notes but are presented here in human readable form.

The ability of each method to recover ADR signal was measured by the ability to rank positively labeled drug-ADR pairs before negatively labeled pairs using Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) and the Average Precision (AP) score using two commonly used ADR reference standards. The first reference standard is the Exploring and Understanding Adverse Drug Reactions (EUADR)³⁹ reference standard, comprising 44 positive and 50 negative drug-ADR pairs for a wide range of drug classes across 10 ADRs. The second is the reference standard developed by Ryan et al for the Observational Medical Outcomes Partnership (OMOP)⁴⁰, now known as the Observational Health Data Sciences and Informatics (OHDSI) program. The OMOP reference standard comprises 165 positive and 234 negative drug-ADR pairs for a wide range of drug classes for 4 serious ADRs of interest. In either case, the examples are manually curated drug-ADR pairs reviewed for known interactions (positive examples) or lack of association (negative examples). After mapping to CUIs and RxCUIs and removal of example pairs for which an entity did not occur in the clinical notes used, 93 and 397 example pairs were retained for the EUADR and OMOP sets respectively. The dropped EUADR example pair was nimesulide with acute liver injury, a positive example in the reference set. Nimesulide has never filed for FDA approval and reassuringly does not appear in the clinical notes used in this study. The two OMOP examples dropped were both positive example pairs: alatrofloxacin with acute liver injury (a prodrug of another example, trovafloxacin, that mapped to the same CUI as that example) and capreomycin with kidney injury (capreomycin did occur in the data but was not retained in the mapping process to RxCUIs). PRR/ROR were ranked according to their values, and aer2vec and SGNS embeddings were ranked according to the sigmoid of the dot product of the trained ADR and drug vectors (input and output weights, respectively). Directional and proximity-based EARP embeddings were ranked according to the maximum and average sigmoid of the dot product of the trained ADR vector looking to the left and to the right, and drug vectors. For EARP_dir this is achieved by indexing the ADR vector by the generated permutation vectors for the right and left directions of the observed term. For EARP_prox this accomplished by indexing the ADR vector by the closest permutation vector to the right (P1 in Figure 1) and left (P_-1 in Figure 1) of the observed term. In either case, the resulting set of sigmoided values are averaged together (denoted μ) or the largest is selected (denoted max).

Results

Performance of each method is shown in Table 1. The best performance on the OMOP and EUADR sets are obtained with the EARP_dir and SGNS models, respectively, with EARP_dir performing best overall as estimated by a weighted average across the sets. On both sets, aer2vec performs substantively better than the ROR/PRR measures. However, the use of narrow context definitions and encoding of directional or proximity information results in further – and larger – improvements, with best performing models attaining to a relative increase in AP over statistical baselines of ~50% and ~55% for the EUADR and OMOP sets respectively.

Table 1.

Performance comparison across tested methods. PRR/ROR were scored according to their ranked values (simply denoted * here). The small variations between PRR and ROR formulas did not change the ordering of the example pairs, and so they resulted in the same AUCs and APs for both sets. Ranking by sigmoid of the dot product of the example pairs [denoted σ(•)] was used for aer2vec and SGNS embeddings. Ranking by the maximum [max(σ(•))] and average [μ(σ(•))] of the left and right permuted ADR and drug pair examples is reported for EARP variants. Average denotes the performance when taking the average weighted by the number of example pairs in each reference standard. Bold values indicate best performance for a given row.

		PRR/ROR	aer2vec	SGNS	EARPdir		EARPprox
		*	σ(•)	σ(•)	max(σ(•))	μ(σ(•))	max(σ(•))	μ(σ(•))
OMOP	AUC	0.484	0.577	0.629	0.645	0.650	0.628	0.638
	AP	0.391	0.477	0.576	0.599	0.604	0.589	0.593
EUADR	AUC	0.530	0.547	0.709	0.647	0.658	0.677	0.677
	AP	0.470	0.537	0.706	0.651	0.663	0.684	0.693
Average	AUC	0.493	0.571	0.644	0.645	0.652	0.637	0.645
	AP	0.406	0.488	0.601	0.609	0.615	0.607	0.612

Notably, EARP based models consistently achieve their best performance when taking the average of the sigmoid of the dot product [μ(σ(•))]. Document level methods PRR/ROR and aer2vec both have lower relative performance than sliding window / locality sensitive methods, regardless of data set or scoring measure. Comparing document level methods, aer2vec performed consistently better than PRR/ROR, consistent with previous findings^11,26. EARP_dir performed better than EARP_prox and SGNS on OMOP, whereas EARP_prox performed better than EARP_dir on EUADR but worse than SGNS. In terms of overall results, calculated by the weighted average of performance on both reference standards, EARP_dir has the best overall performance, followed by EARP_prox, SGNS, aer2vec and then PRR/ROR.

Results are compared to previous work in Table 2. These results are not strictly comparable as Li et al¹³ and Malec et al^15,41 both use different subsets of the OMOP reference standard and different clinical note sources. In this study, we utilize 397 of 399 example pairs in the OMOP reference standard. Malec et al 2018 resolved 396 of the pairs and Malec et al 2021 resolved 163 of the pairs as a result of enforcing more strict inclusion criterion. Li et al utilized 214 of 399 example pairs, again due to strict inclusion criterion. For our study, we required only that an entity occur at least once in the clinical notes to be included. For clinical notes, Li et al used 0.3 million notes and structured data from New York Presbyterian Hospital at Columbia University Medical Center, and Malec et al used 2.2 million clinical notes from UTHSC-H spanning 2004 – 2012. It should also be noted that Li et al used additional sources of information, including claims and FAERS data, to reach a final adjusted AUC of 0.89, supporting the promise of multi-modal data integration to improve performance on this task. Similarly, Malec et al improved performance over their 2018 work utilizing information from the literature to identify confounding variables for modeling of clinical notes in their 2021 paper. Malec et al was included in Table 2 on the account that literature data was used to adjust signal from the clinical notes but does not contribute signal itself. Neither group has thus far deployed their methods on the EUADR reference standard using clinical notes.

Table 2.

AUC performance comparison to other clinical note methods utilizing the OMOP reference standard, with EUADR results presented for additional context. Bold indicates best performance for a given row. PRR/ROR results calculated in this study are presented as a baseline. In the unadjusted results, the best performing method utilizing only information from clinical notes is presented. Adjusted results for comparators are after adjustment via reference set supervised confounding analysis (Li et al) or literature-derived estimates of confounding (Malec et al). Results for Li et al and Malec et al were curated from the publications. This study does not perform any confounding adjustment and solely uses clinical note-derived data. These comparisons are not strictly comparable on account of differing subsets of the OMOP reference standard in addition to different clinical data sources and are presented only for context.

	PRR/ROR	Li (2015)	Malec (2018)	Malec (2021)	This Study
# of Notes Used	4.67M	0.3M	2.2M	2.2M	4.67M
# OMOP Examples	397	214	396	163	397
OMOP	0.484	0.51	0.50	0.55	0.65
OMOP (Adjusted)	-	0.55	0.58	0.65	-
EUADR	0.530	-	-	-	0.71

Discussion

To our knowledge, this is the highest performance achieved on the OMOP reference standard using information solely from clinical notes without any additional sources of information or context. That proximity methods provide a significant 50% lift or more over baseline statistical methods or even document level is a key finding. With the increase of SGNS results over aer2vec, and the overall performance increase of EARP methods, juxtapositional information is a main contributor to recovering signal on these reference standards with NLP extracted clinical concepts. EARP_dir did perform the best overall, indicating that proximity can provide additional signal, though EARP_prox did not yield additional improvements. Notably, this supports the hypothesis that proximity information (e.g. how closely a side-effect is mentioned in a clinical narrative to a putative drug) can improve signal yield from clinical notes in drug-ADR prediction tasks, and may be equally beneficial for other tasks. Interestingly, in further experiments (not shown in Table 1), using only the cosine between the input weight embedding vectors for both drugs and ADRs, as is typically the case with word embedding analyses (where output weights are discarded, prohibiting recapitulation of the training objective), had relatively poor performance compared to the permuted embedding vectors for ADRs and context vectors for the drugs. This is to be expected. The training objective of the SGNS and EARP methods optimize for the sigmoid of the dot product between an embedding vector (positionally permuted in the case of EARP) and its context vectors; using the same objective in the testing procedure should produce the optimal result when estimating the probability of association (i.e., P(drug|ADR)), and this was observed here. Still, as many word vector embedding approaches discard the trained output weights for context vectors, this finding is notable, and further underscores the utility of this fundamentally different way of retrieving encoded information from neural embeddings when modeling observational data (originally demonstrated by aer2vec). Additional research into the potential utility of context embeddings generated during training for other applications may be warranted.

In line with the multi-modal performance improvements from our previous work leveraging FAERS and biomedical literature and the success of Li et al and Malec et al, future work should explore the use of these locality sensitive embeddings with other sources of ADR signal. Additionally, we did not perform grid searching to optimize window radius, negative sampling, or any other hyperparameters for models trained here, nor did we perform any modifications to the CLAMP clinical note preprocessor. It seems possible that optimizing these parameters and/or the text preprocessing could further improve performance. The reference standards themselves are human curated artifacts and subject to potential bias and are limited to a total of 10 ADRs of interest. Further study is warranted as to the generalizability of this method beyond these reference standards. Our work was also confined to a subset of clinical notes collected in the Houston metropolitan area during a fixed time window. It is possible that other clinical note repositories, such as that used by Li et al, would not see the same performance breakdown as observed here. Future work is planned to investigate if these findings hold true across NLP preprocessing tools and across clinical data repositories. Furthermore, retrofitting and subword embeddings have been used to enhance FAERS-based aer2vec performance on this task¹⁰, and could be used in future research for clinical-based aer2vec and EARP models. Li et al used time dependency for their inclusion criterion in preprocessing their clinical notes and Malec et al sought to causally model drug-ADR assertions and reduce confounders, both of which could be complementary to the work presented here. Finally, the methods developed here could have generalization beyond the task of pharmacovigilance, extending the use of clinical note modeling for drug repurposing or other tasks.

Conclusion

This study presents a locality sensitive embedding technique applied to NLP processed clinical notes to achieve state-of-the-art performance for models using clinical notes alone on two ADR reference standards. NLP preprocessing combined with document level statistical approaches in PRR/ROR did not produce appreciable signal when scored against the two reference standards. Document level embeddings of this same data did recover more signal than statistical approaches, consistent with previous research²⁶. While performance variations between specific embedding techniques are present, all locality sensitive embedding techniques substantially outperformed statistical and document level embedding techniques, underscoring the added utility of retaining proximity features.

Conflicts of Interest

Dr. Xu and The University of Texas Health Science Center at Houston have research-related financial interests in Melax Technologies, Inc.