Predicting Natural Product-Drug Interactions with Knowledge Graph Embeddings

Abstract

Natural product-drug interactions (NPDIs) occurring due to concomitant exposure to botanical products and prescription drug therapies could lead to adverse events or reduced treatment efficacy. To better understand and address potential safety concerns, researchers investigate the underlying NPDI mechanisms using in vitro and clinical studies. Given that natural products are complex mixtures of compounds that are often not well characterized, it is important to advance computational methods for novel NPDI research. Biomedical knowledge graphs (KGs) can aid in identifying potential mechanisms to support such research efforts. We evaluated the ability of several KG embedding methods to improve NPDI prediction on NP-KG, a large-scale, heterogeneous, biomedical KG. We found that the ComplEx model outperformed other KG embedding approaches in both intrinsic and extrinsic evaluations. Future work will focus on utilizing the embeddings to identify underlying mechanisms of novel, potential NPDIs.

Introduction

Botanical natural products used for complementary health approaches are complex mixtures of chemicals produced by living organisms¹. Approximately 50% of adults in midlife have reported co-consumption of natural products and drugs, with the prevalence being even higher in older adults (up to 88%) in the US²,³. Botanical natural products differ from conventional drugs, both in terms of the usage and regulatory oversight. While research to date has found many natural products safe for human consumption, studies have also identified risks of harm from natural product-drug interactions (NPDIs) such as adverse events or reduced efficacy during concomitant exposure to natural products and prescription drugs⁴,⁵. For example, the botanical product St. John’s wort (Hypericum perforatum L.) reduces the efficacy of drugs metabolized by the enzyme CYP3A4, including several antidepressants, anti-migraine medications, warfarin, cyclosporine, and oral contraceptives⁶. Recent evidence shows that the herbal substance kratom (Mitragyna speciosa) can also interact with CYP3A4 substrate drugs to precipitate serious interactions⁷.

To determine which NPDIs are clinically relevant, pharmacovigilance practitioners and researchers must evaluate the existing evidence from clinical studies, published case reports, and prior experiments, as well as identify chemical characterizations and constituents of the natural products to address safety concerns⁸,⁹. Since most of the relevant NPDI evidence is present in narrative form within the published scientific literature, practitioners must manually search, synthesize, and interpret the evidence, which is a time-consuming process¹⁰. The synthesized NPDI evidence may also be inaccessible to many users, such as the commercial resources NatMed¹¹ and UpToDate¹². With the large number of botanical natural products available in the market and increasing consumer use¹³, it is crucial to facilitate our understanding of the potential safety concerns.

A biomedical knowledge graph (KG) integrates expert-derived sources of information into a graph data structure, where the nodes represent biomedical entities (such as chemicals, diseases, proteins, genes) and edges represent relationships between the entities¹⁴. Large-scale biomedical KGs have been used for drug-related applications, including drug repurposing and prediction of side effects¹⁵,¹⁶. However, large-scale KGs can often be intractable due to the large number of nodes and edges. Graph representation learning or KG embedding methods, that create low-dimension vector representations of the nodes and edges in the KG, are often utilized for the prediction tasks as they bridge the gap between graph data and traditional machine learning methods¹⁷. These methods are designed for heterogeneous KGs and encode graph structured information through mathematical functions such that the representation in the embedding space reflects the structure of the original graph¹⁸. Downstream tasks with the embedding methods can exploit this numerical representation to simplify computational methods that derive insights from the graph, such as link prediction or KG completion, that aim to predict edges in the KG using the embeddings.

Related work: Prior work in computational discovery of NPDIs using KGs has used both literature-based discovery 19,20 and network methods²¹. It is difficult to compare these studies because they used varying definitions of natural products, for example focusing on broader food items rather than individual natural products or phytoconstituents. Classification models have been developed for supplement-drug interactions using transfer learning with a drug-drug interaction dataset²⁰ and herb-target prediction for traditional Chinese medicine using vector representations of herbs and proteins in a heterogeneous network with node2vec embeddings²¹. Schutte et al.¹⁹ extracted knowledge of dietary supplement-drug interactions from abstracts to develop a literature-based KG using relation extraction methods and applied discovery patterns to the KG to identify drug-supplement interaction mechanisms with shared gene or biological function concepts. Several other studies use a broader definition of natural products for computational investigation. These studies have attempted to predict food-drug interactions in graphs constructed from food-related databases²²,²³, PubMed abstracts²⁴, and DrugBank annotations and chemical structures²⁵. The major challenges in the computational identification of NPDIs involve a lack of gold-standard data on NPDIs and difficulty obtaining representations of natural products and their constituents²⁴. Furthermore, comprehensive evaluations of the KG embedding methods are needed before they can be utilized for novel NPDI prediction for pharmacovigilance and clinical decision support.

In this study, we use graph representation learning methods to predict NPDIs in a natural products-focused KG by casting prediction of NPDIs as an edge prediction task. Prediction is done with a natural products KG, NP-KG, that combines biomedical ontologies, open databases, and full texts of scientific literature related to pharmacokinetic natural product-drug interactions. We use several state-of-the-art embedding models to predict NPDIs and evaluate the methods with a reference dataset of known interactions. The major contributions of this study include the construction of a reference dataset for NPDIs from several gold-standard resources and a comprehensive evaluation of KG embedding methods for NPDI prediction that can be applied to downstream tasks such as prediction and mechanism generation for the discovery of novel potential NPDIs.

Methods

In this section, we describe the KG, and the reference dataset used for the edge prediction task. We also discuss the graph representation learning methods and delineate the experimental setup, including hyperparameter tuning. Then, we present the evaluation strategies to validate NPDI prediction using the embedding methods in NP-KG.

Knowledge Graph: We used a natural products KG, called NP-KG that was constructed with the Phenotype Knowledge Translator (PheKnowLator)²⁶ workflow for disease mechanisms. It was then enriched by combining the graph with a literature graph constructed from relation extraction of full texts of PubMed-indexed scientific articles related to selected natural products²⁷. NP-KG is a large-scale, heterogeneous, directed multigraph that contains biomedical entities, including natural products, diseases, phenotypes, anatomical entities, biological processes, cellular components, molecular functions, proteins, pathways, chemicals, genes, cell lines, and adverse events, from 14 Open Biomedical and Biological Foundry (OBO) ontologies. We used the third iteration of NP-KG which advanced on our prior work²⁷. Edges in this version of NP-KG originate from 17 open databases and 4,529 full texts of scientific literature related to 30 natural products of interest. Constituents of each natural product were extracted from the Global Substance Registration System (G-SRS)²⁸ and the European Medicinal Agency (EMA) herbal monographs²⁹. The constituents were then linked to the natural product nodes in NP-KG through ontology extensions. To use NP-KG for the edge prediction task, the multigraph was pre-processed to collapse the edges between nodes, such that node pairs with multiple edges between them were assigned a unique edge type and unique identifiers were assigned to all edges.

Reference Data: We extracted information from several resources to construct a reference dataset of NPDIs and related adverse drug reactions (ADRs):

• NatMed is a commercial resource that contains Food, Herbs, and Supplements Database with over 1,300 monographs on natural ingredients, including vitamins, herbs, minerals, non-herbal supplements, naturally sourced chemical compounds, and foods. Monographs for natural products include potential NPDIs curated from the literature along with evaluation of the study quality, level of evidence available (such as randomized controlled trial or animal research), and in some cases, the adverse outcome associated with the NPDIs¹¹,³⁰.

• The NaPDI database is an expert curated database maintained by the NaPDI Center that contains results from published case reports, in vitro experiments, and clinical pharmacokinetic studies related to 33 natural products (as of August 2023)⁵.

• The Stockley’s Herbal Medicines Interactions provides curated monographs on herbal medicines and information about interactions of herbal medicines with drugs, where ‘herbal medicines’ are defined as traditional herbals, nutraceuticals, dietary supplements, and some food items. The resource is targeted for use in the clinical setting and contains recommendations based on evidence from clinical trials, animal, and in vitro studies³¹. The publicly available version was last published in 2013.

• The Food Interactions with Drugs Evidence Ontology (FIDEO) version 2 (last updated in February 2023) contains food-drug interactions extracted from scientific abstracts and is publicly available. Although the authors state that FIDEO contains mechanistic information for each interaction along with the study type and level of clinical importance, this information appears to be missing in the current version³².

• The Diet-Drug Interactions Database (DDID) is an expert curated knowledge base for interactions between drugs and herbs or food items from literature³³. DDID contains the effects and evidence for 23,950 interactions between 270 foods, 1,068 herbs, and 1,516 drugs. This information can be browsed and downloaded from their website.

Monographs from NatMed and Stockley’s were annotated to extract NPDIs with related evidence, including protein targets, overall effect of NPDI, related outcomes, and source publications, to a spreadsheet. Publicly available files containing NPDI evidence were downloaded from the NaPDI database, FIDEO, and DDID. Data were processed, deduplicated, and mapped to concepts in NP-KG. Mapping was done with existing identifiers in the above resources, mappings extracted from the OBO ontologies and named entity recognition with the PyOBO library³⁴ followed by manual review. All NPDIs in the dataset were compiled and saved with the original source information, including ADRs, protein targets, source literature information, and overall effect, where available.

KG Embeddings: KG embedding methods generate embeddings in three steps, (1) representation of entities and relations as vectors, (2) definition of scoring functions, and (3) optimization of observed facts (represented as triples in the KG) to learn the final embedded representations³⁵.

Using the terminology of KG embedding methods, a KG is a heterogeneous, multi-relation, directed graph, containing a set of entities e and set of relationships R, defined as KG c e x R x e with triples (h, r, t) E KG. For each triple (h, r, t), h and t are the head and tail entities connected by the relation r. After defining a scoring function f(h, r, t) to measure plausibility, an embedding model is trained to maximize the total plausibility of all triples in the KG³⁵. Negative examples, or corrupted triples, are generated per triple in the KG by randomly sampling from head or tail entities. These are noted as (h’, r, t) or (h, r, t’). Training occurs using the open-world assumption which assumes that the corrupted triples may be either true or false but have a lower plausibility score than the positive triples³⁵. The number of corrupted triples or negative examples is one of the model parameters during training³⁶,³⁷.

Several embedding models are available that differ based on the type of scoring functions, optimization methods, and scoring function domain. Translational distance models are based on learning representations of a graph through distance-based functions and include TransE³⁸, TransR³⁹, TransD⁴⁰, and RotatE⁴¹. Semantic matching models are based on matching the semantics of entities and relations in the embedding space through similarity-based functions and include RESCAL⁴², DistMult⁴³, and ComplEx⁴⁴. In this study, we evaluated the five KG embedding models shown in Table 1 with the related scoring function. The five different embedding methods were chosen to balance performance, efficiency, computational resources, and training time.

Table 1.

Embedding models with scoring function and model type. Scoring function is defined by f for triple with head h, relation r, and tail t.

Model	Scoring function	Model Type
TransE	f(ℎ, r, t ) _ − ||h + r − t||1	Translational
TransR	f(ℎ, r, t ) _ − ||h⊥ + r − t⊥||2	Translational
RotatE	f(ℎ, r, t ) _ − ||h Q r − t||2	Translational
DistMult	f(ℎ, r, t ) = hTdiag(r)t	Semantic Matching
ComplEx	f(ℎ, r, t ) = Re (h Q r Q t)	Semantic Matching

^*h⊥ and _t are projected embeddings of h and t, h^T is the transpose of h, || . ||1 is the L1 norm, || . ||2 is the L2 norm, Q represents the Hadamard product, diag(r) is the diagonal matrix for relation embedding r, and Re denotes the real part of a complex value.

TransE³⁸ was the first model to use a distance function to describe a triple as a translation between h and t through r in the embedding space, such that the embedded entities h and t are connected by r with low error. TransE is the simplest model of the translational distance methods. However, it can only model one-to-one relationships such that all entities linked with relation r have very similar vector representations even if they are totally different entities. Nonetheless, TransE has shown good performance for biomedical KG prediction tasks¹⁶,⁴⁵. TransR³⁹ and RotatE⁴¹ models are extensions of TransE that generate relation-specific vectors to overcome the limitations of TransE. TransR introduces relation-specific vector spaces through projection matrices and then projects the entity representations of h and t (h⊥ and t⊥) for each triple into the relation-specific space for r. The RotatE approach models relations in a complex vector space as rotations from head to tail entities. The RotatE model has been shown to outperform other models in drug discovery predictions using a KG⁴⁶. The DistMult⁴³ model is a semantic matching model that captures pairwise interactions between head and tail entities in the KG in a matrix associated with the relation. However, this approach can only model symmetric relationships such as “is a”. The ComplEx model⁴⁴ extends DistMult to generate entities and relation vectors in complex domain and the scoring function involves the real part (Re(x)) of the complex vectors.

Experimental Setup: In this study, we evaluated the two classes of heterogeneous graph representation learning methods (translational distance models and semantic matching models) for edge prediction in NP-KG. The methods were evaluated for both the prediction of existing edges in NP-KG to measure performance of the embedding models (intrinsic evaluation), and prediction of NPDIs in a reference dataset (extrinsic evaluation). Figure 1 shows an overview of our prediction pipeline, including embedding generation, hyperparameter tuning, and evaluation.

Figure 1.

Overview of the knowledge graph embedding prediction pipeline.

All embedding models were implemented using the DGL-KE (v0.1.2)⁴⁷ library. While some studies argue that the choice of hyperparameters does not have an impact on model performance, there is sufficient evidence in benchmarking studies that data splitting strategies, hyperparameters, and negative sampling techniques do play an important role in improving the usefulness of KG embedding methods³⁶,⁴⁶. Table 2 presents the hyperparameters, default values, and values tested for optimization. Training was conducted on a high-performance computing cluster with 4 A100 GPUs. The embedding models were trained first with default parameters (Table 2) to determine baseline performance. Then, hyperparameter tuning was implemented with 20 random trials to identify the best performing configuration using the scikit-optimize library (v0.10.2)⁴⁸. Each model was trained for 10,000 epochs with batch size equal to 512.

Table 2.

Hyperparameter values and defaults used for tuning and evaluation.

Parameter	Hyperparameter Tuning	Default Values
Embedding dimension	100, 250, 400	400
Learning rate	0.01, 0.05, 0.1, 0.25	0.25
Number of negative examples per KG triple	30, 64, 128	128
Regularization coefficient	2e-08, 1e-09, 2e-06	1e-09
Margin score (gamma)	9, 12, 19.9, 24	19.9

Evaluation: Evaluation of the KG embedding methods was performed in two phases, intrinsic and extrinsic evaluation. In the intrinsic evaluation, a vector similarity ranking task was used to differentiate correct (existing) triples in KG from the corrupted triples. In the extrinsic evaluation, the embeddings that resulted from a given method were used as features for machine learning classification.

Intrinsic evaluation: The intrinsic evaluation was based on the link prediction task modeled as a ranking task to differentiate existing triples in KG from the corrupted triples. Existing edges in NP-KG were split the triples into train (70%), validation (10%), and test (20%) data such that the train, validation, and test data were all connected graphs that contained the same nodes and edge types. The validation data were used for hyperparameter tuning. Final results were calculated using mean rank, mean reciprocal rank, and Hits@K on the test data using the scoring functions.

Mean Rank (MR) is the average rank of the test triples, where smaller values represent better performance.

$$MR=\frac{1}{\left|K_{test}\right|}\sum t\in K_{test}rank\left(t\right)$$

Where $K_{test}$ is the set of test triples. The mean reciprocal rank (MRR) is the mean over reciprocal of the individual ranks, where large values indicate better performance. MRR evaluates the ability of the model to rank the correct answer highly and is less sensitive to outliers and the number of test triples than MR.

Hits@k measures the percentage of edges in which the true triple appears in the top k ranked triples, where we chose k E {1,3,10}.

$$Hits@k=\frac{\left|\left\{t\in K_{test}|rank\left(t\right)\le k\right\}\right|}{\left|K_{test}\right|}$$

Extrinsic evaluation: Embeddings from the models described above were saved and loaded to be used as features in the edge prediction task. Two classifiers – logistic regression and multi-layer perceptron – were trained using the embedding features. The edge prediction task was modeled as binary prediction with positive and negative classes, where positive indicates existence of edge and negative indicates that the edge does not exist. For the extrinsic evaluation of the embedding models, the natural product-drug pairs in the reference dataset were mapped to concepts in NP-KG. NPDIs that already existed in NP-KG as edges and NPDIs with drugs that could not be mapped to nodes in NP-KG were removed from the reference data. The interactions with both negative and positive evidence were not used to train the classifier and treated as a holdout test set for evaluation. The remaining NPDIs were used to train and test the classifier models with 5-fold cross validation and entity embeddings as input features. For the holdout test data, the prediction probabilities for the positive and negative classes were calculated to test the classifier performance. Of note, negative examples in intrinsic evaluation refers to corrupted triples sampled from the training data, whereas negative examples in extrinsic evaluation are interactions in the reference data with negative evidence or no interaction evidence.

To train the classifiers, input features were generated by computing feature vectors through either the concatenation or Hadamard product of the entity embeddings from the KG embedding models (with embedding dimension based on hyperparameter tuning) for each NPDI in the reference dataset. The concatenation method adds each element of the entity embeddings to create the input feature vector and the Hadamard product calculates the element-wise product of the entity embeddings. Using the notations defined above, if h represents the natural product node embedding and t is the drug node embedding, input feature vectors are derived either from concatenation (h ⊕ t) or Hadamard product (h (D t) of the h and t entity embeddings.

The output probability of each interaction in the reference dataset for ‘positive’ or ‘negative’ classes was calculated using the two classifiers and edge feature methods. The classifiers were trained and evaluated with 5-fold cross validation of the training data such that the performance metrics were averaged over the test data. The performance was evaluated for the models using the scikit-learn⁴⁹ (v1.3.1) library. Evaluation metrics for each embedding model used for input features, edge feature method, and classifier type were calculated, including the area under the receiver operating characteristics curve (AUC), area under the precision-recall curve (AUPRC), precision, recall, and F1-score.

Results

NP-KG: NP-KG version 3 contained 1,089,139 nodes and 7,836,115 edges, with 355 unique edge types in the multigraph. After collapsing edges in the multigraph, NP-KG contained 1,089,139 nodes and 7,786,923 edges with 1,343 edge types. Figure 2 shows the visualization of NP-KG node and edge types generated with the t-distributed stochastic neighbor embedding (t-SNE) algorithm⁵⁰ using the GRAPE⁵¹ (v0.2.4) library. The t-SNE algorithm graphically represents embeddings in two-dimensional space with dimensionality reduction such that similar concepts in embedding space are presented by nearby points in the visualization. For the intrinsic evaluation, the graph data was split into train, validation, and test sets with 5,606,586 (70%), 622,953 (10%), and 1,557,384 (20%) edges, respectively. The train, validation, and test sets all contained 1,089,125 nodes and 1,343 edge types.

Figure 2.

Node and edge types in NP-KG. Node types are assigned based on ontology concepts and edge types are relations from the Relation Ontology.

Reference Data: Table 3 shows the data sources in the reference dataset along with the number of interactions and coverage for the selected natural products in NP-KG. The total number of NPDIs across the various sources was 3,280. After removing interactions with drug classes (N=215), mapping to ChEBI identifiers, and deduplication, there were 1,431 unique interactions in the reference dataset for 30 natural products and 521 unique drugs. The reference data contained 1,190 (83.16%) positive interactions and 155 (10.83%) negative interactions, where positive interactions referred to NPDIs with inhibition or induction effect and negative interactions referred to natural product-drug interactions where studies found no measurable interaction. A small number of interactions (86 or 6.01%) had both positive and negative evidence. This was due to studies conducted with different study designs, experimental systems, or at different times.

Table 3.

Sources in the reference dataset with number of interactions in each source.

Data Source	Version/Year	No. of NPs covered out of selected NPs	Total Interactions for the selected NPs	Total Interactions after mapping
NaPDI Database	April 2023	16	349	269
Stockley’s	3rd edition (2013)	25	177	173
FIDEO	February 2023	20	553	553
NatMed	January 2024	30	425	338
DDID	May 2024	28	1798	1595

Intrinsic Evaluation: Table 4 shows the intrinsic evaluation results on the test data with the default model parameters for the embedding models. All models showed moderate performance with the default parameters, with DistMult model performing better than other models.

Table 4.

Performance of embedding models on test data with default model parameters.

Model	MRR	MR	Hits@1	Hits@3	Hits@10
TransE	0.53	51.59	0.43	0.60	0.72
TransR	0.53	52.30	0.44	0.58	0.68
RotatE	0.45	68.40	0.36	0.51	0.62
DistMult	0.56	26.46	0.44	0.64	0.79
ComplEx	0.53	32.97	0.41	0.61	0.75

After hyperparameter tuning of each model with parameter values in Table 2, all models showed improved performance on the test data. Table 5 shows the results of intrinsic evaluation of models on the test set using the best hyperparameter configurations. With the optimal hyperparameters, the ComplEx model with embedding dimension equal to 400 outperformed the other models in terms of MRR, MR, hits@3, and hits@10. The TransR model with embedding size 100 outperformed other models in terms of the hits@1 measure, suggesting that the correct triple is the top ranked triple in 65% of the test data triples.

Table 5.

Performance of embedding models on test data with best hyperparameter configurations.

Model	MRR	MR	Hits@1	Hits@3	Hits@10
TransE	0.68	27.30	0.58	0.74	0.85
TransR	0.71	27.25	0.65	0.75	0.83
RotatE	0.65	37.74	0.58	0.71	0.80
DistMult	0.63	18.49	0.52	0.71	0.84
ComplEx	0.72	16.02	0.63	0.78	0.87

Extrinsic Evaluation: Out of the 1,431 interactions in the reference dataset, 84 were present as edges in NP-KG and were removed from the prediction experiments. After mapping, the training and test data for 5-fold cross-validation included 1,268 interactions, and an additional 73 interactions with both positive and negative evidence as a holdout test set. Table 6 shows the performance of the embedding models with a binary logistic regression classifier and either the concatenation or Hadamard product edge feature method. The RotatE, DistMult, and ComplEx models with concatenated features performed well in the classification task on different metrics, with DistMult achieving higher values for AUC, RotatE achieving higher precision, and the ComplEx model outperforming in terms of recall and F1-score. Concatenation of the embedding features performs consistently better than Hadamard product for all models. The AUPRC is high across all models and edge feature methods. Table 7 shows the performance of the concatenation-based embedding features with a binary multi-layer perceptron classifier. All embedding models achieved comparable performance across the metrics with the multi-layer perceptron classifier and the performance improved when compared to logistic regression classifier, especially for the recall metric. From this result, it is clear that both translational distance and semantic matching models were able to capture useful information about the KG in the embedding features, although there is a possibility of overfitting with the complex classifier due to the small size of the training and test data.

Table 6.

Performance of embedding models in classification of NPDIs in the reference dataset with logistic regression.

Embedding Model	Edge feature method	AUC	AUPRC	Precision	Recall	F1-score
TransE	Concatenation	0.80	0.97	0.87	0.75	0.79
	Hadamard	0.68	0.93	0.63	0.73	0.67
TransR	Concatenation	0.76	0.96	0.86	0.71	0.76
	Hadamard	0.63	0.93	0.77	0.67	0.65
RotatE	Concatenation	0.81	0.97	0.88	0.76	0.80
	Hadamard	0.68	0.93	0.63	0.73	0.67
DistMult	Concatenation	0.82	0.97	0.87	0.83	0.85
	Hadamard	0.72	0.94	0.85	0.75	0.79
ComplEx	Concatenation	0.80	0.97	0.87	0.85	0.86
	Hadamard	0.70	0.94	0.85	0.78	0.80

Table 7.

Performance of embedding models in classification of NPDIs in the reference dataset with concatenated features in multi-layer perceptron classifier.

Embedding Model	AUC	AUPRC	Precision	Recall	F1-score
TransE	0.82	0.97	0.87	0.88	0.87
TransR	0.80	0.97	0.85	0.87	0.86
RotatE	0.83	0.97	0.88	0.88	0.88
DistMult	0.82	0.97	0.88	0.90	0.88
ComplEx	0.83	0.97	0.88	0.90	0.89

For the holdout test set with interactions in the reference dataset that included both positive and negative evidence and were excluded from the training, we calculated the prediction probabilities using the ComplEx concatenated embeddings. The logistic regression classifier resulted in 50.7% cases classified as positive and 49.3% classified as negative edges with a 0.5 probability threshold. The mean probabilities for the positive and negative predicted classes were both 0.50, indicating that the model was unsure of the class to be assigned to these cases. In comparison, the multilayer perceptron model trained with the ComplEx embeddings predicted ‘both’ edges as ‘positive’ more often than ‘negative’ (mean positive prediction probability equal to 0.72).

Conclusion

In this study, we evaluated the performance of several KG embedding methods on a natural products-focused KG, NP-KG, and used the embedding features for the classification of NPDIs with an external reference dataset of known interactions. Prediction with embedding methods in biomedical KGs is notably more complex than other domains as biomedical KGs are sparse, noisy, incomplete, and contain heterogeneous data⁵². This makes finding a one-size-fits-all model or configuration that outperforms across different tasks or datasets challenging. Rather, the performance is contingent on the size and sparsity of the KG, hyperparameter configurations, number of unique relations, data splitting, and negative sampling strategies^45,46,52,53. This underscores the importance of context-specific model selection and tuning.

In our experiments, the ComplEx model outperformed others in terms of MRR in the intrinsic evaluation, and the RotatE, DistMult, and ComplEx models generally outperformed other models in the extrinsic evaluation. These findings were consistent with prior work where the RotatE, and ComplEx model showed superior performance in similar edge prediction tasks, including drug discovery and polypharmacy prediction⁴⁶,⁵³. This may be due to the ability of ComplEx and RotatE models to better handle asymmetric (“drug A inhibits protein B”) and multi-relational edges (e.g., many-to-many links between drugs and genes) that are often prevalent in biomedical KGs. Unlike TransE and TransR models that use simple translations, the ComplEx and RotatE methods can effectively model directionality and complex interactions in KGs. Our study also adds evidence that the embedding model configuration (hyperparameter tuning) has a significant impact on the prediction performance. This is in line with prior research that KG embedding models are not standalone methods but should be considered in combination with the training setup and hyperparameter values for optimal performance⁴⁶. In extrinsic evaluation, the concatenated embedding features for edges proved better than the Hadamard product across all models. This suggests that combining head and tail entities through the concatenation of node embeddings captured the KG features more effectively. Further, a more complex classifier such as the multi-layer perceptron showed improved performance over logistic regression, indicating that non-linear classifiers can better leverage the KG embedding features for classification.

Our overarching goal is to use NP-KG embedding features to predict novel potential NPDIs, particularly from pharmacovigilance signals, to address potential safety concerns and elucidate the underlying mechanisms of interaction. KG mechanism discovery has shown promise through the synergistic use of KG embeddings and discovery patterns⁴⁵,⁵⁴. However, this study has several limitations that need to be addressed. First, we limited the evaluation to a specific set of models due to time and resource constraints. In some cases, simpler (homogeneous models) or more complex (graph neural networks) models have been shown to perform well in biomedical KG prediction tasks⁵⁵,⁵⁶. We did not test the latter because NP-KG currently does not provide the extensive node and edge features required for graph neural networks. Secondly, the lack of gold-standard datasets for NPDIs is a challenge for validation. We addressed this by carefully combining several different datasets for the evaluation.

In summary, we implemented a comprehensive evaluation of KG embedding methods for predicting NPDIs using NP-KG. The ComplEx embedding model, when combined with concatenated features and a non-linear classifier, provides robust performance across both intrinsic and extrinsic prediction tasks. These findings are a step forward in prediction of NPDIs using heterogeneous data from ontologies, databases, and scientific literature. Future work will apply the embeddings to identify plausible explanations for NPDI pharmacovigilance signals. We are also exploring the use of large language models to generate summaries of the predictions, with the aim of presenting plausible mechanisms to researchers in a user-friendly format to facilitate clinical decision making. All data and code from this study are available at https://github.com/sanyabt/npkg-embedding and NP-KG is available at https://doi.org/10.5281/zenodo.12536780.