ARCH: Large-scale Knowledge Graph via Aggregated Narrative Codified Health Records Analysis

Abstract

Objective:

Electronic health record (EHR) systems contain a wealth of clinical data stored as both codified data and free-text narrative notes (NLP). The complexity of EHR presents challenges in feature representation, information extraction, and uncertainty quantification. To address these challenges, we proposed an efficient Aggregated naRrative Codified Health (ARCH) records analysis to generate a large-scale knowledge graph (KG) for a comprehensive set of EHR codified and narrative features.

Methods:

Using data from 12.5 million Veterans Affairs patients, ARCH first derives embedding vectors and generates similarities along with associated $p$-values to measure the strength of relatedness between clinical features with statistical certainty quantification. Next, ARCH performs a sparse embedding regression to remove indirect linkage between features to build a sparse KG. Finally, ARCH was validated on various clinical tasks, including detecting known relationships between entity pairs, predicting drug side effects, disease phenotyping, as well as sub-typing Alzheimer’s disease patients.

Results:

ARCH produces high-quality clinical embeddings and KG for over 60, 000 codified and narrative EHR concepts. The KG and embeddings are visualized in the R-shiny powered web-API ^*. ARCH achieved high accuracy in detecting EHR concept relationships, with AUCs of 0.926 (codified) and 0.861 (NLP) for similar EHR concepts, and 0.810 (codified) and 0.843 (NLP) for related pairs. It detected drug side effects with a 0.723 AUC, which improved to 0.826 after fine-tuning. Using both codified and NLP features, the detection power increased significantly. Compared to other methods, ARCH has superior accuracy and enhances weakly supervised phenotyping algorithms’ performance. Notably, it successfully categorized Alzheimer’s patients into two subgroups with varying mortality rates.

Conclusion:

The proposed ARCH algorithm generates large-scale high-quality semantic representations and knowledge graph for both codified and NLP EHR features, useful for a wide range of predictive modeling tasks.

Graphical Abstract

1. Introduction

The increasing adoption of electronic health record (EHR) systems has provided opportunities for clinical studies and biomedical research [1, 2, 3]. EHR data often cover hundreds of thousands of unique clinical features from both codified data and unstructured clinical narrative notes. To analyze these two types of data simultaneously, the main challenges lie in combining the codified and unstructured data efficiently, representing their covered clinical features meaningfully, and quantifying statistically the presence-absence as well as the strength of relationships between different features.

The integration of codified and unstructured data is driven by their complementary nature, collectively providing a more comprehensive view of a patient’s medical history. While codified data are readily usable due to standardized formats [4], extracting valuable insights from unstructured clinical notes requires natural language processing (NLP) techniques. These extracted NLP concepts, also referred to as clinical concepts of unique identifiers (CUIs) in the Unified Medical Language System (UMLS) [5], provide complementary information to the codified data.

Many studies have shown that incorporating this textual information into analyses can enhance model performance by significant margins [6, 7]. For example, NLP concepts are particularly valuable for capturing drug side effects, as a significant proportion of these effects, such as symptoms, cannot be adequately represented by diagnostic codes [8]. Furthermore, combining codified and unstructured data yields benefits for disease phenotyping. In the United States, a diagnosis code is required by the healthcare provider during the evaluation of a condition. Even if the patient is ultimately diagnosed with a different condition, the initial diagnosis code will remain in the patient’s record and may be misleading if viewed in isolation [9]. It has been shown that prediction models that combine unstructured clinical notes with codified data outperform models that utilize either unstructured or codified data alone [10, 11].

To generate prior knowledge on the relationship among the clinical codes and NLP concepts, a potential solution is to construct a large-scale clinical knowledge graph (KG) on these concepts [12, 13, 14]. Representing EHR concepts with low-dimensional semantic embedding provides a quantitative glimpse into the degree of inter-relatedness of medical entities. These high-quality embeddings can improve the efficiency of downstream applications in biomedical and healthcare research including information retrieval [15, 16, 17], cohort selection [18, 19], and risk prediction [20, 21, 22].

In recent years, word embedding techniques [23, 24, 25] in NLP have been successfully applied for representing clinical concepts in a low-dimensional space. However, many methods do not naturally generate a sparse KG that indicates whether a link exists between entities. In addition, joint representation of large-scale codified and NLP EHR concepts is currently lacking, as summarized in Table 1. Recently, Bai [26] proposed to jointly learn vector representations of medical concepts and words using MIMIC-III data [27]. However, their work was limited in two ways. First, they did not represent words in the clinical notes as CUIs, thus limiting the reproducibility of these representations. Second, the MIMIC-III data only contains 58, 597 in-patient visits, which confines the model performance and cannot infer broader information for outpatients. As a result, their embeddings cannot be used to generate high-quality knowledge graphs capturing general clinical information. To the best of our knowledge, there is no existing work that derives comprehensive embeddings for both codes and CUIs from a comprehensive EHR with both inpatient and outpatient data.

Table 1:

A summary of existing EHR-derived medical embeddings.

Method	Type	Number of concepts
Choi et al. [28] (claims)	CUI	15,905
Finlayson et al. [13] (notes)	CUI	28,394
De Vine et al. [17] (MedTrack)	CUI	59,266
Code2Vec [29]	code	8,477
Med2Vec [30]	code	28,840
KESER [14]	code	14,718
MIKGI [31]	code	13,261
ARCH	CUI&code	51,423 CUIs + 9,586 codes

Generating KG with a large number of entities, however, is challenging for several reasons. First, an efficient computational algorithm is needed to embed all concepts when both the number of concepts and the number of EHR records are large. Second, no existing KG embedding methods provide statistical certainty on whether a link exists between two entities. Most existing KG predicts links via a supervised fashion by optimizing prediction tasks using the labeled links between entity pairs, requiring mapping EHR codes and narrative concepts to existing entity pairs, which itself is a challenging task. In addition, these methods necessitate the use of unlinked entity pairs, which are not readily available. Moreover, it is challenging to handle narrative notes, which are disorganized and not ready to use. But thanks to the development of the Narrative Information Linear Extraction NLP software, as detailed in Appendix S.4, we are able to map clinical terms to CUIs in the UMLS. Then the codified and NLP data can be organized as triplets: (Patient id, date, concept) and are thus aligned.

In summary, there is a great unmet need for an approach that can integrate and summarize these high-dimensional and large-scale clinical data into a KG for studies. In this paper, we will address this need by proposing an Aggregated naRrative Codified Health (ARCH) records analysis which is an efficient statistical algorithm that can generate KG embeddings along with uncertainty measures on the links. With pairwise co-occurrence counts of all EHR concepts and a few simple summary statistics, the ARCH algorithm generates low-dimensional embeddings for each concept and performs large-scale hypothesis testing based on the cosine similarity between these embedding vectors. The connectivity of entity pairs is assessed jointly by controlling for a target false discovery rate (FDR). We validate the clinical utility of the ARCH KG, generated from EHR data from the Veterans Affairs (VA), along with semantic embeddings through downstream tasks including detecting known relationships between entity pairs, predicting drug side effects, disease phenotyping, as well as sub-typing Alzheimer’s disease (AD) patients.

Summary	Description
Problem or Issue	Managing massive and complex electronic health record (EHR) data presents challenges in feature representation and uncertainty quantification for both codified data and narrative notes.
What is Already Known	Existing word embedding techniques in natural language processing (NLP) or based on co-occurrence matrix lack inherent support for sparse knowledge graph (KG) creation and joint representation of large-scale codified and NLP EHR concepts.
What this Paper Adds	This study introduces a method for generating low-dimensional KG embeddings for both codified and NLP EHR concepts, incorporating uncertainty measures on the links, andconstructing a sparse, large-scale KG.

2. Methods

2.1. Generative model for the knowledge graph

Suppose there are a total of $d$ EHR codified and NLP concepts, indexed by $\mathcal{V}=\{1,\dots ,d\}$. The semantic meaning of each concept is represented by a $p$-dimensional embedding vector ${\mathbf{V}}_j$ for $j=1,\dots ,d$. These embeddings are generated from a latent Gaussian graphical model [32]: each column of $\mathbf{V}={\left({\mathbf{V}}_1,\cdots ,{\mathbf{V}}_d\right)}^{\top }\in {ℝ}^{d\times p}$ is independent and identically distributed from $N\left(0,{\mathbf{\text{Θ}}}^{-1}\right)$, where the precision matrix $\mathbf{\text{Θ}}$ embeds the conditional dependency network of the $d$ concepts, $\mathcal{G}=(\mathcal{V},\mathcal{E})$, with the vertex set $\mathcal{V}$ representing all EHR concepts and the edge set $\mathcal{E}\subseteq \mathcal{V}\times \mathcal{V}$ characterizing the conditional dependency between the concepts. Our goal is to learn the KG $\mathcal{G}$ with $\mathcal{E}$ characterized by $\mathbf{\text{Θ}}$ in that $(j,k)\in \mathcal{E}$ if and only if ${\mathbf{\text{Θ}}}_{jk}\ne 0$ or equivalently ${\mathbf{V}}_j$ is conditionally dependent on ${\mathbf{V}}_k$ given all remaining embeddings. We aim to identify $\mathcal{E}$ through testing the set of hypotheses $ℍ=\left\{H_{0,jk},(j,k)\in \mathcal{V}\times \mathcal{V}\right\}$:

$${\text{H}}_{0,jk}:(j,k)\notin \mathcal{E}\text{v.s.}{\text{H}}_{1,jk}:(j,k)\in \mathcal{E},\text{for all}(j,k)\in \mathcal{V}\times \mathcal{V}.$$ (1)

For each patient, we observed the recorded clinical concepts for each visit and the date of each visit. We assume that the observed clinical concepts in the EHR are generated from a latent Markov process driven by the embeddings sampled from the graphical model [33]. In specific, let $w_t$ be the concept recorded at the visit date $t$ and the occurrence probability of concept $j$ is modeled by

$$\mathbb{P}\left(w_t=j\mid {\mathit{c}}_t,\mathbf{V}\right)=\frac{\exp \left(\langle {\mathbf{V}}_j,{\mathit{c}}_t\rangle \right)}{{\sum }_{k=1}^d\exp \left(\langle {\mathbf{V}}_k,{\mathit{c}}_t\rangle \right)},$$

where the latent discourse vector ${\mathit{c}}_t$ represents the embedding of the topic at time $t$ and is generated from an autoregressive model

$${\mathit{c}}_1\sim N\left(0,{\mathbf{I}}_p/p\right)\text{and}{\mathit{c}}_t=\sqrt{\alpha }{\mathit{c}}_{t-1}+\sqrt{1-\alpha }{\mathit{ϵ}}_t,\text{with}{\mathit{ϵ}}_t\stackrel{\text{i.i.d.}}{\sim }N\left(0,{\mathbf{I}}_p/p\right)\text{for}t\ge 2,$$

where $0<\alpha <1$ is the weight parameter. Figure 1 illustrates the generation process. The ${\mathit{c}}_t$ represents the latent topic vector at each time (e.g., phenotype, treatment, lab measurement, etc) and changes gradually. The stochasticity of the topic vector explains the change of health status or disease stage of a patient; while the slow random walk (meaning that ${\mathit{c}}_{t+1}$ is obtained from ${\mathit{c}}_t$ with a small random displacement vector) ensures the nearby features are generated under similar discourses. Under this model, the embedding inner product ${\sigma }_{jk}=\langle {\mathbf{V}}_j,{\mathbf{V}}_k\rangle \equiv {\sum }_{k=1}^p{V}_{jl}{V}_{kl}/p$ can be approximated by the population positive point-wise mutual information (PPMI) between concept $j$ and $k$ [34] :

$${\sigma }_{jk}=\text{PPMI}(j,k)+O(1/d),$$

where $\text{PPMI}(j,k)=\max \left\{0,\log \frac{\mathbb{P}(j,k)}{\mathbb{P}(j)\mathbb{P}(k)}\right\}$, $\mathbb{P}(j,k)$ is the co-occurrence probability of the concept pair $(j,k)$ and $\mathbb{P}(j)$ is the occurrence probability of the concept $j$. Therefore, when the number of concepts $d$ has a larger order than the square root of the sample size used to estimate the PPMI, testing $H_{0,jk}$ can be achieved by testing $\text{PPMI}(j,k)=0$ based on the estimated PPMI.

Figure 1:

Data generation process of the EHR occurrence data, and data analytics pipeline. The embeddings of concepts are generated from a graphical model and the occurrence is then driven by a Markov process.

2.2. ARCH representation learning and graph recovery

For large-scale EHR datasets with a massive number of concepts and patient records, it is both statistically and computationally challenging to infer the network due to the latency and the large number of hypotheses involved. Our ARCH representation learning approach carries out the inference in two steps by (i) first screening for $\mathcal{E}$ by identifying marginally dependent concept pairs with nonzero PPMI, and (ii) inferring about the Gaussian graphical model structure of $\mathbf{\text{Θ}}$ via sparse regression. In the first step of screening, we apply the SURE screening [35] by selecting pairs $(j,k)$ with ${\sigma }_{jk}\ne 0$ after controlling for a desired FDR. In the second step, we further infer the edges from the network $G$ via node-wise regression [36]. As the embedding vectors follow the Gaussian graphical model, the conditional distribution of embeddings is

$${\mathbf{V}}_j\mid {\left\{{\mathbf{V}}_k\right\}}_{k\in {ℂ}_j}\sim \text{N}\left(-\sum _{k\in {ℂ}_j}{\mathbf{V}}_k\left({\text{Θ}}_{jj}^{-1}{\text{Θ}}_{jk}\right),{\text{Θ}}_{jj}^{-1}{\mathbf{I}}_p\right),$$ (2)

where ${ℂ}_j$ is the set of concepts related to concept $j$ obtained from the first prescreening step.

2.2.1. Pre-screening by PMI testing

To form a test statistic for $H_{0,jk}:{\sigma }_{jk}=0$ and estimate $\mathbf{V}$, we first calculated the empirical PPMI as $\mathbb{P}\mathbb{P}\mathbb{M}\mathbb{I}=[\mathbb{P}\mathbb{P}\mathbb{M}\mathbb{I}(j,k)]$, with $\mathbb{P}\mathbb{P}\mathbb{M}\mathbb{I}(j,k)=\max \left\{0,\log \frac{\mathcal{C}(j,k)\mathcal{C}(\cdot ,\cdot )}{\mathcal{C}(j,\cdot )\mathcal{C}(k,\cdot )}\right\}$, where $\mathcal{C}(j,\cdot )$ is the row sum of co-occurrence matrix $\mathcal{C}(j,k)$, and $\mathcal{C}(\cdot ,\cdot )$ is the total sum of the co-occurrence. Details for the construction of $\mathcal{C}(\cdot ,\cdot )$ is given in Section 2.3. We next took an SVD of the empirical PPMI matrix as $\mathbb{P}\mathbb{P}\mathbb{M}\mathbb{I}=[\mathbb{P}\mathbb{P}\mathbb{M}\mathbb{I}(j,k)]=\mathbb{U}\text{diag}\left({\text{Λ}}_1,\dots ,{\text{Λ}}_d\right){\mathbb{U}}^{\top }$, we can estimate $\mathbf{V}$ and population PPMI matrix of $d$ concepts respectively as

$$\begin{array}{l}\tilde{\mathbf{V}}={\left({\tilde{\mathbf{V}}}_1^{\top },\cdots ,{\tilde{\mathbf{V}}}_d^{\top }\right)}^{\top }={\mathbb{U}}^{(p)}\text{diag}\left({\text{Λ}}_1^{\frac{1}{2}},\dots ,{\text{Λ}}_{\text{P}}^{\frac{1}{2}}\right),\\ \widehat{\text{PPMI}}=\tilde{\mathbf{V}}{\tilde{\mathbf{V}}}^{\top }={\mathbb{U}}^{(p)}\text{diag}\left({\text{Λ}}_1,\dots ,{\text{Λ}}_{\text{p}}\right){\left({\mathbb{U}}^{(\text{p})}\right)}^{\top },\end{array}$$

where ${\mathbb{U}}^{(p)}$ being the first $p$ singular vectors of $\mathbb{P}\mathbb{P}\mathbb{M}\mathbb{I}$ with positive eigenvalues. The dimension $p$ can be selected to optimize embedding quality similar to KESER [14] by maximizing the area under the Receiver Operating Characteristics curve (AUC) of distinguishing those known relation pairs from random pairs, where known relation pairs are curated from online sources, detailed in the validation studies in Section 2.4.1.

The estimator $\widehat{\text{PPMI}}$ is close to the population PPMI matrix with a high approximation rate and asymptotically normal, which allows us to approximate ${\sigma }_{jk}$ with ${\widehat{\sigma }}_{jk}={\tilde{\mathbf{V}}}_j^{\top }{\tilde{\mathbf{V}}}_k$. Furthermore, we may form test statistic $z_{jk}={\widehat{\sigma }}_{jk}/{\widehat{s}}_{jk}$ to identify ${\sigma }_{jk}\ne 0$ since $z_{jk}$ follows approximately standard normal distribution under the null hypothesis [34], where ${\widehat{s}}_{jk}$ is an estimated standard error for ${\widehat{\sigma }}_{jk}$, as shown in Figure 1 and detailed in Appendix S.1. To control for multiple comparisons, we performed the Benjamini-Hochberg (BH) procedure under dependence and identified related concept pairs with $z_{jk}$ higher than a BH-controlled threshold as detailed in Appendix S.2.

2.2.2. Sparse embedding regression

The FDR-controlled testing procedure based on ${\widehat{\sigma }}_{jk}$ could serve as a prescreening of related concepts from the large number of concept pairs. To further screen for the most relevant concepts to form ${\mathcal{E}}_j=\left\{k:{\text{Θ}}_{jk}\ne 0\right\}$, we further performed a sparse regression of ${\tilde{\mathbf{V}}}_j$ against all embedding vectors identified as related to the concept $j$ after initial screening, denoted by ${ℂ}_j$, to recover ${\mathbf{\text{Θ}}}_j,·{\text{Θ}}_{jj}^{-1}$ and hence its associated graph structure. Due to the potentially large number of elements identified in the pre-screening stage, we adopted an adaptive elastic-net penalized regression [37], which is a hybrid of a Lasso and Ridge regression:

$${\widehat{\mathit{\theta }}}_j(\gamma ,\lambda )=\arg \underset{\mathit{\theta }}{\min }\left[{\Vert {\tilde{\mathbf{V}}}_j-{\tilde{\mathbf{V}}}^{\text{T}}\left[,{ℂ}_j\right]\mathit{\theta }\Vert }_2^2+\lambda \sum _{l\in {ℂ}_j}\left\{\frac{1-\gamma }{2}{\left({\theta }_l\frac{{\widehat{\sigma }}_{jj}\sqrt{{\widehat{\sigma }}_{ll}}}{{\widehat{\sigma }}_{jl}}\right)}^2+\gamma \left|{\theta }_l\frac{{\widehat{\sigma }}_{jj}\sqrt{{\widehat{\sigma }}_{ll}}}{{\widehat{\sigma }}_{jl}}\right|\right\}\right],$$

where ${\tilde{\mathbf{V}}}^{\top }\left[,{ℂ}_j\right]$ is the submatrix of $\widetilde{\mathbf{\text{V}}}$ corresponding to ${ℂ}_j$. The tuning parameters $\lambda$ and $\gamma$ control the support of ${\widehat{\mathit{\theta }}}_j$ and hence the network structure. We determined the optimal values for the hyperparameters $\lambda$ and $\gamma$ for each target concept $j$ by performing a grid search to balance the external and internal validation losses. Specifically, we computed the average of the internal Akaike information criterion (AIC) loss [38] and an external validation loss, which was obtained using an independent dataset ${\tilde{\mathbf{V}}}^{*}$, as detailed in Appendix S.3.

2.3. EHR data sources and preprocessing

We trained a large-scale ARCH KG using EHR data from the VA Corporate Data Ware-house (CDW), integrating both codified and narrative data from 12.6 million patients with at least 1 visit between 2000–2019. All raw data are processed as detailed in Appendix S.4, and then used to create a co-occurrence matrix by counting the number of co-occurrences within a 30-day window across all patients. To reduce noise, we removed concepts that have less than 3000 occurrences and concept pairs that have less than 1000 co-occurrences. Furthermore, we removed all concepts that co-occur with more than 95% of other concepts as they tend to be overly non-specific. This results in a total of over 61, 000 concepts, out of which 51, 423 are CUIs and 9, 586 are codified concepts.

2.4. Validation analyses

The ARCH KG was validated in four downstream tasks: (1) detecting known similar or related clinical concepts; (2) detecting drug side effects; (3) disease phenotyping; and (4) profiling of patients with AD. For the detection of known relationships and drug side effects, we also compared to embedding vectors from GloVe [25], from node2vec [39], and from several pretrained language models (PLMs). GloVe is an unsupervised learning algorithm that leverages feature-feature co-occurrence statistics to obtain vector representations. We used the same co-occurrence matrix to train embeddings from both ARCH and GloVe. Node2vec, which is a representational learning algorithm designed for networks, optimizes the likelihood of preserving the neighborhoods in a network. We transformed the PPMI matrix into a binary network where an edge exists between concepts $j$ and $k$ if $\mathbb{P}\mathbb{P}\mathbb{M}\mathbb{I}(j,k)>0$. This approach was applied to generate concept embeddings. However, due to its substantial memory requirements, we limited node2vec’s application to codified data only, as running it on the entire dataset would require more than 700GB of memory. The PLM embeddings we compared to are based on Bidirectional Encoder Representations from Transformer (BERT) [40], including Self-aligning pretrained BERT (SAPBERT) [41], BERT for Biomedical Text Mining (BioBERT) [42], and BERT pretrained with PubMed (PubmedBERT) [43]. BERT’s model architecture is a multi-layer bidirectional Transformer encoder, while BioBERT, PubMedBERT and SAPBERT are pretrained on different sources based on BERT. BioBERT is pretrained on both general domain corpora and biomedical domain corpora (PubMed abstracts and PMC full-text articles), PubMedBERT is pretrained purely with in-domain text (PubMed text), and SAPBERT is pretrained on the biomedical KG of UMLS. The language model based embeddings were obtained only based on the description of the EHR concepts (e.g. preferred term for the CUI and code description). To examine the performance of including both codified data and NLP data, we also perform the proposed ARCH algorithm on codified data alone and NLP data alone.

2.4.1. Detecting known relationship pairs

We curated different categories of known relation pairs from online knowledge sources including similar pairs and related pairs as detailed in Appendix S.6.1. For each type of relationship, we calculated the cosine similarities of the embedding vectors of related pairs and those of randomly selected pairs to calculate AUC of the cosine similarities in distinguishing known pairs from random pairs. The dimension of embedding is chosen by optimizing the AUC. We then performed ARCH testing procedure to determine whether a pair of entities are related with FDR chosen at 1%, 5%, and 10%, and reported the power of the ARCH procedure. The power of ARCH procedures refers to the proportion of true positive relationships correctly identified among all known relationships. Since no existing procedures can control FDR, we calculated the power of other algorithms in detecting known relationships by ranking entity pairs according to cosine similarity generated from their corresponding embeddings and then selecting the top $M$ entity pairs as significant, where $M$ is the number of entity pairs selected by ARCH. Among those $M$ pairs, we calculated the proportion of those known to be related as their power. We have also computed the Spearman’s correlation between the algorithm output and domain expert ratings on 507 UMLS concept pairs for semantic similarity and ratings on 515 UMLS concept pairs for semantic relatedness. These reference standards are obtained from [44, 45].

2.4.2. Detecting drug side effect

We first obtained labels from the Side Effect Resource (SIDER)² database of drugs and adverse drug reactions [46] as detailed in Appendix S.6.2, resulting in 128, 220 drug-AE (Adverse Events) pairs. The negative pairs were obtained from [47] and we kept the same number of negative pairs as the number of positive pairs. The AUC and power for detecting drug side effects based on ARCH embeddings or $p$-values were calculated similarly to those for the relation detection. We further evaluated the quality of ARCH embeddings based on the performance of a few-shot supervised model [48] for this task, as detailed in Appendix S.7. We used 1% of the positive and negative pairs to estimate model parameters, another 1% as validation data to select optimal tuning parameters, and the remaining 98% pairs as a test data set for evaluation.

2.4.3. Disease phenotyping

A major bottleneck for conducting translational research studies with EHR is the lack of large-scale precise data on disease outcomes needed for predictive modeling. Most unsupervised algorithms require the specification of relevant features [49, 50, 51, 52, 53]. We next illustrate how the ARCH network can serve as an effective feature selection tool for EHR phenotyping and compare it to the existing KG-based feature selection tool, KESER [14]. KESER is a method that generates embeddings for codified data from EHR data and constructs a knowledge graph through sparse embedding regression. It can serve as both an embedding construction technique and a feature selection tool for codified codes. We compared PheNorm [51] trained with ARCH selected features, PheNorm trained with KESER selected features, the MAP algorithm which only uses counts of the main PheCode and CUI, and healthcare utilization [52], as well as two benchmark methods that use the logarithm of the count of the main disease ICD code plus one (Main ICD Only) and the logarithm of the count of the disease CUI plus one (Main NLP Only) as the disease predictive scores, respectively. PheNorm is a phenotyping algorithm that computes the score of each patient to the target disease by computing an inner product of the patient’s normalized feature vector and a coefficient vector, where the feature vector aggregates the information in the candidate feature set with denoising self-regression via dropout training. We trained these phenotyping algorithms using EHR data from 53, 549 Mass General Brigham (MGB) Biobank participants for 8 conditions: coronary artery disease (CAD), Crohn’s disease (CD), rheumatoid arthritis (RA), ulcerative colitis (UC), Congestive heart failure (CHF), type 1 diabetes mellitus (T1DM), type 2 diabetes mellitus (T2DM) and depression. To evaluate their accuracy, the CAD, CD, RA, UC, CHF, T2DM, T2DM and depression phenotyping algorithms were validated against 187, 138, 154, 127, 114, 540, 285 and 540 labeled observations curated via manual chart review, and the AUCs were reported.

2.4.4. Profiling of AD patient via ARCH embeddings

Semantic representation of the EHR concepts can be linked with patient-level EHR data to represent patient clinical profile [54, 55, 56]. These patient embeddings can then be applied to perform downstream tasks such as identifying “patient like me” [57] and mortality prediction [58]. However, representing a patient’s clinical profile for a specific condition, such as AD, requires the knowledge of other EHR features relevant to AD progression as well as their relative importance [59]. Our ARCH KG serves this purpose in that it can generate embeddings to represent an AD patient. To demonstrate this, we used EHR data of 38, 267 patients with AD diagnosis, collected from the University of Pittsburgh Medical Center (UPMC) over the period 2011–2021. We selected the AD-relevant features and generated patient embeddings using the term frequency-inverse document frequency (TF-IDF) procedure as detailed in Appendix S.8. As an illustration, we applied $k$-means algorithm to cluster patients into two groups using the patient embeddings. We analyzed the mortality risk of the two groups using the Kaplan-Meier (KM) curve of the time from first AD diagnosis to death. We characterized the between group differences in patient profiles with respect to the distributions of AD-related features selected via ARCH.

3. Results

By optimizing the AUC of distinguishing known relation pairs from random pairs as detailed in Section 2.4.1, we set the dimension of embeddings output from ARCH as $p=1500$ to optimize the embedding quality. Visualizations of the ARCH network can be found at https://celehs.hms.harvard.edu/ARCH/, https://phenomics.va.ornl.gov/web/cipher/vistools, which enables users to visualize concepts relevant to a set of target concepts.

3.1. Detecting known relationship pairs

The AUCs and power in detecting known relationships are summarized in Table 2, with detailed accuracy for specific types of relationships provided in Table A4 in Appendix S.10. The embeddings trained by ARCH achieved an AUC of 0.828 for detecting CUI-CUI similar pairs and 0.844 for detecting CUI-CUI related pairs, while GloVe derived embeddings attained lower AUCs of 0.825 and 0.760. Pretrained language model derived embeddings, including PubmedBERT, BioBERT and SAPBERT, achieved much lower AUCs, ranging from 0.572 to 0.759. The ARCH screening procedure had a power of 0.981 for code-code similar pairs and 0.901 for code-code related pairs under the target FDR of 0.05. In contrast, the highest power among the five benchmarks (GloVe, node2vec, PubmedBERT, BioBERT, and SAPBERT) was 0.956 for code-code similar pairs and 0.909 for code-code related pairs.

Table 2:

AUCs and power at FDR = 0.05 in detecting known similar (s) pairs and related (r) pairs with ARCH $p$-values (p) and cosine similarities (c), GloVe, node2vec (n2v), PubMedBERT (Pub), BioBERT (Bio), SAPBERT (SAP) with target FDR being 0.05. For ARCH, we trained using full (f), codified only (code), and CUI only (CUI) features. Glove is trained with full features, node2vec is trained with codified-only features due to memory limitation (node2vec requires more than 700GB memory if trained with full features), and PubMedBERT, BioBERT, SAPBERT utilize the code description of full features. Shown also are the Spearman’s correlation (Corr) between the algorithm output and domain expert ratings for the human curated pairs with similarity and relatedness rankings.

	Type		ARCH						Glove	n2v	Pub	Bio	SAP
			p(f)	c(f)	p(code)	p(CUI)	c(code)	c(CUI)	(f)	(code)	(f)	(f)	(f)
AUC	code-code	s	0.849	0.904	0.867	-	0.906	-	0.790	0.796	0.631	0.603	0.775
		r	0.776	0.797	0.770	-	0.781	-	0.779	0.667	0.584	0.544	0.636
	CUI-code	s	0.943	0.965	-	-	-	-	0.903	-	0.746	0.726	0.908
	CUI-CUI	s	0.864	0.826	-	0.857	-	0.825	0.825	-	0.728	0.641	0.759
		r	0.862	0.844	-	0.855	-	0.839	0.760	-	0.670	0.572	0.712
Power	code-code	s	0.981	0.980	0.962	-	0.962	-	0.955	0.956	0.764	0.681	0.907
		r	0.901	0.897	0.825	-	0.819	-	0.909	0.810	0.757	0.660	0.790
	CUI-code	s	0.925	0.924	-	-	-	-	0.863	-	0.579	0.521	0.786
	CUI-CUI	s	0.896	0.887	-	0.882	-	0.870	0.874	-	0.675	0.526	0.708
		r	0.876	0.865	-	0.855	-	0.841	0.828	-	0.635	0.517	0.547
Corr		s	0.497	0.598	-	0.506	-	0.595	0.310	-	0.120	0.085	0.234
		r	0.478	0.573	-	0.494	-	0.572	0.333	-	0.085	0.035	0.217

When the proposed ARCH model is trained on codified data alone, the power for detecting known similar and related code-code pairs decreases. For example, the power of ARCH $p$-values in detecting code-code related pairs at FDR = 0.05 was 0.901 when using all concepts and 0.825 when using codified data only. Node2vec showed much worse performance in detecting these code-code pairs, with AUCs being 0.956 for similar pairs and 0.810 for related pairs. The Spearman’s rank correlation between the cosine similarities from the ARCH embedding vectors and the manual ratings is as high as 0.598 for similar pairs and 0.573 for related pairs. Other benchmarks, including GloVe and PLM embeddings, result in correlation lower than 0.35.

3.2. Identifying drug side effects

Table 3 shows the AUC-ROC and power of different methods in detecting drug side effects. The unsupervised ARCH embeddings and the screening test $p$-values, which are trained on both structured and unstructured data, achieved substantially a higher AUC of 0.723 and 0.747, compared to those trained on the structured data or unstructured data alone, and those from other benchmarks which ranged from 0.584 to 0.708. With few-shot supervised training, the ARCH embeddings attained an AUC of 0.826 while the AUC of the fine-tuned GloVe embeddings being 0.818, fine-tuned node2vec embeddings being 0.65 and fine-tuned PLM embeddings remained below 0.73. Comparing the power in detecting drug side effects using either codified data or NLP data alone versus both codified and NLP data, we find that jointly analyzing codified and NLP data greatly improved the ability to capture side effects for most drug classes as shown in Figure 2. Furthermore, we show most of the side effects of Levothyroxine and Hydrocodone can be detected by ARCH while a significant fraction of the side effects can only be captured with the help of NLP data. More examples of word-cloud figures are shown in Appendix.

Table 3:

AUCs and the sensitivities of different benchmark methods compared with ARCH for identifying drug side effects. The first and the third blocks show the performance of each method without supervision (AUC(U) stands for AUC with unsupervised embeddings and Power(U) stands for power with unsupervised embeddings), while the second and the fourth blocks show the performance of the method with supervised learning (AUC(S) stands for AUC with supervised embeddings and Power(S) stands for power with supervised embeddings). ARCH has different deformations: p(f) stands for the $p$-value output from ARCH trained with both codified data and NLP data; c(f) stands for the cosine similarity between the embedding vectors output from ARCH trained with both codified data and NLP data; p(code) stands for the $p$-value output from ARCH trained with codified data alone; c(CUI) stands for the cosine similarity between the embedding vectors output from ARCH trained with NLP data alone. n2v stands for node2vec, Pub stands for PubmedBERT, Bio stands for BioBERT and SAP stands for SAPBERT. Glove is trained with full features, node2vec is trained with codified-only features due to memory limitation (node2vec requires more than 700GB memory if trained with full features), and PubMedBERT, BioBERT, SAPBERT utilize the code description of full features.

	FDR	ARCH						Glove	n2v	Pub	Bio	SAP
		p(f)	c(f)	p(code)	p(CUI)	c(code)	c(CUI)	(f)	(code)	(f)	(f)	(f)
AUC(U)		0.760	0.759	0.531	0.588	0.538	0.586	0.708	0.526	0.637	0.620	0.597
AUC(S)		-	0.899	-	-	0.615	0.686	0.818	0.651	0.649	0.627	0.724
Power(U)	0.1	0.599	0.545	0.211	0.451	0.300	0.320	0.577	0.143	0.383	0.346	0.386
	0.05	0.586	0.533	0.206	0.444	0.295	0.312	0.568	0.138	0.371	0.334	0.374
	0.01	0.564	0.510	0.195	0.428	0.286	0.296	0.551	0.130	0.346	0.314	0.352
Power(S)	0.1	-	0.647	-	-	0.437	0.494	0.652	0.243	0.473	0.403	0.462
	0.05	-	0.638	-	-	0.431	0.484	0.643	0.237	0.463	0.389	0.449
	0.01	-	0.618	-	-	0.419	0.466	0.623	0.224	0.441	0.366	0.424

Figure 2:

(a) Sensitivity of detecting drug-side effects pairs with only codified data, that with NLP, and that with both codified data and NLP with ARCH under target FDR 0.05. The word clouds of the side effects of two sample drugs - (b) Levothyroxine on the left and (c) Hydrocodone on the right. The surrounding words describe side effects. The words colored red are detected using codified data only while the words colored orange or red are detected by using both codified data and NLP codes. The words colored by grey are undetected. The size of the words is determined by the cosine similarity with the target drug code.

3.3. Disease phenotyping

Figure 3 shows the AUCs of 8 phenotyping algorithms validated on labeled data from MGB. PheNorm with ARCH selected features performs the best among all methods. The AUCs of the PheNorm algorithms with features selected by ARCH exceeded 0.9 for all 8 diseases and on average were 0.028 ($p$-value $3.30\times {10}^{-5}$), 0.067 ($p$-value $9.87\times {10}^{-12}$), 0.081 ($p$-value $3.29\times {10}^{-11}$), and 0.076 ($p$-value $1.06\times {10}^{-11}$) higher than that of PheNorm with ARCH trained on codified features alone, PheNorm with KESER features, MAP, ICD only and NLP only. The gain in performance is particularly noteworthy for conditions that benefit from NLP features. For example, after applying ARCH in the feature selection step, the AUC of the PheNorm algorithm for depression increased from 0.857 of KESER to 0.927 ($p$-value $2.47\times {10}^{-4}$).

Figure 3:

The AUC of different phenotyping algorithms trained with different feature sets across 8 diseases.

3.4. Profiling of AD patient via ARCH embeddings

The AD cohort consists of about 64.7% female patients, 90.3% white and 7.6% black patients, with an average age of 82 years at first ICD code for AD and an average lifespan of 86 years. K-means clustering of the ARCH-based patient embeddings resulted in two subgroups: a late stage group consisting of 12.3% the patients and a early stage group formed by the remaining patients. As shown in Figure 4, the 5-year survival rate was 42.0% (95% CI: [38.6%, 45.7%]) and 80.9% (95% CI: [80.3%, 81.6%]) for the late and early stage groups, respectively.

Figure 4:

The KM survival curves for the early stage and late stage groups identified via $k$-means clustering of the ARCH patient level embeddings.

Figure 5 highlights the top disease and drug features with the largest differences between the late and early stage groups. The phenotype features associated with the group with a smaller population are common phenotypes at the late stage of AD. Pneumonia is one of the two most serious medical conditions seen in late-stage AD patients [60]; hypovolemia and hypernatremia may be found in association with dehydration, which can occur in impaired late-stage AD patients who are dependent on others for fluid intake [61, 62, 63]. On the other hand, the features that appear more frequently in the early stage group of patients, which are colored blue in the figure, are either common signs or possible causes of AD. Memory deficits begin from the early stage of AD [64], while vitamin deficiency and hypothyroidism are risk factors for AD [65, 66, 67]. As shown in the network of drug features and procedure features, the features ‘atorvastatin’, ‘metformin’, ‘escitalopram’, ‘melatonin’, among others, have been shown to moderate AD or slow down the progression of cognitive impairment in AD patients [68, 69, 70, 71]. Memantine, a type of N-methyl-D-aspartate receptor antagonist, is the only drug approved for use in moderate to severe AD under current AD treatment guideline [72, 73]; Rivastigmine and Donepezil are the drugs approved by Food and Drug Administration (FDA) for AD treatment besides Memantine and two accelerated approval drugs³; all these three drugs are more common in the late stage group of patients. With these references, the clustering of patients is practical and realistic, indicating the good quality of patient embedding based on the feature selection by ARCH.

Figure 5:

The word cloud and the network of (a)(c) phenotype features; and (b)(d) drug features that drive the differences between the two subgroups. The size of the feature is determined by the between-group difference in the average intensity of such a feature. Red-colored features represent higher average intensity in the late stage group and blue-colored features represent higher intensity in the early stage group.

3.5. Computation cost

The main steps for our proposed algorithm are fast. First, the computation time for constructing co-occurrence matrix highly depends on the number of patients and features. However, we are able to create the co-occurrence matrix through distributed computing with the help of code at https://github.com/rusheniii/LargeScaleClinicalEmbedding. For the 12.6 million patients from VA, it cost us 3 days to compute the co-occurrence matrix with the help of 12 machines with 128 cores. Second, the variance estimation step and sparse embedding regression step are much faster if run in parallel. Splitting into 10 jobs, it takes less than 10 hours to finish these two steps.

4. Discussion and Conclusion

Utilizing summary-level EHR data, the ARCH KG learning approach provides a highly scalable method for effectively representing codified and narrative EHR concepts on a large scale, while also recovering their network structure. The proposed ARCH algorithm outputs both cosine similarity and $p$-value to evaluate the relationship between two concepts. The latter ($p$-value) includes uncertainty assessment, making it suitable for reproducible and robust analysis. On the other hand, the former (cosine similarity) is based on similarities between the embedding vectors and lacks a measure of uncertainty, making it more appropriate for predictive tasks. The differences in performance stem from these underlying characteristics. To choose between these variants, one should consider the specific requirements of their use case: $p$-value for scenarios needing reliability and robustness, and cosine similarities for tasks prioritizing prediction accuracy. The $p$-value should be used when reliability and robustness are critical, such as in scenarios where reproducibility is important or when the sample size is small. For example, if you are adding new features to a knowledge graph or working with a new dataset, ARCH($p$) is preferred because it assesses the significance of relationships and is more generalizable. Cosine similarity is more computationally efficient and can be useful for tasks where computational resources are limited. To address varying evaluation requirements, we propose a complementary integration of $p$-values and cosine similarity scores. By normalizing these metrics (e.g., via Min-Max scaling), their combination in downstream tasks can enable a balanced approach, prioritizing $p$-values for statistical rigor and cosine similarities for computational efficiency. This framework enhances ARCH’s adaptability and ensures it meets diverse analytical needs. In conclusion, the VA EHR-derived ARCH embeddings represent the first large-scale EHR embeddings to include both codified and NLP concepts, with the incorporation of NLP concepts proving particularly beneficial in real-world applications and a wide range of predictive modeling tasks such as drug side effect monitoring and disease phenotyping. Additionally, the network structure derived from ARCH is constructed with a statistically guaranteed false discovery rate.

The versatility of the learned ARCH embeddings makes them ideal for a broad range of downstream tasks. These embeddings demonstrate greater robustness than existing PLM-based embeddings. Our semantic representation evaluations and drug side effect prediction studies show that the ARCH embeddings can effectively capture the semantic relationships between EHR entities and concepts. Our results indicate that the ARCH embeddings with few shot training have the potential to achieve high accuracy in KG-related tasks, such as entity matching and relation extraction. Additionally, the ARCH embeddings can serve as pre-trained representations of EHR concepts that can be linked to individual-level EHR data, further improving patient-level prediction tasks, as demonstrated in the AD patient profiling study. Joint representations of both codified and NLP data also enable more comprehensive multi-modal modeling of EHR data, significantly enhancing prediction performance for outcomes that require predictors that are not well-coded.

The use of summary-level data in learning the ARCH network creates an opportunity for collaborative training of knowledge graphs across multiple institutions. This approach can enhance the quality of the trained representation and improve the portability of downstream prediction algorithms. However, co-training ARCH embeddings using multi-institutional data faces a challenge in dealing with coding differences between institutions. Even for institutions that have mapped their local EHR codes to a common ontology, such mappings are often incomplete. Future research needs to explore co-training knowledge graphs for overlapping yet non-identical EHR concepts from multiple institutions based on summary-level data. Furthermore, average embedding across a range of representative contexts may help to improve the stability and quality, thus increasing the ability of serving the downstream tasks. Currently, the ARCH network relies solely on EHR occurrence patterns of concepts, disregarding valuable information contained in their descriptions. Incorporating both occurrence patterns and descriptions through language models is an intriguing avenue for further research in improving the network. The global embeddings for narrative concepts trained with large corpora of full-length biomedical articles have also attained strong results on the UMNSRS dataset [74]. The rank correlation between human annotation vs cosine similarity was 0.62 and 0.58 for embeddings from [74] and 0.598 and 0.573 for ARCH embeddings. More detailed comparisons between these two sets of embeddings warrant further research. However, the results presented in the above section collectively highlight the substantial value of the ARCH framework, not only in providing robust, scalable, and reproducible embeddings for clinical data but also in paving the way for innovative and impactful applications in biomedical research and healthcare delivery, despite limitations seen in specific evaluation sets.

Highlights.

• Problem or Issue: Managing massive and complex electronic health record (EHR) data presents challenges in feature representation and uncertainty quantification for both codified data and narrative notes.

• What is Already Known: Existing word embedding techniques in natural language processing (NLP) or based on co-occurrence matrix lack inherent support for sparse knowledge graph (KG) creation and joint representation of large-scale codified and NLP EHR concepts.

• What this Paper Adds: This study introduces a method for generating low-dimensional KG embeddings for both codified and NLP EHR concepts, incorporating uncertainty measures on the links, and constructing a sparse, large-scale KG.

Data availability

The data that support the findings of this study are available from the Veterans Affairs (VA) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of VA. All original code has been deposited at 10.5281/zenodo.10426582 and is publicly available.