Ontology Enrichment using a Large Language Model: Applying Lexical, Semantic, and Knowledge Network-Based Similarity for Concept Placement

Abstract

Objective:

Ontologies are essential for representing the knowledge of a domain. To make ontologies useful, they must encompass a comprehensive domain view. To achieve ontology enrichment, there is a need to discover new concepts to be added, either because they were missed in the first place, or the state-of-the-art has advanced to develop new real-world concepts. Our goal is to develop an automatic enrichment pipeline using a seed ontology, a Large Language Model (LLM), and source of text. The pipeline is applied to the domain of Social Determinants of Health (SDoH), using PubMed as a source of concepts. In this work, the applicability and effectiveness of the enrichment pipeline is demonstrated by extending the SDoH Ontology called SOHOv1, however our methodology could be used in other domains as well.

Methods:

We first retrieved PubMed abstracts of candidate articles with existing SOHOv1 concepts as search terms. Next, we used GPT-4–1201 to extract semantic triples from the abstracts. We identified concepts from these triples utilizing lexical, semantic, and knowledge network-based filtering. We also compared the granularity of semantic triples extracted with our method to the triples in the SemMedDB (Semantic MEDLINE Database). The results were evaluated by human experts and standard ontology tools for checking consistency and semantic correctness.

Results:

We expanded SOHOv1, which contained 173 concepts and 585 axioms, including 207 logical axioms to SOHOv2, which contains 572 concepts, 1,542 axioms, including 725 logical axioms. Our methods identified more concepts than those extracted from SemMedDB for the same task. While we have shown the feasibility of our approach for an SDoH ontology, the methodology is generalizable to other ontologies with an existing seed ontology and text corpus.

Conclusions:

The contributions of this work are: Extracting semantic triples from PubMed abstracts using GPT-4–1201 utilizing prompt chaining; showing the superiority of triples from GPT-4–1201 over triples from SemMedDB for SDoH; using lexical and semantic similarity search techniques with knowledge network-based search to identify the concepts to be added to the ontology; confirming the quality of the new concepts with human experts.

Statement of Significance

1. Issue:

Adding new concepts to an existing ontology is difficult, error-prone, and time-consuming.

2. What is already known:

LLM-based ontology learning in domains such as biomedicine has not yet led to the hoped-for success in fully automated concept extraction for correct, consistent, and complete ontology enrichment.

3. What this paper adds:

This paper develops a novel pipeline for ontology expansion. We explore prompt chaining with GPT-4–1201 to extract semantic triples from PubMed abstracts in the domain of Social Determinants of Health (SDoH). We present a combined lexical, semantic, and knowledge network-based similarity methodology for extending a pre-existing SDoH ontology.

Graphical Abstract

1. Introduction

Ontologies provide a structured representation of domain-specific knowledge, facilitating information retrieval and data integration, and enabling decision-making systems. However, keeping ontologies up-to-date and enriched with new concepts is a time-consuming process that often requires expert involvement. Ontology enrichment techniques encompass various methods and approaches to improve and expand ontologies. Many techniques have been explored in the past, including leveraging sources such as dictionaries or domain-specific terminologies, incorporating concepts and relations from real-world data, ontology alignment and merging, and crowdsourcing-based ontology enhancement [1–3]. These ontology enrichment techniques can be employed individually or in combination, depending on the specific goals, domain, and available resources. The application of these techniques contributes to the continual improvement and evolution of ontologies, enabling a more comprehensive understanding and representation of knowledge within a given domain.

The scientific literature serves as a vast repository of knowledge, containing valuable information about concepts, relationships, and emerging trends within various domains. By mining the literature, ontologies can be enriched with new concepts and relationships that may not be present in existing knowledge sources. The literature provides a means to access up-to-date and domain-specific knowledge, ensuring that ontologies remain relevant and comprehensive. Additionally, research articles can help validate the correctness and relevance of existing concepts within an ontology. Nevertheless, accommodating the increasing number of concepts from recent articles involves extensive literature reviews. The idea of acquiring semantic information from text dates back to the 1960s with Harris’ distributional hypothesis [4] and Hirschman and Sager’s work [5]; these early approaches focused on sublanguage-specific word classes using syntactic patterns from domain-specific corpora. Semantic relations between entities in a sentence can often be captured based on syntactic patterns. The most common relations include hyponym-hypernym, meronym–holonym (part-whole), and cause-effect. They are often expressed by distinct syntactic patterns [6, 7]. “Syntactic patterns” refer to the rules and structures that govern how words combine to form phrases, clauses, and sentences. These patterns are essential for conveying meaning and to define the language’s grammatical correctness. Most human readers can effortlessly identify the syntactic correctness of a sentence, which is not the case for machine learning-based models that are more suitable for statistical pattern recognition than syntactic pattern recognition. Thus, designing algorithms that can detect these patterns is imperative to perform semantic analysis efficiently. Automatic extraction of such patterns can help with subsequent problems, such as Named Entity Recognition and ontology enrichment [8]. Although manual or semi-automatic techniques for extracting syntactic patterns have been reported in the literature, designing an automatic extraction method for all semantic relations is still a challenging problem.

LLMs such as GPT-4 [9], Llama [10], Claude [11], and PaLM [12] have emerged as powerful tools in natural language processing (NLP) across various domains, having been trained on billions of documents, enabling them to process and comprehend large volumes of text efficiently. This extensive training enables them to identify the semantic rules governing language use and they can generate information across different domains. While automated ontology enrichment is still an unresolved problem, the advent of LLMs allows steps towards this goal. We propose a pipeline that leverages LLMs to automatically extract semantic triples from the biomedical literature, focusing on is-a relationships. Specifically, we utilize OpenAI’s GPT-4 (GPT-4–1201) to automate the inherently labor-intensive and often subjective literature review process by extracting triples from PubMed articles. The resulting triples serve as a valuable resource for concept enrichment in ontologies and for constructing a knowledge graph (KG) [13]. A significant challenge lies in the large volume of extracted triples, which may include duplicates and require filtering. To address this challenge, we propose an automated process integrating multiple techniques to identify and filter parent concepts from the extracted triples. Our approach combines lexical, semantic, and knowledge network-based similarity measures to retrieve the most promising triples for concept enrichment.

To demonstrate the applicability and effectiveness of our proposed pipeline, we use Social Determinants of Health (SDoH) as a case study. SDoH are non-clinical factors that affect the health outcomes of individuals [14]. Addressing the impact of SDoH is a significant step in attaining health equity. Health equity is achieved when all individuals, irrespective of their social circumstances, have the opportunity to achieve their full health potential. Hence, developing an SDoH ontology is crucial for systematically categorizing and recording the complex factors affecting health outcomes. Recently, with the White House releasing the US playbook to address SDoH [15] and growing research on SDoH factors such as digital redlining [16], enriching an SDoH ontology is a promising use case to evaluate our approach. Moreover, in our previous research, we have developed a Social Determinants of Health Ontology (SOHO)¹ [17].

The first version, SOHOv1, was developed in a top-down manner, utilizing a scoping review and PRISMA guidelines; that version covers a broad array of factors but lacks granularity. However, performing a scoping review is resource-intensive in terms of time and human effort. With the increasing number of published medical articles, it is nigh impossible to keep track of new concepts relevant to the domain. As of June 2025, the total number of articles in PubMed on the topic “social determinants of health” over the last four years was 11,437, compared to a total of 17,588 articles since 1997 (actualized on June 10, 2025). These factors make SOHO an ideal seed ontology to illustrate our pipeline.

In summary, we introduce a new approach that utilizes GPT-4–1201 to automate the labor-intensive task of literature review by extracting <subclass, is-a, superclass> triples from PubMed articles. To deal with the large volume of extracted triples and avoid duplication, an automated process that combines various similarity techniques to identify and filter the triples is used. Algorithms are presented to identify missing concepts and perform concept placement in the existing SDoH ontology. This framework of ontology expansion can be applied to any domain where sufficient literature and a “seed ontology” exist and saves the manual effort of performing a scoping review.

2. Background

2.1. Ontology enrichment

With the dynamic nature of the medical domain, enriching the existing domain ontologies with concepts and relationships to capture this evolving knowledge is paramount. Traditional ontology building involves a top-down approach of collaborating ontology developers and domain experts, which is time-consuming [18]. In this work of ontology enrichment, we focus on a bottom-up approach as the process involves enriching the conceptual content of an existing ontology. In recent research on automatic ontology expansion, Elnagar et al. [19] utilized KGs to develop a domain-independent automatic ontology and used association rule mining to check for inconsistencies and eliminate them. They utilized a complex embedding model for KG completion for each triple. According to Elnagar et al. [19], the goal of KG completion is to assign a higher score to a correct triple compared to an incorrect one. Later, they compared the generated ontology with a reference terminology. Hence, reference and benchmark ontologies are needed to train the knowledge completion algorithms before they are used for refinement. That framework challenges medical informatics since the medical field has many specialized terminologies with substantial overlap [20].

Methods based on sentence structure and rules have been employed previously for developing ontologies [6]. To construct syntactic patterns, Kabir et al. [6] developed ASPER, an attention-based supervised learning model that uses a bi-directional Long Short-Term Memory (Bi-LSTM) architecture augmented with an attention mechanism to identify patterns indicating relations such as hyponym-hypernym, cause-effect, and meronym-holonym. Once this edge set was obtained, the authors used a frequent itemset mining (FIM) algorithm called ECLAT to construct the syntactic patterns. The patterns developed are dependent on the collection of training sentences and the thresholds used in ECLAT. Thus, ASPER sometimes fails to extract rare patterns, if sufficient support for them is absent in the training dataset.

Trajanoska et al. [21] demonstrated that meaningful information could be extracted from unstructured data and represented as a KG. They have shown two separate approaches: 1) Using REBEL (Relation Extraction By End-to-end Language generation), which is a generative seq2seq model to translate the raw text into a triple format 2) Using ChatGPT, as a conversational agent. According to the authors, ChatGPT generated whole sentences instead of extracting entities from the articles and linking them with relationships. Hence, the authors refined the prompt to the ChatGPT, specifying explicitly to generate an OWL ontology and to return the result in RDF Turtle format. Then they incorporated additional classes and properties generated by ChatGPT. The KGs generated with the proposed methods were manually reviewed, and the authors concluded that the second approach utilizing ChatGPT with refined prompting created a KG of similar quality as the REBEL approach.

To address the limitation of pre-assumed concept discovery, and to address the focus of relying primarily on atomic rather than complex concepts with logical operators, Dong et al. [22], developed an evaluation framework. This framework focuses on two tasks: discovering mentions of concepts that are not yet included in the knowledge base (KB), what they call out-of-KB mention discovery, and accurately placing these new concepts within the ontology. An ideal dataset for concept discovery and placement, as per the authors, requires a real-world text corpus and a large OWL ontology, with gold-standard directed edges for each out-of-KB concept linked to the mentions in the corpus. In the paper, the authors utilized MedMentions [23], where the concepts were manually and exhaustively linked to the UMLS, and applied KB versioning with SNOMED CT (2014 vs. 2017) to simulate newly emerging concepts. Evaluation using both rule-based and neural network-based methods showed that BLINKout (a BERT-based entity linking model) significantly outperformed rule-based baselines in out-of-KB mention discovery. For concept placement, GPT-3.5 and edge-bi-encoders achieved only average performance in selecting the top-1 from the top-50 edge candidates, highlighting the challenge of placing unseen concepts into complex biomedical ontologies.

Being aware of the limitations of LLMs in terms of token length and of the absence of in-context prompt learning designs, Wei et al. [25] formulated the triples for KG enrichment as a query q. They formulate each query as q = (h, r, ?) or q = (?, r, t) in the link prediction task, where “?” denotes the missing tail or head entity to be predicted. Possible subject or object concepts are ranked using a retriever model. They used the RotatE retriever model and trained it to generate KG embeddings of concepts as vectors, and of relations as rotations in a complex vector space. The model first processes the query independently and generates an ordered candidate entity list, which is then passed to GPT-3.5-turbo to perform a re-ranking of the top m entities returned by the retriever. The authors utilize the FB15k-237 and WN18RR KGs and the RotatE retriever model to evaluate the proposed methods. FB15k-237 is a subset of the Freebase KG, which includes commonsense knowledge about topics such as movies, sports, awards, and traveling. WN18RR is a subset of WordNet, which contains knowledge about English morphology. The model proposed by the authors for KG link prediction performs well for link prediction benchmarks and long tail entities, but the proposed method works mostly for general KGs.

Sun et al. [26] developed a KG generation framework called Doc2KG, utilizing multimodal data such as website data forms, Excel sheets, PDF files, and images. Docs2KG involves two main stages, dual-path data processing and multimodal unified KG construction. The dual-path data processing stage processes the input documents into textual content, images, and tables. The multimodal unified KG construction stage enriches the processed information with structural and semantic relationships. Doc2KG parses heterogeneous and unstructured documents for subsequent integration into a unified KG by combining parsers and document segmentation models.

Mukanova et al. [27] describe the method and technology that they used to extract text corresponding to the semantics of an ontology model. The authors utilize the aiohttp library in Python to extract webpages with data relevant to the ontology under consideration. An LLM, specifically ChatGPT 3.5, was used to extract domain-specific data, for which the authors have shown a sample template. The template consisted of various lists for countries, regions, settlements, etc. GPT 3.5 processed this template to populate the lists with values derived from the text. Then, the Protégé VOWL plugin added each list as classes under the OWL root class “Thing.” This approach would presumably be less effective in Medical Informatics, because in the medical domain the grouping of terms and concepts often is constructed based on complex relationships between them, such as disease classifications, symptoms, treatments, patient demographics, etc.

Dong et al. [28] presented an innovative approach for ontology enrichment using a neural network-based framework that leverages fine-tuned and pre-trained language models (PLM). The paper’s focus is to solve the problem of placing a new concept into an ontology by inserting it into an edge corresponding to a subsumption relationship between two atomic concepts or between one atomic concept and one complex concept. Atomic concepts are simplest building blocks of the ontology, whereas the phrase “complex concepts” refers to concepts that involve at least one logical operator, e.g., negation (¬), conjunction (⊓), disjunction (⊔), existential restriction (∃r.C) or universal restriction (∀r.C). Dong et al. used a three-step process: 1) Edge search encodes a mention (edge or concept) and an ontology edge using two fine-tuned language models to align them in a shared embedding space for concept and edge search, 2) Edge formation and enrichment leverages the ontological structure with the language model-based embedding for candidate retrieval. A list of edges at one and two hops is extracted to calculate the ranking. 3) Edge selection with multi-label classification method, is performed.

Lopes et al. [29] performed a study to classify domain entities into top-level ontology concepts using informal definitions of the domain entities defined during the ontology development process. The authors hypothesized that informal definitions of domain entities can encapsulate enough semantic information to map them to top-level ontology concepts. They explored the possibility of leveraging the similarity in taxonomies and the distributional hypothesis to classify domain entities into top-level ontology concepts, using the informal definitions of these domain entities. The authors perform alignment between OntoWordNet and the BabelNet semantic network, resulting in a dataset containing 65,018 domain entities. They present ways of textually representing domain entities, using only the definiendum, only definiens, or an example sentence. Definiendum (or term) is the shortest way to represent a domain entity, since the definiendum names a domain entity through a combination of a few words, and definiens is the explanatory part of an informal definition. Then the authors explore a classification pipeline to classify domain entities into top-level concepts. The authors utilize the similarity in taxonomies and the distributional hypothesis to support the hypothesis that informal definitions represent semantic information that allows domain entities to be related to top-level ontology concepts. The authors compared two training pipelines for the classification task: (1) fine-tuning a pre-trained language model for classification, and (2) using the language model to generate embeddings of input text, followed by a k-nearest neighbor (k-NN) classifier. Their approach focuses exclusively on definitions present in OntoWordNet and aims to map them to a fixed set of top-level ontology categories.

Nine different LLMs in a zero-shot setting, including Meta LLAMA, BART, and Google Flan-T5, were utilized by Babaie et al. [30] for benchmarking their performance in three main ontology learning tasks. The models were tested for 1) The term typing task i.e., identifying the part of speech; 2) Taxonomy discovery, i.e., discovering taxonomic (hierarchical) relations between types, such as the “is-a” relationship in ontologies; and 3) Non-taxonomic relationship extraction, i.e., extracting semantic relations beyond taxonomic hierarchies, such as causal relationships or part-whole relationships. The empirical results of Babaie et al. indicate that with effective fine-tuning, LLMs can serve as valuable assistants, helping to overcome the knowledge acquisition bottleneck in ontology development compared to foundational models.

Funk et al. [31] describe a general approach toward constructing ontologies using the GPT 3.5 model. The methodology starts by taking a seed concept that determines the domain. All the sub-concepts will belong to the same domain. An iterative querying approach is implemented by repeatedly asking the LLM to provide relevant sub-concepts of concepts already in the hierarchy, and an established traversal algorithm is utilized to place them. Since these concepts may have more than one parent, the ultimately constructed hierarchy does not take the form of a tree but that of a directed acyclic graph (DAG). Since metric evaluation of the precision and recall of the constructed ontologies is difficult because there is no ground truth, the authors performed a purely subjective evaluation based on manual inspection of the constructed ontologies.

2.2. Social Determinants of Health Ontology

The ontology enrichment presented in this work is based on the Social Determinants of Health Ontology (SOHO) developed by Kollapally et al. [32]. We created the first version of SOHO in 2022 in Protégé [14], with 173 classes and 585 axioms across five main categories: Economic instability, Educational access and quality of education, Healthcare access and equality, Neighborhood and built-in environment, and Social and community context. In SOHOv1, we placed 87 concepts under Economic instability, 11 under Educational access and quality of education, 22 under Healthcare access and equality, 22 under Neighborhood and built-in environment, and 31 under Social and community context. The total number of levels is five, with 45% (79/173 concepts) of the concepts at Level 3. SOHOv1 uses only is-a relationships between the concepts, leading to lower semantic relationship richness.

Dang et al. [33] published a Social Determinant of Health Ontology (SDoHO) with 708 classes, 106 object properties, and 20 data properties in 2023. They included temporal concepts, which we did not do in SOHOv1. To ensure the reproducibility of our work in alignment with FAIR principles, we have used a publicly available seed ontology (SOHOv1) and PubMed as data sources. As a baseline for comparison, we used SDoH triples from SemMedDB, providing us with the ability to demonstrate our pipeline’s methodological advantages. Another undertaking to advance this domain is the Gravity Project [34], which represents a collaborative initiative focused on establishing standardized data for SDoH. The provision and maintenance of an SDoH ontology are vital to capture the evolving non-clinical factors that affect patients’ health and to identify the interplay between these factors as they appear in SDoH data collections.

2.3. Semantic triples repository

The SemMedDB database [35] contains semantic relationships from biomedical text captured automatically using the SemRep NLP tool [36]. These relationships are then organized into subject-predicate-object triples, also called semantic predications, originating from unstructured text in PubMed citations, including titles and abstracts. The subjects and objects of these predications are standardized as unique concept identifiers (CUIs) from the UMLS Metathesaurus [37], further categorized by UMLS Semantic Types such as Disease or Syndrome and Pharmacologic Substance. SemRep has processed over 36.8 million citations, leading to nearly 129 million predications. The Semantic MEDLINE application extracts phrases from MEDLINE citations generated through PubMed search statements and presents a visualization based on these semantic relations connected to MEDLINE text sources.

2.4. UMLS Metathesaurus

The Metathesaurus is the most significant part of the UMLS. It is organized by concepts, and it links together synonyms for the same concept from nearly 200 different vocabularies/ontologies. The Metathesaurus is a collection of concepts and relationships established between concept pairs. The latest version 2024AB [38], of the UMLS contains approximately 3.38 million concepts and 16.4 million unique concept names from 187 source vocabularies.

2.5. Text Similarity Approaches

Similarity measures play an important role in text-related research and applications in tasks such as information retrieval, text classification and topic tracking [39].

Lexical similarity comes in three variants, namely edit distance-based, token-based and sequence-based similarity measures. Edit distance-based measures count the number of single-character insertions, deletions, or substitutions needed to transform one string into another. Token-based measures compare tokens between two strings, often using metrics such as Jaccard similarity. Sequence-based measures focus on the longest common subsequence of characters or words between two strings. The Levenshtein distance [40] is an edit-distance-based similarity computation. A token-based similarity metric, e.g., Jaccard Similarity/Jaccard Distance, for comparing strings is based on constituent tokens or words rather than individual characters. The Jaccard Similarity [41] measures the proportion of common elements between two sets relative to the total number of distinct elements in those sets. The resulting similarity score ranges from 0, which means no similarity, to 1, which means perfect similarity. An example sequence-based similarity metric for lexical similarity is the Longest Common Subsequence (LCS) metric [42]. In a sequence-based similarity metric, the focus is more on analyzing and comparing the entire sequence as opposed to a token-based approach. LCS determines to what degree two sequences match by computing the maximum length of a sequence of characters (substring) common to both of them, whether that substring represents the whole or a part of the string values.

While keyword or lexical searches may not always follow strict Boolean matching, they all rely on matching keyword terms or tokens. This approach, even though simple, has its limitations. Semantic similarity considers the meaning of the concepts at hand while comparing the concepts. Semantic similarity can be measured using various approaches, e.g., transformer-based methods, and network-based techniques that capture domain knowledge[43, 44].Bidirectional Encoder Representations from Transformers (BERT) has become a standard building block for training task-specific NLP models [45]. Pre-trained BERT models have been widely used to convert text into embeddings (high-dimensional vectors) that capture the meaning of the text. Domain-specific pre-trained BERT models can significantly improve the performance of various biomedical NLP tasks [46, 47]. The pre-training corpus for PubMedBERT comprises 14 million PubMed abstracts with 3 billion words (21 GB) after filtering empty and short abstracts [48]. Knowledge network-based similarity is a semantic similarity measure based on identifying the degree of similarity between words using the information retrieved from a semantic structure such as the UMLS.

3. Methods

We performed a preliminary analysis of Semantic MEDLINE to visualize the semantic predications related to SDoH keywords and determine whether these predications could be utilized for enriching SOHO. Our analysis of Semantic MEDLINE showed that significant aspects relating to SDoH, such as economic instability, poor housing conditions, neighborhood quality, educational access and quality, along with health care access and quality, were not reflected correctly. Most of the significant concepts in SOHOv1, such as economic instability and neighborhood, and granular concepts such as poverty, addiction, and low income, were not retrieved from SemMedDB.

In SemMedDB, under the Social Determinants of Health category, the source concepts with the highest number of outgoing relationships are “women” (with 39 outgoing “Process_of” relationships), “adult” (with 35 outgoing “Process_of” relationships), and “adolescent” (with 35 outgoing “Process_of” relationships) to their respective target concepts. There are other types of relationships also in SemMedDB. We have attached a selection of triples extracted from SemMedDB as Table T2 in the supplemental file. Thus, SemMedDB had a poor coverage of SDoH concepts. Due to the same reason, we did not use the semantic relationships in SemMedDB to supplement our framework, but used SemMedDB as a baseline to compare our approach to it. According to a published study [49], there are 198,677 SDoH concepts in the UMLS. SemRep relies on concepts from the UMLS Metathesaurus, but only those relevant to biomedical contexts. While the 198,677 concepts might be present in the UMLS, they may not be mapped or selected by SemRep, due to its focus on biomedical predicates. SemMedDB contains over 30 predicates types largely related to categories, such as clinical medicine (e.g. Treats, Diagnoses, Administered_To, Process_Of), substance interactions (e.g. Interacts_With, Inhibits, Stimulates), genetic etiology of diseases (e.g. Associated_With, Causes, Predisposes), and pharmacogenomics (e.g. Affects, Augments, Disrupts) [35].

We will now describe in detail our pipeline that utilizes GPT-4–1201 for enriching the conceptual content of an ontology. While our methodology is described with the SDoH ontology as a use case, it can be extended to any ontology for which domain-relevant documents and a seed ontology are available. The pipeline developed for ontology expansion can be divided into six stages. Figure 1 shows the complete pipeline for this ontology enrichment process. Stages are numbered with red integer numbers. In Figure 1, <C, R, P> refers to the semantic triples generated by GPT-4–1201, where C is the child concept, R is the relationship and P is the parent concept. Correspondingly [P] refers to a list of parent concepts, [C] refers to a list of child concepts and [N^X] refer to the list of concepts N at Level X that are targeted for enrichment. N^X is a generic notation for the node considered for enrichment. Once the lexical similarity is established between elements in [P] and [N^X] (having a lexical composite score above a chosen threshold KL), the corresponding child C of the parent P will be added as a child of the target concept N^X into the ontology during Stage 6. Hence, [(C^LS, P^LS)] is a list of concept pairs to represent a list of lexically matching parents and their corresponding child concepts generated at Stage 3, where the ‘LS’ symbol means lexically similar.

Figure 1:

The complete pipeline for automatic ontology enrichment from a seed ontology using lexical, semantic and knowledge network-based similarity techniques.

All the triples that were not lexically matched will be denoted as <C^S, R^S, P^S>, where ‘S’ means that they are candidates for semantic similarity. Each of the concepts at Level X and each parent concept in the list [P^S] where [P^S] is the list of candidate parent concepts for semantic similarity, will be converted into embeddings using PubMedBERT.

The triples that contain a parent concept P^S having a cosine similarity value with N^X above a given threshold (KS), will be denoted as <C^SS, R^SS, P^SS > (SS=semantically similar). [ (C^SS, P^SS)] is the list of parent and child concept pairs identified after computing semantic similarity between parents and the list of nodes N^X at Level X. Correspondingly [P^S]-[P^SS] is the list of parent concepts with cosine similarities with target nodes N^X below the threshold. In other words, this is the list of concepts that are left over after semantic similarity search. They will be passed to the next stage, i.e., knowledge network-based similarity.

Let <C^K, R^K, P^K > be the semantic triples considered for knowledge network-based similarity, which were not matched either by lexical similarity or semantic similarity. From these parent concepts in the triples, we extract a list of candidate parent concepts for knowledge network-based similarity, denoted as [P^K], which will be used to identify their matching alternative textual representations (atoms) from the UMLS Metathesaurus. The resulting list of concepts are called [P^UMLS]. Similar to semantic similarity we calculate cosine similarity value of [P^UMLS] with N^X and if the cosine value is above a given threshold (KK), these triples will be denoted <C^N, R^N, P^N > (N=Knowledge network-based similar. [(C^N, P^N)] is the list of parent and child concept pairs identified after computing knowledge-network based similarity between parents and the list of nodes N^X at Level X generated at Stage 5. Each child of P^N, i.e., C^N, will be added as child concept of the concept N^X to the ontology, hence enriching the node(s) at Level X in Stage 6. In Stage 6 all the resulting child concepts extracted in Stage 3, 4 and 5 will be added to SOHOv1 to obtain the enriched SOHOv2.

3.1. Stage 1: Extracting SDoH abstracts from PubMed

With SOHOv1 as the seed ontology, the first stage is retrieving PubMed articles relevant to the main SDoH categories (Levels 1 and 2 of SOHOv1). We extracted PubMed articles corresponding to Level 1 of the ontology, which consists of the single concept “SDoH.” Then we continued with the five Level 2 concepts (Table 4), e.g., “Economic instability” and “Educational access and quality.” We used the Entrez package [50] to extract all articles from PubMed that match the Level 1 and Level 2 SOHOv1 concepts. Thus, we made use of the two-level organization of the six concepts. Entrez is a data retrieval system that provides users access to NCBI’s databases, such as PubMed, GenBank, GEO, etc. We created a database, called PubMed-SDoH, with PubMed abstracts grouped under the five major SDoH categories: i) Economic instability, ii) Educational access and quality, iii) Health care access and equality, iv) Neighborhood and built-in environment, and v) Social and community factors, which are children of the overarching SDoH.

Table 4:

The number of concepts added to build SOHOv2 using lexical, semantic and knowledge network-based similarity

Column 1	2	3	4	5	6	7	8	9
SOHOv1 Level 2 concepts	# Levels in SOHOv1	# of concepts in SOHOv1	# of Levels in SOHOv2	# of concepts in SOHOv2 after lexical similarity	# of concepts in SOHOv2 after semantic similarity	# of concepts in SOHOv2 knowl. netbased similar.	#ofnew concepts in SOHOv2	% new concepts in SOHOv2
Economic instability	4	87	5	19	42	112	173	198%
Educational access and quality	2	11	3	11	17	46	74	672%
Healthcare access and equality	2	22	3	11	23	31	65	295%
Neighborhood and built in environment	1	22	2	5	14	18	37	168%
Social and community factors	1	31	2	13	16	21	50	161%
TOTAL	N/A	173	N/A	59	112	228	399	230%

3.2. Stage 2: Utilizing GPT-4–1201 for semantic triple extraction

We used the GPT-4–1201 Chat Completion API (Application Programming Interface) to extract semantic triples from the PubMed abstracts. After iteratively refining the prompts for better results, a four-step prompt chaining technique was utilized to achieve optimal chat responses. <Child, is-a, Parent> semantic triples are extracted from all abstracts in our database PubMed-SDoH grouped by the SDoH category. For example, “ostracization is a negative social outcome” and “nutritional habit is a lifestyle factor” are parts of sentences from PubMed abstracts. The corresponding triples are <Ostracization, is-a, Negative social outcome> and <Nutritional habit, is-a, Lifestyle factor>. The parent concepts are “Negative social outcomes” and “Lifestyle factors,” the relationship is is-a, and the child concepts are “Ostracization” and “Nutritional habits.” The parent concept of every triple (from PubMed) could potentially match a concept in SOHOv1. If this turns out to be the case, then the child of that triple should be added to SOHOv2 as a new concept. The following four-step prompt chaining was used to extract semantic triples with the help of GPT-4–1201. The prompt template can be found in Table T1 in the supplemental file.

Step 1: Semantic triple extraction

Objective: Establish the base extraction by instructing GPT-4–1201 to act as semantic triples extractor.

Implementation: We created a message template [51] with the assistant’s role to input abstracts and output semantic triples. All possible triples are extracted at this step. This extraction relies on the model’s inherent ability to identify semantic triples without requiring manually crafted examples.

Step 2: Extraction of hyponym–hypernym (child – parent) concept pairs

Objective: Enhance the extraction process by focusing on hyponym–hypernym concept pairs.

Implementation: We added an additional assistant’s role [51] within the same template to extract general and specific concept pairs connected by an is-a link, thus refining the semantic relationships that the model targets.

Step 3: Focus on the Social Determinants of Health domain

Objective: Refine the extraction process to information relevant to the Social Determinants of Health domain.

Implementation: We introduced a new assistant’s role dedicated to extracting only those semantic triples that are pertinent to SDoH.

Step 4: Exclusion of non-relevant concepts and output of structured semantic triples

Objective: Enhance the output by eliminating irrelevant concepts and organize the extracted triples for better readability.

Implementation: We added a “system role” [51] to the message template to filter out any concepts not directly derived from the given PubMed abstracts. We also added instructions to eliminate actual numerical data and organizations outside of the US as irrelevant to SDoH. Additionally, we instructed the model to output the semantic triples separated by new lines, improving the readability of the output.

We utilized three consecutive steps of the assistant roles, followed by the system role. The model automatically assumes the assistant role when generating responses based on user queries or following system instructions. This approach allows for progressive refinement of the extracted information, ensuring structured knowledge extraction before applying narrower constraints. We imposed a higher-level filtering mechanism by introducing the system role at the final stage, refining the output to align with the intended scope. This approach aligns with Chain-of-Thought (CoT) reasoning [52], where incremental steps guide the model toward structured responses before applying a more rigid directive at the end.

For the chat completion configuration of GPT-4–1201, we converged on a concise “temperature” parameter equal to 0.2, based on preliminary trials that resulted in the most deterministic and reproducible output. In the experiment, we faced the issue of session timeouts in the Python script with GPT-4–1201. To avoid this, we created a five-second sleep delay before making another request per every 50,000 token requests. A total of 19,728 API calls were sent to GPT-4–1201, resulting in 6,438,653 tokens used for the Level 1 and Level 2 concepts of SOHOv1.

3.3. Filtering the semantic triples using multiple similarity measures

While LLM-based triple extraction can capture rich semantic information, it often results in a high volume of extracted triples. This abundance of triples raises concerns about potential duplicates and the need for filtering to ensure only the most relevant and novel triples are considered for concept enrichment. To automate the filtering process and retrieve the most promising triples for concept enrichment, our approach incorporates multiple similarity measures: lexical similarity, semantic similarity, and knowledge network-based similarity. Even though knowledge network-based similarity search is a semantic similarity search, in our experiments, we performed knowledge network-based filtering separately, utilizing the UMLS Metathesaurus to identify closely related terms of parent concepts present in the semantic triples extracted by the GPT-4–1201 API.

With SOHOv1 having a five-level tree structure, we performed a breadth-first bottom-up traversal for enriching the ontology by retrieving triples. Starting with leaf concepts, we extracted all the concepts at the leaf level and performed similarity-based filtering. At each level, we also removed duplicate concepts from the pool of triples available from the previous phase. Comparing SOHOv1 concepts at each level with the parent concepts in the extracted semantic triples from Stage 2 enabled the expansion of these concepts with their direct descendants (children). The advantage of starting from the leaf nodes is that we add more granular concepts to SOHOv1, and since unique concepts are added to SOHOv1 there is no duplication of the same concepts in the higher tree levels. Figure 2 shows a tree structure exemplifying various levels in SOHOv1. The red concept which is a leaf node in SOHOv1 is retrieved and compared with extracted parent concepts. As shown in Figure 2, after performing each phase of filtering (lexical, semantic, and knowledge network-based as described below), the Level 5 red concept is expanded by adding green concepts as children.

Figure 2:

Excerpt from SOHOv1 with the red node considered for concept enrichment and the green nodes that were added when creating SOHOv2.

3.4. Stage 3: Filtering the semantic triples using lexical similarity

Prior research [53, 54] indicates that a combination of lexical similarity approaches yielded better results compared to one single similarity metric. In this work, we implemented a composite measure of lexical similarity that employs token-based, sequence-based, and edit distance-based distances. We used three different metrics (Levenshtein distance, Jaccard similarity, and LCS)) for lexical similarity search.

To determine reasonable threshold, we plotted the Kernel Density Estimate (KDE) for the Cosine similarity score, Longest Common Subsequence (LCS), Levenshtein distance (L), Jaccard distance (J) and the composite of the latter three scores. The KDE plot provides a smoothed representation of the distribution of data points. The density values on the y-axis of a KDE plot represent the estimated probability density of the data instead of direct counts or frequencies of terms. Since we used three different similarity metrics resulting in three different scores for lexical similarity, we defined a novel composite score to combine the three metrics. In Equation 1, W_J, W_L, and W_LCS are the weights assigned to the three metrics. In this work, we gave a higher weight to the Levenshtein distance, since the concepts with lower Levenshtein distance were lexically close in most cases in our analysis and according to the KDE plot in Supplemental File (Figure S2 (c)) the distribution is not uniform. In Figure S2(c) it starts low, rises gradually, and peaks towards the higher end of the scale on the x-axis. This could indicate that terms have varying levels of dissimilarity, with most pairs being more dissimilar than similar. Hence W_L was set to 2 and the other two weights were set to 1. J, the Jaccard similarity score, has values between 0 and 1. (1-L/M) is the inverse Levenshtein distance; the higher the inverse Levenshtein distance, the better the similarity. We normalize the inverse Levenshtein distance with the maximum length M of the two concepts under consideration. After computing the composite score according to Equation (1) and plotting the KDE, we observed a plateau in Figure 3(a) within the range of 0.7 to 0.94. Since there was no significant variation in performance across the thresholds of 0.7, 0.8, and 0.9, we selected the approximate average value of 0.8 as the threshold.

Figure 3.

(a) Kernel density estimate for Composite Similarity Score; (b) Kernel density estimate for Semantic Similarity; (c) Kernel density estimate for Knowledge network-based similarity. Red-dotted lines in the figures indicate the selected threshold.

$$C=\frac{W_J\text{*}J+W_L\text{*}\left(1-\frac{L}{M}\right)+W_{LCS}\text{*}\frac{LCS}{M}}{W_J+W_L+W_{LCS}}$$ (1)

3.5. Stage 4: Filtering the semantic triples using semantic similarity

In Stage 4, we utilized the PubMedBERT model to convert the parent concepts from the semantic triples that were not matching lexically (i.e., not selected in Stage 3) and the SOHOv1 concepts into embeddings. Then we calculated the cosine similarities between these embeddings. Figure 3(b) shows the kernel density plot of semantic similarity versus the cosine values. The values of the cosine similarity can range from −1 to 1, where 1 indicates two identical vectors, 0 indicates orthogonal vectors, and −1 indicates diametrically opposed vectors, although in the context of BERT embeddings for semantic similarity the values ranged from 0 to 1. A cosine similarity close to 1 indicates high similarity between text embeddings. The density on the y-axis is a smoothed estimate showing where the most clusters of values are located. The distribution of semantic similarity scores visible from 0.6 to 0.85 indicates low variation of semantic distances of the text pairs, however, there is a strong trend towards highly semantically similar concepts. The peak at around 0.78 dominates the plot, indicating that such pairs of concepts should be considered semantically close. Hence a cosine value of 0.72 was selected for extracting semantically similar concepts from Stage 4.

3.6. Stage 5: Filtering the semantic triples using knowledge network-based similarity

In Stage 5, we focused on the parent concepts of the semantic triples extracted from PubMed that were passed on by Stage 4. We extracted all the semantically similar concepts corresponding to the parent concept from the UMLS, using the UMLS API. The first API call used a search endpoint to retrieve the concept unique identifier (CUI) of the source asserted identifier that matches the parent concept of a specific triple. Later, this CUI was utilized to extract the corresponding string(s) using the content endpoint. While the UMLS provides a rich conceptual network with hierarchical (parent-child) and associative relationships, our approach leveraged atoms from the Metathesaurus, rather than explicitly incorporating hierarchical relationships (e.g., parents or children beyond those initially extracted in Stage 4). We converted the corresponding UMLS concepts obtained in the previous step and the SOHOv1 concepts into embeddings using PubMedBERT and then we computed cosine similarities.

Figure 3(c) plots the range of kernel density estimates vs. cosine values for knowledge network-based similarity. The cosine values appear to range approximately from 0.8 to 1.0. The y-axis indicates the density of observations within the dataset for each value of knowledge network-based similarity. There is a first peak between 0.78 and 0.8, which indicates that there is a cluster of concept matches with cosine values in that range. The next peak occurs around 0.84, which suggests that many data points are close to each other according to knowledge network-based similarity. The last sharp peak at 0.9 is remarkable, as it indicates that many parent concepts are identical to SOHOv1 concepts. Hence, for the cosine threshold, we used 0.9 for knowledge network-based similarity.

Having established the threshold values for cosine similarity in both semantic and knowledge network-based filtering, we now introduce an algorithm designed to automate the crucial steps of matching extracted parent concepts and SOHOv1 concepts and appropriately placing child concepts within the ontology. To begin, Algorithm1 takes the seed ontology SOHOv1 as input and extracts the concepts of the level under consideration. Domain specific weights and thresholds for similarity scores are considered as input parameters. To keep track of all the existing children of the parent concept at the current level, a SOHO_children dictionary is created in Line 1. This ensures no duplication of children exists after the enrichment process. Line 2 introduces S_enriched, which is a dictionary with sets as values (a set ensures that the same concept is not added again during expansion) of all the concepts obtained using three level similarity searches. Lines 3–17 describe the algorithm for lexical filtering followed by Lines 18–26 describe the process of semantic similarity and Lines 28–38 define the algorithm of knowledge network-based similarity. Line 12 calculates the composite score for lexical similarity by using Jaccard score in Line 6, Levenshtein distance in Line 7 and LCS distance in Line 8. Lines 21–22 iterate to find the cosine value between embeddings of parent concepts from the extracted semantic triples and embeddings of SOHOv1 concepts. When the cosine value is above the threshold (KS), the children in the semantic triple will be added as children of the SOHOv1 concept after making sure there is no duplication. In a similar manner, Lines 28–31 iterate to extract UMLS concepts similar to each parent concept, and then the extracted UMLS concepts are compared with SOHOv1 concepts. If the cosine value between embeddings is above threshold (KK) we add the child concept of the triple as a child of the SOHOv1 node compared with the UMLS concept. This algorithm follows a breadth-first bottom-up traversal for collecting concepts at each level.

3.7: Stage 6: Human and tool-based evaluation of SOHOv2

The triples generated by GPT-4–1201 contained non-is-a links in addition to is-a links. We modeled non-is-a links as object properties and attributes of concepts as data properties. After extracting the concepts based on the lexical, semantic and knowledge network-based approaches, they were added to SOHOv1. We also added the human-extracted object and data properties along with newly derived concepts to create SOHOv2. The Protégé application was used to enter the new concepts, links and properties, resulting in an .owl file.

An essential aspect of ontology development is the evaluation of the enriched ontology after adding concepts generated in Stages 3, 4 and 5. Hence, we performed a human-based selective evaluation of the ontology, followed by a software tool-based evaluation. Using the evaluators’ feedback, we performed a final revision of the ontology.

A selective evaluation approach using human experts in the field of medical informatics and ontology building was employed to access the quality of SOHOv2 focusing on methodological validation of the pipeline. For the human-centric evaluation, we created an Excel sheet with 100 semantic triples from the new set of concepts added. The human experts (JX, VK and ZH) evaluated the ontology. The three experts were independently given a list of child-parent relationships. We designed 50 out of 100 triples as intentionally wrong triples, i.e., concept pairs that do not share child-parent or grandchild-grandparent relationships were chosen. An example of a correct semantic triple would be <Anxiety over shopping, is-a, Food insecurity>. “Anxiety over shopping” may be caused by an inability to afford necessary food items and can be linked to “Food insecurity.” An incorrect triple would be <Degree of responsibility and accountability at workplace, is-a, Food insecurity>. The incorrect triple has a child concept that is unrelated to food or “Food insecurity” (the parent). We hypothesize that this random sampling method is better than having a human expert review the entire ontology, because they could develop a bias towards claiming all relationships as valid. For human expert evaluation, metrics such as Fleiss’ kappa (κ) [55] are used to understand the percentage agreement between human evaluators. Fleiss’ κ is a measure of inter-rater agreement used to determine the degree of agreement between two or more raters for categorical scales.

The semantic triples were manually verified by cross-checking with the corresponding abstracts from which they were extracted to serve as gold standard. The evaluation sheet allowed the evaluators to look at each semantic triple and record their agreement or disagreement with the triple. They were asked to provide the reasons why they disagreed with specific triples. The inter-rater agreement was computed and is presented in the Results section. After the first round of evaluations, we created a mitigation plan for all the semantic triples for which two among the three raters disagreed. The expanded SOHOv2 was updated based on the mitigation plan. It is notable that the three experts were able to agree to all mitigation plans. Finally, for tool-based evaluation, we utilized the HermiT reasoner [56] in Protégé [57] to evaluate the consistency of the expanded ontology.

4. Results

4.1. Extracting SDoH abstracts from PubMed

Table 1 indicates the number of abstracts extracted per SOHOv1 Level 1 and Level 2 concept. No additional filters were placed in PubMed to find these articles. To limit the context to SDoH, we used an AND operation for each concept in Level 2. The number of full-text abstracts relevant to the title “Social Determinants of Health” in PubMed was 26,084; in total, combining articles from the five Level 2 concepts in SOHOv1, 41,339 abstracts were extracted using Entrez. Subsequently, we eliminated empty abstracts or abstracts with one line of text and passed 16,403 unique abstracts to the GPT-4–1201 API to extract semantic triples. The last row in Table 1 contains values for the Level 1 concept (root) in SOHOv1. Column 4 with the header ‘#Semantic triples from GPT-4–1201’ contains the triples generated by GPT-4–1201. Column 5 ‘#Semantic triples processed during enrichment’ was obtained by removing duplicates from the triples in Column 4, as some triples were generated by more than one published article.

Table 1.

Number of PubMed abstracts retrieved, and the corresponding numbers of semantic triples generated by using the search terms consisting of Level 1 and Level 2 categories with the additional term SDoH (expanded)

Column 1	2	3	4	5
Level 1 and Level 2 Categories of SOHOv1	#Abstracts retrieved using Entrez	#Abstracts sent to GPT-4-1201	#Semantic triples from GPT-4-1201	#Semantic triples processed during enrichment
Economic instability AND SDoH	294	294	5,360	4,998
Educational access and quality AND SDoH	704	523	7,787	7,322
Healthcare access and equality AND SDoH	2,537	2,113	61,007	35,921
Neighborhood and built-in environment AND SDoH	3,734	3,249	50,619	35,739
Social and community factors AND SDoH	7,986	4,746	81,355	24,276
SDoH	26, 084	5,478	178,773	31,320
Total	41,339	16,403	384,901	139,576

4.2. Utilizing GPT-4–1201 for semantic triple extraction

To extract semantic triples from PubMed-SDoH, we used prompt chaining techniques (Section 3.2). On average, 14 semantic triples were extracted per abstract. In total, the number of semantic triples from GPT-4–1201 was 384,901, and after duplicate removal, 139,576 semantic triples were considered for enrichment. Table 2 below shows a sample of semantic triples extracted by GPT-4–1201.

Table 2:

A sample of semantic triples <Child, Relationship, Parent> extracted by GPT-4–1201 from PubMed articles for the “Economic instability AND SDoH” search term.

Column 1	2	3
PMID	PubMed Article Title	Semantic Triples Retrieved from this Article
34739460	From “The Interpersonal Theory of Suicide” to “The Interpersonal Trust”: an unexpected and effective resource to mitigate economic crisis-related suicide risk in times of Covid-19?	<Economic instability, is-a, Economic condition> <Income Decrease, is-a, Economic stressor> <Unemployment, is-a, Economic stressor> <Great Recession, is-a, Economic Crisis> <Repaying Debts difficulty, is-a, Economic stressor> <Home loss, is-a, Financial stressor> <Social hierarchy drop, is-a, Financial stressor> <Poverty, is-a, Economic stressor> <Interpersonal trust, is-a, Protective Factor>
36849477	Patient and Health Care Professional Perspectives on Stigma in Integrated Behavioral Health: Barriers and Recommendations	<Stigma, is-a, Barrier to using health care> <Integrated behavioral health (IBH), is-a, Service in primary care> <Internalizing negative stereotypes, is-a, Barrier to IBH> <Patient-centered communication, is-a, Strategy to reduce stigma> <Empathetic communication, is-a, Strategy to reduce stigma> <Patients' preferred understanding, is-a, Factor in discussions of mental health> <Health care professionals, is-a, Group that can reduce perceptions of stigma>
33172725	Limited treatment accessibility: Implications for alcohol treatment disparities among Mexican Americans living in the U.S.-Mexico border region	<Alcohol-related care, is-a, Treatment for alcohol use> <Alcohol use disorder (AUD), is-a, Health condition> <Outpatient treatment, is-a, Type of facility> <Proximity to treatment, is-a, Factor affecting treatment accessibility> <Border cities, is-a, Area with lower density of treatment> <Density of treatment, is-a, Factor affecting treatment accessibility>

Even though we instructed the GPT-4–1201 model to return only is-a relationships, it also returned non-is-a links such as “is-influenced-by,” “is-effect-of,” “is-associated-with,” etc. We modeled non-is-a links as object properties. We modeled attributes of concepts as data properties. For example, we modeled the triple <Transactional Sex, is-associated-with, HIV Risk> as object property with the source “Transactional sex,” the target “HIV Risk” and the property name “associated_with.” As an example of a data property, we will use the concept “Marital status.” Its data property “Marital Status” has the allowed values “Married,” “Single,” “Divorced,” “Engaged,” “Widowed,” etc. This was based on the following triples obtained from GPT-4–1201: <Widowed, is-a, Marital status>, <Single, is-a, Marital status>, <Married, is-a, Marital status>, etc. These properties were added manually to the ontology after human screening of the triples.

We visualized the semantic triples generated by GPT-4–1201 using network community detection with the Louvain method [58] available in the supplemental file as Figure S1. Figure 4(a) is a readable zoomed out version of the center of the cluster and Figure 4(b) is a cluster from the outer edge of Figure S1. The dots represent concepts, and lines (edges) represent the relationships between the concepts. To reduce complexity, we removed the labels of edges. The red cluster in Figure 4(a) shows SOHOv1 concepts and relationships and the black edges and nodes represents the semantic triples extracted from GPT-4–1201.

Figure 4:

Network community visualization among semantic triples using the Louvain method; (a) The red nodes and edges are SOHOv1 concepts. The black cluster, centered around “Economic instability” consists of nodes extracted by GPT-4–1201. b) Green nodes form clusters of semantic triples generated by GPT-4–1201. All the edges represent is-a relationships, e.g., <Complex emergencies, is-a, Natural disaster emergency>.

4.3. Filtering the semantic triples using similarity measures

To accommodate the benefits of edit-based, token-based, and sequence-based lexical similarity measures, we defined a composite score (Eq. 1) as mentioned in Section 3.3. A composite score of 0.8 was used to filter parent concepts from the GPT-4–1201 extracted semantic triples that are lexically similar to existing concepts in SOHOv1. Subsequently, the children of these filtered parent concepts are added as child concepts of existing SOHOv1 concepts, thereby enriching the ontology, creating SOHOv2. In Table 3, we show some examples with Columns 2 to 4 representing GPT-4–1201 extracted triples and Column 5 showing the existing SOHOv1 concept that is lexically similar to the extracted parent concept. We added the child concepts (Column 2 in Table 3) as the child concepts of the concepts in Column 5 into SOHOv2. A similar filtering process was performed using semantic similarity and knowledge network-based similarity measures with the corresponding threshold values as discussed in Methods.

Table 3:

A sample of the output from lexical similarity analysis showing child concepts of the social and community factors category (Note: IPV = Intimate Partner Violence)

Column 1	2	3	4	5	6	7	8
PMID	Child Concept	Relationship	Parent Concept	SOHOv1 Concept	Matching Term by LCS	Matching Term by Jaccard similarity	Matching Term by Levenshtein
26497097	Alcohol consumptio n	is-a	Behavioral circumstance increasing risk for IPV	Behaviora l risk of IPV	Behaviora l risk of IPV	Behaviora l risk of IPV	Behavioral risk of IPV
26497097	Lack of proper parental guidance	is-a	Consequenc e of IPV on children	Impact of IPV in children	--	--	Impact of IPV in children
26497097	Lack of safety and stability	is-a	Consequenc e of IPV on children,	Impact of IPV in children	--	--	Impact of IPV in children
28662255	Academic failure	is-a	Child and adolescent impacted by IPV	Impact of IPV in children	--	--	Impact of IPV in children
28662255	Depression	is-a	Child and adolescent impacted by IPV	Impact of IPV in children	--	--	Impact of IPV in children

Table 4 summarizes the number of concepts added to create SOHOv2 using the different similarity measures. Columns 2 and 3 show the number of levels and the number of concepts at each level of SOHOv1 categorized under its main category. The remaining columns record the numbers of concepts added using lexical, semantic, and knowledge network-based similarity, the number of levels in SOHOv2 after this addition, the total number of concepts, and the percentage of new concepts. In Table 4, Column 3 represents a total number of concepts in SOHOv1; the total number of concepts is the same as a total number of concepts under the SDoH (Level 1) concept since SOHOv1 Level 2 only contains concepts listed in Table 4, Column 1.

In Table 4, one may notice that most terms were added to concepts in SOHOv1 using knowledge network-based similarity for concept placement. With lexical similarity, we added 14% of the new concepts to SOHOv1. Then, 28% were added using semantic similarity, and 57% of concepts were added by knowledge network-based similarity. The values in Column 8 are computed as the sum of the values in Columns 5, 6, and 7.

4.4: Tool-based and human evaluation of SOHOv2

We performed a tool-based evaluation with the widely used HermiT reasoner. According to HermiT, SOHOv2 is consistent. The design of the ontology inherently promotes consistency, because the ontology only contains single inheritance relationships, hence reducing the risk of contradictions arising. The gold standard for evaluation consisted of 50 related semantic triples and 50 unrelated semantic triples, shuffled in the sheet provided to the evaluators. Thus, the evaluators were given 100 randomized triples. We provided a few sample triples with reasons why they were related or unrelated. This gave them a flavor of the domain ontology.

To reduce the number of disagreements, we devised a mitigation plan for all the triples on which evaluators disagreed. Most of the resulting changes added more granularity to concepts. For example, consider the semantic triple <Crowding at home, is-a, In-hour service>. An example of a clarification question asked by one evaluator was “Not sure exactly what in-hours service refers to. Is this referring to the opposite of out-of-hours service? If so, unrelated to the poor housing situation.” In response, we clarified the child concept to the evaluators as “in-hour counseling service.” The Fleiss κ value was 0.643 for human expert evaluation between the three ontology evaluators, which implied substantial agreement between the three reviewers [55, 59]. Table 5 provides the level of agreement and disagreement between human experts’ evaluations and the gold standard described above. The agreements are calculated with Cohen’s Kappa.

Table 5:

Agreement and disagreement scores between evaluators with respect to the gold standard.

Evaluator	Agreement with correct semantic triples	Agreement with incorrect semantic triples	Disagreement with correct semantic triples	Disagreement with incorrect semantic triples	Cohen’s Kappa w.r.t gold standard
1	39	4	11	46	0.7
2	45	6	5	44	0.78
3	38	1	12	49	0.74

5. Discussion

LLMs are often utilized for answering questions, summarizing documents, translating between languages and completing sentences, etc. In this work, we experimented with how an LLM can be used for concept extraction from research articles. We hypothesized that the generation of semantic triples, utilizing prompt engineering techniques, would be effective. We utilized GPT-4–1201 for semantic triple extraction from PubMed abstracts grouped under various SDoH categories. To filter the semantic triples, we used three techniques: lexical similarity, semantic similarity, and knowledge network-based similarity. We focused on each level of SOHOv1 and compared similarities between the SOHOv1 concepts at the level with the parent concepts in the semantic triple. Once the similarity was found to be above the defined threshold, we placed the child concept from the triple as a child into SOHOv1. This resulted in the expansion of concepts one level at a time. The initial version of SOHOv1 had 173 concepts and the enriched SOHOv2 had 572 concepts, slightly above a two-fold increase. We also utilized the HermiT reasoner and human expert evaluation to ensure the consistency of SOHOv2. The human experts achieved a Fleiss’ kappa agreement of 0.64, which indicates a substantial agreement [59].

In Section 2.1, we have cited relevant work exploring the development of ontologies utilizing large language and neural network models. Medical ontologies have stricter requirements than general ontologies due to the complexity of biomedical data and the need for precise terminology [60]. Creating hierarchies, axioms, and formal relationships between concepts is crucial, which LLMs might only accurately capture with careful curation.

Recent advancements in using LLMs for ontology learning come with specialized needs for tailored prompting strategies, as suggested by Dong et al. [22], who pointed out the limitations of using zero-shot prompt engineering techniques to refine the LLM output. In their paper, they used annotated datasets as a basis for concept discovery and placement, which is different from our approach. In another paper, Dong et al. used narratives in the prompt [28]. Alternatively, Trajanoska et al. [21] have explicitly specified the main categories of the concepts in the prompts. Unlike these approaches, we have utilized a specialized prompting technique, called chain of thoughts, adapting GPT-4–1201 for complex reasoning.

Wei et al.’s model query [25] is similar to the format of the semantic triples we extracted from GPT-4–1201, but it contains additional steps to train a model to generate KG embeddings, followed by utilizing ranking criteria to identify relevant concepts. Our approach uses a tailored prompting technique to extract all available related concepts from the PubMed database, and we have focused on filtering the concepts to ensure rich and semantically related concepts for inclusion. In contrast to Sun et al.’s approach [26] we do not rely on multi-modal data.

Entity recognition tasks in ontology learning are often described as multi-label classification problems to avoid the issue of multiple inheritance. Multi-label datasets derived from top-level ontologies tend to inherit the imbalances from differences in path length from the root, resulting in skewed distributions of instances across different categories [29]. Based on that, the performance of classification models that use such datasets can be negatively affected, and approaches to handle data augmentation may become complex. The pipeline that we have implemented does not perform a multi-label classification, rather the focus is on bottom-up enrichment, which helps us to start with the most granular concepts. Subsequently, we added more general concepts as children of non-leaf concepts.

Traditionally, a top-down approach has been utilized to extend foundational ontologies, and a bottom-up approach to develop domain ontologies [61],[62]. The problem of complex subhierarchies is a known issue of the top-down approach. Thus, it would be possible that two discovered “children” are themselves standing in a parent-child relationship to each other, downgrading one of them to the status of a grandchild. As we worked bottom up from the lowest level of the ontology, the discovered concepts were already fine-grained, reducing the probability of such cases. When we hand-added the discovered new concepts in Protégé, no such cases were noticed. This approach, combined with the composite filtering approach, makes our enrichment pipeline distinct.

The evaluation of semantic triples for the newly added concepts in SOHOv2 was a critical aspect of assessing our methodology. While human experts assessed 100 semantic triples, it is important to note that a total of 383 new concepts were introduced, indicating that not all additions underwent expert evaluation. This selective evaluation approach was chosen to focus on methodological validation rather than on a comprehensive assessment of the entire ontology. While the ideal scenario would involve experts assessing every single triple in the ontology, the practical challenge lies in finding reviewers with expertise in both ontology theory and the domain who would be willing to evaluate such a large number of triples. Hence, we followed prior studies [17,18,63] that relied on similar numbers of concepts for expert evaluation.

We have presented a proof-of-concept for the enrichment pipeline with our SDoH ontology SOHOv1 as a seed ontology and identified concepts from abstracts of published PubMed articles. The necessary changes while adapting our pipeline to another domain would involve 1) Rewriting the prompt and thereby updating the domain in the template, 2) Updating the weights in the composite score for lexical similarity, and 3) Adapting the thresholds for the similarity techniques utilized, which can be obtained by plotting kernel density estimates (Figure 3). The effort involved in the adaptation is limited, compared to the design of the structure of the entire enrichment pipeline.

6. Limitations and Future work

The main limitation of this study was caused by the increased financial burden of token usage with GPT-4–1201. In the future, this could be reduced by additional filtering techniques for the PubMed abstracts, reducing the number considered by the LLM. With open-source LLMs such as Llama 3 becoming increasingly competitive with GPT-4, in the future, we will also explore the use of these models that can be deployed locally. For the next phase of ontology enrichment, we plan to utilize filtering strategies to reduce the number of PubMed articles sent to the LLM model.

The thresholds mentioned in this manuscript for the three similarity techniques may vary for other domain-specific documents used for the extraction of concepts. Hence, we suggest plotting the kernel density estimates for each individual ontology development project to identify threshold values when implementing the pipeline. The manual selection of similarity thresholds and weights represents a potential limitation of the current pipeline. To address this, in future work we will focus on automating this stage by experimenting with several hyper-parameter optimization techniques such as Bayesian optimization, grid search, and random search. We also plan to explore more data-driven approaches such as performing statistical analysis of similarity scores to enhance the robustness and generalizability of our methods. Kernel densities should also be recomputed with every major enrichment of the ontology. Additionally, we will conduct an ablation study to assess the quality of concepts lost across all three similarity techniques. This analysis will also provide insights into whether the pipeline can be optimized to rely on a single similarity measure.

In this paper, we only focused on is-a relationship expansion. SOHOv2 may need to include concepts that share other relationships with the existing SOHOv2 concepts, e.g., by part-whole relationships. In future research, we will investigate meronym–holonym (part-whole), cause-effect, and other relationship patterns. We also plan to compare the semantic triples from various LLM models to analyze the quality differences among the semantic triples extracted. In this manuscript, we have performed only one round of similarity search and placement of concepts. Recursively repeating this process, we would be able to enrich any ontology starting from Stage 3 in the pipeline in Figure 1. Each iteration follows a bottom-up approach. However, the process results in possibly adding a level to the ontology. Since the ontology was evaluated for consistency, we can expand it resulting in SOHOv3.

In future work, we want to completely automate the ontology enrichment process by focusing on eliminating disconnected clusters before applying the similarity techniques. In Figure 4(a) the smallest disconnected cluster has 16 concepts connected by 15 links, whereas in Figure S1 in the supplemental file, in the periphery of the figure, there are clusters with two concepts each connected by one link. Hence, we plan to perform filtering of small clusters of concept pairs by checking the feasibility of a rule-based approach, i.e., identifying the amount of information that is lost by eliminating small clusters. More generally, three hundred twenty clusters are present in Figure S1 in the supplemental file. Filtering the semantic triples based on clustering may reduce the analysis needed to identify potential concepts using similarity searches. Additionally, we might be missing some important semantic triples as it is not guaranteed that the abstracts of articles will always contain all essential information.

We also plan to expand the similarity search from immediate neighbors to longer paths between two concepts in the UMLS and to utilize multi-modal data relevant to a specific, well-defined domain, without limiting ourselves to PubMed abstracts. Examples of additional sources of data for SDoH include forms from SDoH questionnaires, e.g., in the All of us Research workbench [64], and human-annotated simplified PubMed abstracts from the PLABA dataset [65]. We will also extend this pipeline to non-medical domains, e.g., telecommunication to determine whether we can first create a seed ontology from concepts present in 3GPP (a technical specification document in telecommunications) and then enrich it with our pipeline.

7. Conclusions

The study showcases the potential of LLMs for automating the enrichment of ontologies by mining semantic triples from biomedical literature, offering researchers and domain experts a valuable tool for knowledge discovery. It also addresses the challenge of dealing with the large volume of extracted triples by proposing an automated, comprehensive filtering process combining lexical, semantic, and knowledge network-based similarity measures. Through a case study on SDoH and automated approaches for concept placement, the feasibility of the proposed pipeline was demonstrated. Both automated and manual evaluations of the resulting ontology provided confirmation of our approach, validating the effectiveness of the proposed pipeline.

The code of our ontology enrichment algorithms is available in the GitHub repository [66].

Abbreviations

SOHOv1

Social Determinants of Health Ontology Version 1

SOHOv2

Enriched Social Determinants of Health Ontology Version 2

API

Application Program Interface