Evaluating RAG and Non-RAG Pipelines for Concept Discovery in Environmental Health Ontologies

Abstract

The expansion of biomedical ontologies with relevant, high utility concepts remains a significant challenge in biomedical knowledge representation, particularly for rapidly evolving fields like Environmental Determinants of Health (EnDOH). In this work, we evaluate the effectiveness of using LLMs in support of ontology expansion, comparing Retrieval-Augmented Generation (RAG) with non-RAG concept extraction from the medical literature. Candidate concepts were generated across 15 targeted topics using category-specific prompts. The quality of candidate concepts was assessed through semantic similarity to existing EnDOH concepts and sub-hierarchies. This design enables both a comparative analysis of RAG versus non-RAG concept extraction approaches and the identification of topic-level concept alignment with the ontology. Our results quantify the comparative strengths and weaknesses of RAG vs non-RAG concept extraction and offer a replicable methodology for effectively extracting potentially useful candidate concepts from the literature for the purpose of inclusion in biomedical ontologies.

Introduction

Ontologies are foundational tools in biomedical informatics, providing structured vocabularies that enable consistent integration and standardization of health-related data. Recently, there has been renewed interest in ontologies, as evidenced by the awarding of the Morris F. Collen Award of Excellence at AMIA 2025 for ontology-related long-term research¹. However, the process of identifying and incorporating new, high-utility concepts into ontologies remains labor-intensive, error-prone, and subjective². With the recent advancements in large language models (LLMs), capable of natural language generation, there is growing interest in leveraging LLMs and other AI tools to automate and/or enhance ontology development³.

This study explores and compares two approaches to automate concept extraction from domain-specific publications for the purpose of ontology expansion: Retrieval-Augmented Generation (RAG) versus non-RAG^4,5. In the RAG paradigm, the model is augmented with access to an external knowledge source (e.g., domain-specific PDFs of scientific articles, open access books, etc.), which the LLM consults during processing to extract domain-relevant concepts. With PubMed Central making over 8 million publications available without a paywall, there are ample sources for the RAG concept extraction task online. On the other hand, non-RAG methods rely solely on the pre-trained language model’s intrinsic knowledge to propose candidate concepts. We focus our study on the Environmental Determinants of Health (EnDOH) ontology⁶. Our goal is to evaluate whether augmenting language models with RAG retrieval mechanisms may lead to the extraction of more relevant, high-utility concepts for inclusion in the ontology.

To objectively measure concept utility, we compare the concepts produced by the RAG and Non-RAG approaches with the existing EnDOH ontology. This allows us to assess whether RAG derives concepts that are more relevant, grounded in domain evidence and more suitable for systematic incorporation into the target ontology. Specifically, we formulate the following research objectives and questions:

1. Does RAG produce concepts that are more semantically aligned with the existing EnDOH ontology concepts compared to concepts generated without external evidence (non-RAG)?

2. What is the overall impact of RAG vs. non-RAG extraction when evaluated against the existing EnDOH subhierarchies by semantic similarity scoring when considering existing and newly derived categories?

3. Across the chosen 15 environmental health categories, does the RAG pipeline outperform the non-RAG in its alignments with existing EnDOH concepts?

4. What is the quantified difference in average ontology alignment performance between RAG and non-RAG pipelines?

Background

Biomedical ontologies are structured knowledge frameworks that encode domain concepts and the relationships among them (and possibility other information items), thus enabling consistent interpretation, integration, and reuse of biomedical literature across research and clinical applications. They provide controlled vocabularies that support functions such as semantic interoperability between EHRs (Electronic Health Records), automated reasoning, and data integration within and between healthcare organizations. Large repositories, especially the National Center for Biomedical Ontology’s BioPortal, host hundreds of domain-specific ontologies reflecting the breadth of biomedical knowledge⁷. While comprehensive ontologies are highly valuable, their development and maintenance remain a challenging task. Adding new classes requires substantial expert oversight, careful alignment with existing hierarchies, and justification for inclusion to preserve long-term coherence. As previous studies have noted, ontology growth is constrained not by the lack of available relevant concepts, but by the manual processes that require domain experts to evaluate and incorporate them responsibly and at the correct locations, with the correct connections (associations, links, relationships) to other concepts^2,8.

Biomedical ontologies must evolve alongside emerging scientific evidence, yet the process of discovering new candidate concepts remains highly manual and difficult to scale. Experts are required to sift through the scientific literature, identify emerging terms, and determine which concepts can be considered apt for the target ontology. This early-stage concept retrieval phase is a significant bottleneck in ontology development⁹. If the algorithm is missing important concepts at this stage, it will prevent those concepts from ever reaching expert review. The terminological challenges in the early phases of the COVID epidemic provide a textbook example of the importance of complete, up-to-date medical terminologies. (We will use the terms ontology and terminology as substantially equivalent for our purposes.)

LLMs provide a viable mechanism to support automated concept retrieval¹⁰. Modern LLMs demonstrate strong capabilities in synthesizing textual information and identifying domain-specific terms that appear in the scientific literature¹¹. However, when prompted without access to external knowledge sources, an LLM’s suggestion are based solely on the distributed statistical associations encoded in its model weights, which were obtained during pretraining. This intrinsic knowledge is likely to get outdated very rapidly in a fast-evolving field, such as modern medicine (e.g., Machine Learning model drift). This can lead to omissions, hallucinations or generation of overly generic terms that may not reflect the contextual details required for a specific biomedical domain¹².

Retrieval-Augmented Generation (RAG) has emerged as an effective strategy to address these limitations. By grounding LLM responses in a curated set of domain-specific documents such as scientific articles, reports and research papers, RAG enables LLMs to propose concepts directly supported by scientific evidence⁴. This approach aligns with the way human curators identify candidate concepts (which may be expressed by several synonymous terms) by reading, filtering and summarizing the published literature. Comparing RAG-based concept retrieval with a non-RAG retrieval can help researchers to determine whether RAG meaningfully improves the relevance, utility, and domain grounding of concepts for incremental ontology expansion.

Related Work

Existing methods to automate ontology expansion have relied on traditional NLP pipelines that include dictionary matching, statistical co-occurrence methods and rule-based entity extraction^13-15. While these approaches offer partial automation, they have often struggled with ambiguity, synonymous concepts and contextual interpretation. These systems generated large numbers of candidate concepts, but provided limited mechanisms for assessing whether a concept was sufficiently domain-specific or suitable for integration into a target ontology. More recently, hybrid approaches such as Dynamic Retrieval Augmented Generation of Ontologies using Artificial Intelligence (DRAGON-AI)¹⁶ have explored the integration of RAG techniques to dynamically curate and refine ontology content, demonstrating a shift towards LLM-assisted ontology engineering.

Recent studies have utilized machine learning and deep learning models to improve concept extraction and ontology enrichment. Transformer-based models such as BERT and BioBERT have been used to identify biomedical entities and associations (relationships) with greater contextual accuracy, enabling more sophisticated approaches to ontology constructions and expansion^17,18. It has been shown that these models can extract high-quality concepts from the biomedical literature, though their outputs still require human validation and often lack direct alignment with the hierarchical structure of existing ontologies.

With the recent advancements of LLMs such as GPT-5, Llama 4 and Gemini 3, there has been significant interest in the use of generative models to support ontology expansion^19-21. These models can synthesize information across multiple documents, narrow down new concepts, and generate appropriate domain-specific terms. RAG has been introduced as a potential approach to further improve factual grounding by constraining LLM outputs to domain-specific evidence. Yet, little is known about whether RAG provides a measurable advantage over standard prompting for identifying concepts suitable for ontology expansion. This lack of empirical comparison motivates our current study to systematically examine the extent to which RAG and non-RAG pipelines differ in the quality and domain alignment of the candidate concepts they generate for the EnDOH ontology.

Methods

To investigate the differences between RAG and non-RAG concept extraction, our methodology proceeds through the following steps:

Seed Ontology Selection

We chose EnDOH as the seed ontology for this study. EnDOH provides a structured representation of environmental factors that influence human health, covering domains such as air quality, water and sanitation, housing, occupational exposures, and community environments. It was chosen as the baseline ontology because of its direct relevance to environmental health and its role in capturing determinants that are often underrepresented in broader biomedical ontologies such as SNOMED CT, National Cancer Institute Thesaurus (NCIt), Medical Dictionary for Regulatory Activities (MedDRA) and many more^22-24. Moreover, as a relatively new ontology, EnDOH offers a high potential for testing ontology expansion methods, since its coverage is still evolving and many relevant concepts are yet to be incorporated. An additional advantage of choosing EnDOH was that data relevant to EnDOH domains is widely available in the public literature and public datasets, making it suitable for the RAG pipeline²⁵. Non-RAG approaches also benefit from the abundance of sources, since such information is well integrated into modern Large Language Models (LLMs). Lastly, these authors have some familiarity with the area of EnDOH, which increases our confidence in judging the results correctly.

Domain Coverage for RAG and non-RAG concept extraction: EnDOH Aligned Categories

To ensure a comprehensive and domain-specific evaluation of concept retrieval pipelines, we structured our analysis around 15 categories aligned with the EnDOH ontology (Tables 1 and 2). The first nine categories were derived by filtering existing EnDOH concepts with more than two siblings and at least one grandparent, ensuring selection of well populated subhierarchies. We chose these concepts’ parent classes to have a broader and more meaningful scope for analysis. These categories provide a structured baseline for assessing how well each pipeline expands already established ontology subhierarchies.

Table 1.

Ontology Derived Categories

No.	Category	Scope of Category
1	Climate Change and Disease Spread	Captures environmental mechanisms through which climate variability can influence infectious disease dynamics.
2	Occupational and Industrial Exposures	Represents environmental hazards that are encountered in workplace and industrial settings.
3	Housing Conditions and Health	Covers physical and environmental attributes within residential settings.
4	Vulnerable Populations	Identifies population groups with heightened susceptibility to environmental harm.
5	Neighbor and Community Environment	Characterizes community level environmental qualities that influence public health.
6	Biological Exposures	Includes direct exposure to biological hazards originating from environmental conditions.
7	Chemical Exposures	Captures environmental contact with toxic or hazardous chemicals, and organic compounds that impact human health.
8	Disasters and Emergency Preparedness	Consists of environmental health risks arising from natural, technological, or climate driven disasters and the capacity of communities to withstand and recover from them.
9	Emerging Contaminants and Microplastics	Represents newly recognized or evolving environmental pollutants.

Table 2.

Literature derived Emerging Categories

No.	Category	Scope of Category
10	Mental Health and Psychosocial Stressors	Captures environmental contributors to psychological stress, mood disorders, trauma, and cognitive impacts.
11	Policy, Governance, and Risk Communication	Addresses environmental health determinants shaped by regulations and public.
12	Transportation and Mobility Systems	Covers environmental health effects related to transportation infrastructure.
13	Food Systems, Agriculture, and Nutrition Security	Captures environmental pathways linking agriculture and food systems to human health.
14	Radiation and Electromagnetic Exposures	Represents environmental exposure to ionizing and non-ionizing radiation.
15	Thermal Environment and Heat Stress	Reflects health impacts of extreme temperatures.

The remaining categories (10-15) in Table 2 were derived based on recent trends in environmental factors affecting health, identified through open-access journal publications and leading conference proceedings including Nature Climate Change, Journal of Hazardous Materials Advances, The Lancet and others. These emerging categories such as Psychosocial Stressors, Thermal Environment, and Heat Stress and Radiation Exposures represent areas where scientific interest is rapidly growing, but ontology coverage remains limited. Utilizing both seed ontology-derived categories and literature-derived categories enables a balanced evaluation of concept retrieval performance.

Retrieval-Augmented Generation (RAG) Pipeline

Figure 1 shows the RAG pipeline, which should be contrasted with the non-RAG pipeline below in Figure 2. The core of the RAG pipeline is the comparison between the embedded query and the embeddings of the documents retrieved by the RAG search. The result of the comparison is not a fuzzy truth value, but a list of preferred top-k text chunks, from which the LLM subsequently derives EnDOH concepts. The steps before the comparison have the purpose of making the query and the text structurally similar enough to allow the use of cosine similarity. We now explain details.

Figure 1.

RAG based Concept Extraction Pipeline for EnDOH

Figure 2.

Non-RAG Concept Extraction Pipeline for EnDOH

Literature/Document Ingestion: To build a domain-relevant literature set, we systematically gathered domain-specific literature using Google Scholar, prioritizing open-access publications. For each EnDOH aligned category, we first restricted our search to publications within the most recent six months, reflecting the fast-evolving landscape of environmental health research. However, several topics did not yield a sufficient number of openly accessible publications within this narrow timeframe. In such cases, we implemented a sliding-window retrieval approach²⁶. When fewer than ten eligible publications were found with the sliding window approach, we incrementally expanded the search window by an additional one-year interval, repeatedly backtracking until we accumulated a minimum of ten open-access publications per topic. This approach balanced recent publications with availability and ensured a consistent corpus size across categories, allowing us to capture current developments while maintaining adequate literature depth for the RAG and non-RAG pipelines. This literature/document ingestion is summarized in Figure 1.

Text Chunking: After assembling the document set for each EnDOH aligned category, all publications were segmented into overlapping text chunks to enable efficient embedding and retrieval. Each document was divided into contiguous units of 200 words with a 50-word overlap, preserving continuity across boundaries. We heuristically selected a chunk size of 200 words (with overlap), consistent with common RAG implementations that emphasize short, semantically coherent retrieval units that are small enough to support efficient vector embedding^4,27. The overlap acts as a buffer that ensures that key definitions, methodological steps, or result descriptions are not truncated.

Chunk Embedding: Each chunk was then transformed into a high dimensional vector representation using a SentenceTransformer model (all-MiniLM-L6-v2)²⁸. The embedding model was chosen for its strong performance on semantic similarity tasks, computational efficiency, and suitability for large scale document processing pipelines²⁹. This embedding phase as shown in Figure 1, mapped each chunk into a dense semantic space where conceptually related text segments lie in close proximity. Importantly, the model captures domain-level semantics such as environmental exposures and health outcomes that allow retrieval to extend beyond exact keyword matching. To ensure consistency across categories, embeddings were computed using identical preprocessing steps (lowercasing, punctuation normalization, and whitespace standardization).

Semantic Indexing using FAISS: For scalable semantic search, all chunk embeddings were inserted into a Facebook AI Similarity Search (FAISS) index³⁰. FAISS provides efficient approximate nearest-neighbor (ANN) search capabilities, enabling rapid retrieval of the most relevant chunks from thousands of candidates with minimal computational overhead.

We used an index configuration optimized for mid-scale corpora (tens of thousands of vectors), balancing retrieval speed with similarity accuracy. The index was constructed using cosine similarity as the underlying distance metric, given its strong alignment with semantic similarity tasks and its compatibility with the MiniLM embedding space.

Adaptive-k Retrieval of Relevant Chunks: Rather than using a fixed k for all queries, we implemented an adaptive-k retrieval mechanism consisting of three parameters:

• α – minimum number of chunks to retrieve

• β – maximum number of chunks to retrieve

• γ – relative-drop threshold

The parameter α ensures a minimum level of semantic coverage, preventing important contextual information from being overlooked; β sets an upper bound to avoid unnecessary computational cost or retrieval of semantically diluted chunks; γ identifies when similarity scores fall sharply between adjacent chunks, which is a sign that further chunks are no longer topically cohesive. These parameters determine the dynamic expansion or contraction of k based on how sharp the decline in similarity scores is across the ranked list of retrieved chunks.

In our study, we assigned α = 6, β = 20, and γ = 0.10, based on document length heuristics as follows:

• A minimum of six 200-word chunks with a 50-word overlap (~950 words) is sufficient to represent short conference papers and two-page posters.

• Up to twenty chunks (~3,050 words) typically cover conference papers (assuming a page of references and roughly two pages taken up by acknowledgments, figures, captions, and result tables, out of eight pages).

• A 10% similarity drop (γ) captures shifts in semantic relevance without being overly sensitive to minor fluctuations.

This adaptive strategy ensures that retrieval is context-rich, computationally efficient, and aligned with the structural characteristics of the scientific literature.

LLM-Based Concept Extraction from Retrieved Chunks: After adaptive retrieval, the selected chunks were provided to the LLM for category specific concept extraction. We used GPT-5 (gpt-5-chat-latest) for all extraction tasks²⁰. Each category used its own tailored prompt, resulting in 15 distinct prompts, one for each environmental health domain defined in our study. This ensured that concept extraction was aligned with each topic and avoided cross category drift. The LLM synthesized information across multiple retrieved chunks, identifying recurring exposures and health related environmental determinants supported by the literature.

To maintain consistency across categories, all prompts shared a common structure and included explicit instructions designed to reduce hallucinations and overly broad outputs. The instructions emphasized:

• Restricting concepts strictly to those supported by the retrieved text.

• Avoiding definitions, long phrases, or explanatory descriptions.

• Returning exactly one concept per line (RAG extraction mode) or a comma-separated list (non-RAG mode)

• Excluding synonyms unless they represented conceptually distinct entities.

We set the model temperature to 0.5, which we selected as a balance between creativity and factual grounding. This configuration was chosen based on a subsequent submission of ours, where we evaluated temperature effects on domain-specific concept generation and analyzed LLM convergence over multiple iterations³¹.

An example prompt shown here for Category 14: Radiation and Electromagnetic Exposures illustrating the structure and domain-specific guidance used during extraction:

“You are a biomedical expert specializing in environmental determinants of health (EnDOH). From the following scientific text, extract a list of concise, unique concepts related to radiation and electromagnetic exposures. Examples include: UV radiation, ionizing radiation, radon exposure, X-ray exposure, microwave radiation, radiofrequency electromagnetic fields, nuclear fallout, cosmic ray exposure, etc. Only return a comma-separated list of environmental determinants of health–relevant concept names. Exclude social, behavioral, or economic terms and skip all explanations. Text: {text_block}”

We provided only one example here due to space constraints, all 15 prompts and full extraction outputs, along with additional implementation details, are publicly available at: https://github.com/narenkhatwani/RAG-vs-non-RAG-concept-extraction

Non-RAG Pipeline

For the non-RAG concept extraction pipeline, we used the same model configuration as for the RAG pipeline, namely GPT-5 (gpt-5-chat-latest) with a temperature of 0.5, selected to balance creativity and factual grounding. However,

the non-RAG pipeline relied solely on the LLM’s intrinsic knowledge, unlike the RAG pipeline where the model was augmented by text chunks from the literature.

We designed the non-RAG prompts to mirror the structure of the RAG prompts as closely as possible, but without supplying any retrieved text chunks. All category specific instructions, output constraints, and the context around the prompt were retained to ensure methodological equivalence. Hence, each of the 15 environmental health categories used its corresponding category specific prompt, but with the text block placeholder removed. An example prompt for Category 14: Radiation and Electromagnetic Exposures, shown previously in the RAG setting is shown below in its non-RAG setting to illustrate the difference:

“You are a biomedical expert specializing in environmental determinants of health (EnDOH). Create a diverse list of concise, unique concepts related to radiation and electromagnetic exposures. Examples include: UV radiation, ionizing radiation, radon exposure, X-ray exposure, microwave radiation, radiofrequency electromagnetic fields, nuclear fallout, cosmic ray exposure, etc. Only return a comma-separated list of environmental determinants of health relevant concept names.”

This non-RAG setting preserved the conceptual intent of the RAG prompt but removed any scientific literature derived text, enabling us to examine how LLMs produce domain-specific concepts in the absence of external evidence.

Comparative Analysis: RAG vs. non-RAG

Ontology Alignment Evaluation using EnDOH subhierarchies: To compare the conceptual quality of RAG and non-RAG, we evaluated each extracted concept against the structure of the EnDOH ontology. Using EnDOH as a seed ontology we evaluated each extracted concept against the existing EnDOH structure. Using EnDOH as the seed ontology, we followed its existing subhierarchy structure by grouping each Preferred Label under the parent class defined within the ontology, without modifying or reconstructing any part of the original hierarchy. All root-level nodes (e.g., Environmental Determinants of Health, Thing) were excluded to avoid trivial matches. For each parent node, we generated a cluster of its child concepts and precomputed their MiniLM embeddings. This produced a set of embedding-based semantic neighborhoods representing the ontology’s internal structure. Each RAG and non-RAG concept was embedded using the same MiniLM encoder and compared against every EnDOH cluster to identify the closest matching subhierarchy as well as the most similar EnDOH concepts. This alignment procedure enabled a fine-grained assessment of where new concepts “fit” within the existing ontology.

Semantic Similarity Scoring: After aligning each generated concept with its most similar EnDOH subhierarchy, we quantified the match strength using cosine similarity. For every concept, we computed its similarity to all EnDOH labels within a subhierarchy and retained the maximum similarity score as the concept’s ontology-alignment score. We then aggregated these scores across all RAG concepts and all non-RAG concepts. As implemented in our evaluation script, this produced average alignment scores for each method, allowing for a direct comparison of how well each pipeline adheres to the semantic structure of EnDOH.

Because both RAG and non-RAG concepts were evaluated using identical normalization procedures, the same embedding model, and the same cluster comparison logic, any differences in similarity scores reflect the effect of retrieval itself, i.e., whether the presence of evidence-grounded text improves how closely generated concepts align with existing ontology content.

Results

Across the 15 environmental health categories, the RAG pipeline consistently produced sets that were more semantically aligned with the existing EnDOH ontology concepts than those generated by the non-RAG pipeline. As shown in Figure 3, RAG concept extraction outperformed non-RAG concept extraction in 12 out of 15 categories, demonstrating the advantage of grounding concept extraction in retrieved literature evidence. This fulfills Research Objective 3, which raised the question whether RAG outperforms non-RAG across categories. Only three categories (1) Policy, Governance, and Risk Communication, (2) Transportation and Mobility Systems, and (3) Mental Health and Psychosocial Stressors showed slightly higher similarity scores in the non-RAG setting.

Figure 3.

Semantic similarity of RAG vs. non-RAG concepts relative to existing EnDOH concepts.

Overall, the mean semantic similarity for RAG across all categories was 0.5490, compared to 0.4976 for the non-RAG approach, representing a 10.3% relative improvement in semantic alignment with established ontology concepts (addressing Research Objective 2). These aggregate values highlight that RAG concept extraction enhances the model’s ability to generate domain-specific and ontology consistent environmental health concepts. The detailed categorical values supporting these findings are provided in Figure 3. This quantitative comparison also fulfills Research Objective 4, which evaluates the difference in average ontology alignment performance between the two pipelines.

These results collectively indicate that integrating RAG retrieval into the concept extraction workflow meaningfully improves the semantic coherence of generated terms, especially for categories from rapidly evolving literature subdisciplines. The non-RAG model, in contrast, produced narrower “niche” environmental terms, reflecting its reliance on intrinsic knowledge rather than category-specific knowledge gleaned from the literature.

For example, in Category 2: Occupational and Industrial Exposures, the RAG pipeline produced broad and widely recognized exposure concepts such as particulate matter exposure, industrial chemical exposure, asbestos exposure, silica exposure, lead exposure, and pesticide exposure. These concepts are related directly to the standard occupational health hazard domain. On the contrary, the non-RAG pipeline returned narrow, chemical focused concepts such as chromium mist exposure, vinyl chloride exposure, toluene solvent exposure, manganese fume exposure, refractory ceramic fiber exposure, nanomaterial aerosol exposure, and diesel exhaust particulate exposure. While these concepts are accurate, they have a narrow scope where broader hazard categories are needed. This example highlights how RAG concept extraction enables the model to generate more generalizable and semantically appropriate concepts.

Discussion and Limitations

Retrieval Augmented Generation is not universally superior to non-RAG approaches⁵. When a task depends heavily on reasoning, then the consideration of additional sources is not necessarily helpful. Thus, the advantage of using RAG depends on the domain and context of a specific problem. In this paper, we attempted an answer to the question whether RAG is superior to non-RAG in the context of ontology expansion based on the scientific literature. Intuitively, this should be a context where RAG excels, as the black box weights in the LLM network of existing systems are likely to become outdated (drift) very quickly when the human knowledge base, as expressed by the literature, grows exponentially (interpreting findings for Research objective 1).

The relative advantage of RAG should also depend on the domain of its application. Not all biomedical fields are growing at the same rate, and growth is often punctuated by sudden breakthroughs (e.g., mRNA vaccines) that lead to strong growth in specific fields. Environmental Determinants of Health have become a field of interest only recently, thus extensive growth with the concomitant development of new conceptual knowledge, is expected, making the area a good testbed for the RAG vs. non-RAG evaluation. Overall, our hypothesis was born out by the study results in 12 out of 15 categories.

Another limitation of this study was the lack of involvement of human expert reviewers (who were not part of the research team) in vetting the newly derived concepts. Instead, we relied on semantic similarity to existing subhierarchies in the target ontology computed by a widely used metric. It is an ongoing problem in this research area that human experts, who must be fluent in both ontologies and the biomedical subfields of the specific study, are hard to come by and typically expensive.

Lastly, a limitation of this study is the result of the rapid reissuing of branded LLMs. New versions of ChatGPT, Gemini, Llama, etc. are released at a breathtaking speed and frequency. While it is not generally known what source materials new versions are trained on, one may assume that a newer version of a specific LLM would benefit less from RAG than an older version. This has not been investigated in this study.

Conclusions and Future Work

We compared the effectiveness of new concept extraction for the purpose of ontology expansion performed with RAG and with non-RAG pipelines. This extraction was done in the domain of Environmental Determinants of Health. We hypothesized that results would be better with the RAG pipeline accessing the scientific literature. This expectation was supported for 12 out of 15 term categories, while the non-RAG pipeline was superior for two categories: Mental Health and Psychosocial Stressors and Policy, Governance, and Risk Communication. Semantic similarity for RAG varied between 0.37 and 0.66 for different term categories. For non-RAG the range was smaller, extending from 0.44 to 0.56. This addresses Research objective 1.

Naturally, the limitation of this study to one domain makes it impossible to confidently express a conclusion about ontology expansion in general. More studies are necessary in different medical subspecialties, e.g., cardiology, pulmonology, infectious diseases, cancer, etc. In future work, we intend to close a number of gaps mentioned above. Specifically, we aspire to

• Evaluate RAG vs. non-RAG with different LLMs beyond ChatGPT.

• Evaluate RAG vs. non-RAG with different versions of the same LLM, with the expectation that the relative advantages of RAG will decrease for newer releases compared to older ones.

• Evaluate RAG vs. non-RAG in other domains beyond EnDOH,

• Involve human experts external to the core research team in an independent evaluation of the derived new concepts.