Toward total recall: Enhancing data FAIRness through AI-driven metadata standardization

Abstract

Scientific metadata often suffer from incompleteness, inconsistency, and formatting errors, which hinder effective discovery and reuse of the associated datasets. We present a method that combines Generative Pre-trained Transformer 4 (GPT-4) with structured metadata templates from the Center for Expanded Data Annotation and Retrieval (CEDAR) knowledge base to automatically standardize metadata and to ensure compliance with established standards. A CEDAR template specifies the expected fields of a metadata submission and their permissible values. Our standardization process involves using CEDAR templates to guide the GPT-4 in accurately correcting and refining metadata entries in bulk, resulting in significant improvements in metadata retrieval performance, especially in recall—the proportion of relevant datasets retrieved from the total relevant datasets available. Using the BioSample and Gene Expression Omnibus (GEO) repositories maintained by the National Center for Biotechnology Information (NCBI), we demonstrate that retrieval of datasets whose metadata are altered by GPT-4 when provided with CEDAR templates (GPT-4+CEDAR) is substantially better than retrieval of datasets whose metadata are in their original state and that of datasets whose metadata are altered using GPT-4 with only data-dictionary guidance (GPT-4+DD). The average recall increases dramatically, from 17.65% with baseline raw metadata to 62.87% with GPT-4+CEDAR. Furthermore, we evaluate the robustness of our approach by comparing GPT-4 against other large language models, including LLaMA-3 and MedLLaMA2, demonstrating consistent performance advantages for GPT-4+CEDAR. These results underscore the transformative potential of combining advanced language models with symbolic models of standardized metadata structures for more effective and reliable data retrieval, thus accelerating scientific discoveries and data-driven research.

Introduction

Effective data sharing can be improved by adhering to the FAIR guiding principles [1], ensuring that data are findable, accessible, interoperable, and reusable. A key requirement for making data FAIR is high-quality metadata [2], which provides information about their associated data such as their origin, purpose, usage, and the conditions under which the data were collected. These metadata play a pivotal role in facilitating the reproducibility and organization of the data and in enhancing their discoverability. Nevertheless, metadata in scientific repositories are often incomplete, inconsistent, and incorrectly formatted, hindering data discoverability and reuse [3, 4]. Standardizing metadata is therefore a necessary process for ensuring that datasets are properly described and accessible, enabling their reuse, integration, and secondary analysis. However, manually improving metadata quality is a complex and time-consuming process due to variability in community standards and subtleties in applying such standards. In this paper, we introduce an automated approach to standardizing metadata by leveraging structured metadata templates—machine-readable specifications that define required metadata fields and permissible values—and large language models (LLMs), such as Generative Pre-trained Transformer 4 (GPT-4) [5]. We evaluate the effectiveness of our approach through key information-retrieval metrics, primarily focusing on recall, which measures the proportion of relevant metadata records correctly retrieved out of all relevant metadata records. Additionally, we assess precision, the proportion of retrieved metadata records that are relevant, and the F1-score, a balanced measure combining precision and recall. Our study demonstrates that AI-driven metadata standardization significantly enhances these retrieval metrics, greatly improving the discoverability and usability of scientific data. Scientific metadata usually consist of pairs of field names and field values that describe datasets derived from experimental results. In Fig. 1, we show an example metadata record taken from the BioSample repository of the National Center for Biotechnology Information (NCBI) [6]—a public repository that stores metadata for a wide range of biological samples, including biomedical studies and samples from plants, animals, pathogens, and environmental sources such as seawater and soil. In our example medical record, the field name is given as tissue, and the field value is given as lung cancer. Even without knowing the details of the original sample, we can infer that the metadata field value is inaccurate, as lung cancer is not a type of tissue. Searches for scientific datasets primarily involve querying metadata. Consequently, a researcher querying for appropriate tissue values would overlook this record and potentially other useful records during their search.

Figure 1.

Metadata record from BioSample, where the black box highlights a field name–field value pair. In this example, the field name “tissue” is wrongly associated with the field value “lung cancer”. Orange boxes mask identifying information. Retrieved from https://www.ncbi.nlm.nih.gov/biosample/.

A key strategy for improving the quality of scientific metadata involves the development of discipline-specific metadata standards that draw upon both reporting guidelines (that provide standard field names) and ontologies (that provide standard field values). For example, in the functional genomics community, the Minimum Information About a Microarray Experiment (MIAME) guideline specifies essential metadata attributes needed for reproducibility and reusability of microarray data, such as details about the sample that was studied, the experimental design, data processing, and normalization methods. By establishing clear, structured reporting standards, MIAME improved the consistency, comparability, and overall utility of microarray datasets, facilitating more effective data sharing and secondary analysis within the genomics community [7]. Ontologies provide structured, standardized vocabularies that facilitate consistent annotation across datasets, supplying systematized values for the fields enumerated by reporting guidelines. They also provide useful information at different levels of granularity, most suited for complex querying. For example, UBERON [8] is a multi-species anatomy ontology that offers a standardized vocabulary for anatomical structures in animals and is increasingly used to annotate tissue-related datasets. For metadata standards to be effective, however, they must be tailored to specific research domains, as different scientific disciplines require metadata specifications that reflect their unique experimental situations. A promising approach is for research communities to define discipline-specific metadata reporting guidelines and to render those guidelines as structured templates. The Center for Expanded Data Annotation and Retrieval (CEDAR) Workbench [9] offers scientists a collaborative and efficient means to create machine-actionable, structured metadata templates for authoring standards-adherent metadata. A CEDAR template (Fig. 2) is a machine-readable specification of the expected metadata fields and their permissible values, which may be sourced from multiple ontologies or controlled vocabularies available in the BioPortal repository [10]—the most comprehensive, open archive of biomedical ontologies. By encoding well-established reporting guidelines using CEDAR templates, researchers can ensure that metadata are standardized and machine-actionable, ultimately enhancing data discovery and secondary analysis. Recent work by our laboratory [11, 12] has demonstrated that encoding community metadata standards as structured CEDAR templates provides a scalable, automated way to enhance the FAIRness of scientific data by improving metadata quality and standardization.

Figure 2.

A screen capture depicting the CEDAR metadata entry form derived from the template for BioSample. Field names specified in the data dictionary, such as “tissue” and “organism”, are listed in the template. CEDAR recommends options for the “tissue” field from a branch of the Uberon ontology based on frequency of values in the data hosted on CEDAR. The percentages indicate the frequency of usage of a field value over the data samples present in CEDAR.

Automated metadata cleaning inherently involves natural language processing, a challenge for which LLMs have recently proven particularly effective. Contemporary research efforts [13, 14] have explored the use of LLMs to improve metadata quality, including efforts to curate specific fields such as cell line names in BioSample. These studies show that targeted metadata curation with LLMs can substantially improve the quality and usability of key metadata fields. However, the focus of these studies has largely been limited to a small number of fields—such as cell line names—within otherwise well-structured metadata. In contrast, real-world metadata authoring often requires reasoning over entire reporting guidelines with numerous interdependent and heterogeneous fields, presenting additional challenges in scalability and generalizability that prior work has not addressed. This limitation underscores the need for approaches that can operate in more complex metadata environments where multiple interdependent fields must be curated in concert. Our recent work [15] demonstrates that automated multi-field metadata curation is a challenging task for LLMs that may be addressed by augmenting LLMs with knowledge derived from expert feedback. In this paper, we present the quantifiable effects of using structured metadata templates in CEDAR in tandem with LLMs to automatically correct legacy metadata. We evaluate our method on a large dataset of 4,800 samples drawn half from BioSample and half from the Gene Expression Omnibus (GEO) [16], measuring the impact of metadata improvements on search outcomes. The GEO is a public database at the NCBI that archives and freely distributes high-throughput gene expression and other functional genomics data submitted by the scientific community. We also verify whether the benefits of using metadata templates to guide metadata standardization are consistent across different LLMs, including GPT-4, Large Language Model Meta AI (LLaMA-3) [17], and MedLLaMA-2 [18]. While the FAIR principles are widely endorsed for their theoretical benefits, a lack of quantitative analysis has made it difficult to assess their practical effects on dataset searchability. A key contribution of our work is the quantification of metadata retrieval improvements following metadata standardization.

Data description

For our experiment, we used metadata records from two distinct databases: BioSample and GEO. We developed queries (shown in Table 1) to retrieve metadata records from both BioSample and GEO related to three types of cancer: lung cancer, liver cancer, and ovarian cancer.

Table 1.

Queries used to retrieve metadata records from the BioSample and GEO databases and create our source datasets.

Disease	Database	Query
Lung cancer	BioSample	lung cancer[All Fields] AND “human 1 0”[filter]
	GEO	lung cancer[All Fields] AND human[Organism]
Liver cancer	BioSample	liver cancer[All Fields] AND “human 1 0”[filter]
	GEO	liver cancer[All Fields] AND human[Organism]
Ovarian cancer	BioSample	ovarian cancer[All Fields] AND “human 1 0”[filter]
	GEO	ovarian cancer[All Fields] AND human[Organism]

For each of the three types of cancer, we extracted a sample of 800 randomly selected records from each dataset. We initially sampled 1,000 records for each query, removed those with XML formatting errors, and selected the maximum uniform number of well-formatted records across all queries, which was 800. Our test corpus thus comprises 4,800 records: 2,400 from BioSample and 2,400 from GEO.

Methods

Our work aims to enhance the quality of scientific metadata by leveraging LLMs and the CEDAR structured metadata knowledge base. Specifically, we evaluate how improvements in metadata quality affect search-related metrics, with a focus on recall performance.

To automate the correction of metadata records, our first method was to instruct an LLM to use the BioSample data dictionary [19]—which defines allowed metadata field name–field value pairs, formats, and descriptions—to correct the metadata record. This process and data dictionary are illustrated in Fig. 3, and the prompts are described in Appendix A. The GEO repository lacks a similarly detailed data dictionary. Although GEO recommends adherence to the widely adopted MIAME guidelines, these guidelines primarily outline broad metadata reporting requirements—specifying the types of information to include—rather than offering structured descriptions of individual field–value pairs. Consequently, for metadata fields specific to GEO, we reused relevant portions of the BioSample data dictionary.

Figure 3.

GPT-4 correction of BioSample record. The prompt given to GPT-4 includes both the metadata to be corrected and the entire data dictionary to which BioSample records should adhere.

We then designed a second method that uses structured metadata templates built with CEDAR as guiding mechanisms for LLMs to correct metadata. CEDAR templates contain machine-readable restrictions on fields that can assist an LLM in determining when a value is inappropriate for a field (Fig. 4). These restrictions include data-type restrictions (e.g., string, date, and integer), value-set restrictions, and ontological restrictions that ensure that metadata entries are consistent with established standards. Given its comprehensive repository of metadata templates, CEDAR is a natural choice for supporting metadata-related endeavors. For example, as shown in Fig. 2, CEDAR has in its library a BioSample metadata template where the “tissue” field only takes values from the Uberon ontology—a comprehensive multi-species anatomy ontology—thus restricting the user to select standard terms. A dedicated CEDAR template for GEO was not available, so we reused relevant sections of the BioSample CEDAR template.

Figure 4.

GPT-4 correction of BioSample record with CEDAR template. The prompt includes template information and value restrictions.

We evaluated the described framework using different LLMs (GPT-4, LLaMA-3, and MedLLaMA2) and our test dataset. In our test data, we used two versions of each metadata collection—one that is augmented based on a data dictionary (LLM+DD), and another that is augmented based on information in a CEDAR template (LLM+CEDAR). We searched the standardized metadata collections using four representative queries targeting the metadata field “tissue”. In these queries, we adopted the syntax field:value, indicating the metadata field followed by its desired value. Specifically, our queries were tissue:lung, tissue:blood, tissue:liver and tissue:ovary, adjusted for the relevant cohort. The search was based on exact string matching between the standardized metadata field values and the query terms. Specifically, a metadata record was counted as retrieved if the value in its “tissue” field matched the query string exactly, without partial matching, stemming, or synonym expansion. We then measured commonly used search metrics, including precision, recall, and F1-score. The F1-score is the harmonic mean of precision and recall, providing a single summary metric that balances both measures.

Our query strategy employs simple queries designed to expose the workings of current databases. While we have also experimented with more complex queries and observed further gains in recall, increasing query complexity introduces additional linguistic and semantic content that is weakly aligned with the retrieval target, thereby adding noise and making retrieved results harder to interpret. This effect is amplified by the fact that computing ground truth for our task is non-trivial, so distinguishing true improvements from noise-induced fluctuations becomes difficult.

Statistical analysis

All analyses were performed using independent experimental runs. For each condition, we computed the mean performance metric across six runs, where each run represents the aggregate of ~2,400 individual predictions; the run was therefore treated as the independent unit of analysis. Variability is reported using the standard error of the mean (SE), calculated as

where is the standard deviation across runs and . SE values were used for all error bars in the result figures to represent uncertainty around the mean estimate. Because our metrics are bounded and non-negative, asymmetric error limits were applied when necessary to prevent error bars from extending below zero. Statistical comparisons between methods were performed using ANOVA (Appendix C). All analyses and visualizations were conducted using Python (NumPy, SciPy, Matplotlib).

Evaluation metrics

To evaluate how our metadata-correction methods influenced searchability, we first needed to establish a ground truth against which to compare retrieval results. Because no formal gold standard was available, we manually examined the “tissue” values in our test dataset and developed simple rules to assign the correct tissue values (Table 2). For example, “blood” is a commonly found tissue value, accounting for almost 50% of samples from BioSample (Fig. 1). However, the metadata value for blood is often confounded by the methods of sample preparation, such as peripheral blood mononuclear cells (PBMC), whole blood, blood sample, plasma, etc. Hence, we used a heuristic approach (Tables 3 and 4) to assign the correct field value. This approach allowed us to construct a gold standard that served as the best available reference for evaluating the performance of our methods. We manually inspected 5% of the gold standard, representative across queries and databases, and found it complying with our expectations. We considered stemming and other methods for cleaning up the values, but we found them to have no impact on the final cohort precision, as the errors came from mislabeling or unknown synonyms. We evaluated search performance on our two test datasets using standard information retrieval metrics—precision, recall, and F1-score (see Table 2 for definitions).

Table 2.

Annotation rules used to construct our approximate gold standard dataset. These rules are used to assign corrected tissue values in the test dataset.

Input: Metadata record

Output: Assigned tissue label (lung, liver, ovary, blood, plasma, lymph, or unknown)

1. Initialize tissue field label as “unknown”

2. If tissue field contains the word “lung”, set label to “lung”

3. Else if tissue field contains the word “liver” or “HCC”, set label to “liver”

4. Else if tissue field contains the word “ovary” or “ovarian”, set label to “ovary”

5. Else if tissue field contains the word “PBMC” or “blood”, set label to “blood”

6. Else if tissue field contains the word “plasma”, set label to “plasma”

7. Else if tissue field contains the word “lymph”, set label to “lymph”

8. Return the assigned tissue label

Table 3.

Explanation of search metrics: precision, recall, and F1-score.

Metric	Explanation	Formula
Precision	Precision is the ratio of correctly retrieved relevant instances to the total retrieved instances.
Recall	Recall is the ratio of correctly retrieved relevant instances to the total relevant instances.
F1-Score	F1-Score is the harmonic mean of precision and recall.

Table 4.

Definitions of true positives, false positives, false negatives, and true negatives in the context of metadata curation and retrieval for a query of the form tissue = X.

Outcome	Definition in this study
True positive (TP)	A metadata record where the gold-standard tissue label is X  and the curated value produced by the model is also X. The record is relevant and correctly retrieved.
False positive (FP)	A metadata record where the curated value produced by the model is X, but the gold-standard tissue label is not  X. The record is not relevant but is incorrectly retrieved.
False negative (FN)	A metadata record where the gold-standard tissue label is X, but the curated value produced by the model is not  X (including unknown, other tissues, or missing values). The record is relevant but was not retrieved.
True negative (TN)	A metadata record where the gold-standard tissue label is not  X, and the curated value produced by the model is also not  X. The record is not relevant and is correctly excluded.

Results

We found that recall improved substantially across all datasets and methods when we augmented GPT-4 with structured knowledge sources, compared to retrieval using the original baseline metadata. Figures 5–7 show the average recall, precision, and F1-scores. In the BioSample dataset, the average recall increased from 20% with baseline metadata to 21% with GPT-4 guided by the data dictionary (GPT-4+DD), and further to 82% with GPT-4 guided by the Center for Expanded Data Annotation and Retrieval (CEDAR) templates (GPT-4+CEDAR). In GEO, recall improved from 15% at baseline to 3% with GPT-4+DD and to 44% with GPT-4+CEDAR (Fig. 5). When we averaged results across both datasets, recall increased from 18% at baseline to 63% with GPT-4+CEDAR. We also observed an improvement in precision, rising from 58% at baseline to 66% with GPT-4+CEDAR (Fig. 6). As a result, the overall F1-score increased from 24% to 63% (Fig. 7).

Figure 5.

Comparison of average recall values for BioSample and GEO datasets across three result sets: Baseline, GPT-4+DD, and GPT-4+CEDAR.

Figure 7.

Comparison of average F1-score values for BioSample and GEO datasets across three result sets: baseline, GPT-4+DD, and GPT-4+CEDAR.

Figure 6.

Comparison of average precision values for BioSample and GEO datasets across three result sets: Baseline, GPT-4+DD, and GPT-4+CEDAR.

We conducted ANOVA analyses to compare recall values across conditions (baseline vs. GPT-4+DD, GPT-4+DD vs. GPT-4+CEDAR, and baseline vs. GPT-4+CEDAR). All comparisons showed statistically significant improvements (P < 0.01) for BioSample, and the details are available in Appendix C. We could not establish statistical significance for GEO. We also computed effect sizes using Cohen’s d, a standardized measure of the magnitude of differences between conditions. A Cohen’s d value of 0.2 is typically considered a small effect, 0.5 a medium effect, and 0.8 or above a large effect. In our study, we observed large effect sizes (d > 0.8) for the improvements with GPT-4+CEDAR over baseline, indicating that the observed gains are not only statistically significant but also practically meaningful.

We observed that the most significant improvement in overall retrieval (F1-score) and recall occurred when we standardized metadata using GPT-4 and CEDAR (GPT-4+CEDAR). The baseline metadata exhibited poor recall due to poor field-value quality, which made it difficult to retrieve relevant records. By contrast, standardizing metadata with GPT-4 augmented with knowledge from CEDAR templates substantially improved retrieval accuracy and enhanced search performance. Our qualitative analysis showed that declines in precision primarily stemmed from errors introduced by GPT-4 when it processed lengthy or ambiguous metadata. For example, GPT-4 occasionally changed correct tissue values—such as “blood” to “lung”—especially in longer-than-average BioSample records. Mentions of “lung” elsewhere in the metadata likely influenced these misclassifications. We also found that the rule-based labeling approach acted conservatively, often marking tissue values as unknown. In many such cases, GPT-4 correctly extracted the tissue value but was penalized due to inaccuracies in the gold standard.

In our dataset-specific analysis, we saw recall improvements across all queries. In the Liver Cancer dataset, searches for tissue:liver showed substantial improvements in recall, while precision dropped slightly due to a few minor misclassifications. For example, one of the retrieved metadata records was a match for the tissue type “liver”, but our ground truth heuristic missed it because the relevant information appeared in the description field rather than the tissue field. In the tissue:blood query, we observed modest results where we either recorded the same recall values or a small improvement. Similarly, in the Ovarian Cancer dataset, the query for tissue:ovary led to substantial recall improvements with minor variation in precision. The Lung Cancer yielded the most promising results, with the tissue:lung query achieving high improvements in recall. When we compared our two, methods GPT-4+DD and GPT-4+CEDAR, we found that using only the repository’s official data dictionary provided minor enhancements in recall for BioSample and significantly reduced performance for GEO. Since GEO records contain lengthy, free-text metadata values, there is increased potential for the LLM to deviate from the expected metadata values. For a non-small cell lung cancer (NSCLC) sample, GPT4+DD corrected the existing metadata to tissue:NSCLC tumor rather than the expected tissue:lung. In contrast, integrating CEDAR templates consistently resulted in noteworthy improvements across datasets, highlighting the importance of structured templates in the standardization of metadata.

We encountered difficulties when deploying alternative LLMs such as LLaMA-3 and MedLLaMA-2 (see Appendix B). These models frequently produced formatting errors that required extensive post-processing, making them less appropriate for large-scale metadata standardization. In contrast, GPT-4+CEDAR achieved the best balance of accuracy and scalability, consistently outperforming other models in recall and retrieval performance.

Discussion

Our study empirically validates a long-standing but previously untested assumption at the heart of the FAIR data principles: that structured, standardized metadata improve data findability, accessibility, and reusability. While advocates of FAIR have promoted these principles for over a decade, most empirical work has focused on qualitative assessments or proxy metrics such as repository adherence or metadata availability. Our work directly quantifies the impact of metadata structure on dataset retrieval, providing clear evidence that aligning metadata with FAIR principles using structured templates significantly boosts performance in search systems. These findings fill a crucial evidence gap and offer a data-driven foundation for ongoing efforts to implement FAIR practices across the scientific data ecosystem.

Beyond this empirical contribution, our work addresses an emerging consensus in the field of AI: Achieving optimal performance with LLMs requires more than just improved model architecture or increased scale. As LLMs become more powerful, their limitations—particularly in reasoning, precision, and domain specificity—have become more apparent. Increasingly, researchers are recognizing that enhancing LLM performance demands the strategic integration of structured, domain-specific, external knowledge [20, 21]. Our approach exemplifies this shift by embedding knowledge about metadata structures into LLM prompts through reusable, ontology-aligned metadata templates curated by experts. A range of methodologies has been proposed for injecting such knowledge into LLMs, including prompt engineering, fine-tuning with curated datasets, retrieval-augmented generation (RAG), integration with knowledge graphs, and neuro-symbolic approaches. Each of these methods offers a trade-off between accuracy, scalability, and annotation effort. We use template-augmented prompt engineering [22], a scalable and practical strategy for combining the linguistic fluency of LLMs with the semantic rigor of metadata templates.

Template-augmented prompt engineering occupies a unique space among knowledge infusion strategies. Fine-tuning, while effective, requires labor-intensive construction of labeled training data, which is often infeasible in specialized domains such as biomedicine where expert annotation is costly. Another common strategy is RAG, which supplements generation with externally retrieved documents; however, effective RAG deployment typically requires changing how search is performed at the database/indexer level so that the retriever returns the right context. By contrast, our goal is to improve the author-side inputs—templates, field definitions, and prompt formats—so that downstream systems (and users) can obtain higher-quality outputs without requiring modifications to the existing database search infrastructure. These templates provide explicit, context-specific knowledge, guiding LLMs to generate accurate, standardized, and ontology-aligned metadata while reducing hallucinations and inconsistencies. Our method complements current knowledge-infusion strategies by providing a practical, reusable, and low-overhead strategy for augmenting LLMs with trusted domain knowledge, especially in high-stakes biomedical contexts where precision, reproducibility, and adherence to reporting guidelines and community standards (such as FAIR principles) are paramount.

Our findings further highlight the importance of manual review following automated metadata standardization. To support this work practice, we developed a tool that enables researchers to search using corrected metadata while simultaneously viewing the original, uncorrected metadata side-by-side. This interface helps users to compare both versions and aids informed decision-making about metadata trustworthiness, reducing cognitive load during secondary data analysis. Our software is publicly available [23]. Such a tool is especially beneficial for secondary data users, who often need to assess metadata quality and compatibility across heterogeneous sources without detailed knowledge of the original study context.

Our findings are especially relevant in light of the growing interest in making scientific datasets “AI ready”. Although this term is widely used, it often lacks operational definition. We propose that one core criterion for AI-readiness is the availability of structured, standardized, and machine-readable metadata that can reliably support automated data discovery, integration, and analysis. Scientific data repositories increasingly aim to serve as foundational infrastructure for AI-driven research, yet their metadata remains heterogeneous and inconsistently formatted. Our method offers a clear path toward remediation: By applying template-augmented LLMs to existing metadata records, repositories can enhance the semantic quality of their metadata, thus lowering the barrier to secondary analysis and enabling more robust AI applications. Our results thus offer clear, actionable insights for those seeking to prepare datasets explicitly for secondary AI-driven analyses, directly clarifying what operationalizing “AI readiness” entails. Secondary users of data—especially those conducting secondary analyses, exploratory studies, or hypothesis generation—stand to benefit the most from these improvements. These users often lack direct access to the original data collection context, and must rely entirely on metadata to interpret data correctly and to combine datasets from multiple sources. By automating the standardization of metadata into a form readily interpretable by computational tools, including AI systems, our approach significantly reduces preprocessing burdens and facilitates accurate computational analyses. Our empirical findings demonstrate that structured metadata dramatically improves dataset findability, accessibility, and reusability, offering concrete guidance for researchers and repository managers to ensure datasets meet established standards for data reuse. Thus, our work not only supports human-driven secondary analyses but also strengthens computational frameworks that increasingly underpin advanced AI-driven data exploration.

From a performance perspective, recall emerged as the most improved metric across our evaluations—a finding with direct implications for data discovery. In scientific data discovery, maximizing recall is critical: Relevant datasets may be rare, inconsistently described, or hidden behind poor-quality metadata. Hence, secondary data analysts prioritize retrieving as many datasets as possible. Our results suggest that template-augmented prompt engineering can help to surface many datasets that would otherwise be missed when using legacy search systems. This improvement in findability is not merely technical; it has practical consequences for research reproducibility, comprehensiveness, and equity in access to scientific data.

Looking ahead, our work opens multiple pathways for future development. One promising direction is the integration of our method with RAG systems that can dynamically select the most relevant CEDAR template based on user-submitted metadata or search queries. Such a system would bring us closer to fully adaptive metadata correction pipelines that automatically tailor standardization strategies to the context of use. Additionally, we plan to scale our system to process the entire BioSample database—comprising ~5 million records—which would represent the largest effort to date in automated FAIR-aligned metadata remediation. The enhanced database would not only improve discoverability and reuse for millions of samples but also serve as a valuable resource for benchmarking future metadata curation efforts.

Potential implications

Our results speak to broader conversations in AI, knowledge representation, and scientific infrastructure. As LLMs become more integrated into scientific workflows, ensuring that they interact with structured, trustworthy, and domain-aware knowledge becomes paramount. Our work offers a concrete, reproducible model for doing so—one that is grounded in community standards, validated with empirical metrics, and designed with real-world applications in mind. Crucially, this study empirically confirms a foundational yet previously untested claim of FAIR advocates—that structured metadata significantly enhances dataset findability and reusability. This empirical validation offers advocates of FAIR principles robust, quantifiable proof of the effectiveness of metadata standardization—thus addressing a central critique that FAIR’s impact had previously remained speculative. By demonstrating that FAIR-aligned metadata correction improves machine-mediated data access, we provide a missing link between principle and practice—and set the stage for a new generation of knowledge-aware, FAIR-compliant, AI-ready data ecosystems.

Conclusion

Combining LLMs with structured metadata templates is a transformative approach for improving the FAIRness of scientific data. Our findings demonstrate that domain-informed automation—specifically through CEDAR templates—enables substantial gains in metadata quality, directly enhancing data discoverability, accessibility, and reuse. We quantified these improvements through results in improved recall for biomedical datasets. This work shows that effective metadata standardization requires AI grounded in knowledge of metadata standards.

Supplementary Material

giag019_Reviewer_1_Report_original_submission

Reviewer 1 -- 7/17/2025

giag019_Reviewer_2_Report_orignal_submission

Reviewer 2 -- 8/9/2025

giag019_Reviewer_2_Report_Revision_1

Reviewer 2 -- 11/17/2025

giag019_Reviewer_3_Report_original_submission

Reviewer 3 -- 8/15/2025

giag019_Reviewer_3_Report_Revision_1

Reviewer 3 -- 11/27/2025

Appendix A. Prompts used

LLM Prompt.

Given the following sample from [BioSample/GEO]:

Report a new and corrected metadata. Check if the field values and field names make sense. If no match is found for a field name, match it to an ontology. As far as possible, make field values adhere to ontology restrictions.

The following text is the BioSample data description: [biosample_desc]

Output only the corrected record in Python dict format without comments or suggestions. Have only simple key-value pairs in the dict; no nested or complex values.

CEDAR Prompt Template forMetadata Correction.

Prompt:

Given the following sample from [BioSample/GEO]:

Report a new and corrected metadata sample where the following template is as complete as possible:

biosample_accession, sample_name, sample_title, bioproject_accession, organism, isolate, age, biomaterial_provider, sex, tissue, cell_line, cell_subtype, cell_type, culture_collection, dev_stage, disease, disease_stage, ethnicity, health_state, karyotype, phenotype, population, race, sample_type, treatment, description.

Check if the field values and field names make sense. If no match is found for a field name, match it to an ontology. As far as possible, make field values adhere to ontology restrictions.

• tissue: value should be one of the UBERON ontology concept names (e.g., sac, lung, etc.)

• disease: value should be one of the DOID ontology concept names (e.g., arthritis, etc.)

• cell type: value should be one of the CL ontology concept names

• Missing values: use “NA”

Do not provide any explanations. Output only the corrected record in Python dict format.

Appendix B. Fine-grained Analysis

To better understand the behavior of different LLM-based curation strategies, we conducted a stratified evaluation over three cancer-specific subsets (liver, lung, and ovarian), each queried using the tissue attribute (Fig. B1, B2, B3). Across all datasets, the “Original” baseline exhibits high precision but extremely low recall, confirming that non-augmented metadata retrieval retrieves only a small portion of relevant samples but with high correctness. Models fine-tuned with domain dictionaries (“DD”) or ontology grounding alone show marginal improvements in recall and, in many cases, suffer from precision collapse, indicating that lightweight supervision. Notably, LLaMA-based pipelines that incorporate CEDAR metadata grounding provide a moderate recall boost but still fail to balance precision and recall.

Impact of CEDAR grounding and GPT-4

The only configuration that consistently improves both recall and F1-score across all cancer types is the GPT4 + CEDAR pipeline. For liver, lung, and ovarian datasets, this model variant exhibits a marked increase in recall (0.90–0.95 range for BioSample) while retaining strong precision (–0.95), yielding the highest F1-scores overall. The gain is most pronounced in ovarian cancer, where recall increases more than 10-fold relative to the baseline.

Figure B1.

Lung cancer dataset fine grained analysis.

Figure B2.

Liver cancer dataset fine grained analysis.

Figure B3.

Ovarian cancer dataset fine grained analysis.

Appendix C. Statistical significance analysis

1. Findings for BioSample

We examine the recall performance of the different methods on the BioSample cohort below (Table C1, C2, C3, C4, C5).

Table C1.

Assumption check summary for BioSample.

Assumption	Test/rationale	Result	Conclusion
Normality	Shapiro–Wilk on residuals	W = 0.92,	Passed (no evidence of non-normality)
Sphericity	Var(LLM+CEDAR baseline)		Not clearly satisfied; variance notably larger than other differences
Independence	Distinct sub-cohorts (shared only within)	Same samples only within methods; no overlap between sub-cohorts	Passed for subjects

Table C2.

ANOVA Analysis for BioSample.

Test	df	Statistic	p-value	Conclusion
RM-ANOVA	2, 10		0.0001	Significant effect
Friedman	2		0.0087	Significant effect

Table C3.

Post hoc comparisons between methods (BioSample).

Contrast	Raw p-value	Holm-adjusted p	Significant
Baseline vs. LLM+DD	0.8870	0.8870	No
Baseline vs. LLM+CEDAR	0.0032	0.0070	Yes
LLM+DD vs. LLM+CEDAR	0.0023	0.0070	Yes

Table C4.

Assumption check summary for GEO.

Assumption	Test/rationale	Result	Conclusion
Normality	Shapiro–Wilk on residuals	W = 0.97,	Passed (clearly normal)
Sphericity	Var(LLM+CEDAR baseline)		Reasonably satisfied (no strong violation)
Independence	Distinct sub-cohorts (shared only within)	Same samples only within methods; no overlap between sub-cohorts	Passed for subjects

Table C5.

ANOVA Analysis.

Test	df	Statistic	P-value	Conclusion
RM-ANOVA	2, 10		0.439	No significant method effect
Friedman	2		0.10	No significant method effect

Design

For the BioSample dataset, we analyzed recall using a one-way repeated-measures design, with method (baseline, LLM+DD, LLM+CEDAR) as a within-subject factor and six sub-cohorts as subjects. For each sub-cohort and method, we used the mean recall >800 samples as the dependent variable, yielding 18 observations (6 sub-cohorts 3 methods).

Assumption Check

We fit a repeated-measures style linear model with method and sub-cohort as categorical predictors,

and used the residuals to assess model assumptions. Residual normality was evaluated with the Shapiro–Wilk test and quantile–quantile plots. The Shapiro–Wilk test did not indicate a deviation from normality (W  , ), so the normality assumption was considered acceptable given the small sample size.

For the three-level within-subject factor, sphericity reduces to equality of the variances of two independent difference scores. We computed and for each sub-cohort. The sample variances were and ; an approximate F-test for equality of these variances yielded with .

Within each sub-cohort, the same 800 samples are evaluated under all three methods (baseline, LLM+DD, LLM+CEDAR), so observations are intentionally dependent within subjects, as required for a repeated-measures design. Independence is instead assumed between sub-cohorts: Each of the six sub-cohorts is formed from a distinct set of samples and contributes one mean per method, with no overlap in underlying samples across sub-cohorts. Under this design, the independence assumption for the subject factor (sub-cohort) is considered satisfied.

ANOVA analysis

A repeated-measures ANOVA with method as the within-subject factor showed a strong effect of method on recall:

indicating that at least two methods differ in mean recall in BioSample analysis.

Because sphericity was only borderline satisfied and the number of subjects was small, we additionally ran a non-parametric Friedman test on the table of sub-cohort means. The Friedman test was significant (chi-square , ), confirming that the method has a statistically significant effect on recall without relying on normality or sphericity.

Post-hoc comparisons

The repeated-measures ANOVA and the corresponding Friedman test both indicated a significant overall effect on recall. We then did post hoc pairwise comparisons to determine which methods differed significantly from each other.

We conducted paired post hoc comparisons between methods with Holm adjustment for multiple testing. The comparison between baseline and LLM+DD was not significant (Holm-adjusted ), indicating that LLM+DD does not provide a detectable improvement over baseline. In contrast, LLM+CEDAR showed statistically significantly higher recall than both baseline (Holm-adjusted ) and LLM+DD (Holm-adjusted ). Thus, the significant method effect is driven by a clear performance gain of LLM+CEDAR, whereas LLM+DD performs similarly to the baseline method.

2. Findings for GEO

We examine the recall performance of the different methods on the GEO cohort below.

Design

For GEO, we used the same one-way repeated-measures design as for dataset B, with method (baseline, LLM+DD, and LLM+CEDAR) as a within-subject factor and six sub-cohorts as subjects. For each sub-cohort and method, the dependent variable was the mean recall >800 samples, resulting in 18 observations (6 sub-cohorts 3 methods).

Assumption check

We fit a repeated-measures style linear model with method and sub-cohort as categorical predictors,

and used the residuals to assess model assumptions.

Normality

Residual normality was assessed using the Shapiro–Wilk test and Q–Q plots. The Shapiro–Wilk test indicated no deviation from normality (W  , ), and the Q–Q plot showed residuals close to the reference line. Thus, the normality assumption is clearly satisfied for GEO dataset.

Sphericity

For the three-level within-subject factor, sphericity was examined via difference scores and . The sample variances were and , and an approximate F-test for equality of these variances yielded with . This provides no strong evidence against sphericity; we therefore consider the sphericity assumption to be reasonably met.

Independence

Within each sub-cohort, the same 800 samples are evaluated under all three methods, producing the intended dependence structure for a repeated-measures design. Independence is assumed between sub-cohorts, which are constructed from distinct sets of samples and contribute one mean per method without overlap. Under this design, independence of subjects (sub-cohorts) is considered satisfied.

ANOVA analysis

Despite acceptable assumptions, the repeated-measures ANOVA for dataset G did not show a significant overall effect of method on recall. The RM-ANOVA-style linear model yielded a non-significant global F-statistic (F  , ), indicating that the three methods do not differ reliably in mean recall across the six sub-cohorts.

To obtain an assumption-robust check, we also ran a Friedman test on the table of sub-cohort means. The Friedman test was not significant (chi-square , ), again suggesting that any apparent differences between methods are not statistically reliable at the 0.05 level.

Post-hoc comparisons

Because neither the repeated-measures ANOVA nor the Friedman test showed a significant overall effect of method in dataset G, we did not interpret formal post hoc pairwise comparisons as inferentially meaningful. Although LLM+CEDAR attained higher recall than baseline in some sub-cohorts, the pattern of differences was inconsistent (including many zero values and cases where baseline was higher), and the small number of sub-cohorts (n = 6) limited statistical power. As a result, any pairwise differences in G should be regarded as descriptive rather than statistically significant.

Availability of source code and requirements

• Project name: FAIRMetadataCuration

• Project home page: https://github.com/musen-lab/FAIRMetadataCuration

• Operating system(s): Platform independent

• Programming language: Python

• License : MIT License

• Package Management : Python 3.10+ with Jupyter, pandas, numpy, scikit-learn, openai

• Hardware requirement : Standard workstation with 16GB RAM and multi-core CPU; GPU optional for local LLM inference.

Funding

This work was supported in part by grant R01 LM013498 from the U.S. National Library of Medicine.

Data availability

The data supporting the results of this article are available in the Zenodo repository [24].

Competing interests

The authors declare that they have no competing interests.