OmniExtract: an automatic data extraction tool based on large language model and prompt engineering

Abstract

Extracting structured information from documents or scientific papers is crucial for data sharing and retrieval. Recent advances in large language models (LLMs) have demonstrated strong capabilities in language understanding, and a number of LLM-based tools have been developed for extraction-oriented tasks. However, it’s still difficult to find a universal and user-friendly tool for various practical extraction tasks. To address this challenge, we propose OmniExtract, an automatic data extraction tool with user-friendly configuration files that can adapt to various data extraction tasks. OmniExtract employs a prompt optimization method to refine task-specific prompts and achieve high extraction performance. It also supports comprehensive data extraction from both documents and tables, making it applicable to a broad range of data sources. Evaluation results show that OmniExtract obtains a high accuracy ~90% for three datasets. Furthermore, two additional data extraction applications of OmniExtract in real-world scenarios have been presented, achieving an accuracy of 92.21% and ~90% precision and recall, respectively. Specifically, OmniExtract can handle tabular files of various sizes and formats, and achieve over 99% precision and recall on table information extraction tasks. The data reliability performance shows that OmniExtract is a valuable tool for database updating. An online testing service is available at https://ngdc.cncb.ac.cn/omniextract/. The service can be deployed locally with the code in https://github.com/wyb39/OmniExtract.

Introduction

With the advancement of scientific research, a large amount of research results are published as papers or documents, which form the foundation of building domain-specific knowledgebases such as epigenome-wide association study (EWAS) for epigenome research [1], Community-curated human lncRNA database (LncRNAWiki) for long non-coding RNA research [2], and Animal biomarker knowledgebase (BioKA) for biomarker research [3]. However, most of them depend on manual curation, making the knowledge mining process expensive and inefficient, and facing the problem of continuous maintaining. Therefore, using efficient tools to improve the efficiency of data mining has become an urgent demand.

Several methods, such as a semi-automated free-text data system (DECaF) [4] and BERT for materials science curation (MaterialsBERT) [5], have been developed for automatically extracting simple tasks from documents such as Named Entity Recognition. These methods rely on hundreds to thousands of high-quality annotated data and are difficult to generalize to other information extraction tasks, which limits their applicability. The large language models (LLMs) such as ChatGPT [6] show impressive ability in text understanding, and recently several LLM-based extraction methods have been developed and can be broadly categorized into three types [7]: fine-tuned model-based, Retrieval-Augmented Generation (RAG)-based, and prompt engineering-based methods. Fine-tuned approaches, such as an automated framework for information extraction from scientific literature (autoIE) [8], utilize task-specific fine-tuning to extract structured information from large-scale literature collections on molecular sieve synthesis. RAG-based methods improve extraction by incorporating external retrieval, where An information extraction framework based on two RAG methods (HybridRAG) [9] combines vector similarity based RAG (VectorRAG) and subgraph retrieval based GraphRAG (RAG), and a framework for hybrid long documents extraction (AIE) [10] applies document segmentation with RAG retrieval for long-document extraction. Prompt engineering-based methods optimize extraction through carefully designed instructions or workflows. For example, LLM-IE [11] employs template-based prompt optimization, ChatExtract [12] develops a prompt-driven extraction pipeline for materials data (and constructs a materials database), MedPromptExtract [13] integrates prompt engineering with human verification, and a trainable retrieval framework for information extraction (RUIE) [14] trains an LLM to select preferred demonstrations for prompt optimization. Compared with the method based on RAG and fine-tuned model, prompt engineering does not rely on a large amount of high-quality annotated data and can switch models used for information extraction, making it flexibly applicable to various information extraction tasks in different scenarios. However, some limitations still exist. Firstly, most of these methods are focused on specific extracted task and the code or implementation of most methods cannot be obtained, which is difficult to transfer to other tasks. Secondly, open-source methods such as LLM-IE are dedicated to providing a package for building a module that extracts specific information from a given text, and cannot be directly applied to extract target information from raw documents. Furthermore, these methods mainly focus on extracting text information and have limitations to handle extracting tables from supplementary materials. Therefore, there is still a gap in reality application for automatic information extraction that are capable of extracting multiple data types and can be simply transferred with user-friendly configuration.

Here, we present OmniExtract, a universal and automatic tool used for data extraction. OmniExtract is developed based on prompt engineering and LLMs, which has these advanced features: (i) it provides a user-friendly way to extract target information from the original documents by using a web interface or by using the command line and modifying configuration files; (ii) it enables the use of existing curation data (a dataset comprising multiple original documents paired with manually curated structured extraction results) to modify the prompt and get an optimized prompt to extract information, which allows it to adapt to various curation requirements; (iii) it can extract information from text and tables, which allows it to fulfill most of the reality application requirements. OmniExtract is evaluated on three standard corpora using eight various LLMs, yielding a high accuracy of ~90%. Further evaluation on four information tasks across two real scenarios resulted in 92.21% accuracy and ~90% precision/recall. Specifically, OmniExtract can handle tabular files of various sizes and formats, and achieve over 99% precision and recall on table information extraction tasks. These features, as well as the performance, will make OmniExtract as a valuable tool for data extraction which can help to improve knowledge update efficiency.

Results

Overall performance evaluation

We first tested the extraction performance of OmniExtract using different models on the WikiReading Recycled dataset [15] and the Yidu-S4K (https://tianchi.aliyun.com/dataset/144419) dataset. After prompt optimization, the models achieved extraction accuracy rates of 88.6%–93.4% on the WikiReading Recycled dataset and 67%–89% on the Yidu-S4K dataset (Table 1). Most models with a scale exceeding 14B achieved extraction accuracy exceeding 90% for the WikiReading Recycled dataset and 80% for the Yidu-S4K dataset, demonstrating the reliability of the extraction tool (considering the subjective nature inherent to information extraction tasks).

Table 1.

Performance evaluation using WikiReading Recycled and Yidu-s4k dataset.

	WikiReading Recycled accuracy (improvement)	Yidu-s4k accuracy (improvement)
Qwen2.5-7B-Instruct-1 M	88.58% (0.75%↑)	70.33% (38.00%↑)
Meta-Llama-3.1-8B-Instruct	91.50% (3.92%↑)	67.00% (7.00%↑)
Qwen2.5-14B-Instruct-1 M	91.08% (0.92%↑)	83.33% (12.00%↑)
Mistral-Small-24B-Instruct-2501	92.42% (−)	86.67% (3.67%↑)
Qwen2.5-32B-Instruct	91.75% (0.83%↑)	82.67% (0.33%↓)
Llama-3.3-70B-Instruct	92.50% (1.50%↑)	82.33% (5.33%↑)
Qwen2.5-72B-Instruct	93.42% (−)	89.00% (6.33%↑)
Mistral-Large-Instruct-2407 (130B)	93.08% (0.33%↑)	83.00% (1.67%↑)

↑ indicates the accuracy improvement achieved by using the optimized prompt compared to the unoptimized prompt; ↓ indicates the accuracy reduction after using the optimized prompt. Bold values indicate the best result (performance or improvement) in cross‑model comparisons.

With the exception of the two models below 10B (Qwen2.5-7B-Instruct-1M and Meta-Llama-3.1-8B-Instruct), the extraction performance of the other models was generally similar. However, the highest accuracy rates on both tasks were achieved by the Qwen2.5-72B-Instruct model. Furthermore, the extraction performance of the four different Qwen models showed an increasing trend with the parameter count, suggesting that larger-scale models can achieve higher extraction accuracy ceilings, which is consistent with the understanding that model capabilities exhibit logarithmic growth with increasing parameters.

We further analyzed the magnitude of accuracy improvement achieved by the models after prompt optimization. After prompt optimization, eight models achieved an average accuracy improvement of 9.13% on the Yidu-S4k dataset and 1.03% on the WikiReading Recycled dataset, despite the latter’s pre-optimization accuracy being close to 90%. Three models exceeding 70B showed an average improvement of 0.61% on the WikiReading Recycled dataset and 4.44% on the Yidu-S4K dataset. In contrast, four models below 24B exhibited larger gains, with average improvements of 1.40% on the WikiReading Recycled dataset and 19% on the Yidu-S4K dataset. Nonetheless, even for models larger than 70B, prompt optimization could still yield improvements exceeding 6% points for the Yidu-S4K dataset and 1.5% for the WikiReading Recycled dataset, which validates the efficacy of the prompt optimization module.

Next, we used the CriticalCoolingRates dataset [12] to test the models’ effectiveness in extracting multiple materials and their corresponding critical cooling rate values (with different units) from raw literature (Table 2). After prompt optimization, most models performed poorly, achieving extraction precision below 60%. However, the Qwen2.5-72B-Instruct model achieved 88.89% precision and 77.19% recall, with an entity-level precision of 84.85% and a recall of 73.68% (Supplementary Table S1). Considering the challenging nature of this task (original reports indicate both precision and recall were below 80% [12]—though not comparable, this demonstrates the task’s difficulty), these results demonstrate that our tool can achieve acceptable multi-entity extraction performance.

Table 2.

Performance evaluation using CriticalCoolingRates dataset.

	Precision (%)	Recall (%)
Qwen2.5-7B-Instruct-1 M	30.63	63.51
Meta-Llama-3.1-8B-Instruct	15.10	65.26
Qwen2.5-14B-Instruct-1 M	56.53	67.54
Mistral-Small-24B-Instruct-2501a	–	–
Qwen2.5-32B-Instruct	47.19	72.28
Llama-3.3-70B-Instruct	49.94	74.39
Qwen2.5-72B-Instruct	88.89	77.19
Mistral-Large-Instruct-2407 (130B)	52.22	64.04

^aMistral-Small-24B-Instruct-2501 does not have enough context length to handle the whole literature content. Bold values denote the best performance among all compared models.

Furthermore, to evaluate the robustness and stability of extraction performance, we used three random seeds to independently sample subsets from the CriticalCoolingRates dataset and performed prompt optimization and information extraction for each of the seven models mentioned in Table 2. As shown in Supplementary Fig. S1, Qwen2.5-7B-Instruct-1M and Meta-Llama-3.1-8B-Instruct showed relatively poor extraction performance, whereas Qwen2.5-72B-Instruct achieved the best overall results, with no substantial differences observed among the remaining models.

Performance variations across random seeds were observed, likely due to differences in dataset sampling and prompt optimization outcomes. For Qwen2.5-72B-Instruct, the prompt optimization scores were 86.13, 82.41, and 84.36 for seeds 1, 2, and 3, respectively, which were consistent with the ranking of subsequent extraction performance. These results indicate that the quality and typicality of curated data used for prompt optimization may influence the stability of information extraction. Across four independent evaluations, Qwen2.5-72B-Instruct achieved an overall precision of 85.69% ± 2.86% and a recall of 77.38% ± 4.22%.

Ablation study of prompt optimization strategies

To better align extraction tools with implicit criteria of human curation for structurally complex data, we modified the miproV2 method by incorporating human-extracted examples. To evaluate the effectiveness of this modification, the prompt optimization-only method (PromptOptim-0shot), the standard miproV2 method (miproV2-1shot), and the modified method (OmniExtract-1shot) were evaluated on the CriticalCoolingRates dataset using the same random seeds and the Qwen2.5-72B-Instruct model described above. As shown in Table 3, OmniExtract-1shot consistently achieved higher precision and recall than the other two approaches across all independent tests. These results indicate that incorporating human-extracted examples into the optimization process can provide additional guidance for prompt optimization and improve information extraction performance.

Table 3.

Ablation study of three prompt optimization strategies.

	Seed 1 (%/%)	Seed 2 (%/%)	Seed 3 (%/%)
PromptOptim-0shot	75.29/77.04	78.98/71.00	80.91/72.44
miproV2-1shot	75.84/72.89	68.15/71.84	78.71/76.53
OmniExtract-1shot	87.16/82.75	82.58/72.44	84.12/77.14

The values in the table are presented as precision/recall. The 1-shot setting uses a prompt consisting of an extraction instruction and one corresponding information extraction example, whereas the 0-shot setting optimizes the extraction instruction alone without providing any examples. Bold values denote the best performance among the three methods.

Comparison with other tools

We compared OmniExtract with existing methods, whose key characteristics are summarized in Table 4. These tools are designed for different application scenarios, and most accept only text-formatted input, highlighting the advantages of OmniExtract in processing scientific PDF documents and tabular data. Moreover, several of these tools do not provide open-source code, limiting their applicability to customized information extraction tasks. Among the three tools with publicly available source code—ChatExtract, LLM-IE, and RUIE—LLM-IE is the most comparable to OmniExtract because it is designed for general-purpose applications and, unlike RUIE, does not require the training of a reward model. We therefore evaluated OmniExtract and LLM-IE on the aforementioned CriticalCoolingRates dataset using the same Qwen2.5-72B-Instruct model. We started from the same initial prompt, optimized the prompt using LLM-IE’s “Prompt Editor,” and conducted information extraction through the web-interface service provided by the LLM-IE deployment. The results are presented in Supplementary Fig. S2.

Table 4.

LLM-based information extraction tool comparison.

Tool	Input type	Extraction task	Auto prompt optimization	Needs training	Open source
AIE	Text	Hybrid long documents (usually financial report)	No	No	No
ChatExtract	Text	Materials data	No	No	Yes
LLM-IE	Text	General task	Yes	No	Yes
MedPromptExtract	Text	Discharge summary	No	No	No
RUIE	Text	General task	No	Yes	Yes
OmniExtract	pdf|xml|tex|text|xlsx|csv|tsv	General task (especially for scientific files)	Yes	No	Yes

Across the three independent runs, the recall values showed no substantial differences; however, the precision of LLM-IE was ~20% lower than that of OmniExtract. These discrepancies may arise from dataset constraints during prompt optimization, where curated data help select more effective prompts and ensure more reliable optimization. Therefore, the current results suggest that, when previously curated data are available for prompt optimization, OmniExtract can achieve more reliable extraction performance.

Application for the automatic extraction of multiple traits in dog breed standardization

Dogs are among the earliest domesticated animals and exhibit diverse phenotypes. Currently, there are ˃400 dog breeds worldwide [16], and 283 breeds of them have been registered in the American Kennel Club (AKC) with detailed standards for morphologic traits such as body weight, height, tail length, etc. These morphological traits have been selected through long-term artificial breeding, showing significant differences among different breeds and having a strong correlation with the genetic background of specific breeds. Therefore, these traits are valuable data resources for canine genetics research. To investigate the genetics of complex traits for dogs, genome-wide association study (GWAS) is used to research the association with genotype and trait using standard trait value, and several candidate genes have been found [17]. However, since the traits have been provided in the breed registered documents, it will be inconvenient to retrieve the value of multiple traits for each breed. Thus, our goal is to automatically extract phenotypic information from breed standard documents and standardize some important phenotypes, in order to construct a phenotypic dataset convenient for canine GWAS research. OmniExtract is used to automatically extract traits from dog breed standardization files, and the whole extracted traits can be downloaded from iDog 2.0 [18].

To optimize the prompt for extracting traits, a comprehensive dataset containing 39 distinct traits across 62 breeds was manually curated. A simple initial prompt for extracting traits has been constructed, after the prompt engineering tuning, an optimized prompt has been obtained as Fig. 1. The used LLM Qwen2.5-72B-Instruct was configured, OmniExtract was applied to the breed standard files’ extraction. In total, 100 breed standard files were selected for evaluating extraction accuracy. A total of 3900 trait values covering full-body morphological features from head to tail (Fig. 2A) were manually checked, resulting in an extraction accuracy rate of 89.62%.

Figure 1.

The initial prompt and optimized prompt used to extract trait values from breed standardization files.

Figure 2.

Trait distribution extracted from dog breed standard documents and error distribution before and after correction; (A) number and proportion of various trait categories; (B) number of errors in the top five traits before and after correction, and the proportion of error reduction after correction; (C) error distribution before and after correction.

Further, we analyzed the causes of extraction errors. The top five traits with the highest error rates were Head Width, Head Length, Size, Cheek Shape, and Weight (Fig. 2B). The reasons for errors in these five traits are respectively:

1. Semantic confusion: For Head Width and Head Length, the erroneous extraction results often describe the length and width of the skull; for Cheek Shape, the erroneous results often describe the Muzzle (meaning the animal’s snout or nose). We observed that in certain breed standard documents (e.g. Golden Retriever), descriptions labeled as “Head” actually pertain to skull characteristics. Similarly, some documents’ “Skull” sections substantively describe head features. Furthermore, within breed standards, Head and Skull, as well as Cheek and Muzzle, frequently appear in close proximity within the same paragraph. This semantic and positional ambiguity likely contributes to the occurrence of such extraction errors.

2. Failure to meet formatting requirements: For Body Size, we expect outputs to be descriptive keywords such as small, medium, large, or combinations thereof. However, as these keywords are absent in certain breed documents, the model often extracts descriptions of height or weight in an effort to capture comprehensive information. This violates our formatting rules.

3. Generation without consulting the document: Many documents do not contain Weight information for specific dog breeds. However, the model may fabricate weight data based on knowledge from unknown sources. Some of these fabricated weights coincidentally matched data from authoritative sources like AKC.

To verify the above analysis and further improve the extraction accuracy, we rewrote and optimized the prompts for these five traits and extracted the values of them. Compared with the previous results, the error rate of this extraction has decreased by 33.33%–79.17% (Fig. 2B). Combining the results of the two extraction processes, the extraction accuracy has risen to 92.21% (Fig. 2C).

These results indicate that the initial prompt for the information extraction task requires more refined curating standard descriptions. This need arises because data-driven optimization processes struggle to align with implicit human intent—a likely reason why it is difficult to improve the accuracy rate to above 90% or 95% when using simple initial prompts.

Application for comprehensive information automatic extraction in epigenome-wide association study

EWAS has become a powerful approach to identify epigenetic variations associated with biological traits [19]. To provide a comprehensive collection of high quality EWAS associations in support of systematic investigations of complex molecular mechanisms associated with different biological traits, a manually curated knowledgebase EWAS atlas has been constructed and published in 2018 [1]. For ˃6 years, EWAS atlas has consistently maintained and updated the database by manually curated information which is time-consuming and hard-working. To improve the data update efficiency, OmniExtract was used to automatically extract comprehensive information including trait, cohort, and EWAS association information which involves text and table extraction from main text and supplemental materials.

We first apply the OmniExtract tool to the task of extracting trait names and trait types from the abstracts of EWAS-related literature. An EWAS study may involve multiple traits, and according to the requirements of the EWAS Atlas database, these traits need to be classified into five types: cancer, non-cancer disease, phenotype, environmental factor, and behavior. In addition, some traits that are not the target of EWAS research need to be accurately filtered out, which further increases the difficulty of information extraction. We utilized the optimized prompt to assist the extraction task and manually checked the extraction results for 100 articles. This process yielded a precision of 91.03% and a recall of 93.62%. The majority of extraction errors stemmed from either incorrectly extracting non-target traits or failing to extract target traits, as exemplified in Fig. 3.

Figure 3.

Error cases for the target traits extraction.

Further, we utilized the OmniExtract tool to extract cohort metadata (sample size, age range, male proportion, platform, cohort name, ancestry, etc.) from EWAS full-text literature. Considering the complexity of population cohort information and the variable number of cohorts that may be involved in a single EWAS study, this constitutes an exceptionally challenging task. After manual verification, the OmniExtract tool ultimately achieved a precision of 92.19% and a recall of 88.21%. We analyzed the causes of the lower recall rate and identified that one article contained a significantly higher-than-average number of cohorts (17 cohorts for this article, with 6.3 z-score). These cohorts were erroneously omitted during extraction, accounting for 23.44% of all missed cohorts (Fig. 4, pink column). Such corner cases constitute the primary factor for recall rate degradation in similar tasks, while the decline in precision is predominantly attributable to factual errors stemming from the high complexity of information extraction (Fig. 5).

Figure 4.

Distribution of article numbers about error attribute counts of the trait and cohort extraction results.

Figure 5.

Error cases of mistakenly extracted cohort.

The extraction of EWAS association data was completed using OmniExtract’s table processing module. We processed 38 tabular files (xlsx/xls/csv/tsv) containing 116 tables from supplementary materials across 20 publications. Manual verification confirmed that 88.79% of the 116 tables could be classified and extracted correctly (available at https://ngdc.cncb.ac.cn/omniextract/, ewas_table_extract_result.xlsx). Extraction of every attribute from the table reached 99.47% precision and 99.57% recall. The primary cause of errors stems from the limited information provided by table field names, making it challenging to map them to the intended extraction targets in certain scenarios.

The process of extracting EWAS literature information is now available as a Web service (https://ngdc.cncb.ac.cn/ewascuration/). Users can upload the original PDF article and supplementary tables to obtain the extracted data.

Besides, we have developed an online service to facilitate users in completing testing and small-scale information extraction tasks. Users can upload files and provide minimal configuration information to perform information extraction. The service also includes features for exporting and loading configuration information to enhance user convenience. The service is available at: https://ngdc.cncb.ac.cn/omniextract/. We also provide the local deployment code for this service at https://github.com/wyb39/OmniExtract, where users can deploy the service locally and configure different models to perform information extraction tasks.

Discussion

OmniExtract provides a one-step solution for extracting multi-property entities from original documents and tables, encompassing all the functions of file parsing, prompt optimization, and information extraction. OmniExtract has two features: (a) user-friendly: users can utilize the corresponding functions by using a graphical interface or by using the command line and modifying predefined a human-readable data format (YAML) file templates. (b) Stable performance: using mainstream closed-source or open-source models, the extraction accuracy or precision/recall of multi-property entity extraction tasks can reach ~90%, and can increase by a few percentage points after targeted adjustments. Specifically, our tool can handle tabular files of various sizes and formats, and achieves over 99% precision and recall on table information extraction tasks.

More thorough detection and correction of extraction errors are important for improving data quality. Besides the sporadic errors that may arise during document parsing, extraction errors can be broadly categorized into three types: (i) hallucinated outputs generated by the model, (ii) entity-attribute mismatches during the extraction process, and (iii) errors and omissions caused by inconsistencies between model outputs and human curation criteria.

To mitigate model hallucinations, the original evidence corresponding to each extracted attribute can be additionally retrieved after extraction, reducing the impact of unsupported model-generated information. For entity-attribute mismatches, the locations of the extracted information in the original document can be leveraged together with model-based verification to identify and filter such errors. For discrepancies between model outputs and human-defined criteria, in addition to switching to more capable models, the extraction performance of the tool can still be enhanced by better aligning with users’ implicit criteria. The following are some ideas for improving extraction performance, which will be implemented in future versions:

1. Collaborating with LLM as a corrector: As demonstrated in the extraction of cohort information from EWAS literature, the main factor contributing to decreased recall is information loss when handling atypical tasks, while the reason for decreased precision is some factual errors. Therefore, after information extraction, we can use capable models to complete verification and correction for these two situations. Adopting this strategy of small model extraction followed by large model correction can ensure extraction efficiency while improving extraction accuracy.

2. Implementation of context refinement: In the current version, text extracted from raw documents is just roughly chunked before being directly fed into the model for extraction. This approach not only incurs higher computational costs but also allows irrelevant or incorrectly formatted context to adversely affect extraction effectiveness. Therefore, in future releases, we plan to integrate context refinement methods into the workflow to minimize the impact of irrelevant information on model extraction. We plan to enhance the document parsing module by introducing semantic similarity computation or integrating an independent small model to filter out irrelevant content and incorrect formatting. Furthermore, we will add emphasis markers to the relevant content to improve the effectiveness of information extraction.

Additionally, our table extraction feature currently focuses primarily on extracting information from table files, and has not optimized the workflow for extracting table information from PDF files. In future versions, we plan to integrate parsing tools with stronger PDF table parsing capabilities and optimize the table extraction process from PDF files to achieve better extraction results.

Methods

System design

Our goal is to build an information extraction tool that is (a) functionally complete: this tool needs to be able to conveniently extract information from original files without introducing additional tools or modules, (b) user-friendly: users only need to modify the configuration file to use the tool for information extraction, and (c) stable in performance: the tool can achieve good performance when completing different extraction tasks. To achieve the aforementioned goals, we have constructed the information extraction tool named OmniExtract (Fig. 6) using DSPy [20], which is a declarative, modular framework for building artificial intelligence applications that offers efficient prompt optimization functionalities. OmniExtract seamlessly integrates Marker (https://github.com/datalab-to/marker), a document parsing tool, to provide robust support for processing raw PDF documents. OmniExtract comprises four modules: (i) raw document processing, (ii) prompt optimization based on existing data, (iii) parallel information extraction and assessment, and (iv) table extraction workflow. We provide highly encapsulated APIs for each module, enabling the complete process from raw documents to information extraction through simple configuration file modifications.

Figure 6.

System design of OmniExtract.

Raw document processing

For general literature documents (typically in PDF format), we have integrated the Marker document parsing tool, which can quickly and accurately parse documents into LLM-friendly markdown format for subsequent information extraction. Additionally, we have developed a rule-based parsing pipeline for XML format files provided by PubMed Central (PMC) and ScienceDirect, as well as TeX format files from arXiv, which can parse the main content of documents into Markdown format. Furthermore, we have provided a text segmentation method. This method matches section headings within the literature using keyword matching, dividing the document into sections such as the Title, Abstract, Introduction, Methods, and Results. The segmented documents are saved in JavaScript Object Notation, a lightweight data format for text (JSON format). This segmentation provides more precisely scoped context for subsequent information extraction.

For table information extraction, we have also implemented rule-based parsing pipelines. These pipelines clean and convert tabular files (xlsx, xls, csv, and tsv) by removing blank rows and separating table titles. The table content is stored in TSV format, while the table titles are saved to a metadata file. Regarding literature sourced from PMC and ScienceDirect, we have also customized parsing methods to extract the table into a TSV file.

Prompt optimization based on existing data

To improve the performance of information extraction, we have also encapsulated the prompt optimization workflow. We have modified the prompt optimization method miproV2 [21] offered by DSPy to better adapt to information extraction tasks. We have also provided both rule-based and predefined AI evaluation methods to assess the performance of selected prompts during prompt optimization. Based on the provided curated data, users can automatically complete the prompt optimization process by supplying a configuration similar to the information extraction workflow and selecting an appropriate evaluation method.

DSPy offers several prompt optimization methods. Among them, miproV2 [21] can generate few-shot examples and candidate prompts, and select the optimal combination via Bayesian search. This allows it to improve the extraction accuracy by properly utilizing the curated information. However, information extraction tasks tend to directly extract information from given text without requiring a long processing chain. Models can often generate roughly correct answers without needing rejection sampling. Moreover, the extracted information often has implicit format and content requirements, which can be reflected in curated results. Therefore, we modified the miproV2 method and used the curated results instead of few-shot examples generated by LLM for the downstream prompt generation and composition optimization steps.

Prompt optimization methods require providing metrics to evaluate generated results, in order to select the highest-scoring prompts based on existing data. We provide two methods to evaluate the effectiveness of prompt-example combinations:

1. Rule-based method: We establish a set of rules to evaluate the degree of match between extracted results and reference results. For string-type attributes, building upon the evaluation methods provided by DSPy, we additionally employ the Smith–Waterman algorithm [22], Ratcliff–Obershelp algorithm, Jaccard index [23], and Longest Common Subsequence (LCS) algorithm, taking the highest score as the string similarity metric. For other non-list/non-string attributes, we require the extracted result to exactly match the reference answer. For list-type attributes, we calculate the Jaccard similarity as the score.

2. Predefined AI evaluation method: We use an LLM to score the degree of match between the extracted results and the reference results.

Users can utilize the OmniExtract tool to process curated data and the matched original documents, building datasets for prompt optimization. Based on the modified MiproV2 method, the instruction-instance combination could be optimized to enhance the effectiveness of information extraction.

Parallel information extraction and assessment

We have encapsulated a parallel information extraction workflow based on the declarative features of the DSPy framework. This approach transforms the coding effort for building the workflow into configuring input content and the names, types, and descriptions of target information to be extracted. Combined with predefined or optimized prompts, this generates modules required for information extraction. We leverage DSPy’s built-in ParallelExecutor class to process extraction tasks concurrently. Furthermore, adhering to the “LLM as a judge” [24] paradigm, we predefine a scoring component to evaluate extraction results. Users can employ a different model (or the same model used for extraction) to assign confidence scores to the results, enabling efficient filtering of low-quality outcomes.

Table extraction workflow

Extracting information from tables differs from extracting from ordinary text: tables are more structured and have higher information density compared to ordinary text, typically yielding more extracted information. Considering these characteristics, we have designed a dedicated workflow for table extraction. After parsing the table, we transform the original table into a structured table containing extracted information through four steps: (a) table classification—determine whether the table contains target information based on LLM, (b) example table generation—extract target information from the first 10 rows of the table, store it as an example file, (c) agent with reasoning–action–observation cycles (ReAct agent) for table transformation—use a ReAct agent to complete the task of generating a Python script and converting tables into the sample table format, and (d) convert the table into the specified format containing target information—run the script generated in step three and check the result. Users need to specify the prompt words for table classification and example table generation in the configuration file to complete this workflow. Similarly, the extraction performance of this process can be enhanced through the prompt optimization module.

Datasets and evaluation metrics

Three public datasets were used to evaluate the information extraction performance of the tool:

1. WikiReading Recycled dataset [15]: This is a dataset designed for multi-property extraction tasks built from the Wikipedia information. We extracted information for 200 professional athletes from this dataset (including 12 attributes such as name, gender, country, and date of birth) for evaluation. This dataset is used to evaluate the performance of extracting multiple attributes of one entity from a short document.

2. Yidu-S4K dataset (https://tianchi.aliyun.com/dataset/144419): This is a Chinese electronic medical record dataset that includes tasks for medical entity and attribute extraction. We extracted 200 records from the training set of this task for evaluation. This dataset is used to evaluate the effectiveness of multi-property extraction for Chinese.

3. The CriticalCoolingRates dataset [12]: This dataset comprises manually curated critical cooling rate data for materials extracted from the materials science literature, we randomly sampled records from 200 articles from the “MANUAL_raw” section, which contains materials’ names and their corresponding critical cooling rates. This dataset is used to evaluate the performance of extracting multiple entities from original documents.

For the first two datasets, we extract multiple attributes of individual entities from each piece of data and calculate the accuracy rate of the extracted attributes. For the third dataset, we adopt a “two-step” strategy: first parsing the PDF documents of the literature and extracting key paragraphs containing the material’s critical cooling rate, then combining the full text with these key paragraphs to extract the material names and critical cooling rates. We manually verified the extraction results by benchmarking against the dataset.

Furthermore, we demonstrate the application of our information extraction tool in two real-world scenarios:

1. Extracting multiple trait information from dog breed standard documents from the AKC. Relevant data is also used as a demonstration data and is provided together with source code.

2. Extracting trait and cohort information from EWAS literature, and capturing EWAS association data from table files.

To evaluate OmniExtract across diverse information extraction tasks—specifically, extracting entities with multiple attributes from raw documents or tables—we primarily conduct attribute-level evaluation tailored to each task setting (we match extracted entities to ground-truth entities one-to-one based on maximum similarity, and compute evaluation metrics at the attribute level). For tasks that require extracting a single entity from a document, we report accuracy. For tasks that involve extracting an indeterminate number of entities from a document, we report precision and recall.

In addition, for tasks with an indeterminate number of entities, we further perform an entity-level evaluation, where an extracted entity is considered correct only if all of its attributes strictly match the ground truth. The evaluation results for the CriticalCoolingRates dataset and the EWAS information extraction tasks are provided in the supplementary materials (Supplementary Tables S1 and S2).

Models used for evaluation

We selected eight open-source models from three mainstream model series (Llama, Qwen, and Mistral) to evaluate tool information extraction capabilities across different model scales: Meta-Llama-3.1-8B-Instruct [25], Llama-3.3-70B-Instruct [25], Qwen2.5-7B-Instruct-1M [26], Qwen2.5-14B-Instruct-1M [26], Qwen2.5-32B-Instruct [27], Qwen2.5-72B-Instruct [27], Mistral-Small-24B-Instruct-2501 (https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501), and Mistral-Large-Instruct-2407 (https://huggingface.co/mistralai/Mistral-Large-Instruct-2407). Using vLLM [28] (version 0.6.6), we deployed the models as a standalone service on a 4 × A100 machine to provide interfaces for the extraction tool usage.

For the two application cases we provided, we used different models for processing: Qwen2.5-72B-Instruct was applied to trait information extraction of dog breeds. When extracting EWAS-related information, Qwen-Max was used for information extraction from documents. Considering the support for agent capabilities, Qwen3-235B-A22B-Thinking-2507 was used for table extraction.

Key Points

• Zero-code design: An intuitive web user interface eliminates programming requirements; configurations for information extraction are reusable.

• Universal adaptability: Zero- and few-shot learning enable seamless adaptation to diverse extraction scenarios. Native support for scientific documents including PDF (Portable Document Format), XML (a machine-readable format), and LaTeX (a text-based source format), and tabular data including CSV (comma-separated tables), TSV (tab-separated tables), and XLSX (Excel spreadsheets), covers both main text and supplementary materials.

• Robustness in real-world scenarios: The tool’s performance has been proven across a wide range of scientific information extraction tasks, ensuring reliable application in diverse research scenarios.

Author contributions

Yibo Wang (Methodology, Software, Data curation, Writing—original draft preparation, Writing—review and editing), Bixia Tang (Data curation, Writing—original draft preparation, Writing—review and editing, Funding acquisition), Sicheng Wu (Software, Data curation), Shaoqi Bei (Data curation), Xiaojing Gao (Data curation), Yueyue Wu (Data curation), Shuying Liu (Data curation), Huangxia Hu (Data curation), Yuyan Meng (Data curation), Demian Kong (Data curation), and Wenming Zhao (Conceptualization, Writing—review and editing, Funding acquisition).

Conflicts of interest

The authors declare no competing interests.

Data availability

The datasets and list of EWAS related publications for performance evaluation are available at https://ngdc.cncb.ac.cn/omniextract/.

Funding

None declared.

Ethics statement

Not applicable.

Code availability

The source code for OmniExtract is available on Github https://github.com/wyb39/OmniExtract.