A Multidisciplinary Multimodal Aligned Dataset for Academic Data Processing

Abstract

Academic data processing is crucial in scientometrics and bibliometrics, such as research trending analysis and citation recommendation. Existing datasets in this domain have predominantly concentrated on textual data, overlooking the importance of visual elements. To bridge this gap, we introduce a multidisciplinary multimodal aligned dataset (MMAD) specifically designed for academic data processing. This dataset encompasses over 1.1 million peer-reviewed scholarly articles, enhanced with metadata and visuals that are aligned with the text. We assess the representativeness of MMAD by comparing its country/region distribution against benchmarks from SCImago. Furthermore, we propose an innovative quality validation method for MMAD, leveraging Language Model-based techniques. Utilizing carefully crafted prompts, this approach enhances multimodal processing capabilities to evaluate the accuracy of text-to-visual alignments. We also outline prospective applications for MMAD, providing the way for novel research endeavors, including automated caption generation and analysis of trends in figures. Thus, this work signals new research prospects and provides a fertile ground for advances in academic data processing.

Background & Summary

In the contemporary academic landscape, where publishing research serves as a principal avenue for dialogue and recognition, Academic Data Processing (ADP) has emerged as a cornerstone in dissecting research trajectories and mining insights across various disciplines. Notably, its significance is underscored in scientometrics^1–3 and bibliometrics^4–7, fields dedicated to the quantitative analysis of scientific literature. These areas have burgeoned, diversifying into numerous subdisciplines and encompassing a broad spectrum of research focuses and methodologies. This diversification caters to the multifaceted needs and challenges encountered in academia, ranging from recommendation systems^8,9 to summarization^10–13 and content generation tasks^14,15. Despite this expansive growth, the predominant focus has been on the textual facets of academic content, leaving a gap in understanding and utilizing the rich information embedded in visual elements like figures and tables. Text, albeit informative, often falls short of conveying the details captured in these visuals. Recognizing this, there’s a pressing need for methodologies that adeptly align visual data with their textual descriptions, unlocking new possibilities in figure trend analysis, figure classification, and automated caption generation. This approach not only enhances the comprehensiveness of academic data processing but also paves the way for more informed research insights.

Until recently, the incorporation of image data in Academic Data Processing (ADP) datasets was notably sparse, a trend that persisted until before 2020. However, a shift has been observed with some researchers pioneering the extraction and integration of visual content from academic articles for ADP applications. Notable initiatives such as SCICAP¹⁶, ACL-FIG¹⁷ and Uni-SMART¹⁸ have begun to leverage visual data, demonstrating their potential to address complex challenges uniquely resolvable beyond textual analysis. Despite these advancements, the scope of datasets like SCICAP focused on computer science, and ACL-FIG, aimed at computational linguistics and natural language processing, remains relatively narrow. This specificity, while beneficial within its domain, presents hurdles for broadening the application of ADP across diverse academic disciplines. The concentration on niche fields underscores a significant gap in the existing infrastructure, limiting the capacity for cross-disciplinary research and the development of versatile ADP tools to a wider academic audience.

To tackle the challenges of limited scope and domain-specific focus in current academic data processing datasets, we introduce the Multidisciplinary Multimodal Aligned Dataset (MMAD). This dataset represents a significant leap forward by offering an expansive collection of metadata from research papers spanning a wide range of disciplines, coupled with an extensive assembly of visual data, including figures and tables, meticulously extracted from each article. Moreover, MMAD enriches these visual elements with detailed annotations, such as captions and comprehensive descriptive context derived from the text of the articles. This fusion of textual and visual data within MMAD is to significantly broaden the horizons for researchers, enabling more detailed analysis and exploration of academic data across various fields. The key contributions of this dataset are manifold:

1. Dataset Presentation and Processing Procedures: MMAD is uniquely designed to cater to the complexities of academic data. We offer an in-depth overview of the dataset’s structure and the sophisticated processing techniques employed to align multimodal data within MMAD. These processes include the extraction and integration of textual and visual information from a diverse range of academic papers published over the past 20 years, resulting in a rich dataset that spans more than 10 major disciplines and 200 subdisciplines.

2. Validation of Dataset Quality and Reliability: Comprehensive experiments have demonstrated the quality and reliability of our MMAD and proposed methods, achieving over 90% accuracy in recognizing visual and textual objects. Furthermore, we employ a large language model (LLM) for a novel quality assessment method, emphasizing the accuracy of matching textual and visual data. This innovative approach, unprecedented in the field, provides a nuanced evaluation of the dataset’s utility and reliability.

3. Exploration of ADP Tasks and Future Research Directions: We delve into the potential applications of MMAD across various ADP tasks, highlighting its versatility and the opportunities it opens for future research. By examining tasks such as trend analysis, content summarization, and automatic caption generation for figures and tables, we illustrate the dataset’s capacity to propel forward the frontiers of academic research. Furthermore, we outline a roadmap for future investigations, underscoring MMAD’s role in fostering innovative studies in ADP.

Methods

The MMAD dataset includes over a million open-source papers licensed under CC BY, CC BY-NC, CC BY-SA, CC BY-NC-SA and CC0. The overall flow chart to build MMAD is illustrated in Fig. 1.

Fig. 1.

The overall pipeline of generating a sample of MMAD from scratch.

Data Source

We procure the full texts of open-source research papers in PDF format from a diverse range of scientific fields through their Digital Object Identifiers (DOI). The download portals for these documents are accessible via the official web pages of the publishing journals. Initially, we extract the papers’ metadata utilizing the S2ORC-DOC2JSON tool in the link of “https://github.com/allenai/s2orc”, a sophisticated software developed by the Allen Institute for AI, designed for the conversion of PDF documents into JSON format. In instances where specific metadata elements are absent, we employ the CrossRef API to complete the information. This API serves as a comprehensive metadata repository offered by publishers upon the initial DOI deposit of an article. As an illustrative case, the metadata for the paper bearing the 10.1186/s40634-014-0017-6can be directly queried using the following link of “https://api.crossref.org/works/10.1186/s40634-014-0017-6”. Further information on leveraging the CrossRef API is detailed in the link of “https://api.crossref.org/swagger-ui/index.html”.

Data Processing

As illustrated in Fig. 1, a typical sample in the MMAD database comprises three key components: a PDF document of a research paper, a corresponding JSON file, and multiple PNG image files. This subsection is dedicated to detailing the procedure for assembling such a comprehensive sample starting from an individual PDF document.

PDF Paper Access

The PDF document is obtained from a specified open-access publishing platform and is stored directly in the PDF folder, bypassing any form of preprocessing.

Metadata Extraction

The metadata for a research paper is predominantly extracted from the PDF file utilizing the S2ORC-DOC2JSON tool. To guarantee the integrity and quality of the data, the CrossRef API is employed as an auxiliary resource for validating the extracted metadata.

Visual information

We leverage the capabilities of the ‘fitz’ module from the PyMuPDF toolkit in the link of “https://pymupdf.readthedocs.io/en/latest/module.html” to facilitate the conversion of PDF documents into PNG format. This step also involves the extraction of the relative positions and contents of all text blocks within the document. Subsequently, for the extraction of figures and tables, along with their respective positions in the converted PNG documents, we deploy a pre-trained YOLOv5 model¹⁹. This model is tailored specifically for the extraction of visual information, ensuring precise identification and localization of graphical elements.

Caption Extraction for Figures or Tables

For each identified figure or table, we determine its closest text block—extracted via the ‘fitz’ module—by calculating the Euclidean distance. The content of this proximate text block is then considered as the caption for the figure or table. Subsequently, this caption is incorporated into the JSON file.

Descriptive Context

Grobid²⁰, a machine learning library, is utilized for its robust capabilities in extracting, parsing, and re-structuring raw documents into more accessible formats. In our methodology, Grobid plays a pivotal role in extracting all paragraphs from the PDF document, aiming to preserve semantic completeness within the descriptive context. Following this extraction, we apply regular expressions as a precise method to align captions with their respective descriptive contexts. For an in-depth exploration of the matching process, Algorithm 1 is provided, showcasing the pseudo-code that elucidates the algorithmic steps undertaken.

Algorithm 1

Retrieve Visual Data Description from a given paper.

Data Management

Using the paper with the DOI 10.1186/s40634-014-0017-6 as an illustrative example, we replace the ‘/’ and ‘-’ characters in the DOI string with ‘_’ and designate it as the basename to ensure compliance with filename conventions. For this paper, the basename is equal to 10.1186_s40634_014_0017_6. Then the PDF file of this paper will be named as ${basename}.pdf and saved to /path/to/MMAD/pdf. Similarly, the metadata information of this paper will be stored in ${basename}.json saved to /path/to/MMAD/json. We demonstrate all keys in ${basename}.json, as well as their datatype and explanations, in Table 1. All visual information extracted from the paper is saved to /path/to/MMAD/visual_data/${basename} with PNG format. The PNG images are named following the format of ${basename}.pdf_${page}_${count}.png, where page and count represent the page number on which the visual information appears and its sequence on that page, respectively. The entire structure of MMAD is shown in Fig. 2.

Table 1.

Reference of metadata keys.

Field Name	Field Type	Description
title	str	Title of this paper.
doi	str	The digital object identifier of this paper.
authors	list[author]	A list of authors contributing to this paper.
author.name	str	Name of the author.
author.org	list[str]	Institution(s) of the author.
author.sequence	str	The author’s order among all authors.
abstract	str	Abstract of this paper.
url	str	The link to download this paper.
publisher	str	Name of this paper’s publisher.
journal	str	Name of the journal that contains this paper.
year	str	The year when this paper is published.
license	list[str]	The license(s) which this paper is under.
subject	list[str]	The field(s) which this paper is in.
figures	list[figure]	All the visual data in this paper.
figure.figureX_url	str	Name of this visual data file.
figure.figureX_acption	str	Caption of this visual data in the paper.
figure.figureX_info	list[figureX_info_X]	All the contexts that match this visual data in the paper.
figure.figureX_info_X	str	The Xth context that matches this visual data in the paper.

Fig. 2.

Dataset structure of MMAD.

Data Records

Within the MMAD root directory, three subfolders are designated: pdf, json, and visual_data. Utilizing DOI as a reference, MMAD encompasses a total of 1,100,810 samples. It is made freely available for public use and academic research endeavors²¹. Given that the included papers are distributed under Creative Commons licenses—namely CC BY, CC BY-NC, CC BY-SA, and CC BY-NC-SA—the MMAD is accordingly governed by the CC BY-NC-SA license to align with these terms.

Technical Validation

Comparison with other ADP datasets

As detailed in Table 2, MMAD presents distinct advantages over existing academic data processing datasets:

1. Large Scale: MMAD is a comprehensive multimodal ADP dataset, encompassing over a million samples. Each sample comprises a PDF document, a JSON file, and a collection of PNG files encapsulating visual data.

2. Multidisciplinary Scope: Spanning a wide array of research fields, including but not limited to General Medicine, Applied Mathematics, and Computer Science Applications, MMAD offers a diverse disciplinary coverage. The distribution of disciplines within MMAD is illustrated in Fig. 3. Notably, given that articles may be categorized under one or multiple subjects as per CrossRef API data, the cumulative count of disciplines surpasses the dataset’s sample count.

3. Content Alignment: Setting MMAD apart from other multimodal ADP datasets is its capability to not only extract visual content, such as figures and tables, from academic papers but also to synchronize these elements with their respective captions and the surrounding descriptive text extracted from the article’s paragraphs. This enriched content alignment is incorporated into the corresponding JSON files.

Table 2.

Comparison of different Academic Content Mining datasets.

Dataset Name	Year	License	Size	Metadata	Citation Data	Visual Data	Visual Data Description	Research Field(s) Covered	Downstream Tasks
RefSeer28	2015	CC BY-NC-SA	1.0M	✓	✓	✗	None	Multidisciplinary	Citation recommendation8
Scholarly Dataset 229	2015	CC BY-NC	100K	✓	✓	✗	None	Computer Science	Paper Recommendation9
Scientific Papers30	2018	Apache License 2.0	348K	✗	✗	✗	None	Multidisciplinary	Document summarization10,11
arXiv CS31	2019	CC0* (metadata)	1.7M	✓	✓	✗	None	Multidisciplinary	Trend analysis32,33, document classification34, text generation35, document summarization36
ScisummNet37	2019	CC BY-SA	1.0K	✗	✓	✗	None	Multidisciplinary	Document summarization12,13
OAG V238,39	2019	ODC-BY	381M	✓	✓	✗	None	Multidisciplinary	Language model training40, citation network study41
S2ORC42	2020	ODC-BY	8.1M	✓	✗	✗	None	Physics, Math, Computer Science	Language model training43,44, document summarization45
SciGen14	2021	CC BY-NC-SA	24.9K	✗	✗	✗	None	Computer Science	Reasoning-aware data-to-text generation15
SCICAP16	2021	CC BY-NC-SA	290K	✗	✗	✓	Figure/Table captions	Computer Science, Machine Learning	Scientific figure analysis46, caption generation24–26
unarXive47	2023	CC BY-SA	165K	✓	✓	✗	None	Multidisciplinary	Citation recommendation48, knowledge graphs49, citation context analysis50
ACL-FIG17	2023	CC BY-NC	56K	✓	✗	✓	Figure/Table captions + parsed descriptive context from main body of each paper	Natural Language Processing, Computational Linguistics	Multi-label classification
MMAD (Proposed)	2023	CC BY-NC-SA	1.1M	✓	✗	✓	Figure/Table captions + parsed descriptive context from main body of each paper	Multidisciplinary	Figure/Table caption generation, document summarization, document classification, paper recommendation, multimodal figure analysis

(*CC0 is applied to all metadata in arXiv dataset³¹, while the articles have multiple licenses. See more details at https://info.arxiv.org/help/license/index.html).

Fig. 3.

Distribution of disciplines in MMAD.

Validation of Visual and Textual Object Recognition and Matching

As illustrated in Fig. 1, the visual charts in our MMAD dataset were identified using the YOLOv5 model. To validate the accuracy of the model’s identification, we randomly sampled 1,000 papers from the dataset as test samples, which contained a total of 15,733 pages (processed as image files). There were 9,091 actual charts, and the results from the model’s identification were compared with the actual labels. The YOLOv5 model achieved a recall rate of 94.91% and a precision rate of 98.08%.

Additionally, to assess the matching success rate between the extracted charts and their corresponding captions, we conducted a random sampling with manual verification. We performed 100 random samples from the dataset, each consisting of 1,000 articles (approximately 5,000 to 7,000 chart-caption pairs). The evaluation resulted in an accuracy rate of 93.47% (95% CI: 93.35% - 93.59%).

Distribution of country/region

A fundamental approach to evaluating the quality of a dataset involves analyzing the distribution of specific attributes and comparing these findings with established benchmarks. In the case of MMAD, examining the distribution of publications across different countries/regions serves as a practical method for quality assessment. The SCImago Journal & Country Rank, accessible to the public, offers a comprehensive portal featuring journals and country scientific indicators derived from the Scopus database²². This resource provides what can be considered the “ground truth” for the volume of publications by country.

In this subsection, we undertake a comparative analysis of the publication volume rankings by country/region within the MMAD dataset against those found in SCImago. This comparison aims to affirm MMAD’s broad coverage and the accuracy of its distribution in mirroring real-world publication patterns.

1. In Experiment 1, we focus on a subset of MMAD, specifically selecting papers published in 2022. Define X = x_ii = 1^N and Y = y_ii = 1^N as the rankings of countries/regions in MMAD and SCImago, respectively. The Pearson correlation coefficient, ρ, is employed to assess the correlation between X and Y, serving as an indicator of the linear relationship’s strength and direction between these two variables. The coefficient ρ ranges from -1 to 1, where -1 indicates a perfect negative linear relationship, 1 denotes a perfect positive linear relationship, and 0 signifies no linear correlation. With cov(X, Y) representing the covariance of X and Y, and DX denoting the variance of X, we calculate ρ = 0.933, as derived from Equation (1).

$$\rho =\frac{cov\left(X,Y\right)}{\sqrt{DX\cdot DY}}$$ 1

• 2.In Experiment 2, utilizing the rankings X and Y derived from Experiment 1, we apply the least squares method to model the linear relationship between them. Our objective is to demonstrate the reliability of the MMAD rankings. We aim to show that the trend line closely approximates the identity function, which would indicate a strong alignment between MMAD’s country/region rankings and those from SCImago. This comparison and its results are visually represented in Fig. 4, where an approximation to the identity function would affirm the credibility of MMAD’s data.
• 3.Experiment 3 examines the evolution of academic publication rankings for Argentina and Pakistan over a ten-year period, spanning from 2013 to 2022. This analysis aims to illuminate the trends and shifts in the publication landscape for these countries, offering insights into their academic progress and prominence. The visualization capturing these decade-long trends and ranking fluctuations is illustrated in Fig. 5.

Fig. 4.

Country/region rankings of MMAD and SCImago in 2022. (ρ = 0.933).

Fig. 5.

Similar ranking trends of particular countries between MMAD and ScImago over the past decade (2013-2022). a, Ranking trend of Pakistan (decreasing). b, Ranking trend of Pakistan (decreasing).

The results of Experiment 1 reveal a Pearson correlation coefficient (ρ) of 0.933 between rankings X and Y, signifying a strong positive linear relationship nearly approaching unity. This high degree of correlation suggests a robust alignment between the datasets’ rankings. In Experiment 2, the linear relationship is further modeled with the equation y = 0.962x + 5.553, as illustrated in Fig. 4. This equation underscores the consistency of our experimental findings with the anticipated ranking congruence.

Moreover, Experiment 3’s visualization, depicted in Fig. 5, showcases the decade-long ranking trends for Argentina and Pakistan. These trends align closely with the SCImago data, reinforcing the reliability of our analysis. Collectively, these outcomes provide substantial evidence of the MMAD dataset’s comprehensiveness and representativeness. The MMAD dataset effectively mirrors the essential attributes of open-source academic publications, establishing its value for academic data processing research.

GPT-4 & Expert Evaluation

A pivotal element in evaluating the MMAD’s quality is the precise parsing of textual context associated with visual information. Presently, the field lacks a standardized quantitative method for assessing this particular aspect. Traditional manual evaluations, while effective, are notably resource-intensive and time-consuming. In response to this challenge, we propose an innovative solution: an LLM-based method for assessing the alignment quality between textual and visual information. This approach harnesses the advanced multimodal text-image comprehension capabilities of GPT-4²³, offering a novel pathway to quantitatively measure the MMAD’s quality.

Our test set is organized into three distinct groups to facilitate a comprehensive evaluation: the fully relevant group, the article-relevant group, and the irrelevant group. Each group contains one hundred samples, ensuring a balanced representation across the categories. A standard sample within these groups comprises three elements: a figure, its caption, and a paragraph. While the figure and caption are consistently well-matched across all samples, the compatibility of the paragraph varies according to the group classification.

1. Fully Relevant Group: In this category, the paragraph is perfectly aligned with both the figure and its caption, presenting a perfect match and demonstrating direct relevance.

2. Article-Relevant Group: Samples in this group feature paragraphs that, although originating from the same article as the figure and caption, do not explicitly reference the content of the figure’s caption. This setup tests the subtleties of relevance within the context of the same document.

3. Irrelevant Group: This group is characterized by paragraphs that are sourced from completely unrelated articles. The inclusion of this category aims to assess the model’s ability to distinguish obvious contrasts in relevance.

In pursuit of precision and consistency across our evaluation process, we carefully crafted a set of prompts. These prompts are uniformly applied to all test samples, ensuring a standardized basis for our analysis.

In this task, we are assessing the quality of the context and how well it aligns with the figure’s caption and the figure image. The context may contain additional details related to the figure. Please evaluate the context based on multiple criteria. Please note that your score could be any integer between 0 and 100. Don’t let the scoring get too rigid. You can use scores like 95, 84, 76, etc. You should give a certain integer, 93 for example, as your score rather than a range.

Criteria for Context Quality:

Relevance (Does the context directly refer to the image as well as its content?):

• score 81 to 100: The context directly refers to some contents in the captain of the image.

• score 61 to 80: The context is strongly related to the figure while not mentioning information from the figure caption directly.

• score 41 to 60: The subject of the context is related to the figure while not directly talking about it.

• score 21 to 40: The context describes the same research field as the figure.

• score 0 to 20: The context has nothing to do with the figure.

Completeness (Does the context provide enough information to help illustrate the function of the figure in this article?):

• score 81 to 100: Reader can totally understand the figure’s role in this article.

• score 61 to 80: Reader can almost understand the figure’s role in this article.

• score 41 to 60: Reader can almost understand the figure’s role in this article but needs thinking for a while.

• score 21 to 40: Reader can hardly understand the figure’s role in this article.

• score 0 to 20: The figure is totally not relevant to this context.

Conciseness (Is the context concise and free from unnecessary details or verbosity?):

• score 61 to 80: Almost every line in the context serve for the figure.

• score 41 to 60: There exist some lines in the context that are not relevant to the figure.

• score 21 to 40: Nearly most of the lines in the context are not relevant to the figure.

• score 0 to 20: The figure is totally not relevant to this context.

Clarity (Is the context provided well clarified?):

• score 81 to 100: The context is well organized.

• score 61 to 80: Minor flaws exist in the writing of the context, but do not detract from the reading.

• score 41 to 60: Minor flaws exist in the writing of the context but can be understood after thinking.

• score 21 to 40: There are some flaws in the context that detract from reading.

• score 0 to 20: Cannot learn any information from this context.

Scoring: Please give me a final average score based on each of the criteria listed above, and the weights of relevance, completeness, conciseness and clarity are 0.35, 0.3, 0.3 and 0.05. You should not provide explanations for your above four scores. Only give me the four-section scores and the final average score.

Caption:

Context:

To verify the reliability of the evaluations conducted by LLMs, we randomly selected a subset comprising 30 samples from each of the three groups, totaling 90 samples. An expert was then enlisted to score these samples based on the previously defined criteria. Concurrently, GPT-4 was also tasked with evaluating the same subset of samples. Let X = x_ii = 1³⁰ represent the scores assigned by the expert, and Y = y_ii = 1³⁰ denote those given by GPT-4.

Acknowledging the potential for difference or bias in interpreting the prompts between the expert and GPT-4, we normalize both X and Y scores based on their respective means and variances. For instance, given the mean (μ) and variance (σ²) of X, the normalization of X is executed as follows: $\tilde{X}=\frac{X-\mu }{\sigma }$. This normalization process ensures a fair comparison by adjusting for scale and dispersion differences between the two sets of evaluations. The distribution of the normalized scores across the three subgroups is depicted in Fig. 6, providing a visual representation of the evaluation outcomes from both the expert and GPT-4.

Fig. 6.

Expert VS. GPT-4 in scoring three subgroups. a, Fully relevant group. b, Article relevant group. c, Irrelevant group.

We analyze the correlation between the normalized scores, $\tilde{X}$ and $\tilde{Y}$, from the three subgroups by plotting trendlines. Notably, the slope of the linear regression line—determined using the least squares method—corresponds to the Pearson correlation coefficients of the two variables post-normalization. This alignment underscores a methodological precision in assessing the relationship between the scores. As depicted in Fig. 6, the trendlines indicate a strong agreement between the expert’s evaluation scores and those derived from GPT-4 across all subgroups. This congruence validates the reliability of utilizing GPT-4 for the quantitative assessment of the MMAD dataset. Such findings not only underscore the efficacy of GPT-4 in mimicking expert judgment but also highlight its potential as a scalable and objective tool for evaluating complex datasets like MMAD.

Upon evaluating all 300 samples with the predefined prompts using GPT-4, the resultant scores are presented in Fig. 7. The analysis reveals a distinct scoring pattern by GPT-4 based on the relevance of the matches. Specifically, samples within the fully relevant group received the highest scores, reflecting their direct correlation between text and visual elements. The article-relevant group was assigned lower scores on average, accompanied by a broader range of variations, indicating a nuanced recognition of relevance. Meanwhile, the scores for the irrelevant group were consistently low, underscoring GPT-4’s ability to discern and penalize the lack of relevance between the provided text and visual content.

Fig. 7.

GPT-4 evaluation scores distribution.

In summary, leveraging the carefully designed prompts, we have shown that GPT-4 possesses the capacity for effectively conducting quantitative evaluations of dataset quality. Through a detailed process of sampling and analysis, our investigation confirms the high degree of alignment between visual and textual information within the MMAD dataset. The textual content not only matches well with the associated visual elements but also exhibits qualities of completeness and conciseness.

Usage Notes

This study represents a pioneering effort to extract and align textual and visual information from academic papers, aiming to create a substantial dataset. Distinct from existing academic datasets that may focus solely on metadata or isolated figures, our approach is comprehensive. We endeavor to collate not only the metadata and figures within a paper but also to precisely identify and correlate descriptions of visual information found in the text with their corresponding image files and captions. This method marks a significant advancement in the field of academic data mining, offering a richer, more integrated dataset for research and analysis.

MMAD can be used to accomplish numerous downstream tasks in a wide range of fields, including but not limited to:

1. MMAD supports most classic ADP tasks that only need either textual or visual information, such as document summarization^{10– 13} and figure caption generation^{24– 26}.

2. The MMAD dataset paves the way for groundbreaking research by merging textual and visual data from academic papers, offering fresh perspectives on scholarly productivity, the influence of visual content on citations, and the dissemination of research findings. By exploring the interplay between text and visuals, researchers can discover how visual elements contribute to the academic impact and visibility of studies, trace patterns of visual communication within the scientific community, and examine the potential for new scholars to adopt established visual representation techniques. This approach promises to deepen our understanding of academic practices and the evolving landscape of scholarly communication.

3. Thanks to advancements in generative artificial intelligence, the MMAD dataset offers exciting opportunities for numerous generative tasks. By fine-tuning multimodal generative models, such as miniGPT-4 and OpenFlamingo, with the meta and visual data from highly cited academic papers, we can leverage AI to generate figures, tables, and captions based on given titles and abstracts. Beyond creating visual content, these fine-tuned models can assess whether a new manuscript contains suitable descriptive texts for its tables and figures, offering suggestions for enhancing the manuscript’s written quality. This integration of AI opens new horizons for improving academic writing and research presentation, making sophisticated visual and textual content generation more accessible to researchers across disciplines.

4. MMAD can be leveraged to facilitate predictive tasks within the academic realm. By analyzing citation records and features extracted from a corpus of scholarly articles, it is feasible to construct a predictive model. This model is capable of estimating the research impact of newly published content, thus offering a quantitative assessment tool for evaluating the potential influence and reach of academic contributions in their developing stages.

MMAD has a few limitations to be noted below:

1. The diversity in popularity across academic disciplines introduces an imbalance in the distribution of paper subjects within the dataset. This difference necessitates tailored adjustments to the MMAD framework to prevent result bias towards subjects with higher popularity. Citation information is not included in MMAD, because this work mainly focuses on the relation between textual and visual information. Researchers may refer to abstract database API, such as Web of Science and Scopus, to adapt MMAD to more downstream tasks about citation.

2. Due to YOLO-V5’s detection accuracy limitations, a small number of images may not be accurately identified. Additionally, incidental extraction of visual content like logos or journal covers/back covers might occur. Such unintended visuals, including logos and back covers, will lack captions in their JSON files. Additionally, our dataset includes only the PDFs and images of the papers without the full text, which represents a limitation of the current dataset. To address this limitation, we plan to gradually incorporate structured text content of the papers in future versions of the dataset.

3. In our task, we utilize an LLM-based approach to assess the alignment quality between textual and visual information. For such complex evaluations involving multiple steps and varied metrics, Chain-of-Thought (CoT)²⁷ reasoning offers an innovative way to enhance model performance. By decomposing the problem and employing step-by-step reasoning, CoT allows for a more systematic analysis of the relationships between different metrics. This will be one of the methods we aim to explore in future research.

Author contributions

H.S. and Z.W. conceived the idea. H.X., Y.W. and J.L. analyzed the data. H.S., Z.W., J.L and H.X. drafted the first version of the manuscript. J.L. revised the manuscript. H.X. prepared the materials. H.S., Z.W. and J.L supervised the project. All authors have read and approved the final version of the manuscript.

Code availability

The implementation of the MMAD framework is structured around two principal modules, each corresponding to a distinct phase of the MMAD creation process: the extraction of paper metadata and the identification and collection of visual data from the papers. The primary input for these modules consists of PDF documents sourced from open-access journals, alongside their respective DOIs. In the “Extracting Paper Metadata” module, the functionality is encapsulated within the ‘doi_api_update_data.py’ script. This script is responsible for leveraging DOIs to retrieve and update the metadata of the papers. The second module, “Identifying and Gathering Visual Data and Their Information,” employs the ‘yolo_process_papers.py’ script. This script is tasked with detecting and aggregating visual content embedded within academic papers, utilizing the capabilities of the YOLOv5 model. Supplementary to these modules, the code repository houses additional resources including pretrained YOLOv5 model weights and associated implementation code. Furthermore, a collection of utility scripts, such as extract_figures_info.py, is provided to facilitate various tasks within the MMAD framework. All required Python libraries are detailed in “requirements.txt”. The source code extensively commented for clarity, is available at this link of “https://github.com/LukaHongyi/MMAD” and was developed using Python 3.9.

Competing interests

The authors declare no competing interests.