Evaluating LC-HRMS metabolomics data processing software using FAIR principles for research software

Abstract

Background

Liquid chromatography-high resolution mass spectrometry (LC-HRMS) is a popular approach for metabolomics data acquisition and requires many data processing software tools. The FAIR Principles – Findability, Accessibility, Interoperability, and Reusability – were proposed to promote open science and reusable data management, and to maximize the benefit obtained from contemporary and formal scholarly digital publishing. More recently, the FAIR principles were extended to include Research Software (FAIR4RS).

Aim of review

This study facilitates open science in metabolomics by providing an implementation solution for adopting FAIR4RS in the LC-HRMS metabolomics data processing software. We believe our evaluation guidelines and results can help improve the FAIRness of research software.

Key scientific concepts of review

We evaluated 124 LC-HRMS metabolomics data processing software obtained from a systematic review and selected 61 software for detailed evaluation using FAIR4RS-related criteria, which were extracted from the literature along with internal discussions. We assigned each criterion one or more FAIR4RS categories through discussion. The minimum, median, and maximum percentages of criteria fulfillment of software were 21.6%, 47.7%, and 71.8%. Statistical analysis revealed no significant improvement in FAIRness over time. We identified four criteria covering multiple FAIR4RS categories but had a low %fulfillment: (1) No software had semantic annotation of key information; (2) only 6.3% of evaluated software were registered to Zenodo and received DOIs; (3) only 14.5% of selected software had official software containerization or virtual machine; (4) only 16.7% of evaluated software had a fully documented functions in code. According to the results, we discussed improvement strategies and future directions.

1. Introduction

Metabolomics is the systematic study of small molecules (metabolites) within cells, biofluids, tissues, or organisms, and it has been widely used in clinical studies (Zhang et al., 2020). Liquid Chromatography – High Resolution Mass Spectrometry (LC-HRMS) is a popular data acquisition technique due to its high sensitivity and specificity (Zhang et al., 2020). Data processing is the first computational step of a metabolomics study and is very important for downstream analysis, such as statistical analysis and data interpretation (Du et al., 2022). Therefore, our study focused on LC-HRMS metabolomics data processing steps. The FAIR Principles – Findability, Accessibility, Interoperability, and Reusability – were proposed to promote open science and reusable data management, and to maximize the benefit obtained from contemporary and formal scholarly digital publishing. To date, numerous software has been developed for LC-HRMS metabolomics data processing (Spicer et al., 2017) such as XCMS (Smith et al., 2006; Tautenhahn et al., 2012), MZmine (Pluskal et al., 2010), and MS-DIAL (Tsugawa et al., 2015); however, there is limited information on the FAIRness of LC-HRMS metabolomics data processing software.

To enhance the propensity of data and other digital objects for sharing and reuse by humans and at scale by machines, the FAIR Principles (Findable, Accessible, Interoperable, and Reusable) originated in the Netherlands during the 2014 Lorentz Workshop “Jointly Designing a Data FAIRport”. Following consultation via the Future of Research Communications and e-Scholarship (FORCE11), the FAIR Principles with 15 detailed guiding principles were published by Wilkinson et al., in 2016 (Wilkinson et al., 2016). With such an arresting and rhetorically useful acronym, the FAIR Principles have gained greater uptake than earlier encapsulation of these ideas (Directorate-General for Research and Innovation (European Commission), 2018). To date, FAIR Principles have been implemented in several areas such as precision oncology data sharing and bioinformatics data management (Mayer et al., 2021; Vesteghem et al., 2020). The adoption of FAIR Principles does not only require FAIR data, but also other FAIR digital objects such as software (Chue Hong et al., 2022). FAIR for Research Software (FAIR4RS) Working Group (Katz, Barker, et al., 2021) was established in 2020 to develop community-endorsed FAIR Principles for Research Software (Chue Hong et al., 2022). In the released FAIR4RS Principles, “Findable” (F) means the software and its metadata can be easily found by humans and machines; “Accessible” (A) means the software and its metadata can be retrieved via standardized protocols; “Interoperable” (I) means software can interoperates with other software; and “Reusable” (R) means software can be understood, modified, built upon, or incorporated into other software (Chue Hong et al., 2022). Clinical studies conducted with FAIR research software are associated with better transparency, reproducibility, and reusability (Barker et al., 2022), leading to more useful research and better translational potential (Goodman et al., 2016). However, the FAIR4RS Principles is a general description that only depicts a continuum of features that moves a research software closer to that goal (Wilkinson et al., 2019). Since FAIR speaks to machine-actionable operations, FAIR digital objects should be amenable to unambiguous forms of validation and evaluation, and a practical interpretation of the FAIR4RS Principles and the development of detailed evaluation guidelines are needed for the implementation (Wilkinson et al., 2019). In the field of LC-HRMS metabolomics data processing, a detailed implementation solution for FAIR4RS Principles has not been investigated yet.

The purpose of this study is to implement FAIR4RS Principles in the evaluation of LC-HRMS metabolomics data processing software. Through a systematic review, we identified relevant LC-HRMS metabolomics data processing software. Next, we evaluated them using criteria related to FAIR4RS Principles, which were based on published papers regarding best practices of research software and internal discussions (XD, MAD, HY, DJL). Each of the criteria was also mapped to the corresponding FAIR4RS Principles. To remove potential ambiguity, detailed evaluation guidelines were also developed and refined through discussion. The goal of our analysis is focused on identifying strategies to improve the FAIRness of software for LC-HRMS metabolomics data processing.

2. Methods

2.1. Study selection process

As illustrated by Fig. 1, we followed PRISMA guidelines for literature review (Liberati et al., 2009). Steps include keyword search, duplicate removal, title-abstract scanning, full-text review, and tool extraction. Search databases include Web of Science (WOS), PubMed, and Embase. The literature search strategy was established by consulting a librarian. As shown by Table S1, search terms have words related to LC-HRMS metabolomics, software, reproducibility, and LC-HRMS data processing. During the Boolean search, we restricted search fields to title and abstract, as well as subject headings for some keywords, and only studies published during the past 5 years as of 2021 August were included. During the paper screening, Covidence software was employed to help with the process (Covidence - Better Systematic Review Management, n.d.). We extracted potentially related software from eligible papers based on sentences around their names in the paper or descriptions in the tool’s paper if the tool has a publication. Then, the software was selected and evaluated based on eligibility and evaluation criteria via reading all their available documentation, publications, and code repositories. Notably, each paper was screened by two reviewers and each software was evaluated by two reviewers. All criteria could be considered as a question like “does the paper/software has a specific feature”. When there is a disagreement between the two reviewers, the reviewer said the paper/software had the feature that would need to show evidence and persuade the other reviewer, then the final label of a specific criterion was decided based on the result of the persuasion. Detailed guidelines for resolving discrepancies were illustrated in Table S3.

Fig. 1.

Consort diagram for the literature review and the computational tool extraction. The screening has two phases: literature screening and tool screening. During literature screening, 1396 papers published in the past 5 years were obtained through keyword search, 70 papers were finally included as relevant ones. In tool screening phase, 122 potentially relevant software were extracted from the 70 papers. We added two software that were recommended by an expert but were not mentioned in the 70 papers. All 124 software were reviewed in more detailed by reading other available resources online such as documentations and code repositories. 61 software were finally considered eligible for final FAIR4RS review.

2.2. Inclusion and exclusion criteria

2.2.1. Related steps

We refer LC-HRMS metabolomics data processing to all steps after data acquisition and before statistical analysis. We documented 13 relevant steps for each software tool. A detailed list and descriptions of the steps are included in the supplementary material Table S2 along with literature references (Clasquin et al., 2012; DeFelice et al., 2017; FillPeaks-Methods, n.d.; Liu et al., 2020; Mayer et al., 2013; Smith et al., 2006; Zhou et al., 2012). We created a controlled vocabulary for these steps based on ontologies and literature. The steps were categorized into four main categories:

1. Data preparation: Steps included in this category happen immediately after data acquisition. These steps make the data usable with computational software in the lab. File format conversion and parameter optimization are the only two steps involved.

2. Feature generation: This category represents processes of generating features and their intensity values in the peak table. Nine steps are included in this category: mass detection, chromatogram building, deconvolution, peak grouping, retention time alignment, peak filling, ion annotation, and batch effect correction.

3. Quality control: Evaluating the analytical variability of the data to make sure the acquired data have good quality for downstream analysis.

4. Metabolite annotation: The process of determining the identity or chemical structure of a metabolite underlying a group of peaks, methods that are used to enhance the proximation result are also included in this category.

2.2.2. Paper screening

The goal of paper screening was to select papers that mentioned or used software related to LC-HRMS metabolomics data processing. Therefore, during title-abstract scanning, we excluded studies based on the following criteria:

1. We excluded papers that were not written in English.

2. We excluded papers whose instrumental analysis did not include LC-HRMS. For example, studies used gas chromatography-mass spectrometry (GC-MS) or nuclear magnetic resonance (NMR).

3. We excluded papers that were not about metabolomics, such as papers focusing on proteomics.

4. We excluded papers that did not evaluate computational steps. For instance, some studies only focused on sample processing steps.

5. We excluded reviews that did not describe specific LC-HRMS metabolomics software.

During the full-text reading stage, we used information provided within the paper and used the following exclusion criteria for further filtering:

1. The paper did not include a computational tool.

2. No tool in the paper was about metabolomics.

3. Some software in the paper was about metabolomics but not related to LC-HRMS data processing steps.

2.2.3. Tool screening

For initial screening, the relevance of a certain tool was judged according to its context in the paper. Software for which at least one reviewer thought it could perform related steps was included. During tool screening, we further investigated whether a tool had a certain function based on its documentation. For example, if a tool did not contain a tutorial regarding how to do quality control, we would think it did not have the function; this applied even if a researcher might figure out a way to do quality control with the tool. Then, the software was further filtered by reading publicly available documentation, publications, and code repositories. The exclusion criteria were:

1. The tool was no longer available.

2. The tool was produced by a commercial company.

3. Functions included in the tool were not related to the steps in Table S2.

4. The tool was an extension of an existing tool and did not have its webpage or documentation.

2.3. Information gathering process

To extract information manually, the full texts of selected articles were read by two reviewers. Relevant software included in these articles was deduplicated and evaluated by two reviewers for general information - name, first release date, supported operating systems, major programming languages, literature citation, supported mode (i.e., web-based/standalone/plug-in), and open-source or not - about the tool and specific FAIR4RS related criteria. Data processing steps a software tool can perform were extracted by going through the documentation as well as literature containing relevant information (Li et al., 2018; Misra, 2018, 2021; Misra et al., 2017; Misra & Mohapatra, 2019; O’Shea & Misra, 2020; Stanstrup et al., 2019). To ensure the correctness of the annotation of steps that a software tool can perform, we first did the annotation by ourselves, then contacted all senior authors of included software via emails for confirmation, correction, and comments. We also sent a follow-up email to software authors that did not respond within five days. The software authors we contacted were recorded in Table S4. FAIR4RS-related evaluation criteria were extracted by going through publications related to research software best practices, and additional criteria were added based on internal discussions (XD, MAD, HY, DJL). Each selected evaluation criterion was then assigned to one or more FAIR4RS categories through discussions between two bioinformaticians (XD and MAD) and one reproducibility expert (HY). The detailed evaluation guideline for each criterion was created and refined through discussion between the two reviewers (XD and FD) to make sure the description was not ambiguous. We screened the latest version of each tool at the time of evaluation, the time spanned from October 2021 to August 2022. During the evaluation process, each criterion was assigned a value of “TRUE”, “FALSE” or “NA” based on publicly available documentation and code repositories. “TRUE” represents that the tool met the criterion; “FALSE” meant the tool did not meet the criterion; “NA” meant the criterion does not apply to the tool. For example, if the tool did not have the command-line option, then the criterion of “include help command” would be not applicable (NA).

2.4. Synthesis of results

The percentage of criteria fulfillment of each tool was calculated by dividing the number of “TRUE”s by the number of applicable criteria of the tool (i.e. ignoring any NA values). We also calculated the percentage of software that met each criterion, which was calculated by dividing the number of “TRUE”s by the number of software that could apply the criterion. For example, a tool may have 45 applicable criteria out of the total 47 criteria, and 35 criteria were labeled “TRUE”, then the percentage of criteria fulfillment would be 35/45*100%=77.8%. This can identify FAIR4RS-related criteria that merit further attention from the authors of each tool as well as future developers.

3. Results

3.1. Paper screening

As illustrated in Fig. 1, the initial search generated 1396 records and 954 articles after removing duplicates. The title-abstract screening reduced the number to 87 articles. During the full-text screening, 17 articles were excluded: 4 did not include a computational tool; 8 mentioned computational software but they were not about metabolomics; and 5 included metabolomics software but the software was not for LC-HRMS data processing. Paper screening record exported from Covidence is included in the supplementary material Table S4.

3.2. Tool screening

We extracted 122 potentially related software from the 70 eligible papers, then added two relevant software (EI-MAVEN, MS-FLO) we knew but were not mentioned in the 70 papers. That said, the two software were not mentioned in the notes produced by reviewers (Table S5 -> sheet “full_text_included” -> column “Notes”) during the full-text review. The number of software was reduced to 61 after reviewing their documentation in detail. In terms of function annotation, senior authors of 49 out of the 61 included software responded to our inquiry emails. General information and detailed evaluation results of the 61 selected software (Adusumilli & Mallick, 2017; Agrawal et al., 2019; Alonso et al., 2011; Broeckling et al., 2014; Brunius et al., 2016; Bueschl et al., 2017; Cai et al., 2015; Capellades et al., 2016; Chokkathukalam et al., 2013; Chong & Xia, 2018; Clasquin et al., 2012; Creek et al., 2012; Davidson et al., 2016; De Livera et al., 2018; DeFelice et al., 2017; Del Carratore et al., 2019; Dührkop et al., 2019; Fischer et al., 2022; Franceschi et al., 2014; Gatto et al., 2021; Giacomoni et al., 2015; Guo et al., 2021; Helmus et al., 2021; Huan & Li, 2015a, b; Huang et al., 2014; F. Huber, Ridder, et al., 2021; Hughes et al., 2014; Jaitly et al., 2009; Ji et al., 2017, 2019; Kantz et al., 2019; Kuhl et al., 2012; Kutuzova et al., 2020; Li et al., 2017; Libiseller et al., 2015; Liggi et al., 2018; Lommen, 2009; Loos, 2016; Mahieu et al., 2016; Müller et al., 2020; Olivon et al., 2018; Palarea-Albaladejo et al., 2018; Pang et al., 2021; Pluskal et al., 2010; Protsyuk et al., 2018; Ridder et al., 2012; Ross et al., 2020; Röst et al., 2016; Ruttkies et al., 2016; Shen et al., 2019; Smith et al., 2006; Tautenhahn et al., 2012; Teo et al., 2020; Tsugawa et al., 2015, 2016; Uppal et al., 2013, 2017; Weber & Viant, 2010; Yu et al., 2009; Zhou et al., 2014) was included in the supplementary material Table S5. Venn diagrams regarding technical properties were illustrated in the supplementary material Figure S1. In the 63 excluded software, 8 were excluded because they were not available, 24 were excluded due to their association with commercial companies, 28 were excluded because functions were not relevant, and 3 were excluded because they were not independent (i.e., an extension of another selected tool). Here “independent” means the tool has its documentation or webpage. For example, we consider MetaboAnalystR a different tool from MetaboAnalyst since MetaboAnalystR has its own GitHub page and webpage. However, we considered CSI:FinderID part of SIRIUS since we did not find CSI:FinderID to have its documentation at the time we did our evaluation. Similarly, MSnBase and CAMERA are included in the xcms software tool, but they were considered “independent” in our study since they had their webpage and documentation. All excluded software and detailed reasons for exclusion were shown in the supplementary material Table S6.

3.3. FAIR4RS evaluation

FAIR4RS evaluation results were illustrated in Fig. 2 and Fig. 3. As illustrated in Table 1, we summarized 47 FAIR4RS-related criteria in total, among which 41 were from the literature regarding research software best practices, and 6 were based on internal discussions (XD, MAD, HY, DJL). At the time of our evaluation, the minimum, first quartile, median, third quartile, and maximum percentages of the fulfillment of criteria were about 21.6%, 39.5%, 47.7%, 57.9%, and 71.8%. Two software (OpenMS and patRoon) fulfilled no less than 70% applicable criteria. Ten software met 60–70% applicable criteria, they were MSnbase, EI-MAVEN, smartPeak, Spec2Vec, SIRIUS, ProteoWizard-msConvert, W4M, xcms, MetaboAnalystR, and MAGMa. Additionally, we found seven software (OpenMS, patron, EI-MAVEN, W4M, MetaboAnalystR, MS-DIAL, MetaboAnalyst) could perform the greatest number of steps (9 out of 13) involved in LC-HRMS metabolomics data processing. Three software were powered by artificial intelligence (AI) techniques (Spec2Vec, DeepMASS, and No_NAME), and they fulfilled 62.2%, 47.4%, and 26.3% of our evaluation criteria. We also found most of the included software was open source except MetaboAnalyst, MetAlign, and XCMS-Online, and none of the closed-source software was ranked within the first quartile of all included software.

Fig. 2.

Summary of basic results. Functions associated with each tool were displayed on the left side. Percentages of evaluating criteria that a software tool fulfilled are shown in the middle of the figure. The right panel indicates the percentage of criteria fulfillment of each tool. Additionally, names of software powered by AI techniques are highlighted in red, and software that are closed source are highlighted in black.

Fig. 3.

Line chart reflecting the criteria fulfillment of each category. X-axis represents tool names, ranked by overall percentage of fulfillment from left to right. Y-axis on the left side stands for percentage of fulfilled criteria. Overall fulfillment is represented by the black solid line. Fulfillments of findability, accessibility, interoperability, and reusability categories are represented by blue, yellow, green, and red dash line separately. Software are ranked by their release time, which is represented by the grey line along with the secondary y-axis.

Table 1.

Information about evaluation criteria

Criteria	FAIR4RS category (Chue Hong et al., 2022; Hasselbring et al., 2020; Lamprecht et al., 2020)	Evaluation guideline
Register to Zenodo and get DOI (Georgeson et al., 2019)	F1.2, A1.1, A2	Whether the software has been deposited to Zenodo (https://zenodo.org/) and got a DOI.For example, some software may have a Zenodo badge on the GitHub repo. Bioconductor DOI does not count since Bioconductor assigns DOI to only the current version. Only using GitHub does not count, since the GitHub URL, which relies on alterable domain name resolution, does not guarantee the persistence of identifiers (Berrios et al., n.d.).
Have a license (Georgeson et al., 2019; Jiménez et al., 2017)	R1.1	If the license appears in any documentation of the software, then this would be TRUE.
Provide a command-line option (Georgeson et al., 2019)	I1, R3	Does any documentation teach users how to use the tool with a command-line (shell programming language, which can be used in a console)?
Use conventional input and output (Georgeson et al., 2019)	I1, I2, R3	The software did not create a data format meanwhile forcing users to use the new format as the input/output, is this correct?
Output log independently (Georgeson et al., 2019)	R	Does the software output the progress logging or error logging to an independent place (e.g., a file for non-web-based software, or on a website for web-based software)
Document exit status (Georgeson et al., 2019)	I	Does the documentation describe the meanings of different exit codes produced after running the software? (e.g., the README file on this page https://github.com/bionitio-team/bionitio-python/search?q=exit+status)
Have continuous integration (Georgeson et al., 2019; Hunter-Zinck et al., 2021)	I, R	Does any documentation or publication of the software mention continuous integration? Has the software applied a tool like GitHub Action, Travis CI, Circle CI, etc.? If any answer to the above two questions is true, this would be TRUE, otherwise FALSE. If the software is not open source, then this would be NA.
Have software testing (Georgeson et al., 2019; Hunter-Zinck et al., 2021)	R3	Does any documentation or any part of the code (or code comment) mention software testing (e.g., unit test, integration test, etc.)? If the software is not open source, then this would be NA.
Provide official software containerization or virtual machine (Georgeson et al., 2019)	I, A1, R	Has the software been packaged with a virtual machine or software container such as Docker or Singularity? Tips: software with containerization might have a file named “Dockerfile” or provide a link to Docker Hub. If the software is not open source, then this would be NA.
Have standard packaging (Georgeson et al., 2019; Heil et al., 2021)	F4, A1, I, R	If the software is an API, has it been packaged? If it has, users should be able to install it with one command line within the programming language like “pip install” or “install.packages”. If the software is not a plug-in, put NA
Have version control for code (Heil et al., 2021; Jiménez et al., 2017; Ram, 2013)	A1	Does the tool’s code use software like GitHub, GitLab, or BitBucket to do version control? The repo should be created by the author, instead of being created by others or a system bot (e.g., CRAN bot). If the software is not open source, then this would be NA.
Have automated code quality checks (Georgeson et al., 2019)	R3	Does the tool use a tool like “Codacy” or “quality gate” to report code quality? If the software is not open source, then this would be NA.
Have code coverage assessment (Aghamohammadi et al., 2021; Hunter-Zinck et al., 2021)	R3	Does the software use a tool like SonarCloud (https://sonarcloud.io/) or Codecov (https://about.codecov.io/) to report the percentage of code coverage? If the software is not open source, then this would be NA.
Provide example input data (Leprevost et al., 2014)	R1, R3	Are any example data provided? The example data may be in the documentation or the code repo.
Provide example results (Zhao et al., 2012)	R1, R3	Has any output result (produced by running the software with the provided example data) been displayed in the documentation or the code repo? If the software did not provide any example data, this would be “NA”
Provide issue tracking (Leprevost et al., 2014)	R1.2	Is there any issue tracking tool used? Such as GitHub Issue (software that has a GitHub/GitLab/BitBucket should fulfill this criterion as long as the repo was created by the author). There is also some other issue-tracking software (see if the documentation/publication mentions any other issue-tracking software if GitHub/GitLab/Bitbucket was not used).
Have user documentation (Georgeson et al., 2019)	R1	Is there documentation for users?
Have developer documentation (Georgeson et al., 2019)	R1	Is there documentation for developers, which has information about how to contribute to the tool?
Have an installation guide in documentation (Karimzadeh & Hoffman, 2018)	R1	Does the documentation teach you how to install the software?
Include dependencies and version numbers in documentation (Karimzadeh & Hoffman, 2018)	I2, R2, F4	Do the documentation mention dependencies (e.g., operating system, programming language, packages, etc.) and some of their versions?
Provide a quick start in documentation (Lee, 2018)	R1	Does the documentation teach you how to run the software with provided example data? If example data were not provided, put NA.
Provide a configuration guide in documentation (Karimzadeh & Hoffman, 2018)	I2, R	Each software tool may have a bunch of parameters that users need to set up. Does the documentation talk about the meaning of the parameters (including the input and output of the software)? Software that only provides a GUI version do not applicable (i.e., put NA)
Have FAQ in documentation (Karimzadeh & Hoffman, 2018)	R1	Does the documentation have an FAQ?
Have a searchable forum (Karimzadeh & Hoffman, 2018)	R	Does the documentation mention any forum for the software?
Have a mailing list in documentation (Karimzadeh & Hoffman, 2018)	R	Does the documentation mention emails that you can contact?
Have a change log in documentation (Lee, 2018)	R1.2	Does the documentation include a changelog (i.e., a document that describes changes and time of changes made by contributors, such as “NEWS” or “releases”, etc? GitHub commit does not count, since it might not be accurate and a person can upload code written by others to GitHub and has one commit record for multiple important changes)?
Have a citation guide in documentation (Lee, 2018)	R1, F3	Does the documentation mention how to cite the software or publications associated with the software?
Have historical contribution record in documentation (Lee, 2018)	R1.2	Does the documentation have a part that lists all contributors? Contributors shown on the right side of the GitHub page do not count, since this contributor section is created by GitHub automatically, but people may upload code or documents written by others to GitHub.
Have fully documented functions in code (Lee, 2018)	I, R	Randomly check the code written by the authors and see if the function includes comments that describe the function and its input and return values. If the software is not open source, then this would be NA.
Include help command (Lee, 2018; Seemann, 2013)	R1	Does the documentation mention a command line that can provide information about how to use the software? If the tool does not provide a command-line argument, then put NA.
Have a comment including the creator’s name in the code (Karimzadeh & Hoffman, 2018)	R1.2	Randomly check the code written by the authors and see if it mentioned the creator’s name. This would be “NA” if the software is not open source.
Have a comment including the creation date in the code (Karimzadeh & Hoffman, 2018)	R1.2	Randomly check the code written by the authors and see if it mentioned the date of creation. This would be “NA” if the software is not open source.
Have a comment including the creator’s email in the code (Karimzadeh & Hoffman, 2018)	R1.2	Randomly check the code written by the authors and see if it mentioned the creator’s email address. This would be “NA” if the software is not open source.
Have version control for documentation (Lee, 2018)	R1	Does any of the documentation associated with the software use a version control system such as GitHub?
Have web-based documentation (Karimzadeh & Hoffman, 2018)	F4	Is any of the documentation web-based (i.e., not in a compressed file that you need to download and decompress, or you can only see them after installing the software via an executable file), which makes it easier for being found by a search engine?
Have information on potential errors and warnings as well as ways to resolve them in documentation (Karimzadeh & Hoffman, 2018)	R1	Does the documentation mention any commonly appeared errors and ways to resolve them? This is usually in the FAQ section but may also be in other parts of the document.
Explain file format in documentation if a new format is created (Karimzadeh & Hoffman, 2018)	I2, R1	If any new file format was created by the authors, does the documentation have explanations of the new format? If no new file was created, put “NA”.
Provide graphical instruction if the software has a GUI (Karimzadeh & Hoffman, 2018)	R1	If the software has GUI (web-based software count GUI software), does the documentation include a screenshot (visualization) to explain the software? Put “NA” if the software does not have GUI. Notably, we considered all web-based tools having GUI.
Have information about supported OS in documentation (Karimzadeh & Hoffman, 2018)	R2	Can you get the information about which operating system is supported from the documentation? This is only applicable to software with a standalone mode.
Include any semantic annotation with controlled vocabulary underlying ontologies in the documentation (Palmblad et al., 2019)	F2, I, R1, R3	Does the documentation mention any part of the software is described with controlled vocabulary associated with an ontology?
Have a name	F2	Does the software have a name?
Have a publication (Romano & Moore, 2020)	F2, F4, R1	Does the software have a scientific publication?
Have the capability of consuming a config file containing the parameter setting information	I2	Can the software consume a config file including parameter setting information? This does not apply to software that only has plug-in mode.
Provide interpretation of the independent log file in documentation	R1	Does the documentation tell you what the inside of the output log file looks like?
Have an output log file containing parameter setting information	R1	Does the output log file include parameter setting information? If the software does not output an independent log file (or log information that is shown on a webpage from a web-based tool), put “NA”.
Have an output log file containing the version of the software	R1	Does the output log file include the version information of the software being used? If the software does not output an independent log file (or log information that can be downloaded from a web-based tool), put “NA”.
Have an output log file containing loaded modules/dependencies during the execution	R2	Does the output log file include the loaded modules/dependencies of the execution? If the software does not output an independent log file (or log information that can be downloaded from a web-based tool), put “NA”.

The table includes the name (left), FAIR categorization (middle), and description (right) of each criterion.

Figure 4 represented the percentage of criteria fulfilled in each category. Notably, some criteria were related to multiple FAIR4RS categories. An overview of the evaluation results in terms of each category is illustrated below:

Fig. 4.

Percentage of software that fulfilled each evaluation criterion. X-axes are values for percentage of fulfillment, and y-axes list all evaluation criteria. There are 4 categories and 47 criteria in total. (A) Blue bars represent findability related criteria; (B) yellow bars represent criteria related to accessibility; (C) green bars represent criteria related to interoperability; and (D) red bars stand for criteria regarding reusability.

1. Findability: We identified eight criteria that were related to findability. Among the eight criteria, “Have web-based documentation” had the best fulfillment of 100%. “Register to Zenodo and get DOI” was associated with the least fulfillment of ~ 6% and “Include any semantic annotation with controlled vocabulary underlying ontologies in the documentation” had the lowest fulfillment of 0%.

2. Accessibility: We identified four criteria that were corresponding to accessibility. Among the four criteria, “Code has version control” had the greatest fulfillment of over 80%. However, “Provide official software containerization or virtual machine” had ~ 14% fulfillment, and “Register to Zenodo and get DOI” was associated with the least fulfillment of ~ 6%.

3. Interoperability: We identified 12 criteria that were about interoperability. Among the 12 criteria, “Use conventional input and output” and “Explain file format in documentation if a new format is created” had the best fulfillment of 100%. Nevertheless, “Have fully documented functions in code” had ~ 17% fulfillment, “Provide official software containerization or virtual machine” had ~ 14% fulfillment, “Document exit status” had ~ 2% fulfillment, and “Include any semantic annotation with controlled vocabulary underlying ontologies in the documentation” had 0% fulfillment.

4. Reusability: We identified 41 criteria that were related to reusability. Among the 41 criteria, three criteria (“Use conventional input and output”, “Have user documentation”, and “Explain file format in documentation if a new format is created”) had 100% fulfillment. However, we found 13 criteria with a fulfillment of < 20%: “Have FAQ in documentation”, “Have developer documentation”, “Have information on potential errors and warnings as well as ways to resolve them in documentation”, “Have fully documented functions in code”, “Have an output log file containing loaded modules/dependencies during the execution”, “Have code coverage assessment”, “Provide official software containerization or virtual machine”, “Have a searchable forum”, “Have comment of creator’s email in code”, “Have historical contribution record in documentation”, “Have comment of creation date in code”, “Have automated code quality checks”, “Document exit status”, and “Include any semantic annotation with controlled vocabulary underlying ontologies in the documentation”.

3.4. FAIRness over time

Table S5 had information of the release time of every included tool. We observed that there was only one included tool was released in 2006 (xcms), 2008 (MZmine), and 2021 (DaDIA), and no tool was released in 2007. Multiple included software was associated with a release time within each year between 2009 and 2020. Figure 5 is a scatter plot representing the change of FAIRness through time. Pearson’s correlation (Schober et al., 2018) indicated: (1) there was no significant (p < 0.05) FAIRness improvement across time; (2) findability has a trend of decrease; (3) all four categories were positively correlated with a statistical significance, and interoperability and reusability had the highest coefficient. To investigate what factors might contribute to the decrease of findability, we did a multiple linear regression using criteria related to findability and found four factors were significantly and positively correlated to findability: (1) have a name; (2) have a citation guide in documentation; (3) register to Zenodo and get DOI; (4) have dependencies and version numbers in documentation.

Fig. 5.

FAIRness trend across time. X-axis are years representing selected software? first release times, y-axis stands for the averaged %fulfillment of software released in each year. (A) represents the relationship between findability and tool release times; (B) represents the relationship between accessibility and tool release times; (C) represents the relationship between interoperability and tool release times; (D) represents the relationship between reusability and tool release times; (E) represents the overall FAIR4RS criteria fulfillment and tool release times. Results of Pearson?s correlation are also included in each sub-figure; (F) includes information regarding correlations among categories.

4. Discussion

The application of FAIR Principles in the metabolomics field has largely focused on data sharing and management (Mendez et al., 2019; Rocca-Serra & Sansone, 2019; Savoi et al., 2021). More recently, the FAIR Data Principles have been translated into research software (Hasselbring et al., 2020; Katz, Gruenpeter, et al., 2021; Lamprecht et al., 2020), and the first version of FAIR4RS Principles was released in May 2022 (Chue Hong et al., 2022). In this study, we used FAIR4RS-related criteria to evaluate 61 selected data processing software, including 58 open-source software and 3 closed-source software. Most (41 out of 61) included criteria were related to reusability, which is consistent with previous findings (Wolf et al., 2021), and we also found the four categories were positively related to each other. Notably, open-source code is not required by FAIR Principles (Katz, Gruenpeter, et al., 2021), thus, it was not one of our evaluation criteria. To date, no study has been conducted to investigate how to implement FAIR4RS Principles in the metabolomics field. Therefore, our study extends previous work by implementing FAIR4RS Principles to assess LC-HRMS metabolomics data processing software.

4.1. FAIRness improvement strategies

The primary findings from our evaluation revealed that semantic annotation of key information was associated with 0% fulfillment among all evaluated software, notably, the criterion is related to findability, interoperability, and reusability. Semantic annotation of key information includes functions, input and output data type, and format. Describing key information using controlled vocabulary underlying an ontology makes it readable and discoverable by both humans and machines (Lamprecht et al., 2020). Although we noticed that many software such as XCMS has been registered to bio.tools (Ison et al., 2019) that include semantic annotation of software, none of the official documentation mentioned this feature at the time of our evaluation. Notably, semantic annotation does not only improve the findability of the tool but also enables machines to find other similar software based on the annotation (Lamprecht et al., 2021). Semantic annotation of data processing software can be used to find relevant software to develop workflows automatically (Lamprecht et al., 2021), such systems have been used for analyzing proteomics data (Kasalica et al., 2021; Palmblad et al., 2019) and DNA sequence data (Zheng et al., 2015). Therefore, adding semantic annotation information to the official documentation may improve the FAIRness of LC-HRMS metabolomics data processing software dramatically.

We also found three other criteria that were related to multiple FAIR categories for which only a small percentage of software was compliant. Firstly, only about 6% of software was registered to Zenodo and received a digital object identifier (DOI). As explained in Table 1, unlike other identifiers such as GitHub page URL or the DOI assigned by Bioconductor, Zenodo assigns each version of the software a persistent and distinct DOI (Berrios et al., n.d.; van de Sandt et al., 2019), facilitating findability and accessibility. Notably, we also found the criterion of registering software to Zenodo contributes to the decrease in findability over time. The FAIR4RS Working Group created a community on Zenodo to ensure the FAIRness of research outputs (Chue Hong et al., 2022). Notably, GitHub now has an integration of Zenodo as a third-party tool to help users with referencing and citing content (Referencing and Citing Content, n.d.). The low percentage of fulfillment (6%) is also expected since we observed most of the included software are R packages (39 out of 61 as illustrated in Figure S1). Unlike other popular repositories for R packages such as GitHub, R-Force, CRAN, and Bioconductor (Decan et al., 2015); users cannot install an R package deposited to Zenodo in the console using “install.packages”. Therefore, to improve software FAIRness, we recommend future developers deposit packages to both Zenodo and an R repository. We also recommend Zenodo developers provide a wrapper or guide regarding how to install R packages using a Zenodo URL. Additionally, we found only about 15% of software provided an official containerized version, which is relevant to accessibility, interoperability, and reusability. Providing a containerized version enables users to execute the software on different machines smoothly and without worrying about the installation of dependencies (Georgeson et al., 2019; Senington et al., 2018). In terms of R-based software, containerization including only a single package may not be that useful since users may want to use multiple packages for the analysis. The RofMassSpectrometry Initiative (Rainer et al., 2022; RforMassSpectrometry, n.d.) and metaRbolomics Toolbox in Bioconductor (Stanstrup et al., 2019) are currently two popular ecosystems for R-based metabolomics software. Therefore, a community-wide containerized ecosystem including all commonly-used and mature R-based software for the metabolomics community would be very helpful. Another poorly fulfilled criterion is “fully documented functions in code”, which is related to interoperability and reusability. Documenting input, output, and errors that a function may raise makes it easier for users to inspect the software and learn how the software can interact with other software (Lee, 2018). Therefore, in addition to semantic annotation, our results also demonstrate the FAIRness of LC-HRMS metabolomics data processing software also needs: (1) registering to Zenodo and getting version controlled DOIs, (2) providing official software containerization or virtual machine, and (3) providing fully documented functions in code.

Advances in academic publishing of software have required checklists to ensure the quality of submissions, such as the Journal of Open Source Software (Review Checklist — JOSS Documentation, n.d.). Our results revealed that 59 out of 61 software were associated with a peer-reviewed publication. Checklists have been used to promote transparent reporting of metabolomics studies (Considine & Salek, 2019; Fiehn et al., 2007; Goodacre et al., 2007; Snyder et al., 2014; Sumner et al., 2007). Our recent work proposed a checklist to promote reproducible computational analysis of clinical metabolomics research (Du et al., 2022). The evaluation guideline in this study can also be used as a checklist by journals for promoting FAIR4RS Principles. A recent study shows that a direct request from the editor for research resource identifiers (RRID) is a more effective way for users to provide RRID than merely writing it in the journal’s author instructions document (Menke et al., 2020). Similarly, we expect a direct request from the journal editor regarding some criteria in our evaluation guideline will be a more effective way for FAIRness improvement than just providing a checklist in the author’s guideline. Additionally, some journals such as Analytical Chimica Acta now require an assessment of the tool by an independent advanced user, who is not a software developer or bioinformatician (Analytica Chimica Acta | Journal | ScienceDirect.Com by Elsevier, n.d.). In summary, our evaluation guideline can be used during peer-review as a checklist to improve the FAIRness of software associated with a scientific publication.

4.2. Strengths, limitations, and future works

Our study has several strengths. We followed PRISMA guidelines to extract literature in the past 5 years (from 2016 to 2021) that might include a metabolomics data processing tool. Our evaluation checklist synthesized FAIR4RS-related criteria from existing literature regarding best software practices, including reproducible software development practices (Heil et al., 2021; Jiménez et al., 2017; Ram, 2013), best practices for scientific software (Hunter-Zinck et al., 2021; Leprevost et al., 2014; Zhao et al., 2012), best software documenting practices (Karimzadeh & Hoffman, 2018; Lee, 2018), best practices for command-line software (Georgeson et al., 2019; Seemann, 2013), and best software testing practices (Aghamohammadi et al., 2021). We also filtered out criteria that were not required by FAIR4RS Principles such as open-source (Katz, Gruenpeter, et al., 2021). Therefore, our checklist is more comprehensive than existing checklists, and more focused on FAIR4RS Principles. Additionally, our checklist is more specific than the general FAIR4RS Principles and with a detailed evaluation guideline. We used multiple reviewers to mitigate subjective bias. We extracted related software from selected papers, and read their publicly available documentation, publications, and code repositories for a comprehensive FAIRness evaluation. The FAIR4RS criteria were synthesized from the literature and additional criteria were added via discussions among authors (XD, MAD, HY, DJL), and the criteria can also be used to guide future FAIR LC-HRMS metabolomics software development. Each criterion was assigned to one or more FAIR4RS categories based on a discussion between two bioinformaticians (XD and MAD) and one reproducibility expert (YH). During the evaluation, two reviewers (XD and FD) went through the official documentation, publications, and code repositories of the 61 included data processing software based on an agreed evaluation guideline. More importantly, our evaluation method leads to a meaningful result. The result indicates that findability decreases over time, which is expected since people usually try to publish first and then build up nicer documentation afterward when resources are available, which means it takes time after the release for findability to increase. Although we only evaluated non-commercial LC-HRMS metabolomics data processing software, the criteria may also be suitable for evaluating all types of software. To date, this is the first study to provide an implementation solution for using FAIR4RS Principles in the metabolomics field.

However, the results of this study should be considered along with some limitations of our experimental design. First, our evaluation was based only on publicly available materials including documentation and code. Secondly, we only checked the software’s official documentation but did not check the software’s machine-readable metadata (e.g., XCMS has machine-readable metadata in the software ontology (Malone et al., 2014)), which might be added by experts other than the software’s authors. Additionally, we considered all evaluation criteria equally important without weighting them. Furthermore, we included only software that was mentioned in metabolomics papers or metabolomics software review papers published in 2016–2021, thus, the list of tools may not be completed and some relevant software published very recently might not be included, such as MS2DeepScore (F. Huber, van der Burg, et al., 2021) and MS2Query (Jonge et al., 2022). The controlled vocabulary and categorization of data processing functions shown in Table S2 were produced by looking through literature (Clasquin et al., 2012; De Vos et al., 2007; FillPeaks-Methods, n.d.; Libiseller et al., 2015; Liu et al., 2020; Mayer et al., 2013; Smith et al., 2006; Vitale et al., 2022; Zhou et al., 2012) and discussing internally, and we admit that such a categorization is never easy. So, our function categorization may not be perfect and recognized by the entire metabolomics community, and software authors may have a slightly different understanding of our categorization and descriptions when responding to our inquiry. For instance, one software author emailed us saying functions like mass precision enhancement (Huber et al., 2022) should be part of the data processing steps. Notably, our study only focuses on evaluating the FAIRness of software using relevant qualitative properties, but FAIRness is just one of many considerations when selecting the software for a research workflow. For example, some software authors responded to us by saying the speed of data processing, the capability to process large-scale data, and the efficiency of hardware usage are very important aspects (Gatto et al., 2021). We did not use real-world data to get results using the included software and make a comparison. Our main focus is the software itself, and we did not consider factors (e.g., the type of the mass spectrometer) that may affect the development of the software. Additionally, it is essential to assess the quantitative performance of software using annotated LC-HRMS dataset and library (Hao et al., 2018). The FAIRness of the LC-HRMS dataset, spectral library, and software for metabolite proximation are important for the FAIRness of the entire workflow. Therefore, a future review regarding studies and available datasets for such benchmarking would be a very useful resource for FAIR LC-HRMS metabolomics workflows.

5. Conclusion

We presented a comprehensive FAIRness evaluation for existing non-commercial LC-HRMS metabolomics data processing software. We evaluated 61 qualified software with 47 criteria related to FAIR4RS. Our result indicated that no software had perfect FAIR4RS compliance (i.e., fulfilled 100% applicable criteria). The maximum of criteria fulfillment was 71.7%, meaning all evaluated software had considerable space for improvement. We also identified criteria that were poorly fulfilled for each category along with detailed strategies for improvement. We believe our study can serve as a guideline to create FAIR research software for LC-HRMS metabolomics data processing software.

Table 2.

Multiple linear regression for criteria related to findability

Criterion (missing%)	Coefficient	P-value
Have a name (0%)	23.72	0.00
Have a citation guide in documentation (0%)	14.87	0.00
Register to Zenodo and get DOI (0%)	11.63	0.00
Have dependencies and version numbers in documentation (8.2%)	7.25	0.00
Have a publication (0%)	5.87	0.17
Have standard packaging (39.34%)	1.08	0.36
Has web-based documentation (0%)	0.00	0.03
Include any semantic annotation with controlled vocabulary underlying ontologies in the documentation (0%)	−0.00	0.00

The table highlights multiple linear regression results for criteria related to findability. The first column includes criterion names, and the second and third columns indicate the coefficient and p-value of each criterion. Notably, The R² of the regression model is 0.75 and the adjusted R² is 0.72.