An Introduction to the Benchmarking and Publications for Non-Targeted Analysis Working Group

Abstract

Non-targeted analysis (NTA) encompasses a rapidly evolving set of mass spectrometry techniques aimed at characterizing the chemical composition of complex samples, identifying unknown compounds, and/or classifying samples, without prior knowledge regarding the chemical content of the samples. Recent advances in NTA are the result of improved and more accessible instrumentation for data generation, and analysis tools for data evaluation and interpretation. As researchers continue to develop NTA approaches in various scientific fields, there is a growing need to identify, disseminate, and adopt community-wide method reporting guidelines. In 2018, NTA researchers formed the Benchmarking and Publications for Non-Targeted Analysis Working Group (BP4NTA) to address this need. Consisting of participants from around the world and representing fields ranging from environmental science and food chemistry to ‘omics and toxicology, BP4NTA provides resources addressing a variety of challenges associated with NTA. Thus far, BP4NTA group members have aimed to establish consensus on NTA-related terms and concepts, and to create consistency in reporting practices by providing resources on a public website, including consensus definitions, reference content, and lists of available tools. Moving forward, BP4NTA will provide a setting for NTA researchers to continue discussing emerging challenges and contribute to additional harmonization efforts.

Graphical Abstract

Introduction

Non-targeted analysis (NTA), also referred to as “non-target screening” and “untargeted screening,” among several other related terms, is a theoretical concept that can be broadly defined as the characterization of the chemical composition of any given sample without the use of a priori knowledge regarding the sample’s chemical content. Some NTA experiments focus on the discovery of unknown chemicals using first principles and careful evaluation of experimental data. Other NTA experiments (often termed “suspect screening analyses”) aim to rapidly identify known chemicals using suspect lists with experimental data (e.g., reference spectra and metadata). Yet others aim to classify samples using detected chemical profiles (containing both unknown and identified chemicals). For the purposes of this Perspective, we will focus on non-targeted analysis conducted with gas or liquid chromatography coupled to mass spectrometry, with high resolution mass spectrometers being the most commonly used instrumentation. Applications of NTA include, but are not limited to, analysis of naturally occurring materials,¹ manufactured chemicals and materials,² manufactured consumer products (e.g., drugs³ and their interaction with environmental exposures in precision medicine,⁴ medical devices,⁵ and tobacco⁶), environmental media,⁷ food,⁸ and biological samples.⁹ A harmonized NTA framework will help facilitate various research objectives, including discovery,¹⁰ forensics,¹¹ hazard-based prioritization,¹² and support of regulatory decision-making.¹³

Interest in NTA applications has grown rapidly in response to the commercial availability of advanced mass spectrometers and novel data analysis tools, as evidenced by the steady increase in published studies on this topic over the past two decades (Figure 1). Accompanying its rapid development, there is growing recognition of the need to develop approaches and methodologies to both promote and assess the quality and confidence of NTA results. For example, Hites and Jobst recently wrote that “the criteria for reproducibility routinely applied to quantitative analyses are not as well defined for non-targeted screening,” and noted that more evaluation is necessary to assess the accuracy, precision, sensitivity, selectivity, and reproducibility of NTA.¹⁴

Figure 1.

Google Scholar trend analysis with two sets of search terms: 1) “nontarget analysis” OR “nontargeted analysis” OR “non-targeted analysis” OR “non-targeted screening” OR “nontargeted screening” AND “mass spectrometry” (bottom; blue striped) and 2) “untargeted screening” OR “untargeted analysis” AND “mass spectrometry”(top; orange solid). The search analysis was performed on February 8^th, 2021.

For targeted, quantitative analysis, there are existing guidelines for detection and quantification that are accepted by the research community. Compared to traditional targeted analysis of specific compounds, NTA is an emerging field with a lack of standardization and limited capabilities to assess and communicate performance, both of which impede the broader implementation and acceptance of NTA data. Meaningful evaluation of an NTA study’s performance is predicated on the existence of harmonized terminology and clear guidance about best practices for analysis and reporting results. Therefore, harmonized NTA guidance is imperative to promote high quality data and allow inter-study comparisons. Measuring how well a method identifies chemicals from the vast chemical universe is challenging; accurately communicating performance is arguably more difficult, since varying levels of identification confidence can be assigned.¹⁵ Moreover, in contrast to targeted methods, the presence or absence of individual compounds across a sample set is difficult to validate, limiting evaluation using a traditional confusion matrix (i.e., true/false positives/negatives) and associated performance metrics (e.g., precision, accuracy, false discovery rate). There are examples of NTA applications, such as sample classification, where sample classes can be clearly bounded and therefore confusion matrices and other performance metrics are calculable.¹⁶ However, there are still challenges associated with developing robust data analysis strategies and quality assurance/quality control (QA/QC) models for all types of NTA studies.¹⁷ For example, variability in NTA results can be artifacts driven by the selected preparation techniques, instrumentation, software, and user settings, rather than true sample differences, making it difficult to compare methods and results between instruments and/or laboratories.¹⁸ Although some recent studies have developed and implemented sample QA/QC procedures, there are no generally accepted community-wide QA/QC guidelines for NTA performance evaluation.^19–22

Previous efforts by other professional organizations in related research fields (e.g., metabolomics,^23–25 mass spectral data generation,²⁶ and non-targeted screening of water²⁷) have established guidelines and protocols for reporting results and determining method performance. Alternative methods of quantifying NTA method performance have been proposed by various researchers^28–31 but have not been widely adopted by research communities. Many government agencies have provided NTA guidelines for reporting compound identification.^32–34 However, it is challenging to establish guidelines generalizable to all NTA studies, as they can be specific to certain methods, matrices, and/or analytes. There remains a need for consistent definitions, as well as broadly applicable guidance for creating NTA studies and reporting method performance. Easily accessible, centralized recommendations and their widespread adoption are critical to enable accurate and reliable reporting and will facilitate the implementation of NTA beyond the research community.

Formation of the Working Group

In August 2018, the U.S. Environmental Protection Agency (EPA) convened a meeting to discuss the interim results from EPA’s Non-Targeted Analysis Collaborative Trial (ENTACT).³⁵ During the workshop, participants engaged in breakout discussions around several topic areas. Two topics (“Publication Issues” and “Proficiency Testing/Reference Methods”) merged into one discussion group, with members continuing to meet on a monthly basis. The group, now called the Benchmarking and Publications for Non-Targeted Analysis Working Group, or BP4NTA, currently has 100 North American and European members (February 2021). Members hail from three employment sectors, with about 25% from academia and industry, each, and 50% from government. Membership is voluntary with no formalized process except by written expressed interest through contacting group members or via the website. The group has garnered interest through presentations at relevant scientific conferences, including the American Society for Mass Spectrometry (ASMS), American Chemical Society (ACS), Society for Environmental Toxicology and Chemistry (SETAC), and Pittcon Conference & Expo, and by individual members’ research networks.

The overarching goals for BP4NTA are directly related to the community’s needs to:

• harmonize and/or standardize approaches and reporting practices, to the degree possible and practical;

• improve determination, calculation, and communication of performance metrics;

• share best practices (including QA/QC) within the NTA community; and

• improve the transparency and reproducibility of peer reviewed NTA studies.

During the workshop, and in subsequent meetings, the working group established short-term and long-term goals to address these needs. The short-term goals include:

• publishing a list of commonly used NTA terms, concepts, and performance calculations, with accompanying consensus definitions;

• designing and releasing a public study reporting tool to aid the design of NTA studies and the review of research proposals and manuscripts; and

• collating resources for new NTA researchers traversing the learning curve.

Long term goals for the working group include continuing to address gaps in data, methods, and computational tools within the community and moving the NTA field toward measurable standards for proficiency testing of non-targeted analytical laboratories. To facilitate these aims, the group plans to continuously build and maintain coalitions and communications with other groups that have similar interests, such as the NORMAN Network [https://www.norman-network.net/], the Metabolomics QA & QC Consortium [mQACC; https://epi.grants.cancer.gov/Consortia/mQACC/], and the Metabolomics Standards Initiative in Toxicology [MERIT; https://doi.org/10.1038/s41467-019-10900-y].

Development of NTA Reference Content and Tools

To achieve the short-term goals conceived by the first BP4NTA meeting, the members focused on the production of four primary deliverables:

1. a glossary of useful NTA-relevant terms, based on current literature and personal NTA experience;

2. reference content to support concepts mentioned in the definitions;

3. an interactive tool to aid researchers in consistently and transparently reporting NTA methods and results; and

4. a public website for novice and veteran NTA researchers to access the products described above and to gain knowledge of current and emerging NTA concepts.

In this paper, overviews of these four products are provided, although we refer readers to a concurrent manuscript on the Study Reporting Tool (the third deliverable) ³⁷ for additional details, and the website (the fourth deliverable) ³⁶, which can be accessed directly. The formats of these deliverables were designed to be dynamic and evolve with feedback and technological advances from the NTA community.

NTA Glossary.

To support a clear communication of methods and results by NTA researchers, reviewers, and journal editors, the members identified a collection of NTA terms for which harmonized definitions were needed. Arriving at consensus definitions proved challenging (even the term “non-targeted analysis” itself is often debated within the community, see Figure 1 caption), further demonstrating the need for harmonization. In total, the group defined 34 high-level terms that span all aspects of NTA workflows including sample collection/preparation, data acquisition/processing, results, and performance assessment. A subset of these terms is listed in Table 1. The final product of this effort is a comprehensive glossary of clear and concise definitions that have been reviewed by the working group and can be found on the BP4NTA website.³⁶ The BP4NTA definitions and reference content (subsequently described) are applicable to a variety of research fields. While experimental aspects such as sample types, complexity, and data quality may vary, the underlying framework is similar.

Table 1.

A subset of consensus definitions determined by BP4NTA using current literature and NTA expertise.

Term	Definition	Example
Feature	A set of grouped, associated m/z-retention time pairs (mz@RTs) that represent a set of MS1 components for an individual compound (e.g., an individual compound and associated isotopologue, adduct, and in-source product ion m/z peaks) or a single mz@RT if no such associated components are observed. A feature is represented as a tensor of observed retention time, monoisotopic mass, and intensity (e.g., peak height or peak area). Associated MS2 product ions may also be grouped with the MS1 components of a feature during HRMS data processing, depending on the software algorithms. The term “molecular feature” may also be used. The definition has been adapted from Schulze, Jeon, et al. 38	Feature as an individual mz@RT: m/z 287.8595 [M]+ at RT 12.1 min with an abundance of 1.52e7 Feature as a set of multiple mz@RT: m/z 287.8595 [M]+ at RT 12.1 min with an abundance of 1.52e7, with associated mz@RT pairs at m/z 251.8828 [M-HCl]+, and m/z 289.8566 [37Cl1M]+
Annotation of a feature	The attribution of one or more properties or molecular characteristics to an MS1 feature (or component thereof, such as an isotopologue or adduct), fragment ion, or MS/MS product ion.	Molecular formula (C6H6Cl6) Suggested product ion substructure (m/z 252 = loss of HCl, C6H5Cl5+)
Identification of a feature	The attribution of a specific compound, within a stated identification scope (or at a stated confidence level), to a detected feature(s), when the annotated components, features, fragments, or product ions provide enough evidence.	The feature was identified as hexachlorocyclohexane (specific compound) with no ability to determine individual stereoisomer (stated scope).

NTA Reference Content.

To support the use of cohesive terminology across diverse NTA research groups, the working group developed reference content that addresses key study design considerations and supplements the consensus definitions. The reference content was organized according to the chronology of a typical NTA study. For each section, working group members gathered and distilled relevant literature to describe current, published NTA practices. In addition, recommendations regarding good practices for performing and reporting NTA research are offered. For example, the Study Design section includes recommendations for describing and using blanks and quality control samples to enable assessment of background detections and analytical performance. Likewise, the Data Processing & Analysis section provides detailed recommendations on method aspects that, if reported, should promote reproducibility, including software tools, versions, and settings used to detect features and a comprehensive description (e.g., size, content, access date) of the mass spectral database(s) used for annotating features.

Study Reporting Tool.

The working group also developed the NTA Study Reporting Tool (SRT), a stand-alone, interactive tool for assessing the quality of reporting in NTA studies. The SRT was created to help: (1) current NTA researchers report their study methods and results in a consistent manner, and (2) reviewers and editors consistently and rigorously evaluate the content of NTA proposals and research manuscripts. The SRT was organized in the same structure as the reference content. Additionally, the SRT can be used to guide study design or as an educational framework for less-experienced NTA researchers. By aligning the organizational structure of the SRT and the reference content described above, researchers can readily use the two resources in tandem. The SRT is further described and evaluated by Peter, Phillips, et al. ³⁷ in a concurrent manuscript.

BP4NTA Website.

For the aforementioned information and tools to be useful, they must be widely available and amenable to updates given the rapid advancements in NTA technology. Therefore, the BP4NTA website [https://nontargetedanalysis.org/] was developed (Figure 2),³⁶ to provide public access to the glossary, reference content, SRT, and other NTA-related information without requirements for working group membership. In the online reference content, users can click on the organizational headers (identical to those in the SRT) to directly access detailed reference content on the relevant topic. The SRT webpage includes downloadable versions of the interactive tool and portals for community feedback. Beyond these materials, the website contains an extensive list of external NTA resources (including NTA software tools and online databases), a page for NTA job opportunities, information about the membership and history of the BP4NTA group, and the BP4NTA blog (which contains occasional updates on NTA-relevant news and publications). In addition to its accessibility, the website format allows ongoing updates to BP4NTA materials by members of the working group as the field evolves and provides a more interactive user experience than a single, static document. We believe this compilation of online resources will be extremely helpful for both beginning and experienced NTA researchers, as well as journal editors and peer reviewers who encounter manuscripts and proposals that include NTA.

Figure 2.

Screen shot of the Additional Resources/Software Tools page of the BP4NTA website³⁶ showing the website’s main menu (top) and a list of NTA software that is available for NTA researchers to explore (access date: 11/10/2021).

Challenges, Recommendations, and Outlook

The resources developed thus far by BP4NTA provide valuable information for experienced and novice NTA researchers alike. As new NTA methods (both analytical and computational) are developed, it is critical to assess their performance against current best practices. Such assessments can require running samples on multiple instruments or through multiple data analysis pipelines, yielding both time and cost considerations. To meet this challenge, the community should push for (1) open sharing of raw NTA data so new computational approaches can be compared using identical data, (2) additional assessment studies of blinded samples with known composition (akin to ENTACT^{39, 40} and CASMI⁴¹), (3) adoption of common reporting formats and transparent sharing of metadata, (4) adoption of common QA/QC practices, and (5) the development of comparison tools to support automated performance assessment.

Additionally, the NTA community would benefit from a better understanding of the relationship between identification confidence and false discovery rates or the probability of presence. However, to date, there has not been a rigorous assessment to correlate these concepts. To help meet this challenge, the community should (1) openly report the analytical, computational, and subject-matter expert based evidence used for each compound identification, (2) determine methods to directly calculate false discovery rate (FDR) and identification probabilities, and (3) add additional dimensions of evidence that may ultimately lead to feature sets that are unambiguous (i.e., unique to each compound in the entire molecular universe). The community is shifting to address this gap quickly, with decoy libraries and approaches similar to proteomics-inspired FDR assessments now appearing in literature.^28–31 Furthermore, the combination of ion mobility spectrometry (including gas-phase chiral separation⁴²) and cryogenic infrared spectroscopy^43–45 with traditional LC-MS/MS, as well as microcrystal electron diffraction⁴⁶ methods, are emerging tools for providing unique identifiers for molecules.

Moreover, we need to continue developing novel methods for identifying compounds. “Standard-free” identification makes use of computational prediction of molecular properties to build identification libraries without the use of chemical standards. This approach is an improvement on traditional approaches, such as the use of acquired mass spectral libraries, because authentic chemical standards are not available for the vast majority of small molecules.^{47, 48} However, the community has yet to standardize a consistent approach to include in silico libraries as part of identification evidence and to implement these libraries into existing databases. Currently, there is not a clear method to combine information from multiple instruments and multiple libraries into a single identification score or probability. To address this challenge, the community should move to (1) determine the best approach for utilizing this information to bolster the evidence for molecular presence, (2) continually improve predicted property accuracy, and (3) form a deeper understanding of experimental error associated with this evidence (computational or empirical).

Finally, one of the shortcomings of NTA approaches is the inability to provide fully quantitative data (i.e., concentration) for identified compounds, when relative abundance is not sufficient. This is largely due to the absence of authentic standards and methodological compromises necessary to detect a broad swath of chemical space. Accurate semi-quantitative approaches have proven challenging due to the wide disparity in analytical response factors.^{49, 50} Existing studies have focused on using modeling⁵¹ or surrogate compounds (i.e., isotopically labelled spiked compounds) to predict response factors for compounds identified with NTA.^{52, 53} To address this challenge, the community should: (1) develop semiquantitative estimation models or quantitative measurement models, (2) understand the estimation error and determine proper guidelines for reporting how concentration was determined, and (3) standardize the detection limit concept in NTA (e.g., instrument specific limit of detection (LOD) vs. NTA method LOD) when detected compounds are not available as standards.

Moving Forward

Because the BP4NTA working group is organized and operated by volunteers, we feel it is wise to fit ourselves within a larger scientific society structure that will allow a mechanism for recruiting efforts and maintenance of membership rolls. While no official collaborative relationship has been established, representatives from BP4NTA have engaged in discussions with other groups (e.g., mQACC, NORMAN, Compound Identification Development Cores (CIDC)), and our efforts are intended to complement ongoing harmonization efforts by other NTA-related organizations.

Representing a significant effort by working group members, the SRT is an important tool that we believe, with widespread adoption, will benefit a variety of roles within the field. To that end, in addition to this manuscript and a thorough evaluation of the SRT,³⁷ we are reaching out to journal editors to explain the tool, its intent and limitations, and the potential benefit to authors, reviewers, and editors.

Emerging technologies for NTA are routinely developed and there is no intention that the reference content initially presented by this communication or the BP4NTA website will remain static. As the working group receives feedback from the NTA community, and as needs for NTA harmonization evolve, the website will be regularly maintained and updated. To provide input or suggestions, researchers are encouraged to use the form on the website (or the comment box on the SRT page for SRT-specific feedback) or contact the corresponding author of this communication. As NTA techniques advance, it is expected that recommendations for comprehensive reporting will transform accordingly. Therefore, the SRT will remain flexible and evolve with changing technology. The SRT available on the website will be updated annually, although static versions of the document will remain available for download.

While it seems to be a herculean effort, the harmonization of NTA methods and results reporting should have a substantial impact on the quality and outcomes of NTA research. These powerful and ever-advancing capabilities, data processing tools, and molecular databases/libraries will encourage more researchers to enter the field and apply NTA methodologies to unique research problems, making it important to create and share resources such as those from BP4NTA and related groups. While much work remains, the information and tools provided by this group can improve the communication of work being accomplished by the community. Through our efforts, we hope to empower researchers with tools and knowledge for presenting reproducible, transparent, and impactful NTA research.