A Review of the Scientific Rigor, Reproducibility, and Transparency Studies Conducted by the ABRF Research Groups

Abstract

Shared research resource facilities, also known as core laboratories (Cores), are responsible for generating a significant and growing portion of the research data in academic biomedical research institutions. Cores represent a central repository for institutional knowledge management, with deep expertise in the strengths and limitations of technology and its applications. They inherently support transparency and scientific reproducibility by protecting against cognitive bias in research design and data analysis, and they have institutional responsibility for the conduct of research (research ethics, regulatory compliance, and financial accountability) performed in their Cores. The Association of Biomolecular Resource Facilities (ABRF) is a FASEB-member scientific society whose members are scientists and administrators that manage or support Cores. The ABRF Research Groups (RGs), representing expertise for an array of cutting-edge and established technology platforms, perform multicenter research studies to determine and communicate best practices and community-based standards. This review provides a summary of the contributions of the ABRF RGs to promote scientific rigor and reproducibility in Cores from the published literature, ABRF meetings, and ABRF RGs communications.

INTRODUCTION

Advances in biomedical research–an increasingly complex collaboration of both basic and clinical science–are driven by improvements, innovations, and breakthroughs in technology. The application of technology requires significant expertise, as well as implementation of best practices for scientific rigor and transparency, and the acquisition of equipment, instruments, and reagents to make it possible.¹⁻⁴ This increasing sophistication, combined with significant financial investment, has been the impetus for the growth of Shared Research Resource Laboratories (Cores) in academia, government, and industry.⁵⁻⁷ Shared research resources are highly valued for making efficient use of research funds and broadening access to advanced technologies.⁸⁻¹¹ Working with Cores, the research community can more effectively promote rigorous research practices, quality technical training, and collaborative research.⁶ The Association of Biomolecular Resource Facilities (ABRF) is an international scientific society dedicated to advancing shared research resource core laboratories through research, communication, and education.¹² ABRF represents over 700 members from 19 countries working within or in support of Cores in government, academia, research, industry, and commercial settings, representing a collaborative knowledge base to advance the adoption of more rigorous, reproducible, and transparent research practices.

Recently, well-publicized allegations of the inability to reproduce published biomedical research¹³⁻¹⁶ have elicited discussions and examination within the research community and among public stakeholders.¹⁷⁻²¹ Federal agencies that support research activities, including the U.S. National Institutes of Health (NIH) and the National Science Foundation, have implemented new policies to address these concerns and improve communication of critical experimental details within the research community and to the public.²²⁻²⁴ The National Academy of Sciences also released guidelines on Research Reproducibility and Replicability in Science.²⁵ In response, the ABRF Committee for Core Rigor and Reproducibility conducted a survey to gain information on how NIH initiatives on advancing scientific rigor and reproducibility influenced current shared resource services and new technology development and to identify the challenges and opportunities related to implementation of new reporting requirements.²⁶ The survey results supported that core scientists are deeply invested in supporting transparency and scientific reproducibility. By employing best practices, protecting against cognitive bias in research design, employing full transparency in reporting experimental details and authentication of key resources, Cores directly addresses 2 of the 4 areas of improvement identified by the NIH.^{22, 26} As responsible stewards for the majority of research data produced at many institutions, with 94% of all core scientists trained in the operation and performance of instrument and equipment in their laboratories, Cores have a fundamental role in scientific excellence.^{5−7, 26} Recognizing this, federal granting agencies have already made significant investments in Cores through direct and indirect mechanisms, with the goal of providing cutting-edge technologies and expertise to individual scientific investigators.²⁷

Collectively, and synergized through membership in ABRF, core scientists contribute to best practices, data reproducibility, and consensus guidelines for standard operating procedures, data standards, and quality analysis and quality control (QC).^{28, 29} ABRF was founded on these premises,³⁰ its mission to both define and support best practices and scientific excellence and reproducibility in biomedical discovery through research, communication, and education. The hallmarks of ABRF, distinguishing it from all other professional scientific societies, are the ABRF-sponsored multisite Research Group (RG) studies. RG study participants prepare more effectively for change in the analytical environment and stay current with best practices for scientific rigor and reproducibility. In conjunction with the ABRF web-based discussion forum (http://list.abrf.org/), both contribute to the education of resource core directors, scientists, administrators, customers, and interested members of the scientific community.

Because scientific research advances are driven by the temporal nature of technology, scientific rigor is the foundation that drives data quality. Reviewed here are the ABRF RG contributions and current efforts in supporting these tenets, spanning protein primary structural analysis to single-cell transcriptomics.

ABRF RGs

The concept of developing test samples for Cores to use for methodology and proficiency assessment was initiated in 1986 at the Sixth International Conference on Methods in Protein Sequence Analysis Research Resource Facility Satellite Meeting (which later evolved formally into what we now know as the ABRF). A group of attendee scientists with an interest in assessing the reproducibility and rigor among laboratories performing Edman sequencing and amino acid analysis distributed test samples to 103 laboratories, thus establishing the first ABRF RG.³¹ Today, the ABRF RGs are organized by ABRF members who are experts in their respective disciplines and who design studies relevant to the prevailing state of the technology to inform scientific rigor in research performed across national and international academic, government, and industry laboratories. These objectives are met by designing and distributing study materials through several means: 1) provide a protocol for self-evaluation of scientific accuracy and procedural efficiency; 2) contribute improvements to instrumentation, methods, and instrument operation/maintenance by identifying limitations or common problems encountered in laboratories in the field; 3) determine best practices for standard operating procedures, and 4) define realistic expectations for both core scientists and users.³² Through participation in the multisite RG studies, participants are more effective at addressing inevitable changes, improvements, and upgrades in the analytical environment and remain current with best practices for scientific rigor and reproducibility.³³

The first decade of ABRF RG studies focused on the advancement of technologies and education of the scientific community on the best methods and strategies for the hot topic of research focus at the time—protein primary structure and function analysis. These early RGs [Edman Sequencing RG (ESRG), Protein Sequencing RG (PSRG), and related Internal Sequencing RG studies] provided rich detail on the scientific rigor demanded for the cutting-edge technologies of this era.³⁴⁻⁵³ A critical adjunct to the ESRG was the Proteomic Standards RG (sPRG), which focused on designing and producing peptide standards.⁵⁴⁻⁵⁷ The sPRG produced 3 synthetic peptides that were registered and certified as reference standards, which is important work done in collaboration with the National Institute of Standards and Technology (NIST). A significant asset to the scientific community at large, these peptides are commercially available.⁵⁸

A cornerstone of quantitative protein characterization, amino acid analysis was an early methodology and technology focus for ABRF. The amino acid analysis RG members detailed advances in the detection and sensitivity of amino acid analysis, tracing technology platform evolution, and ion-based chromatographic methods to mass spectrometry analyses.⁵⁹⁻⁷⁴ And the PSRG was instrumental in bringing improvements to synthetic chemistry methods, most significantly leading efforts for research and methods for introducing fluorenylmethyloxycarbonyl chloride amino acid chemistry (FMOC) to the forefront.⁷⁵⁻⁹⁰

Later RG efforts and publications focused on determining best practices for analyzing post-translational modifications (Carbohydrate Analysis RG, Glycoprotein RG)^{91, 92} and establishing quality measurement standards for analyzing molecular interactions (protein-protein; protein-ligand) (Molecular Interactions RG).⁹³⁻⁹⁷

Individually and as a valuable resource in aggregate, the ABRF RG studies remain highly referenced for technology applications and methodology. New studies continue, driven by new technologies and the need for robust methods and protocols and building on the success of the early RGs. The current ABRF RGs encompass 4 broad, technology-based categories integral to modern bioresearch: 1) genomics, 2) bioinformatics, 3) mass spectrometry (proteomics, metabolomics), and 4) imaging (microscopy and flow cytometry) and are reviewed here.

RESULTS

Genomics

The field of genomics has exploded over the past 40 yr, with the rapid development of next-generation technologies spawning multiple generations of sequencing platforms, each improving on cost, speed, and data quality, even portability, as exemplified by recent sequencing data performed on the International Space Station.⁹⁸ Although these technology advances have significantly impacted our understanding of biology and disease, the sheer volume of new data that these technologies produce have created tremendous challenges related to data sharing and data management. As a consequence, standardized procedures and data formats as well as comprehensive quality management considerations are at the forefront.⁹⁹

The current Genomics RGs (GRGs) represent 5 overlapping subspecialties representative of critical genomics based technology and applications: 1) next-generation sequencing (NGS), 2) Genome Editing RG (GERG), 3) DNA Sequencing RG (DSRG), 4) GRG, and 5) Metagenomics RG (MGRG). The predecessor genomics-related RGs researched methods and applications for microarrays, fragment analysis genotyping, and nucleic acid analysis [Nucleic Acids RG (NARG)], finally merging to form the current GRG. The mission of all the genomics-related RGs is to analyze, establish, and disseminate standards and best practices for Cores in the fast-evolving field of genomics technologies and applications.

Next-Gen Sequencing RG

The NGSRG is dedicated to educating ABRF members and scientific community in support of scientific rigor and data reproducibility, identifying optimal methods and strategies for NGS projects as well as performance evaluation of the ever-evolving NGS platforms. To that end, the 2018 ABRF-NGS study was a coordinated multi-RG effort by 4 RGs [NGSRG, DSRG, GRG, and Genomics Bioinformatics RG (GBIRG)] and over 20 Cores. The long-term goals of the study are to optimize the detection of genetic variation with the latest sequencing tools, establish a community resource for self-evaluation and self-improvement, and improve sequencing performance while evaluating existing and emerging protocols and platforms. Previous RG studies focused on RNA sequencing (RNA-seq), using standard Microarray Quality Control Consortium (MAQC) total RNA samples combined with External RNA Controls Consortium synthetic spike-in RNA.^{100, 101} This RNA profiling phase of the ABRF-NGS Study included the Illumina HiSeq 2000/2500 and MiSeq (Illumina, San Diego, CA), Roche 454 GS FLX+ (Roche, Basel, Switzerland), Life Technologies Ion Torrent PGM and Proton (Thermo Fisher Scientific, Waltham, MA, USA), and Pacific Biosciences PacBio RS (Pacific Biosciences, Menlo Park, CA) platforms. The Phase 1 ABRF-NGS Study (2012–2015) assessed sequencing accuracy, absolute and relative expression levels, RNA splice junction detection, and differential expression detection between samples. In a similar fashion, the current Phase 2 ABRF-NGS Study (2016–2020), performed in collaboration with the NIST Genome in a Bottle Consortium,¹⁰² the U.S. Food and Drug Administration’s Sequencing Quality Control Consortium¹⁰³ and other sequencing community stakeholders will utilize standardized reference genomic DNA samples.

High-throughput RNA-seq greatly expands the potential for genomics discoveries, but the wide variety of platforms, protocols, and performance capabilities has created the need for comprehensive reference data. Here we describe the ABRF-NGS study on RNA-seq: Replicate experiments across 15 laboratory sites were performed using reference RNA standards to test 4 protocols (poly-A–selected, ribo-depleted, size-selected, and degraded) across 5 sequencing platforms (Illumina HiSeq, Life Technologies PGM and Proton, Pacific Biosciences RS, and Roche 454).¹⁰⁴ The results showed high intraplatform (Spearman rank R > 0.86) and interplatform (R > 0.83) concordance for expression measures across the deep-count platforms but highly variable efficiency and cost for splice junction and variant detection between all platforms. For intact RNA, gene expression profiles from ribosomal ribonucleic acid (rRNA) depletion and poly-A enrichment were similar. In addition, rRNA depletion enabled effective analysis of degraded RNA samples, new approaches for correcting batch effects in large-scale studies¹⁰⁵ and more efficient detection of splice isoforms.¹⁰⁶ These RG studies provide a broad foundation for cross-platform standardization, evaluation, and improvement of RNA-seq^{107, 108} and have led to other standards in genomics, metagenomics, and multiomics.¹⁰⁹

Genome Editing RG

With increasing demand for CRISPR/Cas9 technology, new or existing Cores have adapted their services to fit this technology into their workflows. In the GERG 2017 survey,¹¹⁰ plasmid format and lipofection delivery were favored among cell culture users. The 2018 GERG study evaluated the reproducibility of indel formation rates by comparing guide RNA format and delivery methods across participating labs. Various configurations of guide RNA and Cas9 components can be used for editing cells, including 1) a plasmid expressing both the guide RNA and Cas9, 2) Cas9 protein combined with a synthetic single guide RNA, and 3) Cas9 combined with a synthetic 2-part guide RNA. The study evaluated the cutting efficiency at 3 different guide RNA targets based on the guide RNA format using 2 delivery methods (lipofection or nucleofection transfection methods). The study results suggest the ribonucleoprotein (RNP) format for the guide RNA and Cas9 is gaining in popularity in combination with nucleofection delivery. The GERG RG is finalizing the results of the study to determine the most reproducible method (Sergison, E., Regan, M., Delventhal, K., Gurumurthy, C., Kmiec, E., Pruett-Miller, S., Dahlem, T., Marsischky, G., unpublished results).

DNA Sequencing RG

The DSRG conducts studies to assess the capabilities of DNA sequencing technologies, protocols, kits, and reagents; provides a means of self-evaluation for sequencing technologies to evaluate their own performance; disseminate the findings while still relevant; and promote communication and cooperation among laboratories that perform Sanger sequencing and NGS. The ABRF NARG, a precursor to the DSRG, first conducted a comparison of DNA sequencing technologies in 1995.^111–113 In the late 1990s, the ABRF DSRG established a series of long-term studies, called in aggregate The Never Ending Story, designed to evaluate the performance of Sanger sequencing instruments.¹⁰⁸ The DSRG 1997 study involved sending a guanine-cytosine (GC)-rich DNA template to Cores that offered DNA sequencing as a service.¹¹⁴ The objective was to evaluate whether chemical additives, altered thermocycling conditions, and analysis methods could improve the sequence obtained from a GC-rich template. A total of 134 Cores participated in this study. The study demonstrated that manual review and editing of data generated by automated sequencing instruments had the most impact in improving sequencing accuracy, using the automated analysis algorithms available at that time. Continuation of this study resulted in definition and dissemination of best practices for sequencing high GC templates.¹¹⁵

The second in The Never Ending Story series assessed the state-of-the-art in DNA sequencing to create a publically available quality control resource. Sequencing groups submitted unedited sequence for a common standard template [i.e., bacterial plasmid cloning vector pGEM template with the M13 (−20) forward primer] using the instruments, reagents, and protocols common in their core and also included all metadata (e.g., chemistry type and concentration, instrument type and run conditions, and analysis algorithms and methods). Data were collected electronically and analyzed based on sequencing accuracy, read length, signal sensitivity, enzyme dilution tolerance, and ease of use. The aim of establishing this readily updatable, online benchmark resource for self-evaluation was to promote high standards and reproducible results for DNA sequencing in core laboratories. More than 300 Cores participated in this study.¹¹⁶

The Never Ending Story continued with DSRG studies in 1999, 2000, and 2001, evaluating the outcomes of different DNA sequencing methods on sequencing both standard and difficult templates to expand the web-based quality control resource and produce a defined standard test array of difficult-to-sequence templates.¹¹⁷ The Never Ending Story continued in 2003, 2005, 2006, and 2008, with DSRG studies that evaluated and defined best practices for sequencing through difficult-to-sequence DNA templates containing extensive repetitive sequences.¹¹⁸ In 2019, The Never Ending Story was rebooted with a DSRG study on Sanger-based sequencing best practices in response to the release of new Sanger sequencing dye terminator chemistries. This study was a cross-site evaluation of Sanger sequencing chemistries, designed to evaluate the performance of sequencing both standard and difficult-to-sequence templates using a variety of legacy and new-to-market Sanger sequencing chemistries and provide guidance to Cores in regard to modifying existing sequencing protocols to further the quality and robustness of their Sanger sequencing production pipelines.^119−129

Recent DSRG studies expanded the rigorous assessment of sequencing reagents to NGS modalities as well, with an eye toward implications for the selection of kits for specific experimental contexts. Multisite projects included the following: a robust evaluation of small RNA NGS library sequencing kits with respect to accurate representation of microRNAs (miRNA) differential expression, retention of small RNA and other noncoding RNAs (ncRNAs), technical replication, conservation of material, and ease of use across 11 different ABRF member sites (Herbert ZT, Thimmapuram J, Xie S, et al., unpublished results); an assessment of ribosomal reduction protocols for human/mouse/rat RNA upstream of NGS sequencing in order to determine effectiveness at retaining messenger RNA (mRNA) and ncRNA components of a total RNA specimen for sequencing while minimizing the intervening and noninformative rRNA and transfer RNA (tRNA) components¹³⁰; and a comparison of methods designed for high throughput enzymatic fragmentation of DNA prior to next-gen library generation to assess resulting library complexity, bias in sequence and genomic context, and site-to-site consistency (DSRG posters and presentations). These and an array of RG studies with broad participation have driven improvements in quality, accuracy, and reproducibility for DNA and RNA sequencing provided by Cores.^131−147

Genomics RG

The GRG members provide both academic and industrial scientists useful information and guidance in the use of various microarray, NGS, and other genomic platforms and applications in their research. Earlier RGs that focused on microarrays, fragment analysis genotyping, and nucleic acid analysis (NARG) have merged into the GRG. These RGs performed foundational studies on the evaluation and application of microarrays^148−160 and fragment analysis.^{161, 162}

More recent GRG studies have focused on single-cell genomics, a field that is rapidly evolving in both platform technology and analytical methodology. To assess variability and determine best practices across various leading platform technologies, the 2017 GRG study evaluated the key technologies for single-cell RNA-seq. The platforms chosen were Takara’s iCell8 (Mountain View, CA), Fluidigm’s C1 AutoPrep (South San Francisco, CA), 10X Genomics Chromium Controller (Pleasanton, CA), and Illumina/BioRad’s ddSEQ joint venture. A well-characterized triple-negative breast cancer cell line was distributed to multiple labs for analysis in duplicate across all platform technologies to assess correlation with bulk RNA-seq data, assess reproducibility, and evaluate concordance of expression results using mock vs. drug exposure. Results further understanding of the limitations of each technology, furthering more rigorous experimental design in platform selection (Fournier, C., Reyero-Vinas, N., Ashton, J., Jen, J., Boswell, S., Chittur, S., Mason, C., Rehrauer, H., Steen, R., unpublished results). The 2018 GRG study, performed in collaboration with the American Natural History Museum, assessed accuracy and reproducibility of several long-read sequencing platforms, evaluating PacBio Sequel, Oxford Nanopore Technologies GridION, 10X genomics linked read technology, and Illumina’s MatePair chemistry. An outcome goal of this study was the creation of a full genome reference of an endangered parrot for conservatory purposes (Fournier, C., Reyero-Vinas, N., Ashton, J., Jen, J., Boswell, S., Chittur, S, Mason, C., Rehrauer, H., Steen, R., unpublished results).

Metagenomics RG

The MGRG was created in response to the rapidly growing field of metagenomics by a team of scientists with backgrounds in microbiology, genetics and genomics, bioinformatics, oceanography, geochemistry, planetary sciences, climate research, and extremophile research.¹⁶³

Early ABRF studies on metagenomics were performed by the DSRG and GRG.¹⁶⁴ International reference samples used and characterized by the MGRG are used to assess the performance of impact of protocols like whole-genome amplification¹⁶⁵ and help individual laboratories compare their local results with those of the larger research community or improve clinical implementation of “precision metagenomics”¹⁶⁶. The tested, titrated bacteria/fungal mixtures are now continuously used in a wide range of environments¹⁶⁷ for both genetic and epigenetic applications.¹⁶⁸ Reference standards facilitate the development of much needed peripheral reagents including high-performance DNA extraction kits, complex enzyme mixes for microbial lysis, nucleic acid-free sample concentrators, and bioinformatic pipelines. Improvements in metagenomic methods will ultimately benefit from the availability of standardized reference samples that represent the range of organisms potentially present in samples from the field.¹⁶³

The MGRG has initiated a novel microbiome project Extreme Microbiome Project¹⁶⁹ to characterize organisms from extreme environments around the world. Further goals of the MGRG include assembling microbial standards and characterize shortcomings of current metagenomic techniques¹⁷⁰ and comparison to older techniques in microbiome research like 16S rRNA sequencing¹⁷¹ including optimizing DNA extraction protocols, library synthesis methods for different NGS platforms and developing best approaches for bioinformatics. The RG also works very closely with the appropriate vendors of the field to help advance this technology for metagenomics while developing low-input RNA-seq to further enable metatranscriptomics.^{172, 173} The development of a bacterial counting platform (similar to simple low-cost mammalian cell counters), in partnership with specific vendors, is needed for research and clinical metagenomic quality control requirements. This enumeration device will be validated against standard techniques such as flow cytometry, microscopic, and light scattering chromatography (LSC) techniques.

Bioinformatics

Advances in the technologies and informatics used to generate and process large biologic data sets (“omics data”) are promoting a critical shift in the study of biomedical sciences and a need for interdisciplinary data integration strategies to support a better understanding of biologic systems. Analysis of genomics, transcriptomics, and proteomics data are still primarily analyzed individually with distinct approaches generating monothematic rather than integrated knowledge. Computational methods for data management, algorithms for statistical pattern inference and recognition, and data integration are necessary for the integrated or constructionist view of biology. Coupled with the increase in metabolomics, epigenomics, and pharmacogenomics data needs, the ABRF members have formed 2 RGs to exemplify and evaluate computational methods for bioinformatics (data exchange and management) and quantitative mathematical modeling to meet current needs in data analysis and sharing.

Genomics Bioinformatics RG

The goals of the GBIRG are to 1) provide collaborative bioinformatics and bio-information technology (bio-IT) support for ABRF genomics-focused RG studies, 2) explore collaborations with the ABRF proteomics bioinformatics focused RGs, and 3) investigate questions of interest to all genomics-focused bioinformatics and bio-IT Cores by creating surveys of bioinformatics and bio-IT Core management and funding models, conducting studies of computational biology analysis tools and data management methods issues, and identifying best practices. Most recently, GBIRG has supported a multiplatform assessment of transcriptome profiling using RNA-seq with the ABRF Next-generation Sequencing Study group.¹⁰⁴

Proteome Informatics RG

The mission of the ABRF Proteome Informatics RG (iPRG) is to educate ABRF members and the scientific community on best application and practice of bioinformatics toward accurate and comprehensive analysis of proteomics data. The iPRG members actively support and participate in the development and advancement of new algorithms, software tools, and strategies for proteome informatics with the goal of both educating and introducing these technologies to the membership. The iPRG research studies^174−178 have typically started by generating ground truth data sets to challenge and benchmark commonly used algorithms and statistical methods in proteomics. Several of these data sets have seen reuse independently from the iPRG study for which they were generated, demonstrating the lasting value of reference data from well-designed experiments.^179–181 All past studies, including the data sets and their documentation, are archived and available on the ABRF website.¹⁸²

Proteomics, Metabolomics, and Proteomics Standards RGs

Edman sequencing dominated the early RG studies, the only method at the time for determining the primary structure of proteins. With the advent of mass spectrometric instruments, methods evolved to provide rapid and sensitive qualitative and quantitative analysis of biomolecules (proteins, peptides, oligosaccharides, metabolites, lipids, DNA, and RNA). The PSRG and Protein Identification RG supplanted the ESRG.

Significant advances in mass spectrometry (MS) platform technology and informatics data analysis increased sensitivity and throughput, supplanting earlier chemistry-based technologies.¹⁸³ The PSRG studies focused on N- and C-terminal sequence analysis of proteins by any technology yielding information about the termini of proteins.^184−197

The current Proteomics, Metabolomics, and Proteomics Standards RGs studies are designed to examine reproducibility and other capabilities of emerging techniques to establish best practices for scientific excellence. The Mass Spectrometry RG, which preceded the current Proteomics, Glycoprotein, and Metabolomics RGs (MRGs), contributed the Delta Mass utility website still employed today (Delta Mass). However, our current knowledge that more than 21,000 human genes¹⁹⁸ may code 1 million¹⁹⁹ or more proteoforms leaves ample room for development.²⁰⁰ New methodologies, coupled with transcriptomics data and MS/MS peptide sequence analysis at the subpicomole level have enabled a multiomics approach to protein identification for biologic and disease studies within a given biologic sample. Recent advances in mass spectrometry have clearly revolutionized the studies of post-translational modifications and include the development of specific strategies to preferentially enrich modified amino acids via covalent modifications incorporating affinity tags.

Proteomics RG

The PRG is dedicated to sharing knowledge about the analysis of proteins in support of scientific rigor and data reproducibility. The 2018 PRG study is focused on enabling mass spectrometry–based proteomics laboratories to use Data-Independent Acquisition (DIA) technology. For this study, the PRG reached out to laboratories around the world. Sixty-four laboratories from 16 countries (and 20 U.S. states) were provided with a test sample set, protocols, and resources to facilitate the use of data independent acquisition (DIA). Forty-one participants (64% of sample recipients) deposited the raw data for PRG to analyze. Almost half of the participants in the study were new to DIA (Jagtap, P., Herring, L., Midha, M., Martin, R., Neely, B., Phinney, B., Shan, B., Stemmer, P., Wang, Y., unpublished results). The previous 2 studies conducted by the PRG were based on detection and quantitation of low-abundance proteins in a highly complex sample. In the 2016 PRG study, data submitted from study participants was used to measure intralaboratory variation in liquid chromatography-mass spectrometry (LC-MS) performance to determine the types of QC procedures implemented in proteomics laboratories and identify the elements of system design/setup that correlate with variability. The results showed variability in the identification of lower-abundance spiked-in proteins from different laboratories and demonstrated a significant advantage of performing fractionation on complex samples to detect proteins at an extremely low concentration. Unexpectedly, when identification files provided by the participants were reanalyzed for validation of self-reported values, significant differences in participant-reported and study-validated values were found.²⁰¹

The 2017 PRG study was a member-only study on the quantification of unidentified low-abundance proteins with mass spectrometry spectra (MS1) data and bioinformatics tools. In this follow-up of the 2016 PRG study, retention time and accurate mass of peptides with relatively high concentration (500 fmol in 25 ug cell lysate) were employed to quantify the same peptides in samples with low concentration (20 fmol in 25 μg cell lysate). Four software programs (2 open source and 2 commercial) were used to analyze data set from Orbitrap Fusion, Q Exactive, and Orbitrap Velos instruments (Thermo Fisher Scientific). All evaluated software programs extracted quantitative information from MS1 spectra that did not yield peptide spectral matches in samples with low concentrations of spike-in proteins. False quantification of peptides in the zero spike-in sample was observed. This was attributed to carryover between runs and misassignment of noise in the signal.²⁰²

Proteomics Standards RG

The mission of the sPRG is to promote and support the development and use of standards in proteomics and committed to identifying and implementing technical standards for accuracy, clarity, and consistency, supporting ABRF’s commitment to scientific rigor and reproducibility. Examples of technical standards include, but are not limited to, reference materials, data sets, conditions, and procedures that give proteomics researchers and analysts’ independent criteria to evaluate their abilities to produce predictable, consistent results.⁵⁴⁻⁵⁸ The sPRG strongly supports ongoing efforts for standardization of the recording and reporting of proteomics experiments. The 2018 sPRG Study delivered a new heavy-labeled phosphopeptide standard and a designed study to test the ability of proteomics labs to detect endogenous phosphopeptides (Herren, A., Lee, K., Searle, B., Patel, B., Leib, R., Chien, A., Hawke, D., Koller, A., Isovev, G., Neeley, B., unpublished results).

Metabolomics RG

Metabolomics is the comprehensive profiling of metabolites and other small molecules. The large structural diversity of these compounds makes both comprehensive profiling and identification challenging. With nuclear magnetic resonance (NMR) and MS being the major platforms used, there are a variety of approaches, including untargeted profiling, targeted approaches, and fluxomics. Challenges include identification of metabolites, how they change in relation to a biologic perturbation (e.g., drug, diet, disease) and the biologic significance of these changes.

The goals of the MRG are 1) to educate research scientists and resource facilities in the analytical approaches and management of data resulting from comprehensive metabolite studies and 2) to promote the science and standardization of metabolomic analyses for a variety of applications.²⁰³ The 2013 MRG study assessed the ability of laboratories to conduct successful untargeted and targeted metabolomics analyses. The study sample was human plasma spiked with different amounts of metabolite standards in 2 groups to emulate a typical metabolomics study. Results from the 2013 MRG study highlighted current challenges in the field that include missing benchmarks for comparing different methodological approaches and analytical platforms to enable cross-laboratory reproducibility.²⁰⁴

The 2016 MRG study focused on metabolomics data analysis methods—bioinformatics and statistical approaches for pre- and postprocessing of global profiling data sets, and assessed interlaboratory reproducibility, a major concern that we tried to assess by providing study participants with already-acquired mass spectrometry data. The task was to detect feature level differences between 2 groups to shed light on the contribution of data pre- and postprocessing methods on metabolomics analysis results. A major conclusion from this study identifies proper data preprocessing as a critical step of the data analysis workflow when using untargeted metabolomics profiling.²⁰⁵

We believe that consolidation of validated metabolomics methodologies and benchmarking standards of use and reporting will further augment routine and widespread practice of this powerful and cost-effective technology with many applications in the life and biomedical sciences.

Light Microscopy and Flow Cytometry RGs

Two diverse technology areas focused on capturing images of cell populations, single cells, cells represented in tissues and organelles, entire organisms, and imaging at the molecular level (which may or may not be within a cell) are represented in the Light Microscopy RG (LMRG) and Flow Cytometry RG (FCRG). Few technologies are more widespread in modern biologic laboratories than optical imaging techniques using light emitted through fluorescence or bioluminescence, and new imaging techniques include optical coherence tomography, multiphoton microscopy, total internal reflection fluorescence provide ever-increasing ability to monitor biologic phenomena with higher resolution, specificity, dimensionality, complexity, and scale, all while maintaining viability and biologic relevance. These imaging modalities are increasingly multiparametric and rely heavily on computational approaches, which are in many cases nearly as important as the optics, not only for automating and optimizing image acquisition but also for visualizing and analyzing the data.

Light Microscopy RG

Relative to the other technologies represented by the ABRF, Light Microscopy and Flow Cytometry are unique in their involvement of the individual researcher and therefore unique in their challenges to reproducibility. Light Microscopy Cores typically train individual researchers who then use the core instruments independently to collect their data, in contrast to other types of Cores in which trained Core staff perform or oversee most data collection. Light Microscopy Core users may range widely in expertise, from novice undergraduate and graduate students to more highly experienced postdocs and staff scientists. Moreover, many research laboratories may have their own microscope systems (potentially including “advanced” instrumentation such as confocal and superresolution systems) and may conduct their entire study without ever setting foot in a Core. Finally, light microscopy is, by its nature, an interactive technology in which the user is intimately involved in the data collection process. Microscope users select which regions of a specimen to image and have expansive control over the appearance of those images. Experienced microscopists are well aware of the ethical pitfalls of having such control; equally important, however, are its impacts on reproducibility.²⁰⁶

Given the highly individualized and highly decentralized nature of light microscopy–based research, improving reproducibility necessitates providing both Core and individual microscope users with easily accessible standards for characterizing their instruments and protocols. The LMRG promotes scientific exchange between researchers using microscopy in Cores to increase our general knowledge and expertise and conducts multisite experiments to establish light microscopy standards. The first 2 LMRG studies focused on developing quality assurance tests for light microscopes, including the characterization of objective lenses, the accuracy of spectral detection and separation, illumination stability and uniformity, and characterization of the point spread function.^207−210

A third LMRG study of microscope resolution, distortion, intensity quantification, and signal-to-noise as a function of depth in 3D is being finalized, with the goal of creating a 3D biologically relevant test slide and imaging protocol to assess 1) system resolution and distortions in 2D and 3D, 2) dependence of intensity quantification and image signal-to-noise of the microscope on imaging depth, and 3) dependence of the microscope sensitivity on imaging depth. The test sample consists of a mixture of fluorescence microspheres imbedded in a 120-um-thick layer of CyGel (Cy10500; BioStatus, Loughborough, United Kingdom) with a refractive index of 1.37 closely matched to biologic tissue. Double-sided adhesive 18-mm square spacers with a well (9-mm diameter, 120-μm deep) were used for the sample preparation (70327-8s; Electron Microscopy Sciences, Hatfield, PA, USA). The mixture of microspheres includes 1 μm Orange, 2.5 μm Green 20% brightness, 2.5 μm Green 100% brightness, 6 μm Far Red 36%, 6 μm Far Red, and 15 μm Blue Core/Orange Ring (Kubow, K., Abrams, B., Ammer, A., Arvanitis, C., Callahan, L., Cole, R., Dragavon, J., Itano, M., Mezzano, V., Pengo, T., Powers, J., Sanders, M., Wee, E., unpublished results).

All 3 LMRG studies have resulted in microscopy standards as well as easily accessible protocols for the characterization and verification of microscope systems. The next LMRG study will focus on reproducibility issues in image analysis. As with the acquisition of microscope images, their subsequent analysis would also typically be performed entirely by the end-user and therefore presents a major reproducibility challenge.

Flow Cytometry RG

The FCRG was formed with the goal of providing key information related to the art of flow cytometry, including its cross-technology applications to genomics, proteomics, and other Core-related areas.^{211, 212} Flow cytometry is a broad term describing the rapid measurement of large numbers of cells individually using light-scatter and fluorescence detection to analyze cell characteristics (size, intracellular pH, membrane potential) and intracellular cellular components (DNA, protein, calcium, cell surface receptors) and distinguish based on these parameters for isolating cell populations, critical for downstream analysis of cells. Mass cytometry has recently emerged as a form of flow cytometry where lanthanides are used as the label on cells that are measured in a time of flight mass cytometer. Cell sorting uses flow cytometry to separate cells based on protein expression labeled with fluorescent tags and has many downstream applications such as functional assays. In other experimental workflows, cells sorted will be assessed for gene expression, such as single-cell analysis (10X Genomics, Fluidigm C1). The first FCRG study investigated the impact of cell sorting on cell health using a range of instrument types and configurations across multiple sites, as assessed by cell viability, proliferation, and gene expression. Variables such as sorting vs. not sorting, high vs. low pressure, and the presence vs. absence of UV light were analyzed with RNA-seq and microarray in Jurkat cells, primary mouse B cells, and mouse ES cells. Information garnered through this multisite study provided guidelines for sorting cells upstream of other technologies including genomics and proteomics.²¹³ A second study surveyed flow cytometry Cores for sorter cleaning practices; a subset of study participants submitted sheath fluid samples for testing. The study results showed that the majority of sorters had significant endotoxin contamination, little to no RNase, with bacterial concentrations quite variable. There was no correlation found between sorter cleanliness and any surveyed variables, including sorter age, cleaning practices, date of last preventive maintenance, sheath source, or known recent contamination. Because a number of sorters assayed in the sorter cleanliness study were contaminated with endotoxin, effectiveness of an H₂O₂ cleaning procedure was tested to assess removal of the endotoxin from the sorters. Sheath fluid samples collected before and after cleaning were tested with an Limulus amebocyte lysate (LAL) quantitation kit and it was determined that the contamination was only partially mitigated. Also, the endotoxin levels reached precleaning levels within a few weeks of the sterilization. The results of the second study will be published in combination with the endotoxin removal results (Thornton, S., Bowen, S., Bispo, C., Hassel, C., Abshari, M., Adams, D., Bergeron, A., Brundage, K., Cochran, M., Del Rio Guerra, R., Dwyer, K., Harley, R., Holmes, L., Loof, N., Meyer, M., Niziolek, Z., Saluk, A., unpublished results).

Increasingly, investigators ask Flow Cytometry Cores to sort fixed cells for RNA isolation either in bulk or at the single-cell level. The 2018 FCRG study performed a systematic evaluation of the reported fixation methods prior to sorting to determine the impact on the purity, quality, and RNA yield from sorted cells (Thornton, S., Bowen, S., Bispo, C., Hassel, C., Abshari, M., Adams, D., Bergeron, A., Brundage, K., Cochran, M., Del Rio Guerra, R., Dwyer, K., Harley, R., Holmes, L., Loof, N., Meyer, M., Niziolek, Z., Saluk, A., unpublished results).

Interest groups

The ABRF supports a fifth category of Interest Networks, which facilitate discussion and review for key technology areas of interest that may be considered for RG status. The Antibody Technology Interest Network is dedicated to sharing its collective knowledge about generating, producing, purifying, fragmenting, and conjugating antibodies, with ABRF members and within the larger scientific community. The Antibody Technology Interest Network holds online sessions and workshops to facilitate the discussion of antibody use and application, immunization strategies, fusion protocols, screening strategies, antibody production purification and labeling, antibody applications (e.g., Flow Cytometry, Microscopy, Chromatin immunoprecipitation combined with sequencing (ChIP-Seq), in vivo, in vitro applications), and laboratory organizational and fiscal structures.

CONCLUSIONS

The research enterprise includes numerous stakeholders: universities and other research institutions that educate, employ, and train researchers; the federal and industrial sponsors of research, journal, and book publishers; and scientific societies.^{1, 5, 6, 23} These stakeholders can act in ways that either support or undermine the integrity of research.²³ As presented here, ABRF sponsorship of multicenter research studies promote community-based standards and provide a sustainable framework for sharing best practices in methodology, standard operating procedures, and data management.²⁶ The ABRF RGs, led by highly trained scientists, provide an extremely valuable yet undervalued service to the research community by helping to provide clarity on approaches and best practices for experimental design and data analysis. The ABRF RGs collectively address enigmatic technology-based problems to answer complex biologic questions, ranging from determining best practices for cytometric cell sorting for optimal RNA extraction to partnering with instrumentation manufacturers to improve MS methods for protein and metabolome studies.

At an institutional level, Cores generate the majority of research data at many institutions so their role in maintaining needed expertise and generating quality data is considerable and represents a central repository for knowledge management, with deep expertise and knowledge of the strengths and limitations of the technology and applications.³⁻⁷ Core science inherently supports transparency and scientific reproducibility through unbiased acquisition, minimizing interoperable variability and promoting transparent processes and reporting (detailed experimental materials and methods) for publications and grants.⁵⁻⁷ Data provenance is assured—detailing who performed what experiment on which instrument, instrument standardization and maintenance, quality assurance/QC (required controls, standards, documentation and tracking of buffers, reagents, components, lot numbers, version, expiration dates), location of source data and shared data (curation in compliance with Data Storage Standards for Research Core Laboratories, OMB Circular A-110 and NIH GDS Policy)—and essential for research integrity.²³⁻²⁷ Recognizing this, federal granting agencies have already made significant investments in Cores through a variety of direct and indirect mechanisms, with the goal of providing cutting-edge technologies and expert consultation to individual scientific investigators.²⁷

The shared goals of the NIH, other research stakeholders, and research institutions are more likely to be achieved when core scientists and research scientists work together to identify and minimize risk to research data, thereby improving research quality, rigor, and reproducibility. The ABRF supports data management policies and deposition of data/methods to prevent digital meddling, either through repositories specific for techniques (i.e., https://flowrepository.org) or general data repositories sponsored by the Center for Open Science (https://osf.io). Engaging RG core scientists as technology editors and reviewers²⁶ will promote and support rigorous, transparent, and reproducible research and the responsible conduct of research. Scientists, funding sponsors, and institutions are encouraged to foster an atmosphere of quality data management and to give credit for data sharing by the data author.²¹⁴ This ensures that the available data set follows a set of guiding principles to make data findable, accessible, interoperable, and reusable (FAIR).²¹⁵ FAIR Guiding Principles instruct that the data and metadata meet criteria of findability, accessibility, interoperability, and reusability are standard practice.^{214– 216} Changes by scientific journals to improve reporting transparency includes providing generous length limits for methods section, and the use of a checklist during editorial processing to ensure the reporting of key methodological, and analytical information to reviewers and readers.^{16, 19, 20}