Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Sep 5.
Published in final edited form as: Nat Methods. 2021 Dec;18(12):1423–1426. doi: 10.1038/s41592-021-01162-y

QUAREP-LiMi: A Community Endeavour to Advance Quality Assessment and Reproducibility in Light Microscopy

Ulrike Boehm 1,*, Glyn Nelson 2,*, Claire M Brown 3,*, Steve Bagley 4, Peter Bajcsy 5, Johanna Bischof 6, Aurelien Dauphin 7, Ian M Dobbie 8, John E Eriksson 9, Orestis Faklaris 10, Julia Fernandez-Rodriguez 11, Alexia Ferrand 12, Laurent Gelman 13, Ali Gheisari 14, Hella Hartmann 15, Christian Kukat 16, Alex Laude 17, Miso Mitkovski 18, Sebastian Munck 19,20,21, Alison J North 22, Tobias M Rasse 23, Ute Resch-Genger 24, Lucas C Schuetz 25, Arne Seitz 26, Caterina Strambio-De-Castillia 27, Jason R Swedlow 28, Roland Nitschke 29,30,+
PMCID: PMC9443067  NIHMSID: NIHMS1833228  PMID: 34021279

Abstract

The community-driven initiative ‘Quality Assessment and Reproducibility for Instruments & Images in Light Microscopy’ (QUAREP-LiMi) wants to improve reproducibility for light microscopy image data through quality control (QC) management of instruments and images. It aims for a common set of QC guidelines for hardware calibration, image acquisition, management and analysis.

Introduction

Over the past decade, challenges with reproducibility in science have come to the forefront and tools to improve it are being developed in several areas. The discussion focuses on many aspects, such as the need to address the crisis from the perspective of antibody validation1,2, cell line authentication3,4, and more recently, artificial intelligence5,6. This has led to challenges in science as the problem is immense7,8, and solutions to improve the situation are complex9,10. Community-driven initiatives to improve reproducibility by developing standards for antibody validation11 and cell line authentication12–14 have created more awareness around the problem and have given researchers the tools they need to start to solve it. This has increased the dialog among scientists and, ultimately, will lead to more solutions for reproducibility.

The hardware and software components of light microscope systems are complex and vary widely from one to another. Many light microscopy-based imaging modalities generate complex and multidimensional images (e.g., 3D, multiple colors, millions of pixels). This complexity makes the development of generalized and widely accepted protocols and guidelines for quality control (QC) in light microscopy experiments challenging. Many individual efforts have aimed to improve the situation in light microscopy, but up until now, there has been no concerted effort by the global light microscopy community.

However, the time is right to take action. Researchers are increasingly using light microscopy to publish quantitative data rather than for qualitative observations. Therefore, it is vital that the performance and limitations of the microscope systems used are routinely measured and fully understood. Only with this knowledge can the data captured be reliable, robust, accurate and reproducible. Management of QC is essential and should be mandatory for both spatial measurements (e.g., morphology, size, distance) and quantitative fluorescence intensity measurements (e.g. expression level, protein activity, local concentration) to ensure microscopes are calibrated and stable over time to safeguard repeatability and accuracy of measurements. Moreover, imaging data are becoming more broadly shared for community analysis and made available in public image data repositories15,16. If image data is broadly shared, it is essential that information about how the data was generated is well documented and that images are of high-quality. Therefore, the equipment upon which the images are captured should be subjected to robust ‘quality control’ standards.

Numerous meetings and discussions around the topic of microscopy QC and standards have taken place over the last decade at various venues, e.g., Focus on Microscopy, German BioImaging, Global BioImaging Exchange of Experience, BioImaging UK Light Microscopy Facility, Association of BioMolecular Resource Facilities (ABRF) and European Light Microscopy Initiative (ELMI) meetings. However, these community efforts have not been coordinated at the global level, have not focused on accepted guidelines from the International Organization for Standardization (ISO), and have not been through rigorous community-driven peer review.

In 2019, as part of the Core Facility Day at the European Light Microscopy Initiative (ELMI) meeting, a survey was launched and completed by 225 microscopists from around the world about common practices for the management of microscope QC. The comprehensive survey highlighted that microscope QC procedures across the community are inconsistent in their nature and frequency. Facilities perform different quality tests, use dozens of different standard samples and software tools, metrics, and protocols and perform checks on different timelines (e.g., weekly, monthly, annually) or not at all. The main barrier to frequent in-depth quality checks and standardization was the lack of human resources to perform them. They are typically done manually, take a considerable amount of time and tie up both imaging facility staff time and instruments. In addition, other barriers were identified as a lack of: widely adopted and agreed upon guidelines; access to and consensus for standard samples and protocols; and robust training for how to perform QC tests.

In 2018, Global BioImaging (GBI) published ”Common international recommendation for quality assurance and management in open access imaging infrastructures“. However, these are general guidelines that do not include in-depth protocols. More recently, ISO published the first international standard for confocal microscopy: ISO 21073:2019 - Confocal microscopes — Optical data of fluorescence confocal microscopes for biological imaging. Based on that standard, BioImaging UK has recently published a companion paper providing protocols and recommended tools to implement ISO 21073:2019 titled “Interpretation of Confocal ISO 21073: 2019 confocal microscopes: Optical data of fluorescence confocal microscopes for biological imaging- Recommended Methodology for Quality Control”17.

QUAREP-LiMi formation

The ELMI survey and the highlighted challenges demonstrated a clear need for a global joint initiative involving microscopists from different fields and other key stakeholders including imaging scientists, image analysts, standards organizations, microscope manufacturers, funding organizations and publishers. In response, a community-driven initiative titled ‘Quality Assessment and Reproducibility for Instruments & Images in Light Microscopy’ (QUAREP-LiMi) was formed in April 2020. It aims to improve reproducibility for light microscopy image data through quality control (QC) management of instruments and images. Its ultimate goal is to agree on a common set of QC guidelines for hardware calibration, acquiring and managing microscope images and related software. The tangible outcomes of QUAREP-LiMi include protocols, metadata models and tools (automated if possible) to ensure reproducible, reliable, sharable and easily searchable images and scientific research results. QUAREP-LiMi is currently made up of 256 individuals (as of March 23, 2021) from 24 countries with members from academia, industry, standards organizations, and scientific publishers. Many members are drawn from well-established national and international networks, including but not limited to: German BioImaging (GerBI), Microscopie de Fluorescence Multidimensionnelle (RT-MFM), BioImaging UK, the Royal Microscopical Society (RMS); Euro-BioImaging ERIC, Global BioImaging and BioImaging North America (BINA).

Working Groups

The QUAREP-LiMi initiative is organized into a number of focused working groups (WGs) which address areas/topics important for microscope hardware and image data QC management, including data analysis and presentation. Figure 1 presents the major topics and aims covered by the QUAREP-LiMi initiative in their WGs. Their aims are to drive consensus on the use of standard samples and software tools and agree on the metrics to be measured and reported. They will develop and publish training and standard operating procedures (SOPs) to ensure wide adoption by microscope custodians and users. The overarching aim of the WGs is to develop and recommend straightforward yet comprehensive protocols and robust samples that can be used to streamline and eventually automate the QC process.

Figure 1.

Figure 1.

Open in a new tab

Overview of topics and aims covered by QUAREP-LiMi initiative

QUAREP-LiMi aims to define standard operating procedures and tools for device monitoring, benchmarking and quality management of light microscopy instruments and images, that are aligned with existing ISO standards, as well as to develop in liaison with ISO or other stakeholders (corporate partners, funders and publishers) a community based consensus around microscope and image data QC.

QUAREP-LiMi currently comprises 11 WGs, each led by a chair and co-chair. Each WG is formed from members of the global microscopy community. WGs meet virtually, typically once a month, and their agenda, minutes, and current working protocols can be found on the QUAREP-LiMi webpage (https://quarep.org/). Final protocols and guidelines will be published in journals as open access. Current activities focus on widefield and confocal laser scanning microscopy platforms but will later be expanded to other imaging modalities. The WGs 1 to 6 will acquire image data at multiple laboratories to test the reproducibility of samples, data acquisition, and data analysis tools to develop robust protocols.

WG1 Illumination Power

Focus: Metrics and tools to measure microscope light source power and stability on different time scales.

Current Activities: Establish a protocol for measuring the illumination power and stability during both short- and long-term image acquisition sessions, using calibrated external power sensors.

WG2 Detection System Performance

Focus: Metrics and tools to measure and report detection system (e.g. camera, photomultiplier tube, avalanche photodiode) performance.

Current Activities: Standardize the characterization of the detection system performance (including the emission light path) and create accepted procedures and protocols for monitoring it over time. Definition of universal, externally measurable parameters applicable to any type of detection system.

WG3 Uniformity of Illumination Field – Flatness

Focus: Define a set of protocols, tools, and guidelines to assess the uniformity of illumination across the microscope field-of-view and allow for correction.

Current Activities: Develop protocols and tools based on a consensus, to measure and correct field nonuniformity in single images or tiles of images of a large sample that have been stitched together.

WG4 System Chromatic Aberration and Co-Registration

Focus: Metrics and tools to measure chromatic shifts in x, y, and z and protocols to allow for co-registration correction.

Current Activities: Use multi-colored beads or similar preparations to measure co-registration accuracy and develop protocols to correct for chromatic aberrations and align images in x, y, and z.

WG5 Lateral and Axial Resolution

Focus: Metrics and tools to measure and report lateral and axial microscope resolution limits.

Current Activities: Sample preparation, image acquisition, and data analysis protocols for samples of sub-resolution fluorescent beads or similar preparations for monitoring resolution over time.

WG6 Stage and Focus - Precision and Other

Focus: Metrics and tools to measure and report stability and precision of motorized stage platforms including sample holders and microscope focus drives.

Current Activities: Define key terms and develop measurement standards, testing protocols and performance benchmarks to evaluate xyz-movement in terms of stability, reproducibility, and repeatability.

WG7 Microscopy Data Provenance and QC Metadata

Focus: Develop guidelines defining what ‘data provenance’ and QC metadata should be reported for distinct types of imaging data.

Current Activities: The 4D Nucleome (4DN) imaging WG and BINA quality control and data management WG have developed a tiered set of Microscopy Metadata guidelines and a suite of extensions of the Open Microscopy Environment (OME) Data Model that scale with experimental complexity and requirements, and are tailored to enhance comparability and reproducibility in light microscopy. Establish a coordinated outreach strategy to achieve a wide community consensus around the proposed metadata specifications.

WG8 White Papers

Focus: Publish White Papers to communicate and seek cooperation from the microscopy community to raise awareness and promote QUAREP-LiMi short- and long-term goals.

Current Activities: The first white paper is published on arXiv18. The WG is now focused on raising awareness about the white paper and QUAREP-LiMi with various stakeholders including: (1) Prospective new members; (2) Imaging scientists and bioimage analysts; (3) Group Leaders/Principal Investigators; (4) Research scientists; (5) Scientific publishers; (6) Leads (CEO/directors) of companies and commercial application specialists, and (7) Prospective funders to support the work of this initiative.

WG9 Overall Planning and Funding

Focus: Coordination and promotion of QUAREP-LiMi, seek funding opportunities and engage and liaise with stakeholders.

Current Activities: Formalizing publication and authorship guidelines, engaging with corporate partners, standardization organizations, scientific publishers, and funding bodies. Develop and update webpage, tools database and tools to keep WGs organized and running efficiently.

Long-term Goals: (1) Ensure that the output of QUAREP-LiMi achieves maximum impact; (2) Seek to obtain buy-in from microscope manufacturers; (3) Obtain funding from national bodies, scientific publishers and learned societies; (4) Keep stakeholders informed and share information and (5) Coordinate all WGs and future meetings.

WG10 Image Quality

Focus: Define image quality parameters and their weighted impact based on experiment types and microscope modalities to create tools to evaluate the quality of individual images.

Current Activities: Defining weighted image quality parameters and assigning experiment- and microscope-specific QC rating for individual images.

Long-term Goals: Integration of community agreed -upon image quality metrics in Image Metadata.

WG11 Microscopy Publication Standards

Focus: Develop guidelines and best practices to ensure quality Microscope Metadata and microscopy methods reporting.

Current Activities: (1) Inform scientific publishers of methods reporting standards and align them with the recommendations of QUAREP-LiMi; (2) Facilitate the involvement of technical reviewers for microscopy-based data; (3) Promote and increase the appropriate acknowledgement and co-authorship of imaging scientists and imaging facilities in publications; (4) Encourage publishers to compel authors to make raw imaging data available and (5) Propose minimum standards for microscope-based figure quality.

Stakeholders and Beneficiaries

QUAREP-LiMi comprises many stakeholders that are beneficiaries of more rigorous and reproducible microscopy image data and also need to be part of the solution. Stakeholders include: (1) Microscope users and custodians (facility and non-facility) who can be assured of high-quality image data and access to community developed and agreed upon guidelines, recommendations, tools, and protocols; (2) Researchers who will benefit from well-maintained microscopes and the ability to reproduce data from the literature and collaborators; (3) Scientific publishers who will see improved quality of microscopy data upon which scientific conclusions are based and publications that are more reliable for other researchers leading to more significant scientific discovery and greater trust in scientific output; (4) Funding bodies who will be rewarded with a better return on their investments due to higher data quality and will also benefit from data sharing and consequent new discoveries without the need to repeat additional costly experiments due to low quality; (5) Microscope manufacturers who will benefit from a better knowledge of the instruments’ performance in the field, allowing for predictive instrument service and technical improvements for future instrument development; (6) Standards organizations who can work efficiently and effectively with the global community to gain concensus, develop quality standards and be part of the solution through promotion and implementation.

Invitation to Join QUAREP-LiMi

The QUAREP-LiMi initiative depends on the input of the international microscopy community. This includes academics, industry, funders, standards agencies and scientific publishers. Jointly developed community-driven recommendations and guidelines are essential for them to be accepted and adopted by the majority of microscopists. For more in-depth information about the QUAREP-LiMI initiative, please see the White Paper18. If you are interested in actively working in the QUAREP-LiMi initiative to improve the quality of light microscopy imaging, please sign up here: https://quarep.org/contact/.

Acknowledgements

C.M.B has been funded in part by grant number 2020-225398 from the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation. The work of C.S-D-C. was supported by NIH grant # U01CA200059 and by grant #2019-198155 (5022) by the Chan Zuckerberg Initiative as part of their Imaging Scientist Program. R.N. was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) grant number Ni 451/9-1 - MIAP-Freiburg. We thank somersault18:24 BV (Leuven, Belgium) for help with Figure 1 and Thao Do (Allen Institute, Seattle, WA, USA) for the design of the QUAREP-LiMi logo.

Footnotes

Competing interests

The authors declare no competing interests.

References:

  • 1.Baker M Reproducibility crisis: Blame it on the antibodies. Nature 521, 274–276, (2015). [DOI] [PubMed] [Google Scholar]
  • 2.Bordeaux J, Welsh A, Agarwal S, Killiam E, Baquero M, Hanna J, Anagnostou V & Rimm D Antibody validation. Biotechniques 48, 197–209, (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Eckers JC, Swick AD & Kimple RJ Identity Crisis - Rigor and Reproducibility in Human Cell Lines. Radiat Res 189, 551–552, (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Kniss DA & Summerfield TL Discovery of HeLa Cell Contamination in HES Cells: Call for Cell Line Authentication in Reproductive Biology Research. Reprod Sci 21, 1015–1019, (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Hutson M Artificial intelligence faces reproducibility crisis. Science 359, 725–726, (2018). [DOI] [PubMed] [Google Scholar]
  • 6.Gibney E This AI researcher is trying to ward off a reproducibility crisis. Nature 577, 14, (2020). [DOI] [PubMed] [Google Scholar]
  • 7.Clements JC Is the reproducibility crisis fuelling poor mental health in science? Nature 582, 300, (2020). [DOI] [PubMed] [Google Scholar]
  • 8.Franca TF & Monserrat JM Reproducibility crisis in science or unrealistic expectations? EMBO Rep 19, (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Fanelli D Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proc Natl Acad Sci U S A 115, 2628–2631, (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Franca TFA & Monserrat JM To Read More Papers, or to Read Papers Better? A Crucial Point for the Reproducibility Crisis. Bioessays 41, e1800206, (2019). [DOI] [PubMed] [Google Scholar]
  • 11.MacNeil T, Vathiotis IA, Martinez-Morilla S, Yaghoobi V, Zugazagoitia J, Liu Y & Rimm DL Antibody validation for protein expression on tissue slides: a protocol for immunohistochemistry. Biotechniques, (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Marx V Cell-line authentication demystified. Nat Methods 11, 483–488, (2014). [DOI] [PubMed] [Google Scholar]
  • 13.Freedman LP, Gibson MC, Ethier SP, Soule HR, Neve RM & Reid YA Reproducibility: changing the policies and culture of cell line authentication. Nat Methods 12, 493–497, (2015). [DOI] [PubMed] [Google Scholar]
  • 14.Yu M, Selvaraj SK, Liang-Chu MM, Aghajani S, Busse M, Yuan J, Lee G, Peale F, Klijn C, Bourgon R et al. A resource for cell line authentication, annotation and quality control. Nature 520, 307–311, (2015). [DOI] [PubMed] [Google Scholar]
  • 15.Ellenberg J, Swedlow JR, Barlow M, Cook CE, Sarkans U, Patwardhan A, Brazma A & Birney E A call for public archives for biological image data. Nat Methods 15, 849–854, (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Sansone SA, McQuilton P, Rocca-Serra P, Gonzalez-Beltran A, Izzo M, Lister AL, Thurston M & Community FA FAIRsharing as a community approach to standards, repositories and policies. Nat Biotechnol 37, 358–367, (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Nelson G, Gelman L, Faklaris O, Nitschke R & Laude A Interpretation of Confocal ISO 21073: 2019 confocal microscopes: Optical data of fluorescence confocal microscopes for biological imaging- Recommended Methodology for Quality Control. arXiv, (2020). [Google Scholar]
  • 18.Nelson G, Boehm U, Bagley S, Bajcsy P, Bischof J, Brown CM, Dauphin A, Dobbie IM, Eriksson JE, Faklaris O et al. QUAREP-LiMi: A community-driven initiative to establish guidelines for quality assessment and reproducibility for instruments and images in light microscopy. arXiv, (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES