Quality assurance and quality control processes: Summary of a metabolomics community questionnaire

Abstract

Introduction

The Metabolomics Society Data Quality Task Group (DQTG) developed a questionnaire regarding quality assurance (QA) and quality control (QC) to provide baseline information about current QA and QC practices applied in the international metabolomics community.

Objectives

The DQTG has a long-term goal of promoting robust QA and QC in the metabolomics community through increased awareness via communication, outreach and education, and through the promotion of best working practices. An assessment of current QA and QC practices will serve as a foundation for future activities and development of appropriate guidelines.

Method

QA was defined as the set of procedures that are performed in advance of analysis of samples and that are used to improve data quality. QC was defined as the set of activities that a laboratory does during or immediately after analysis that are applied to demonstrate the quality of project data. A questionnaire was developed that included 70 questions covering demographic information, QA approaches and QC approaches and allowed all respondents to answer a subset or all of the questions.

Result

The DQTG questionnaire received 97 individual responses from 84 institutions in all fields of metabolomics covering NMR, LC-MS, GC-MS, and other analytical technologies.

Conclusion

There was a vast range of responses concerning the use of QA and QC approaches that indicated the limited availability of suitable training, lack of Standard Operating Procedures (SOPs) to review and make decisions on quality, and limited use of standard reference materials (SRMs) as QC materials. The DQTG QA/QC questionnaire has for the first time demonstrated that QA and QC usage is not uniform across metabolomics laboratories. Here we present recommendations on how to address the issues concerning QA and QC measurements and reporting in metabolomics.

Introduction

Metabolomics is a scientific approach applied to the systems analysis of metabolism [Dunn 2011] operating in microbes, plants and animals [Furusawa 2013; Kusano 2015; Cheng 2015]. The discipline of metabolomics is less than 20 years of age [Oliver 1998] although the roots are much older [Pauling 1971]. Metabolomics studies typically use a pipeline from experimental design through analytical measurements (sample preparation and data acquisition) to bioinformatics processing (data processing and statistical analysis) [Brown 2005]. The validity of and confidence in the biological conclusions resulting from this pipeline are highly dependent on the quality of the procedures applied during the metabolomics study. The appropriate application of quality assurance (QA) and quality control (QC) processes are important but are often overlooked in metabolomics. In targeted metabolite studies, guidelines are available to guide the scientist in some aspects of the process including the most frequently applied Food and Drug Administration (FDA) guidelines for bioanalytical method validation [http://www.fda.gov/downloads/Drugs/Guidance/ucm070107.pdf.] as well as other materials [Garfield 2000; Hibbert 2007; Westgard 2008; Booth 2015]. However, there are currently no clear guidelines for untargeted metabolomic studies.

The Metabolomics Society’s mission includes ‘To promote the growth and development of the field of metabolomics internationally” [Metabolomics Society website]. To address this mission, several scientific task groups have been established to act for the community in areas requiring international community consensus. One of these is the Data Quality Task Group (DQTG) chaired by Drs. Daniel Bearden and Richard Beger. The DQTG promotes robust QA and QC in the metabolomics community through increased awareness via communication, outreach and education, and through the promotion of best working practices [Bearden 2014; Metabolomics Society task group website]. One objective of this task group is to define the current application levels of QA and QC processes in both targeted and untargeted studies across all applications in metabolomics. To complete this objective, the task group operated a questionnaire for 2 months (August – September 2015) via the SurveyMonkey website (https://www.surveymonkey.com), which was advertised via e-mail alerts, Metabolomics Society web pages, Twitter and MetaboNews newsletters. The questionnaire included 70 questions covering demographic information, QA approaches and QC approaches and allowed all respondents to answer a subset or all of the questions. All responses are available in the supplementary information and on the Metabolomics Society website [13]. Here we will summarize the most important information and facts derived from the questionnaire and a number of important recommendations.

The respondents

• 97 respondents

• 36 % were principal investigators (PIs) or group leaders, 14 % were staff scientists, 20 % were post-doctoral researchers and 19 % were PhD students.

• 11 % of respondents had less than 2 years of experience in metabolomics with 31 % having greater than 8 years experience.

• The respondents applied metabolomics in a diversity of different applications and many respondents worked across multiple disciplines including clinical sciences (65 %), toxicology (35 %) and systems biology (45 %).

• 70 % responded as working in a combination of a biological/chemical laboratory and data processing/bioinformatics.

• Greater than 70 % of respondents worked with cells, biofluids and tissues and investigated microbes (42 %), plant (34 %), mammals (62 %) and humans (76 %). 73 % and 88 % of respondents applied targeted and untargeted assays, respectively, with 34 % applying NMR spectroscopy in their studies, 83 % applying liquid chromatography-mass spectrometry and 50 % applying gas chromatography-mass spectrometry.

• 74 % of respondents investigated less than 200 samples in a typical biological study and 63 % studied less than 5000 total biological samples each year.

Training

Quality processes include training and competence assessment to ensure a minimum quality-level is associated with processes involving staff. 65 % of 94 responses defined that they operate in an environment with no in-house training program and 74 % were not required to be involved in ongoing continuous professional education. In environments where training was conducted (33 responses), professional staff (49 %) and post-doctoral/graduate staff (36 %) were the major providers of training. Where training is provided, only 21 % of instrument operators have to pass a certification test after training, with 57 % applying professional staff to perform the assessment. 79 % of 85 responses do not operate in an environment where there was a requirement to pass a certification test after training. 73 % of 33 responses applied periodic checks of professional practice with 58 % of checks performed by professional staff as indicated by 33 responses.

Standard Operating Procedures

The mistakes that can be introduced into metabolomics experiments through improper or inconsistent pre-analytical or analytical procedures may cause the data to be inaccurate or invalid, and this may lead to erroneous findings and conclusions. For examples see [Gika 2008; Bernini 2011; Kamlage 2014; Dunn 2012]. Consistent procedures as simple as pipetting, balance usage, sample cross-contamination control, proper preparation of solvents and sample extraction techniques all contribute to the veracity of the analytical measurements and should be thoroughly documented in Standard Operating Procedures (SOPs) and enforced in training programs. For long-term studies or interlaboratory studies, SOPs are essential for communicating well and ensuring the consistency of the data.

Eighty-seven respondents answered questions related to SOPs. SOPs were available in the laboratories of 76 % of respondents with 58 % developed in-house and a further 37 % developed from in-house and published methods. When investigated in more detail, 90 % of respondents had access to SOPs for sample extraction, 53 % for sample storage, 75 % for analytical instruments, 52 % for assessment of data from QC samples and 33 % for deciding when QC data from instrumental analysis has failed and defining how to correct the instrumental data. As a matter of concern and shown for 84 responses was that 70 % of respondents did not have access to a protocol for independent review of quality-related results (Figure 1A) and 80 % did not have access to a written protocol of QA review criteria (Figure 1B).

Figure 1.

A) Responses to “Do you have a protocol for independent review of quality-related results?”; B) Responses to “Do you have a written protocol for QA review criteria?”

Sample measurement validation

The majority of respondents (82 responses) validate sample measurements with 73 % using repeat sample extractions and analyses, 87 % performing repeated analysis of the same sample and 54 % analyzing a historical sample periodically (Figure 2). 88 % of 80 responses analyze a blank sample with extraction performed as for biological samples. Blank samples were analyzed either at the start and end of the analytical batch (28 %), at regular intervals (44 %) or randomly (21 %) as defined in 68 responses. 78 % of respondents operated a process to reduce carryover (80 responses) and 91 % randomize the order of sample analysis (80 responses). 94 % operated instrument condition checks and 79 % of 80 responses did not apply standard reference materials (SRMs); when applicable, 47 % applied a SRM once or less than once a day and 16 % greater than once per day. Methods for reporting of QC data were variable in the 80 responses collected; 34 % reported precision measurements for each metabolite, 45 % report a single range of precisions for all metabolites, 24 % report QC data on a boxplot, 56 % visualize QC samples on a PCA scores plot and 56 % provide a descriptive statement of the QC results.

Figure 2.

Average response to “Do you validate your project sample measurements with: (Check all that apply)?”

QC samples

Of 80 responses, 83 % of respondents applied pooled project materials and 48 % applied standard reference materials (SRMs) as QC materials. This contradicts the results for SRM use as defined above in the sample measurement validation section. Figure 3 illustrates how often QC samples were applied for different processes including the assessment of consistency in sample preparation (80 %) and chromatography column integrity (76 %). Importantly, 59 % of respondents applied replicate extractions and 69 % applied replicate analytical measurements with 85 % analyzing individual samples and 15 % analyzing a single pooled sample.

Figure 3.

Average responses to “What types of QC materials do you routinely use in analytical measurements for metabolomics projects? (Check all that apply)?

Data storage

Of 84 responses, 89 % store data in an archive, with 95 % of data storage being performed in an in-house archive. A lower percentage (73 %) archived QA/QC data.

Inter-laboratory comparisons

Of 82 responses, 33 % had participated in an inter-laboratory comparison study and 48 % were interested in participating in a future inter-laboratory comparison.

Laboratory accreditation

Of 85 responses, 89 % were not required to meet laboratory accreditation and 74 % were not voluntarily attempting to meet any accreditation.

Biggest issues in QA and QC implementation and processes

The most frequent comments related to the currently regarded biggest issues in QA and QC are detailed below:

• Training including staff turnover and lack of training available outside the organization

• SOP formalization, consistency and maintenance including reported changes to published methods (for example papers published in Nature Protocols)

• Ensuring routine compliance to SOPs and QA processes

• Insufficient control over sample collection and sampling consistency

• Inadequate availability of reference standards, isotopically labeled compounds, QC samples and SRMs

• Providing a balance between QA/QC and sample throughput

• QC does not contribute to assessment of output by the wider community and there is a need for true standards across the community

• A global strategy for QA/QC and its review is required

• Establishment of QC acceptance criteria as currently there is a lack of reported QC results and acceptance criteria

• Additional measures beyond pooled QC samples

Key conclusions and recommendations

1. The level of training, both in-house and external to the organization, is low; 65 % of responses replied that they operate in an environment with no in-house training program. 74 % of responses were not required to be involved in ongoing continuous professional education.

   Recommendation: Enhance training focused on QA and QC available as online and face-to-face courses (for example, the Birmingham Metabolomics Training Centre operates a 2-day course focused on QA and QC processes).

2. 76 % of respondents applied SOPs. However, 70 % of respondents did not have access to a protocol for review of quality and 80 % did not have access to protocols focused on a review of quality processes.

   Recommendation: Appropriate agencies and the Metabolomics Society should provide guidance on quality assurance processes and their review; develop consensus processes through specialist meetings and reports.

3. The majority of respondents validate sample measurements, apply sample blanks, apply protocols to minimize sample carryover and randomize the analysis order of biological samples.

   Recommendation: To provide education to the metabolomics community, with an emphasis on early career scientists, on sample measurement validation, and to provide continuing education to ensure these good practices continue.

4. 83 % of respondents applied pooled project materials and 48 % applied standard reference materials (SRMs) as QC materials. 59 % of respondents applied replicate extractions and 69 % applied replicate analytical measurements.

   Recommendation: To provide education to the metabolomics community, with an emphasis on early career scientists, on usage of quality materials, and to provide continuing education to ensure these good practices continue.

5. 79 % of respondents did not access SRM materials.

   Recommendation: To communicate with the metabolomics community to define the types and volumes of SRMs required.

6. 33 % had participated in an inter-laboratory comparison study and 48% were interested in participating in a future inter-laboratory comparison.

   Recommendation: To communicate with the metabolomics community to define the types and frequency of inter-laboratory comparison exercises and encourage independent agencies to support inter-laboratory exercises.

7. 89 % of respondents were not required to meet laboratory accreditation and 74 % were not voluntarily attempting to meet any accreditation.

   Recommendation: To investigate the requirement for laboratory accreditation with the regulatory agencies, funding bodies, the Metabolomics Society and the metabolomics community.

8. There is little incentive for laboratories to improve their QA/QC practices, especially given the non-trivial costs associated with a thorough QA/QC program.

   Recommendation: Recognizing the need to provide further incentive for laboratories to improve overall QA/QC practices, expert panels should be convened to develop workable, practical QA/QC recommendations and guidelines. One possible mechanism is a workshop currently being planned for later in 2017 that will define appropriate QA/QC frameworks that may be adopted widely in laboratories and, possibly, by funders, data repositories and scientific publishers.

Disclaimer

The opinions expressed in this publication do not necessarily represent those of the U.S. Food and Drug Administration or the National Institute of Standards and Technology. Certain commercial equipment, instruments, or materials are identified in this paper to specify adequately the experimental procedure. Such identification does not imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the materials or equipment identified are necessarily the best available for the purpose. Reference to the content of this paper for commercial advertising purposes with regard to commercial equipment, instruments, or materials is prohibited (15 CFR 200.113).