Practical Guidance to Implementing Quality Management Systems in Public Health Laboratories Performing Next-Generation Sequencing: Personnel, Equipment, and Process Management (Phase 1)

Quality standards as part of an effective quality management system (QMS) are the cornerstone for generating high-quality test results. Next-generation sequencing (NGS) has the potential to improve both clinical diagnostics and public health surveillance efforts in multiple areas, including infectious diseases. However, the laboratories adopting NGS methods face significant challenges due to the complex and modular process design.

ABSTRACT

Quality standards as part of an effective quality management system (QMS) are the cornerstone for generating high-quality test results. Next-generation sequencing (NGS) has the potential to improve both clinical diagnostics and public health surveillance efforts in multiple areas, including infectious diseases. However, the laboratories adopting NGS methods face significant challenges due to the complex and modular process design. This document summarizes the first phase of quality system guidance developed by the Centers for Disease Control and Prevention (CDC) NGS Quality Workgroup. The quality system essentials of personnel, equipment, and process management (quality control and validation) were prioritized based on a risk assessment using information gathered from participating CDC laboratories. Here, we present a prioritized QMS framework, including procedures and documentation tools, to assist laboratory implementation and maintenance of quality practices for NGS workflows.

BACKGROUND AND METHODS

Next generation sequencing (NGS) has contributed to the improvement of both clinical diagnostics and public health surveillance efforts (1). However, public health laboratories implementing NGS methods face a number of challenges (2, 3). These challenges include (i) a multistep process with many potential sources of variation, (ii) difficulty setting up and troubleshooting protocols, (iii) Significant personnel training requirements, (iv) frequent updates to sequencing platforms, chemistries, and software, and (v) difficulty assessing the quality of materials and data as they progress through the NGS workflow.

To address the challenges of developing and implementing quality NGS methods within Centers for Disease Control and Prevention (CDC) laboratories, the CDC NGS Quality Workgroup was formed in 2015. Workgroup formation was concomitant with a survey sent to members of the CDC NGS community to gather information on challenges faced by laboratories developing and implementing NGS methods. This workgroup brought together laboratory scientists, bioinformaticians, and quality managers from across the agency for a multidisciplinary and consensus-based approach. The goal of the workgroup was to leverage best practices from across the agency and to develop practical guidance for laboratories implementing NGS within a quality management system (QMS).

The NGS Quality Workgroup wanted to ensure that any NGS-specific guidance could be integrated into existing laboratory QMSs. Many laboratories utilize the 12 quality system essentials (QSEs) identified by the Clinical and Laboratory Standards Institute (CLSI) as their QMS framework (4): organization, customer focus, facilities and safety, personnel, purchasing and inventory, equipment, process management, documents and records, information management, nonconforming events management, assessments, and continual improvement. These 12 QSEs are the foundational components of a QMS and align with quality system elements described in the International Organization for Standardization’s ISO 15189 requirements (5). Therefore, the NGS Quality Workgroup chose to adopt the CLSI QSE framework to develop procedures, guidance, and tools to assist laboratories in implementing and maintaining quality practices for their NGS workflows. The NGS quality guidance developed and presented here should easily integrate into most foundational laboratory QMSs. The guidance provided here is not intended to set regulatory precedents or to address biosafety requirements but, rather, to serve as a resource to public health and clinical laboratories adopting and implementing NGS technologies.

The workgroup’s approach started with reviewing the 12 QSEs to identify basic processes and needs that both apply broadly across NGS workflows and are highly impacted by novel technologies. Comments from the initial survey together with a risk assessment to identify the components most likely to impact NGS quality were used to prioritize the areas of greatest need. While all 12 QSEs are required for a fully functional QMS, the challenges within the three QSEs of personnel, equipment, and process management (particularly quality control [QC]) were identified to pose the most immediate risk to NGS quality. To mitigate the identified risks, the workgroup developed standard operating procedures (SOPs) and forms as practical guidance applicable to a variety of NGS applications.

At the time of the survey, most of the NGS Quality Workgroup member laboratories used the Illumina MiSeq, with a small minority using the Ion Torrent Personal Genome Machine (PGM) sequencing platform. Although much of the documentation developed by the workgroup is tailored to these instruments, documents for the Oxford Nanopore Technologies MinION were added in 2019. It became apparent through workgroup discussions that it would be difficult to provide extensive specificity in the documentation since NGS technology is used for a variety of applications. Due to the multitude of NGS workflows, documents were crafted in Microsoft Word format allowing customization. The workgroup products (Table 1) are designed to provide a firm foundation upon which individual laboratories may build to develop assay-specific documentation.

TABLE 1.

Documents developed by the CDC NGS Quality Workgroup

Document no.	Title	Type	QSE
1	Ion PGM sequencer competency assessment form	Form	Personnel
2	Ion PGM sequencer competency assessment SOP	SOP	Personnel
3	MiSeq competency assessment form	Form	Personnel
4	MiSeq competency assessment SOP	SOP	Personnel
5	Ion PGM sequencer training form	Form	Personnel
6	Ion PGM sequencer trainer designation form	Form	Personnel
7	Ion PGM sequencer training SOP	SOP	Personnel
8	MiSeq employee training form	Form	Personnel
9	MiSeq trainer designation form	Form	Personnel
10	MiSeq training SOP	SOP	Personnel
11	MinION training SOP	SOP	Personnel
12	MinION employee training form	Form	Personnel
13	Ion Chef preventive maintenance log	Log	Equipment
14	Ion Chef preventive maintenance SOP	SOP	Equipment
15	Ion OneTouch 2 preventive maintenance log	Log	Equipment
16	Ion OneTouch 2 preventive maintenance SOP	SOP	Equipment
17	Ion OneTouch ES preventive maintenance log	Log	Equipment
18	Ion OneTouch ES preventive maintenance SOP	SOP	Equipment
19	Ion PGM equipment error log	Log	Equipment
20	Ion PGM in-use equipment daily maintenance log	Log	Equipment
21	Ion PGM in-use equipment weekly maintenance log	Log	Equipment
22	Ion PGM power-off equipment maintenance log	Log	Equipment
23	Ion PGM preventive maintenance wash flowchart	Job aid	Equipment
24	Ion PGM sequencer preventive maintenance SOP	SOP	Equipment
25	MiSeq equipment error log	Log	Equipment
26	MiSeq in-use equipment maintenance log	Log	Equipment
27	MiSeq preventive maintenance SOP	SOP	Equipment
28	MiSeq preventive maintenance wash flowchart	Job aid	Equipment
29	MiSeq standby equipment maintenance log	Log	Equipment
30	Ion PGM system equipment preinstallation checklist	Job aid	Equipment
31	MiSeq equipment preinstallation checklist	Job aid	Equipment
32	Vendor-performed IQ/OQ cover sheet	Form	Equipment
33	Ion PGM sequencer software update evaluation SOP	SOP	Equipment
34	Ion PGM sequencer software update form	Form	Equipment
35	MiSeq software update evaluation SOP	SOP	Equipment
36	MiSeq software update form	Form	Equipment
37	NGS QC guidance for Illumina Workflows	SOP	Process Management
38	Bioinformatics QC workflows	SOP	Process Management
39	Sequencing QC SOP	SOP	Process Management
40	Preanalysis QC SOP	SOP	Process Management
41	Assembly QC SOP	SOP	Process Management
42	NGS QC guidance for MinION 1D workflows	SOP	Process Management
43	NGS QC guidance for MinION rapid sequencing workflows	SOP	Process Management
44	MGS methods validation SOP	SOP	Process Management
45	NGS methods validation plan template	Form	Process Management
46	NGS methods validation report template	Form	Process Management

QSE: PERSONNEL

Table 2 outlines significant challenges and risks associated with personnel responsible for performing NGS processes. The NGS process is complex and requires a substantial time commitment by staff to understand the sequencing process and perform sequencing proficiently. This is compounded by the novelty of the technology; in many cases current staff have to gain knowledge of the NGS technology and process on their own as there may be no staff with prior experience to provide training. When implementing NGS technology, organizations need a basic understanding of the system requirements before equipment installation. Following installation, laboratory personnel must be proficient in maintaining and using the equipment.

TABLE 2.

Challenges and risks identified and mapped to a QSE

QSE and challenge	Risk	Relevant documents (no.)
Personnel
Time commitment and resources to train personnel are required.	Staff who are not fully trained and proficient are more likely to make errors.	5–12
A mismatch between existing skill sets and needed skill sets in personnel who will perform NGS may be present.	Inexperienced staff will have difficulty troubleshooting NGS processes.	1–12
Equipment
Many NGS instruments require accommodations to ensure physical environment requirements are met (e.g., minimal vibration and specific temperature ranges).	Laboratories not aware of the requirements before instrument arrival face delays and unplanned costs.	30–32
NGS equipment requires regular user performed preventive maintenance.	If preventive maintenance is not performed according to schedule, the sequencing results may be compromised.	13–29
Frequent updates are released for NGS hardware and software requiring verification of method performance or potentially revalidation.	Frequent verifications and validations require substantial time and are costly; laboratories risk an out-of-control process if the updates are not tracked and documented.	33–36
Process management
There are several different processes that comprise the overall NGS workflow.	There are multiple opportunities for error to enter the workflow.	37–43
The NGS workflow uses multiple sets of reagents at different cost points.	An error early in the process can result in the use of expensive reagents downstream and poor quality sequence data outputs.	37, 42–43
Bioinformatic analysis requires the time of highly skilled personnel and computational resources.	Poor quality sequence data fed into the analysis pipeline may consume computing resources and personnel time that increase unnecessary resource expenditures.	37–43
Validation of NGS assays with multiple targets are complex to design and time consuming to perform.	Incomplete understanding of assay performance can result in poorly understood results being reported to patients.	44–46

To address these challenges, the NGS Quality Workgroup developed standard training SOPs and accompanying training forms. Based on experience and a literature review, workgroup members identified key sources of information and included these references in the training documents. These resources provide a foundation of knowledge for the NGS technology and processes. Designed to be flexible for each individual laboratory’s needs, the SOPs (Table 1, documents 5 to 12; see also File S1 in the supplemental material) outline foundational knowledge and a process for training that involves four steps: (i) read to understand, (ii) observe the process, (ii) perform the process under supervision, and (iv) perform the process independently. At the CDC, laboratories that are standing up an NGS process for the first time have the option to work with NGS Quality Workgroup members to observe the process. This knowledge sharing can be extended to other laboratories through either webinars or collaboration among laboratory networks.

Laboratories performing testing subject to Clinical Laboratory Improvement Amendments (CLIA) regulations must perform annual competency assessments of their personnel in addition to providing training. The competency assessment documents (Table 1, documents 1 to 4; File S1) address the six criteria included in the CLIA competency assessment. The six criteria are the following: (i) direct observation of routine patient test performance, including patient preparation, if applicable, specimen handling, processing, and testing; (ii) monitoring the recording and reporting of test results; (iii) review of intermediate test results or worksheets, quality control records, proficiency testing results, and preventive maintenance records; (iv) direct observation of performance of instrument maintenance and function checks; (v) assessment of test performance through testing previously analyzed specimens, internal blind testing samples, or external proficiency testing samples; (vi) assessment of problem-solving skills (6). Performing competency assessment ensures that personnel have the appropriate skills required to successfully perform NGS methods. The NGS Quality Workgroup identified key tasks and knowledge necessary to successfully perform the NGS process. These items are prepopulated in the competency assessment SOP and form (provided in File S1) for the benefit of laboratories performing NGS methods that are subject to CLIA.

QSE: EQUIPMENT

Table 2 outlines challenges and risks associated with NGS equipment. Location and physical environment requirements for NGS equipment can be extensive. Based on workgroup member experience and the requirements provided by equipment manufacturers, the NGS Quality Workgroup developed a preinstallation checklist (Table 1, documents 30 and 31; File S2). This checklist assists laboratories in identifying and/or retrofitting laboratory space that meets the requirements for the operation of NGS equipment. The checklist provides laboratories with a tool to prepare laboratory space ahead of time for the receipt of NGS equipment and avoid delays and unexpected costs.

Installation qualification and operational qualification (IQ/OQ) are often performed by the vendor at the time of installation. There may be an additional cost for IQ/OQ that needs to be considered during equipment purchase budget planning. NGS Quality Workgroup members noted that the equipment was purchased for research applications and test development with the intent to transition to clinical applications after test validation. This led to instances where the equipment performance level at installation was not sufficiently recorded for validation. To prevent this from happening in the future, workgroup members developed a form (Table 1, document 32; also File S2 in the supplemental material) to document vendor-performed IQ/OQ that captures the initial data needed and can serve as a cover sheet for the documentation provided by the vendor at installation.

Upon installation of NGS equipment, users are expected to regularly perform preventive maintenance. Many laboratories are not aware of the extent of the required preventive maintenance and/or do not have personnel trained to perform the maintenance. The NGS Quality Workgroup developed SOPs, forms, and flowcharts to standardize the preventive maintenance process across laboratories (Table 1, documents 13 to 29; File S2). The minimum intervals required by the manufacturers are specified in each SOP.

The NGS field is rapidly evolving, which results in frequent hardware and software update releases. As laboratories attempt to keep up with the releases, it is challenging to maintain a change control process that documents updates and traceability of results. In particular, the NGS Workgroup identified software updates as a challenging area to control. Based on these experiences, the workgroup developed an SOP and form (Table 1, documents 33 to 36; File S2) that take a risk-based approach to software updates. Low-risk updates, such as changes to functionalities within the software that a particular laboratory does not make use of, may require only the running of external positive and negative controls to verify performance. Medium-risk updates, such as updates that minimally impact the fluidics of the instrument or the data processing but are not expected to impact the final results, would likely require a verification consisting of testing a subset of samples used during the original validation. High-risk updates that are expected to impact the final output would require a more extensive verification. In each case the results of the verification run are analyzed, and any discrepancies with the original validation data require further investigation and testing. The software update forms capture the risk assessment results, decisions, and resulting required action.

The provided documents will not prepare laboratories for the challenges involving the massive amount of data generated by this equipment; future working group efforts will involve the QSE information management. The workgroup plans to develop guidance on the NGS data file transfer integrity, on NGS data retention and storage, on recording bioinformatic analyses tools and parameter settings, and on recording bioinformatics pipeline versions.

QSE: PROCESS MANAGEMENT

Table 2 outlines significant challenges and risks associated with process management when NGS processes are performed. Process management covers the life span of the specimen and all associated processes from sample receipt to sample reporting. Process management requires establishing quality control steps and using the quality control data to inform decisions to continue or suspend the NGS workflow for a given sample. Recognizing when to suspend further testing of a sample earlier in the process saves time and lowers the cost of testing. Instead of completing the NGS workflow and recognizing a suboptimal outcome through poor-quality sequence data, users are able to identify poor quality and quantity of extracted DNA or poor-quality library material and make the decision earlier to suspend further testing. For laboratories that are in the process of developing and implementing NGS methods, process controls can support troubleshooting by identifying the portion of the process responsible for the suboptimal results.

The NGS Quality Workgroup addressed process controls with a series of SOPs (Table 1, documents 37 to 43; see also File S3 in the supplemental material) that outline quality control checkpoints during both the sample preparation (wet laboratory) and data analysis/bioinformatics (dry laboratory) portions of the workflow. Fig. 1A and B represent examples of general wet-laboratory and dry-laboratory NGS workflows, along with accompanying quality control checkpoints. The quality control checkpoints serve as decision points to determine whether to continue the process. The checkpoint metrics provide information on the quality of the material during the sample preparation process or the quality of the data during the bioinformatics portion of the workflow. The initial focus of the workgroup was on isolate-based approaches. Thus, laboratories performing sequencing directly on primary specimens will need to establish additional process controls for presequencing protocols, such as target enrichment.

FIG 1.

NGS workflow with QC checkpoints in the wet laboratory (A) and dry laboratory (B).

Process management efforts also included method validation. Laboratories subject to CLIA regulations must establish performance characteristics of NGS methods for use in diagnostics (2). Challenges include (i) how to apply traditional validation definitions and metrics to NGS data and methods and (ii) how to determine the appropriate samples to include in the validation.

Cognizant of the many and varied applications of NGS, the workgroup largely focused on the intended use of whole-genome sequencing for the qualitative characterization of bacterial isolates. The developed framework outlines considerations for laboratories in a methods validation SOP and provides validation plan and report templates (Table 1, documents 44 to 46; File S3). The SOP contains definitions and provides specific examples of traditional validation metrics applied to NGS data. Additionally, the SOP outlines criteria laboratories can use to identify the validation samples, with examples provided, to ensure thorough understanding of assay performance. Criteria for selecting positive samples include genetic diversity, frequency of expected submissions, and public health impact. Criteria for selecting true-negative samples are genetic similarity and, if applicable, symptomatic or phenotypic similarity and derivation from healthy individuals (e.g., normal flora). Because these NGS methods validation documents are intended primarily for qualitative NGS methods to characterize bacterial isolates, the workgroup plans to develop a validation framework for detection assays in primary specimens, including metagenomics.

DISCUSSION

The NGS Quality Workgroup identified key challenges faced by laboratories implementing NGS methods and addressed them using a prioritized QMS framework with a focus on personnel, equipment, and process management. The workgroup provided generalized solutions and guidance in the form of SOPs and documents that laboratories could adopt and edit for individual needs. The workgroup framework provided a multidisciplinary and consensus-based structure necessary to develop solutions and provide guidance to laboratories performing NGS.

Although the impact of workgroup documents and guidance to address the key challenges identified has not yet been formally assessed, positive outcomes have been noted. Multiple laboratories have adapted these documents to their workflows. By using comprehensive equipment installation checklists with the necessary information in one place, workgroup members observed fewer occurrences of overlooked requirements during the planning period. Further, the equipment resources were particularly useful to perform and document preventive maintenance and manage software updates. Using process management to assess the quality of materials and data as they progress through the NGS workflow, both new and experienced NGS users have adopted formal QC checkpoints. The QC checkpoints were also used to establish acceptance criteria for material and data transferred between different laboratories.

Notably, NGS technology and methods are continuously changing as new instrumentation and reagents become available. While the supplemental materials provided here have been tailored to specific technologies, they may also act as templates that can be easily and quickly updated to account for new methods and instrumentation.

A fully developed QMS should address the remaining nine QSEs (organization, customer focus, facilities and safety, purchasing and inventory, documents and records, information management, nonconforming events management, assessments, and continual improvement). Additionally, a robust QMS exists in an environment of continuous improvement, and thus additional work is expected to be completed on the three QSEs detailed here. Through continuing input and collaboration from laboratory scientists, bioinformaticians, and quality managers, the NGS Quality Workgroup continues to develop guidance and documentation to support a holistic QMS approach to NGS. Guidance and solutions to address the remaining QSEs and any additions to the existing three QSEs are planned for release in subsequent publications, with current activities including data handling, analysis and storage, and method validation.