Standardization of Nucleic Acid Tests for Clinical Measurements of Bacteria and Viruses

Abstract

Nucleic acid-based tests for infectious diseases currently used in the clinical laboratory and in point-of-care devices are diverse. Measurement challenges associated with standardization of quantitative viral load testing are discussed in relation to human cytomegalovirus, BK virus, and Epstein-Barr virus, while the importance of defining the performance of qualitative methods is illustrated with Mycobacterium tuberculosis and influenza virus. The development of certified reference materials whose values are traceable to higher-order standards and reference measurement procedures, using, for instance, digital PCR, will further contribute to the understanding of analytical performance characteristics and promote clinical data comparability.

INTRODUCTION

Molecular approaches, such as nucleic acid (NA) amplification-based tests (NAATs) and sequence analysis, are increasingly replacing conventional microbiological methods, such as pathogen propagation in culture and techniques for the detection of antigens, for (i) viral load quantification, (ii) the detection of pathogenic viruses and bacteria, (iii) monitoring viral and bacterial resistance to therapeutic agents, and (iv) monitoring transmission across communities, as they often enable faster, more accurate, or more sensitive measurements (1, 2). However, commercial and in-house NAATs for infectious diseases utilize different technologies, reaction chemistries, and calibration materials, leading to demonstrable variability in the test results in terms of (i) numerical values (e.g., numbers of genome copies) for quantitative methods or (ii) presence or absence of the pathogen for qualitative methods (3–5).

The In Vitro Diagnostics Directive (IVDD) in Europe has promoted assay standardization in the clinical laboratory community, through requiring manufacturers to provide information about the traceability of their calibrators (for definitions of terms in italics, see Table 1), in compliance with ISO 17511 (6). The concept of metrological traceability in clinical chemistry and laboratory medicine has been comprehensively discussed in several articles (7–9). For small-molecule measurements in clinical chemistry, such as those of blood glucose, creatinine, cortisol, and electrolytes, reference measurement procedures of high metrological quality are available which relate the quantity value of the calibrators and reference materials used in a calibration hierarchy traceable to the International System of Units (SI) as described in ISO 17511 (6, 8). However, for the majority of the more complex biomeasurements, including those based on NA testing and infectious pathogen detection, reference materials whose values are traceable to the SI and reference measurement procedures (with well-understood uncertainty) are not yet available. ISO 17511 indicates recognition of the fact that, for measurements of, for example, viral NAs (for human cytomegalovirus [CMV], human immunodeficiency virus [HIV], and hepatitis B virus [HBV]), typically WHO international standards (IS) are available, which are classified as international convention calibrators since the assigned values (in international units) are arbitrarily defined and not based on a calibration hierarchy corresponding to an independent and stable reference system (6).

TABLE 1.

Definitions of metrological terms

Term	Definition	Reference
Analyte	Component represented in the name of the measurable quantity (example: in the quantity “genomic copies per ml plasma,” the analyte is “genomic copies”)	6
Calibrator (calibration material)	Reference material whose value is used for the independent variable in a calibration function	6
Higher order	Category of items forming part of a metrological framework above manufacturer/end-user methods, e.g., functions provided by reference laboratory or national measurement institute	7
International convention calibrator	Calibrator whose value of a quantity is not metrologically traceable to the SI but is assigned by international agreement	6
Matrix	All components of a material system except the analyte	6
Metrological traceability	Property of the result of a measurement or the value of a standard whereby it can be related to stated references, usually national or international standards, through an unbroken chain of comparisons all having stated uncertainties	6
Metrology	Science of measurement and its application	49
Reference material	Material, sufficiently homogeneous and stable with reference to specified properties, which has been established to be fit for its intended use in measurement	49
Reference measurement procedure	Measurement procedure accepted as providing measurement results fit for their intended use in assessing measurement trueness of measured quantity values obtained from other measurement procedures for quantities of the same kind, in calibration, or in characterizing reference materials	49

In this review, current sources of variability in method performance, sample type, and calibration materials influencing quantitative measurements are discussed in relation to three viruses (human cytomegalovirus, BK virus, and Epstein-Barr virus [EBV]) monitored in transplant recipients and immunocompromised patients and in relation to Pneumocystis jirovecii, where cutoff values to discriminate between colonization and disease causation are needed. Furthermore, Mycobacterium tuberculosis and influenza virus were selected as model pathogens and to highlight measurement parameters such as the limit of detection (LOD) and determination of cutoff values, which are important for quality control (QC) of qualitative assays. Both quantitative and qualitative tests would benefit from the implementation of a metrological framework of reference measurement procedures and reference materials by facilitating assessment of methodological performance and characterization of clinically used calibration and QC materials.

QUANTITATIVE METHODS

Concordance in quantitative viral DNA results between laboratories and between serial samples from a patient within the same laboratory is a prerequisite for definition of generally accepted clinical thresholds for viral infection and for monitoring disease initiation and progression. Sources of inter- and intralaboratory variability are discussed below with a focus on human cytomegalovirus (CMV) DNA load testing, together with examples from studies of BK virus and Epstein-Barr virus (EBV). Genome copy number measurements of Pneumocystis jirovecii are also included to illustrate further areas of testing where there is scope for standardization.

CMV viral load quantification.

In most cases, CMV infection is asymptomatic or manifests with nonspecific symptoms such as lymphadenitis, flu-like symptoms, and fatigue in the immunocompetent patient. CMV can give rise to congenital infections, while in immunocompromised patients such as organ transplant and HIV/AIDS patients, CMV infections can lead to pneumonia, encephalitis, retinitis, hepatitis, gastroenteritis, and colitis. Besides prophylactic antiviral treatment, preemptive treatment, involving serial laboratory monitoring for evidence of CMV replication, is a commonly used technique for antiviral therapy (1, 10). When viral load measurements reach a predictive threshold, the antiviral therapy is initiated (1). Clinical application of NAATs, most frequently real-time quantitative PCR (qPCR), allows less-laborious and more-sensitive monitoring of clinically significant viral loads than classically used immunofluorescence-based pp65 antigenemia tests (1). Linked with preemptive treatment, qPCR plays a crucial role in prevention of CMV disease and in reducing the duration of antiviral therapy through monitoring of CMV DNA levels in blood and alterations in their levels over time, reflecting viral kinetics. While CMV DNA cutoff values are defined within an individual laboratory for a specified patient group and assay to guide the initiation and termination of antiviral therapy, sources of methodological variability contribute to the current absence of a generally accepted viral load threshold value (1, 10).

Factors contributing to interlaboratory variability in CMV viral load quantification.

The use of different sample types (for example, whole blood and plasma), such that viral particles (the analyte) are present in different backgrounds (or “matrices”) (6), can lead to more than 10-fold differences in cutoff values (10). Moreover, recently reviewed surveys of laboratories showed a high degree of variability in CMV load reporting due to the range of detection reagents, amplification target genes, extraction methods used in clinical diagnosis, and type of calibrator (3). Variability in the efficiency of NA extraction and amplification leads to biases in measurements, which may lead to underestimation of viral load (11). Different calibrators can significantly influence slope and R² parameters of the calibration function due to their variable commutability (12), which can impact the final decisions on administration of preemptive treatment by leading to bias between methods and consequently to over- or underestimation of viral load. Furthermore, CMV assays are performed using different qPCR platforms differing in their thermal cycling parameters and detection systems, which have been observed to cause up to 1.5-fold (0.2 log₁₀) differences for the same set of samples (13). As there is no standardized interlaboratory threshold for initiation of treatment, it is generally recommended that the same NAAT assay and the same laboratory should be utilized to monitor patients over time (1).

These combined sources of variability contribute to a lack of interlaboratory concordance in viral load measurements. Data collected from an external quality assessment (EQA) scheme performed by INSTAND e.V. in 2014 showed that, for a series of samples with CMV concentrations of between 2 × 10³ and 1.5 × 10⁵ copies/ml, less than 20% of 55 participating laboratories were able to quantify all four samples within ±0.25 log₁₀, while the quantifications by 12% of laboratories fell outside ±0.8 log₁₀ (14). Up to 1,000-fold (3.0 log₁₀) differences were also obtained in other interlaboratory comparisons due to various combinations of extraction and detection methods, with commercial tests being less prone to variation than in-house ones (3, 4).

Traceability of values produced by fully automated CMV tests.

The availability of fully automated tests for CMV quantification that employ DNA extraction followed by qPCR has enabled an improvement in measurements for both approaches, possibly as a result of reducing the bias during manual sample transfer between steps. The Cobas AmpliPrep/Cobas TaqMan CMV Test (CAP/CTM CMV) (Roche) became the first U.S. Food and Drug Agency (FDA)-approved test for quantification of CMV from human plasma, while the CE-marked Abbott Real Time CMV Assay (Abbott) has been widely used for CMV quantification from whole blood and human plasma in the last few years. Both automated tests have already been calibrated using the 1st WHO IS and show good concordance (15). However, differences in the sensitivity of the two tests have been observed (15), which warrants traceable measurement results with respect to the LOD with well-defined reference materials in the long term.

Factors contributing to interlaboratory variability in viral load quantification of EBV and BK virus.

EBV, like CMV, is a member of the family Herpesviridae and causes infectious mononucleosis and different malignancies, including posttransplant lymphoproliferative disorders (PTLD) (16). BK virus, a member of the family Polyomaviridae, can cause nephropathy in kidney transplant recipients (17).

Variability in the cutoff values reported for BK and EBV infections is partially due to the number of sample types used for analysis: BK DNA levels are monitored in urine and plasma (17), while EBV DNA is measured in cell-associated and plasma fractions (18). Comparability of data from molecular assays for these pathogens is additionally hampered by the variety of calibrators and extraction and qPCR methods in use (3, 19, 20). In recent EQA schemes for virus genome detection, 38% of participating laboratories managed to quantify all samples in an EBV set within ±0.25 log₁₀, while for BK virus the success was even lower (25%). More than 10% of laboratories failed to provide results within ±0.8 log₁₀ (21, 22). Furthermore, 300-fold (2.5 log₁₀) differences for EBV and 30,000-fold (4.5 log₁₀) differences for BK virus were found between laboratories involved in interlaboratory assessment (3). In addition, discordance in BK viral load measurements can arise from sequence differences in the six subtypes of the virus (23). A standard reference material for BK virus is being developed by the National Institute of Standards and Technology (NIST) which aims to address this genotypic variation (http://www.nist.gov/mml/bmd/genetics/bk-virus.cfm).

Diagnostic cutoff values are required to discriminate between colonization and active disease.

As molecular methods such as qPCR are capable of detecting NA at very low levels, setting of cutoff values in the target copy number is relevant in order to inform whether the presence of a pathogen is the primary cause of an infection or reflects host colonization. For example, setting of copy number/ml thresholds in qPCR measurements of the opportunistic fungus Pneumocystis jirovecii in bronchoalveolar lavage samples was shown to improve the sensitivity and specificity of this assay in diagnosis of Pneumocystis jirovecii pneumonia (24).

QUALITATIVE METHODS

The aim of qualitative molecular methods is to discriminate between the presence and absence of a pathogen so that appropriate treatment is given when the presence of the pathogen is confirmed. Definition of the LOD and assurance of the specificity of the test to the intended bacterial or viral species/strain are paramount for this type of assay as discussed for M. tuberculosis and influenza virus.

Diagnostic tests for M. tuberculosis.

Diagnosis of tuberculosis (TB) is a lengthy procedure, and treatment is often initiated on an empirical basis. Delays in accurate diagnosis can lead to mistreatment and failure to identify multidrug-resistant tuberculosis (MDR-TB). While commonly used sputum smear microscopy allows rapid analysis of sputum samples, it is not specific and is insensitive, with a LOD of ∼10⁴ bacilli ml⁻¹ sputum (2). Smear-negative results require verification by bacterial culture, which has the advantage of a superior LOD, with automated liquid culture (Bactec MGIT 960; Becton Dickinson) demonstrating a detection limit of <10 organisms (25). However, culture-based methods suffer from the disadvantage of a long turnaround time (14 to 42 days) to confirm negativity in the MGIT system (Becton, Dickinson) due to the low growth rate of M. tuberculosis. Additionally, the decontamination process required for sputum culture and the use of selective antibiotics result in killing of a proportion of the M. tuberculosis present. Furthermore, results can be invalidated by overgrowth of other microorganisms.

NAATs for diagnosis of TB and confirmation of drug resistance offer improved sensitivity compared to microscopy, with a more rapid turnaround time than culture-based tests.

A key development in molecular diagnosis of TB is the Xpert MTB/RIF test, an automated system that combines purification of NA from clinical samples with detection of target DNA and mutations causing rifampin (RIF) resistance by nested qPCR in a single cartridge. This test has been widely evaluated following its endorsement by the WHO in 2010 (2). A number of molecular methods have also been applied to TB diagnosis, including strand-displacement amplification (SDA), the basis of the ProbeTec ET DTB test (Becton Dickinson) and loop-mediated isothermal amplification (LAMP) (26). However, molecular tests cannot completely replace phenotypic tests as not all mutations leading to drug resistance are known.

Problematic determination of diagnostic sensitivity and specificity for TB NAATs.

The LOD of the diagnostic test and its specificity for the M. tuberculosis complex versus nontuberculous mycobacterium (NTM) species are key factors in method performance. The diagnostic sensitivity and specificity of NA-based tests are commonly evaluated against results of culture or clinical diagnosis as a reference (26, 27). However, these metrics are dependent on the bacterial load associated with the patient cohort, which tends to be lower in HIV/AIDS patients (2), and on the prevalence of NTM (5), leading to large differences in the reported sensitivity of the Xpert MTB/RIF test (2, 28). More specifically, the Xpert MTB/RIF test demonstrates high specificity and sensitivity with smear-positive specimens, but sensitivity may be as low as 60% in smear negatives (28, 29). The accuracy of detection of RIF resistance is also variable, with a recent systematic review estimating sensitivity to be between 90% and 97% (29). While direct-comparison studies of different NAATs with the same sample set enable their relative performances to be evaluated (5), the lack of sample commonality hampers the long-term comparability of studies.

Diagnostic tests for influenza.

Influenza viruses, members of the family Orthomyxoviridae, are associated with infections causing substantial morbidity, mortality, and economic burden (30). Symptoms are typical of a large and diverse group of respiratory pathogens, posing a major challenge to diagnostic laboratories. Methods allowing rapid and reliable virus identification and determination of the subtype are crucial in investigating and managing outbreaks and monitoring the emergence and spread of strains of high pathogenicity (31). In terms of patient care, reliable diagnosis of influenza may aid the decision to start antiviral therapy in high-risk hospitalized patients or to avoid unnecessary use of antibiotics. Classical influenza virus diagnostics based on culture and immunofluorescence is time and labor-intensive and lacks desired sensitivity. Compared with conventional methods, NA-based assays offer superior sensitivity, specificity, and turnaround time and for these reasons have been widely implemented for detection of influenza viruses (31, 32).

Studies have shown various levels of sensitivity of NAATs for influenza diagnostics (32–34). The efficiency of both RNA extraction and subsequent amplification can contribute to differences with regard to LOD. Use of different specimen types and RNA extraction protocols significantly influences the performance of the diagnostic procedures (35). In addition, molecular diagnostics of RNA viruses depends on reverse transcription (RT)-conventional PCR or RT-qPCR, a two-step process highly influenced by the choice of reagents (36). Another variable is the type of the calibration material used to estimate the LOD expressed in RNA copy numbers. Calibration materials include genomic RNA from a known virus strain, in vitro-transcribed RNA, plasmids, and cDNA (37, 38); however, the impact of the type of calibrant used has not been investigated.

Determination of the assay sensitivity in terms of viral particles is usually performed using viral culture counted with electron microscopy, 50% tissue culture infective dose (TCID₅₀), or plaque assay. All of these methods are intrinsically associated with high uncertainty. The reported LODs of RT-PCR assays for influenza A and B viruses measured using dilutions of known virus titer typically span 2 orders of magnitude (32, 34, 39, 40). While the differences in the sensitivities of the above-mentioned examples cannot be directly compared due to different calibrators used for their evaluation, a few studies have assessed the sensitivity using a common sample pool allowing more-objective comparison. A recent EQA scheme using a sample set with different influenza A and B strains showed that 15% of the participating laboratories failed to detect all samples when they were using combined qualitative genome detection of influenza A and B viruses (41). Furthermore, an RT-qPCR method developed by the U.S. Centers for Disease Control and Prevention (CDC) to diagnose pandemic influenza A/H1N1 virus infections was compared with a commercial RNA extraction method coupled with RT-qPCR (RealStar system), demonstrating that 62.1% of samples identified as influenza A virus positive using the CDC method were found to be negative by the RealStar system. The RealStar system, however, showed improved sensitivity with regard to the subtype-specific target compared with the CDC method (33).

DEVELOPMENT OF REFERENCE MEASUREMENT SYSTEMS FOR PATHOGEN NAATS

Development and implementation of a reference measurement system for NA measurements of infectious agents will help to overcome some of the issues of measurement comparability highlighted above for quantitative and qualitative NAATs in microbiology through establishing metrological traceability of end-user measurement results, so that these can be compared “through time, distance, and different measurement procedures” (6). The roles of reference materials and reference measurement procedures within the reference measurement “framework” or “system” (6, 7, 9) are summarized below.

Traceability of reference materials for pathogen NAATs.

Higher-order reference materials carrying values traceable to the SI or another internationally accepted reference (IU) are key to improving the comparability and reducing the variability of quantitative viral load testing (12) by underpinning reproducible characterization of manufacturer's calibrators and assessment of the performance limits of microbiological tests with semiquantitative or qualitative outputs. The recent release of the 1st WHO IS for CMV and EBV will hopefully lead to better definition of viral load values with respect to clinical outcomes (42), as has been observed following the implementation of international standards for HIV and HCV (3). However, the recent adoption of the standards has not yet led to significant improvements in the comparability of results gathered through EQA schemes (V. James; presented at the 24th SoGAT meeting, Ljubljana, Slovenia, 8 to 9 May 2013). The development of reference measurement procedures for enumeration of viral particles and genome concentrations using well-characterized methods of high precision and accuracy would provide a reproducible definition of the measurement units and enable better comparison between batches of standards with values currently assigned in conventional units (IU), i.e., would improve long-term stability of a measurement system.

In the case of qualitative NAATs, the development of reference materials would provide quantitative measures of analytical sensitivity, which would enable the comparison of different platforms as well as the monitoring of platform performance longitudinally and between laboratories and countries. In addition to assigning concentration values of the analyte traceable to stable references, such as the SI, and demonstration of homogeneity and stability, it is necessary to demonstrate commutability of reference materials in order to ensure that the standard behaves similarly to typical clinical samples when processed using different routine methods (12).

dPCR within a potential higher-order reference measurement procedure.

In order for reference materials to provide better insight into method performance as well as interlaboratory measurement result equivalence, higher-order reference measurement procedures are also required to allow better traceability of values assigned to reference materials (6). Digital PCR (dPCR), a single-molecule amplification approach based on limiting dilution, is a potential higher-order reference measurement method for infectious disease NA testing as it offers a quantification approach which, unlike the majority of qPCR-based NAATs, does not rely on the use of a calibrator containing the analyte subject to measurement (43). Microfluidic circuit and droplet-based dPCR formats are available, among which BioMark arrays (Fluidigm) and QX100 and QX200 (Bio-Rad) are the most extensively tested platforms for microbial analysis. dPCR has been shown to be a highly precise measurement technique, especially for relatively concentrated materials (44), and has demonstrated good accuracy for certification of standards for genetically modified organism testing (45). dPCR showed good discrimination between methicillin-resistant and methicillin-sensitive Staphylococcus aureus isolates; thus, it might enable better discrimination of different genotypes of M. tuberculosis (46). Furthermore, it has been successfully applied in a one-step format for the quantification of RNA viruses showing increased tolerance to inhibitors and higher accuracy at lower concentrations than qPCR (47). Some limitations in the clinical sensitivity of dPCR have been observed which may be linked to restricted sample volume inputs for dPCR platforms (44). However, this may be improved by concentration of master mixes (48) and/or sample extracts.

The INFECT-MET project will offer metrological support.

In summary, NAATs have improved diagnosis of pathogenic viruses and bacteria in the last decade. This review highlights that the implementation of reference materials and reference measurement procedures and improved characterization of NAATs would benefit molecular diagnostics in both clinic- and field-based settings. Several efforts with the aim of increased standardization are ongoing worldwide, including studies by the Standardization of Genomic Amplification Techniques (SoGAT) Working Group and Quality Control for Molecular Diagnostics (QCMD), through the development of new reference materials and EQA schemes. However, metrological support of the diagnostic methods is still needed in order to define typically used IS in terms of the quantities being measured (e.g., DNA copy numbers) and ultimately to provide traceability to the SI system of units. The “Metrology for Monitoring Infectious Diseases, Antimicrobial Resistance, and Harmful Micro-Organisms” (INFECT-MET) project, run under the European Metrology Research Programme (EMRP), aims to address measurement issues that impact the monitoring of infectious diseases, antimicrobial resistance, and harmful microorganisms. The project involves nine partners from five member countries of the European Union. The ultimate goal is to improve the robustness, accuracy, comparability, and traceability of end-user measurement results through the development of higher-order methods and reference materials for infectious disease molecular diagnostics.

Biography

Alison Devonshire read Natural Sciences (Biochemistry) at Queens' College, Cambridge (United Kingdom). She worked as a clinical research associate implementing phase II and III trials based in Germany and the United Kingdom before studying for her Ph.D. in the transcriptional regulation of genes involved in metabolic diseases at the University of Surrey. In 2008, Alison joined the Molecular and Cell Biology group at LGC, which is the United Kingdom National Measurement Institute (NMI) for bioanalytical measurements. She specializes in nucleic acid measurements using techniques such as real-time PCR and digital PCR and their application in clinical diagnostics and cell-based assays. In her current role as Science Leader in Nucleic Acid Metrology, Alison is working on projects addressing the measurement challenges of standardization of molecular methods for cancer biomarkers and infectious diseases through the development of reference materials and reference methods.