Accuracy of quantitative viral secondary standards: a re-examination

ABSTRACT

Interlaboratory agreement of viral load assays depends on the accuracy and uniformity of quantitative calibrators. Previous work demonstrated poor agreement of secondary cytomegalovirus (CMV) standards with nominal values. This study re-evaluated this issue among commercially produced secondary standards for both BK virus (BKV) and CMV, using digital polymerase chain reaction (dPCR) to compare the materials from three different manufacturers. Overall, standards showed an improved agreement compared to prior work, against nominal values in both log₁₀ copies/mL and log₁₀ international unit (IU)/mL, with bias from manufacturer-assigned nominal values of 0.0–0.9 log₁₀ units (either copies or IU)/mL. Standards normalized to IU and those values assigned by dPCR rather than by real-time PCR (qPCR) showed better agreement with nominal values. The latter reinforces prior conclusions regarding the utility of using such methods for quantitative value assignment in reference materials. Quantitative standards have improved over the last several years, and the remaining bias from nominal values might be further reduced by universal implementation of dPCR methods for value assignment, normalized to IU.

IMPORTANCE

Interlaboratory agreement of viral load assays depends on accuracy and uniformity of quantitative calibrators. Previous work, published in JCM several years ago, demonstrated poor agreement of secondary cytomegalovirus (CMV) standards with nominal values. This study re-evaluated this issue among commercially produced secondary standards for both BK virus (BKV) and CMV, using digital polymerase chain reaction (dPCR) to compare the materials from three different manufacturers. Overall, standards showed an improved agreement compared to prior work, against nominal values, indicating a substantial improvement in the production of accurate secondary viral standards, while supporting the need for further work in this area and for the broad adaption of international unit (IU) as a reporting standard for quantitative viral load results.

INTRODUCTION

Quantitative testing for blood-borne viruses is required standard-of-care for highly immunocompromised patients, particularly those who have undergone transplant. The value, actionability, and portability of such measures of DNAemia have been limited by the suboptimal uniformity of the results among testing laboratories. Early studies showed very poor quantitative consensus when samples were tested in multicenter studies for analytes including cytomegalovirus (CMV), Epstein-Barr virus, and BK virus (BKV), among others (1–4). The disharmony of results was thought likely due to many factors (5), but a primary concern was the lack of existing international quantitative standards by which to calibrate these assays. As such standards began to be produced (6–8), and as commercial entities made secondary standards available to the clinical testing community, it was anticipated that an inter-assay agreement would see rapid, marked improvements.

While improvements were seen, based on the results of both external proficiency testing and multicenter studies, the lack of inter-assay agreement has remained a challenge (9, 10). The reasons for the continuing disharmony of results can again be attributed to numerous sources of variability, including differences in sample preparation methods, primer-probe design, and other elements of assay design, which are in turn inter-woven with commutability of assay-standard systems, a necessary element for quantitative accuracy and agreement. Beyond these potential issues, it became clear that secondary standards must accurately reproduce the primary international reference material upon which they are based and show minimal bias from their assigned nominal values. Earlier work showed that this was not the case for then commonly available CMV standards and suggested that such inaccuracies might be limited by using digital polymerase chain reaction (dPCR) as a means of normalizing material to international unit (IU) (11). Here, we re-examine this issue among commercially produced secondary standards for both BKV and CMV, looking for both progress and remaining opportunities to improve the fidelity of these critical assay components.

MATERIALS AND METHODS

Quantitative standards

Commercial, whole virus, secondary standards of human BK virus (BKV) and cytomegalovirus (CMV) were obtained from Bio-Rad Laboratories (Exact Diagnostics; Bio-Rad, Fort Worth, TX), Antylia Scientific company (ZeptoMetrix LLC; Buffalo, NY), and Thermo Fisher Scientific, Inc (AcroMetrix; Fremont, Ca), while the World Health Organization (WHO) International Standards for BKV (Code: 14/212) and CMV (Code: 09/162) were purchased from The National Institute for Biological Standards and Control (NIBSC, Potters Bar, Hertfordshire, UK). Detailed information for CMV and BKV standards is listed in Tables 1 and 2, respectively. The Bio-Rad BKV product is labeled for In Vitro Diagnostic use, while all other commercial standards are labeled for Research Use Only. The materials from Bio-Rad and the CMV product from Thermo Fisher were provided with manufacturer-assigned nominal values (hereafter referred to as “nominal values”) assigned in IU. The copy numbers for these products were provided by the manufacturers for this study with the caveat that this was not their intended use. The materials from Antylia were provided with nominal values assigned in copies and with a conversion factor to IU listed in the package inserts. Experimentally derived results by dPCR were largely in agreement with nominal values provided by NIBSC for both WHO standards (Table 3).

TABLE 1.

Characteristics of cytomegalovirus (CMV) standards

Manufacturer	Bio-Rad (Exact Diagnostics)	Antylia (ZeptoMetrix)	Thermo Fisher Scientific (AcroMetrix)	NIBSC
Product name	CMV Verification Panel	NATtrol Cytomegalovirus Linearity Panel	AcroMetrix CMVtcPanel	First WHO International Standard for Human Cytomegalovirus (09/162)
Viral strain	Merlin	AD-169	AD-169	Merlin
Matrix	EDTA Plasma	Purified protein matrix	EDTA Plasma	Lyophilized equivalent of 1 mL solution of 10 mM Tris-HCL (pH 7.4) and 0.5% human serum albumin
Extractable (y/n)	Yes	Yes	Yes	Yes
Manufacturer nominal value unit assignment	Five 10-fold concentrations provided, 2.60 to 6.60 log10 IU/mL	Five 10-fold concentrations provided, 2.70 to 6.70 log10 copies/mL	Five 10-fold concentrations provided, 2.48 to 6.48 log10 IU/mL	6.70 log10 IU/mL
Means of viral load assignment (primary quantitative measurement)	ddPCR (Droplet Digital PCR)	qPCR	qPCR	Consensus value assignment (32 laboratories)

TABLE 2.

Characteristics of BK virus (BKV) standards

Manufacturer	Bio-Rad (Exact Diagnostics)	Antylia (ZeptoMetrix)	Thermo Fisher Scientific (AcroMetrix)	NIBSC
Product name	BKV Verification Panel	NATtrol BK Virus Linearity Panel	AcroMetrix BKV Panel	First WHO International Standard for BK virus (14/212)
Viral strain	Dunlop, subtype Ia	Gardner	Not disclosed	BKV 1b-2
Matrix	EDTA Plasma	Purified protein matrix	EDTA Plasma	Lyophilized equivalent of 1 mL solution of 10 mM Tris-HCL (pH 7.4), 0.5% human serum albumin, and 0.1% D-(+)-Trehalose dehydrate
Extractable (y/n)	Yes	Yes	Yes	Yes
Manufacturer nominal value unit assignment	Five 10-fold concentrations provided, 2.30 to 7.30 log10 IU/mL	Five 10-fold concentrations provided, 4.00 to 8.00 log10 copies/mL	Five 10-fold concentrations provided, 2.70 to 6.70 log10 copies/mL	7.20 log10 IU/mL
Means of viral load assignment (primary quantitative measurement)	ddPCR (Droplet Digital PCR)	qPCR	qPCR	Consensus value assignment (33 laboratories)

TABLE 3.

Comparison of experimentally derived values (dPCR) versus nominal values for WHO standards

Nominal values in log10 IU/mL	Experimentally-derived results in log10 copies/mL Mean (Standard Deviation)
	Bio-Rad	Roche
CMV WHO standard
5.7	5.59 (0)	5.39 (0.01)
4.7	4.56 (0.01)	4.42 (0.02)
3.7	3.55 (0.02)	3.34 (0.08)
2.7	2.62 (0.08)	2.19 (0.15)
BKV WHO standard
5.2	5.34 (0)	5.15 (0)
4.2	4.44 (0.02)	4.27 (0.03)
3.2	3.4 (0.08)	3.13 (0.03)
2.2	2.02 (0.17)	1.87 (0.3)

Nucleic acid extraction

An aliquot of 0.4 mL of each standard sample was extracted into a 0.12 mL eluant using EZ1 Virus Mini Kit v2.0 on an EZ1 Advanced XL instrument with a virus card v2.0 (Qiagen, Germantown, MD) according to manufacturer’s recommendation. The extracted DNA solution was aliquoted into two of 0.06 mL per vial and frozen at −20°C to keep the same freeze/thaw for all tests.

Digital polymerase chain reaction (dPCR)

Bio-Rad QX200 and QXDx droplet digital PCR system (Bio-Rad, Pleasanton, CA)

The dPCR reaction mixture consisted of 12.5 µL of a 2× ddPCR Supermix (Bio-Rad), 0.2 µM of RealStarASR BKV (or CMV) Primers, 1 µL of RealStarASR BKV (or CMV) Probe (altona Diagnostics GmbH, Hamburg, Germany), and 10 µL of nucleic acid extract in a final volume of 25 µL. The entire reaction mixture was used to produce droplets on the QX200 Auto-Droplet Generator (Bio-Rad). After processing, the droplets were collected in a 96-well PCR plate (Eppendorf, Germany) and amplified on a C1000 Thermal Cycler (Bio-Rad) using a thermal profile of beginning at 95˚C for 10 minutes, followed by 40 cycles at 94˚C for 30 seconds and 58˚C for 60 seconds, 1 cycle at 98˚C for 10 minutes, ending at 12˚C. The plate was read on the Droplet Reader (Bio-Rad) at a rate of 32 wells per hour. The data were analyzed using QuantaSoft analysis software (QuantaSoft Version 1.7.4.0917, Bio-Rad), and the results were reported as copies per milliliter of original sample.

Digital LightCycler system (Roche Diagnostics International Ltd, Rotkreuz, Switzerland)

The dPCR reaction mixture consisted of 10 µL of a 5× Digital LightCycler DNA Master reagent, 0.2 µM of RealStarASR BKV (or CMV) Primers and 2 µL of RealStarASR BKV (or CMV) Probe (altona Diagnostics GmbH, Hamburg, Germany), and 10 µL of nucleic acid extract in a final volume of 50 µL. A total of 45 µL of dPCR reaction mixture was pipetted into one of the eight sample intake ports of a high sensitivity nanowell plate (Roche Diagnostics). The plate was loaded on a Digital LightCycler Partitioning Engine to produce the partitions. After partitioning, the plate was transferred to a Digital LightCycler where the PCR was performed and the results were analyzed. Thermal profile of PCR for both viruses starts at 95°C for 10 minutes, followed by 40 cycles at 94°C for 30 seconds and 58°C for 60 seconds, and one cycle at 98°C for 10 minutes, ending at 12°C. Raw data were exported and analyzed using Digital LightCycler Development Software (Version 1.0.59, Roche Diagnostics). The results were reported as copies/milliliter of the original sample.

Statistical analysis

Mean and standard deviation were reported to describe the numeric variables. The conversion of dPCR values from copies/milliliter to IU utilized a linear relationship between the measured results in copies/milliliter and the nominal values in IU/milliliter within the WHO dilutions. Bias was defined as the difference between the mean measured (or normalized) results and the nominal values for each concentration of standard. Overall bias represented the mean bias across multiple concentrations for each manufacturer. The Wilcoxon signed-rank test was used to test whether the bias was significantly different from 0 (unbiased). Simple linear regression analysis was conducted to investigate the relationship between the normalized measured and nominal values, as well as the measured values of between dPCR measures. The resulting slope indicated the strength of the quantitative association. P-values were two-sided and considered to be statistically significant at 5% level. All analyses and figures were conducted in SAS version 9.4 (Cary, NC).

RESULTS

CMV standards

Bias compared to nominal values was first calculated based on nominal log₁₀ copies/mL provided by each manufacturer, for CMV showing a negative bias of detected DNA concentration by dPCR of approximately 0.35–0.4 log₁₀ copies/mL for Exact and 0.2–0.8 log₁₀ copies/mL for AcroMetrix (Acro) standards (the latter showing a greater difference in results depending on which dPCR platform was used). A positive bias of approximately 0.5 log₁₀ copies/mL was seen for ZeptoMetrix (Zepto) material (Fig. 1A; Table S1). The bias was similar across the full concentration gradient of tested standards (Fig. 2) and persisted to some degree but markedly reduced in magnitude when the secondary standards were normalized to IU and compared to nominal log₁₀ IU/mL, with a bias as low as 0.02 log₁₀ IU/mL seen with Bio-Rad reagents run on the Roche platform (Fig. 1B; Fig. S1; Tables S2 and S3). The comparison of values detected by the two dPCR platforms were largely in agreement but showed a persistently lower concentration of detection by the Roche platform for the Acro material across the full linear range of tested materials (Fig. 3; Table S4).

Fig 1.

Overall bias between the measured results and the nominal values provided by each manufacturer for CMV secondary standards in log₁₀ copies/mL (A) and bias between the normalized measured results and the nominal values in log₁₀IU/mL (B) as measured on each platform (x-axis).

Fig 2.

Regression analysis of nominal values of CMV in log₁₀ copies/mL compared against measured values on each platform (Bio-Rad and Roche digital PCR systems).

Fig 3.

Regression analysis of measured values of Roche compared against measured values of Bio-Rad for CMV.

BKV standards

Bias compared to nominal values was first calculated based on nominal log₁₀ copies/mL provided by each manufacturer, for BKV showing a negative bias of detected DNA concentration by dPCR of approximately 0.3–0.4 log₁₀ copies/mL for Exact and 0.1 log₁₀ copies/mL for Acro standards (Fig. 4A; Table S5). A positive bias of approximately 0.6–0.9 log₁₀ copies/mL was seen for the Zepto material. The bias was similar across the full concentration gradient of tested standards (Fig. 5). Notably, only one product (Exact) was labeled with nominal values in log₁₀ IU/mL, and one (Zepto) included a conversion factor to assign IU values. Bias was reduced and minimal when the Exact secondary standard dPCR results were normalized to IU and compared to nominal log₁₀ IU/mL, but bias was greater with the Zepto product (Fig. 4B; Fig. S2; Tables S6 and S7). The comparison of values detected by the two dPCR platforms were largely in agreement (Fig. 6; Table S8).

Fig 4.

Overall bias between the measured results and the nominal values provided by each manufacturer for BKV secondary standards in log₁₀ copies/mL (A) and bias between the normalized measured results and the nominal values in log₁₀IU/mL (B) as measured on each platform (x-axis).

Fig 5.

Regression analysis of nominal values of BKV in log₁₀ copies/mL compared against measured values on each platform (Bio-Rad and Roche digital PCR systems).

Fig 6.

Regression analysis of measured values of Roche compared against measured values of Bio-Rad for BKV.

DISCUSSION

The results here indicate a marked improvement in agreement between viral quantitative standards and a reduction in bias from nominal values since this issue was last examined in 2015 (11). In that study, mean bias for CMV was 0.6 to −1.0 log₁₀ IU/mL, while here, most CMV standards were within 0.2 log₁₀ IU/mL of nominal values, with a bias range of −0.48 to 0.06. The adaptation of dPCR for value assignment and normalization to IU may be factors to which this improvement can be attributed. The improved trueness and inter-manufacturer consistency of these materials are critical to improve the test performance and interlaboratory agreement of patient testing results, required for result portability, common interpretation of the literature, and the determination of consensus thresholds for result interpretation and clinical decision-making (12). There remains some degree of bias from nominal values which must continue to be addressed.

Most CMV standards normalized to IU showed bias values less than 0.2 log₁₀ IU/mL and therefore likely of minimal clinical significance. The copy number designations showed greater bias. The copy number designations for the Acro and Zepto BKV materials (both value-assigned by qPCR) were somewhat disparate. Acro material was within 0.1 log₁₀ copies/mL of nominal values, and Zepto showed a positive bias of 0.6–0.9 log₁₀ copies/mL. Exact material (value-assigned by dPCR) showed a negative bias of 0.2–0.4 log₁₀ copies/mL. The latter material showed negligible bias when looked at in log₁₀ IU/mL while Zepto material again showed a positive bias. dPCR has been shown to have improved accuracy and precision over qPCR, and some have used it as a reference method for DNA quantitation, with a potential for initial value assignment, for normalization of secondary standards to IU, and for lot-to-lot normalization of primary standards (13–15). dPCR was advocated for its use in our prior work to improve the accuracy of secondary standards, and the present study indicates that the adaptation of such methods can help improve trueness and reduce bias in such materials (evident here principally in the BKV results); their use should continue to be encouraged.

Nonetheless, dPCR has its own limitations, as shown particularly by the bias results for CMV, with differing bias seen for two of the standards, depending on which dPCR system was used, despite the use of common extraction eluates and primer-probe sets. It has been noted previously that, while more robust than real-time PCR, dPCR can be vulnerable to issues including PCR dropout wherein the target in some partitions fails to amplify (16). This can only serve to help emphasize the need for thoroughly optimizing and validating any method used for value assignment, assessment, or clinical testing (17). While such cautionary notes do not reduce the potential value of dPCR in this setting or obviate the recommendation for its use by manufacturers, they do raise questions regarding the best means of primary value assignment for international standards, wherein true reference methods such as mass spectrometry (18) might be utilized as an even more reliable tool. The differences in bias seen here between platforms, primarily for CMV, do not change our conclusions but cannot be definitively explained. One could speculate that the use of differing viral strains among secondary standard manufacturers could result in variable primer and probe binding affinities. The differences in stringency between dPCR mastermixes specific to each system could then affect the risk of PCR dropout differentially. The platforms also differed from one another in that Bio-Rad uses droplet-based partitioning, while Roche uses hard partitions. It is possible that the droplet-based partitioning is less susceptible to PCR dropout in the presence of longer amplicons. As such, these differences in the system design might affect accuracy and bias, particularly for larger viruses, such as CMV. An evaluation of these possible explanations could not be performed here as primer/probe sequences and mastermix compositions were proprietary and unavailable for further analysis. It should be noted that only a single lot of each product could be tested for this study. Other lots may show some differences in bias, based on manufacturers’ acceptable product variability for any given concentration of standard material.

This study was also limited by its inclusion of only two of the viruses for which WHO and secondary standards are available. It is reasonable to suspect that the data here are indicative of broader trends, but reference materials for other viruses or manufacturers, as available, will need individual evaluation. Only three of the secondary standards evaluated here (from two manufacturers) are sold with primary value assignment in IU. Zepto material is sold with values assigned in copies, accompanied by a multiplier for conversion to IU. The other copy number assignments were provided by their manufacturers (Bio-Rad and Thermo) for this study with the caveat that this was not their intended use, and that accuracy was not assured outside of their specific IU designation. In fact, less bias from nominal values was seen for these materials when reported in IU compared to the results reported in copies. The commercial availability of secondary standards with nominal IU designations facilitates the use of IU as a common reporting unit. Where such materials are not available, clinical laboratories must assume the task of normalizing to IU, potentially exposing the process to more sources of uncertainty and the introduction of further bias. The relative effect on trueness when using a multiplier to assign IU values across different assays that may be employed by end users is also uncertain.

The data here indicate progress in the production of accurate secondary, quantitative viral standards. Further adaptation of dPCR and perhaps other reference methods for the development of both primary and secondary standards can be expected to bring further improvement. Increasing availability of secondary standards normalized to IU could also reduce potential sources of variability, while also enabling clinical laboratories to more universally adopt IU as a reporting standard. The continued improvement of reference materials for quantitative measures of viral DNA and the broad adaptation of IU for result reporting are essential to improve testing accuracy and interlaboratory agreement, allowing the potential clinical value of quantitative assays of viral nucleic acid to be more fully realized.

SUPPLEMENTAL MATERIAL

The following material is available online at https://doi.org/10.1128/jcm.01669-23.

Tables S1 - S8; Figures S1 and S2. jcm.01669-23-s0001.docx.

Bias and regression values of measured vs nominal values in both copies and IU. Regressions of viral loads normalized to IU on both testing platforms.

ASM does not own the copyrights to Supplemental Material that may be linked to, or accessed through, an article. The authors have granted ASM a non-exclusive, world-wide license to publish the Supplemental Material files. Please contact the corresponding author directly for reuse.