Abstract
Three kits (Roche AMPLICOR human immunodeficiency virus type 1 [HIV-1] Monitor, Chiron enhanced-sensitivity bDNA, and Organon Teknika NASBA HIV-1 QT) and two in-house assays (from National Genetics Institute and Baylor College of Medicine) were compared with a blinded panel. The results were evaluated as to intra-assay sensitivity, precision, and ability to detect differences in a dilution series.
The quantification in plasma of human immunodeficiency virus type 1 (HIV-1) RNA is a useful test for predicting clinical outcome, evaluating disease progression, and monitoring the effect of antiretroviral treatment on HIV-1-infected patients (6, 9). The primary reason for undertaking this project was to evaluate the performance of selected methodologies or laboratories for monitoring the effect of antiretroviral therapy on HIV-1 viral load in clinical research. Three manufacturers of HIV-1 RNA kits and an independent laboratory agreed to participate in a single round of proficiency testing on a panel of plasma specimens. The laboratories performing the tests were chosen by the manufacturers. Enhanced-sensitivity bDNA assays (ES-bDNA) were performed at the Chiron Reference Testing Laboratory (Emeryville, Calif.), Roche AMPLICOR HIV-1 Monitor tests were carried out at Novum, Inc. (Pittsburgh, Pa.), and NASBA (nucleic acid sequence-based amplification) HIV-1 QT assays were performed at Advanced Bioscience Laboratories, Inc. (Kensington, Md.). National Genetics Institute, or NGI (Culver City, Calif.), an established independent laboratory, used its in-house method. Because the panel samples were prepared at the Center for AIDS Research Virology Core Facility at Baylor College of Medicine (BCM), we have also included in this report the results obtained with the in-house method that was used to establish their nominal concentrations.
Summarized in Table 1 are the differences among the five assays. The ES-bDNA assay for HIV-1 RNA is a second-generation assay developed by the Chiron Corporation (Emeryville, Calif.) and incorporates changes in probe design and reagent composition (3). The distinctive feature of the NGI in-house assay was the use of four different numbers of PCR cycles (1). The AMPLICOR HIV-1 Monitor kit, manufactured by Roche Molecular Systems (Branchburg, N.J.), employed Thermus thermophilus polymerase, a recombinant enzyme capable of carrying out both reverse transcription (RT) and PCR (7). Assays based on NASBA technology developed by Organon Teknika (Durham, N.C.) were carried out under isothermal conditions and resulted in an RNA amplification product (11). The BCM method employed the Qiagen viral RNA kit (Santa Clarita, Calif.) for sample preparation and chemiluminescence assay of PCR products (Digene Diagnostics, Inc., Silver Spring, Md.) (5).
TABLE 1.
Methods employed for quantification of HIV-1 RNA
| Item | Parameters for:
|
||||
|---|---|---|---|---|---|
| Chiron ES-bDNA | NGI (in-house) | Roche AMPLICOR HIV-1 Monitor | Organon Teknika NASBA HIV-1 QT | BCM (in-house) | |
| Standards (Source: kit or in-house) | External standards (kit): single-stranded DNA encoding gag and pol genes | External standards (in-house): dilution series prepared from pooled HIV-1-positive sera and plasma | Internal standard (kit): poly(A)-linked RNA containing conserved and rearranged sequences within the gag region | Internal standards (kit): 1,500-nucleotide RNA homologous to the gag/pol region, modified over a 20-nucleotide sequence | External standards (in-house): dilution series prepared from plasma spiked with strain HXB3 virions |
| Sample volume; sample preparation | 1.0 ml, in duplicate; sedimentation of HIV-1 at 23,500 × g/1 h, protease K digestion | 200 μl; extraction with guanidinium thiocyanate-phenol-chloroform, alcohol precipitation | 200 μl; precipitation from guanidinium thiocyanate reagent, isopropanol wash | 100 μl; lysis in guanidinium thiocyanate reagent, adsorption to activated silica | 140 μl; spin column preparation with a commercial kit |
| Amplification | Signal amplification by means of hybridization of target region to probe, preamplifier, and bDNA molecules | cDNA synthesis with random hexamers, PCR with four different cycle numbers with primers SK38 and SK39 or SK145 and SK431 | Coamplification of standards and sample HIV-1 RNA by means of RT-PCR with SK462 and SK431 | Coamplification of standards and sample by means of reverse transcriptase-, T7 polymerase- and RNase H-mediated reactions with primers conjugated to T7 promoter | RT-PCR with primers SK38 and 5′ biotinylated SK39 |
| Detection | Chemiluminescence | Scanning densitometry | Spectrophotometry | Electrochemiluminescence | Chemiluminescence |
HIV-1 standards and panel samples were prepared with normal acid-citrate-dextrose (ACD) plasma from individual blood donors who tested negative for anti-HIV-1, anti-HIV-2, anti-hepatitis B core, hepatitis B surface antigen, anti-hepatitis C virus, and anti-human T-cell leukemia virus type 1 (The Gulf Coast Regional Blood Center, Houston, Tex.). The primary standard was prepared by spiking negative plasma with supernatant fluid from a coculture of peripheral blood mononuclear cells from an HIV-1-infected patient (4). The secondary standard and panel samples were prepared by spiking negative plasma with HIV-1 strain HXB3 (5). Calibration of the secondary standards against the primary standard was based on 10 observations performed in five separate experiments with a between-runs coefficient of variation (CV) of 11.8%. The panel consisted of duplicate samples at 2.0 × 106, 1.0 × 106, 500,000, 130,000, and 65 HIV-1 RNA copies per ml; triplicate samples at 65,000, 13,000, 6,500, 1,300, 650, and 130 HIV-1 RNA copies per ml; and four samples of negative plasma. Aliquoting of samples was performed under optimal conditions to ensure sample uniformity. Specimens were shipped to testing sites on dry ice, and testing was performed in a blinded fashion at all sites, including the BCM laboratory.
Presented in Table 2 are the results of HIV-1 RNA quantification for each sample as reported by the testing sites, compared with the nominal HIV-1 RNA concentrations. In comparisons of those results that fell within the dynamic range claimed for each method (listed in a footnote to the table), the differences between the means of results for replicate samples that were obtained with the Chiron, NGI, BCM, and Roche assays generally fell within the same order of magnitude.
TABLE 2.
Quantification of HIV-1 RNA in 32 plasma specimens by five different methods
| Nominal no. of estimated HIV-1 RNA copies/mla | No. of HIV-1 RNA copies/ml estimated byb:
|
||||
|---|---|---|---|---|---|
| Chiron ES-bDNA | NGI (in-house) | BCM (in-house) | Roche AMPLICOR HIV-1 Monitor | Organon Teknika NASBA HIV-1 QT | |
| 2,000,000 | (>1,600,000) | 1,300,000 | 2,050,000 | (958,023) | (19,000,000) |
| 2,000,000 | (>1,600,000) | 1,100,000 | 2,450,000 | (882,538) | 8,800,000 |
| 1,000,000 | 1,050,000 | 650,000 | 1,860,000 | (836,158) | 4,900,000 |
| 1,000,000 | 1,172,000 | 600,000 | 1,350,000 | 573,264 | 5,300,000 |
| 500,000 | 548,900 | 450,000 | 450,000 | 329,052 | 4,400,000 |
| 500,000 | 463,300 | 450,000 | 585,000 | (1,291,354) | 3,700,000 |
| 130,000 | 129,500 | 85,000 | 132,500 | 193,356 | 1,400,000 |
| 130,000 | 159,200 | 100,000 | 132,500 | 100,474 | 940,000 |
| 65,000 | 82,060 | 75,000 | 69,500 | 176,522 | 1,100,000 |
| 65,000 | 79,950 | 75,000 | 105,000 | 60,774 | 460,000 |
| 65,000 | 81,260 | 45,000 | 42,500 | 79,388 | 500,000 |
| 13,000 | 19,200 | 10,000 | 5,475 | 16,084 | 250,000 |
| 13,000 | 14,360 | 10,000 | 17,250 | 55,181 | 130,000 |
| 13,000 | 10,950 | 9,000 | 13,300 | 48,605 | 160,000 |
| 6,500 | 5,260 | 6,000 | 8,350 | 7,498 | 66,000 |
| 6,500 | 6,030 | 6,500 | 5,425 | 10,188 | 91,000 |
| 6,500 | 6,022 | 6,000 | 9,100 | 8,187 | 1,000,000 |
| 1,300 | 823 | 1,500 | 1,630 | 5,982 | 5,300 |
| 1,300 | 1,008 | 350 | 1,000 | 1,321 | 13,000 |
| 1,300 | 793 | 500 | 2,400 | 1,631 | 11,000 |
| 650 | 680 | 800 | 500 | 1,162 | 7,700 |
| 650 | (<500) | 600 | 640 | 1,911 | 8,700 |
| 650 | (<500) | 500 | 1,200 | (206) | 12,000 |
| 130 | (<500) | 300 | (<500) | 421 | (<4,000) |
| 130 | (<500) | 150 | (<500) | (123) | (<4,000) |
| 130 | (<500) | 200 | (<500) | (221) | (<4,000) |
| 65 | (<500) | 200 | (<500) | (341) | (<4,000) |
| 65 | (<500) | 150 | (<500) | (188) | (<4,000) |
| 0 | (<500) | Negative | (<500) | Negative | (<4,000) |
| 0 | (<500) | Negative | (<500) | Negative | (<4,000) |
| 0 | (<500) | Negative | (<500) | Negative | (<4,000) |
| 0 | (<500) | Negative | (<500) | Negative | (<4,000) |
As defined by standards produced at BCM.
The dynamic ranges of the assays claimed by the manufacturers or laboratories were as follows (in copies per milliliter): for ES-bDNA, 500 to 1.6 × 106; for NGI, 100 to 2.0 × 106; for BCM, 500 to 2.5 × 106; for Roche, 400 to 0.75 × 106; for NASBA, 4,000 to 15 × 106. Reported results that fell above or below the claimed dynamic range are in parentheses.
The data in Table 2 were evaluated with respect to the ability of the assays to detect the known differences in HIV-1 RNA concentration present in the dilution series (Fig. 1). Nominal and estimated HIV-1 RNA concentrations were expressed as logarithms to the base 10. For each method, the mean of the results for each group of two or three panel samples with the same nominal HIV-1 concentration was calculated and represented as a point in the appropriate panel in Fig. 1. Groups that contained results falling outside the claimed dynamic range were omitted. For example, the ES-bDNA assay results for the three samples with 500 nominal HIV-1 RNA copies per ml and two samples with nominal copy numbers of 2 × 106 were omitted. If an assay method correctly estimated the relationship of the various samples in the dilution series, a log-log plot of the estimated mean HIV-1 RNA concentrations against the nominal concentrations should produce a straight line having the same slope as the line of equivalence, shown in each panel as a diagonal straight line. The data set exhibiting the closest approximation of a straight line parallel to the line of equivalence was obtained by ES-bDNA. With perfect accuracy, the line representing the assay results should fall on the line of equivalence. This criterion must be applied with discretion, since the nominal concentrations of the panel samples are at best only estimates of an unknown value, the true HIV-1 RNA concentration. The results obtained with the Chiron, NGI, BCM, and Roche assays fell closest to the line of equivalence. These four assays provided estimates of HIV-1 RNA concentrations that generally agreed with each other and with the nominal concentrations of the BCM standards. In this sense, these assays were accurate. The slope of the line for the NASBA test also paralleled the line of equivalence but was displaced from it by approximately one log10.
FIG. 1.
Linearity of HIV-1 RNA estimates performed on serially diluted spiked specimens by different assay methods. Assays employed were ES-bDNA, NGI, BCM, AMPLICOR HIV-1 Monitor, and NASBA HIV-1 QT. The straight diagonal line represents the equivalence of nominal and estimated HIV-1 RNA concentrations. Estimated concentrations for the dilution series that were obtained for a given assay are shown as connected points. Each point represents the mean of two or three estimates. If one or more of the reported results for a point fell outside the dynamic range of the assay, that point was omitted.
The performance characteristics of the five assays are summarized in Table 3. The calculations of correlation, CV, and standard deviation (SD) were based solely on those results that fell within the claimed dynamic ranges. Correlation between the observed and nominal concentrations was close to 1.000 for all five assays, although the results obtained by the NASBA assay were consistently higher. Intra-assay precision was expressed by the CV and as the logarithm of the SD to the base 10 (log10 SD), a calculation that conveniently allowed comparison of SDs when the range of values covered several orders of magnitude. The ES-bDNA assay showed the smallest mean CV. An intra-assay SD of 0.15 to 0.20 log10 RNA copies per ml is required in order to achieve the degree of precision that will provide 90% power to detect a fivefold difference in RNA concentration between two samples (12). The ES-bDNA, NGI, and BCM assays achieved this degree of precision. The duplicate log10 SD was lower than the triplicate log10 SD for all methods. This result was attributable to the composition of the panel. Samples tested in duplicate were preponderantly those with high HIV-1 RNA concentrations, while samples tested in triplicate were those with lower concentrations. Of the five assays, the ES-bDNA assay showed the highest overall intra-assay precision, with a log10 SD of 0.050 to 0.054.
TABLE 3.
Intra-assay performance characteristics of five methods for quantification of plasma HIV-1 RNA
| Performance characteristic | Chiron ES-bDNA | NGI (in-house) | BCM (in-house) | Roche AMPLICOR HIV-1 Monitor | Organon Teknika NASBA HIV-1 QT |
|---|---|---|---|---|---|
| Correlation with nominal concentrations (no. of pairs) | 0.997 (19) | 0.990 (28) | 0.975 (23) | 0.957 (16) | 0.960 (21) |
| Mean CV (%) | 12.1 | 20.9 | 29.0 | 51.9 | 41.8 |
| CV (range) | 1.3–27.9% | 0–79.8% | 0–49.9% | 16.2–87.5% | 5.1–138% |
| Log10 SD: | |||||
| Duplicate | 0.050 | 0.043 | 0.059 | 0.201 | 0.038 |
| Triplicate | 0.054 | 0.126 | 0.193 | 0.232 | 0.261 |
| False-positive results (no. of results/total) | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 |
| Lower limit of detection (L)a: | |||||
| Claimed | 500 | 100 | 500 | 400 | 4,000 |
| Observed | 650 < L < 1,300 | <65 | 130 < L < 650 | <65 | Not evaluated |
| Upper limit of detection (U)a: | |||||
| Claimed | 1.6 × 106 | 2.0 × 106 | 2.5 × 106 | 0.75 × 106 | 1.5 × 107 |
| Observed | 1.0 × 106 < U < 2 × 106 | 2.0 × 106 < U | 2.0 × 106 < U | 0.13 × 106 < U < 0.5 × 106 | Not evaluated |
HIV-1 RNA copies/ml.
HIV-1-negative samples were correctly identified in all five assays, and no false positives were found; thus, the specificity was 100% for all five methods. It was demonstrated in the experiment shown in Fig. 1 that there was little or no difference between the nominal HIV-1 RNA concentrations and the estimates of HIV-1 RNA concentration obtained by the Chiron, NGI, BCM, and Roche methods. It was thus possible to compare the dynamic range claimed for each of these four methods to the observed lower and upper limits of detection, abbreviated L and U, respectively. The observed L for ES-bDNA was higher than the claimed L; the observed H was consistent with the claimed H. NGI’s observed L and U exceeded its claimed dynamic range, while BCM’s observed L and U were consistent with its claims. The observed L and U of the AMPLICOR HIV-1 Monitor were both lower than those claimed by the manufacturer.
This study underscores the importance of the use of well-defined standards for assigning nominal HIV-1 RNA concentrations to panel materials. In a previous comparison of the AMPLICOR HIV-1 Monitor and NASBA assays, both methods produced estimates below the nominal copy numbers (10). Compared to Monitor, NASBA has yielded higher estimates of HIV-1 RNA concentrations in two of three studies (2, 8, 10).
This article describes the first performance comparison of ES-bDNA with other assays for HIV-1 RNA. Among the five laboratories that used ES-bDNA, the NGI assay, the BCM assay, Roche HIV-1 Monitor, and NASBA HIV-1 QT, the laboratories performing the first four tests named provided results that were generally consistent with each other. However, interlaboratory variation with the same kit is known to occur and may be greater than that which occurs with different kits (12). In other evaluations, the NASBA assay has performed well (2, 8, 10). The reason for the approximately 10-fold-higher values provided by the NASBA assay in the present study is being investigated. This event appears to be isolated, since many laboratories have successfully adopted the NASBA method (12). In our opinion, the laboratory may be more important than the assay method in the selection of a testing site for clinical trials.
Acknowledgments
We thank the following persons for their cooperation: John Todd and Pam Johnson, Chiron Corporation; Jay Weiss, Andrew J. Conrad, and Lawrence M. Blatt, NGI; Lisa Cosentino, Novum, Inc.; Beverly Dale, Roche Molecular Systems; Joe Romano, Advanced BioScience Laboratories, Inc.; and Michael Cronin, Stuart P. Geiger, and Richard J. Carroll, Organon Teknika Corp.
The primary standard for HIV-1 RNA was prepared in the Eugene B. Casey Hepatitis & HIV Research Center and Diagnostic Laboratory at BCM in connection with a study (4) supported by a National Institutes of Health grant (AI-82517). Additional support was provided by the Center for AIDS Research (CFAR) Virology Core Facility grant (P30 A136211-04/05) awarded to BCM.
REFERENCES
- 1.Conrad A J, Schmid P, Syndulko K, Singer E J, Nagra R M, Russell J J, Tourtellotte W W. Quantifying HIV-1 RNA using the polymerase chain reaction on cerebrospinal fluid and serum of seropositive individuals with and without neurologic abnormalities. J Acquired Immune Defic Syndr Hum Retrovirol. 1995;10:425–435. doi: 10.1097/00042560-199512000-00005. [DOI] [PubMed] [Google Scholar]
- 2.Coste J, Montes B, Reynes J, Peeters M, Segarra C, Vendrell J-P, Delapporte E, Segondy M. Comparative evaluation of three assays for the quantitation of human immunodeficiency virus type 1 RNA in plasma. J Med Virol. 1996;50:293–302. doi: 10.1002/(SICI)1096-9071(199612)50:4<293::AID-JMV3>3.0.CO;2-3. [DOI] [PubMed] [Google Scholar]
- 3.Kern D, Collins M, Fultz T, Detmer J, Hamren S, Peterkin J K, Sheridan P, Urdea M, White R, Yeghiazarian T, Todd J. An enhanced-sensitivity branched-DNA assay for quantification of human immunodeficiency virus type 1 RNA in plasma. J Clin Microbiol. 1996;34:3196–3202. doi: 10.1128/jcm.34.12.3196-3202.1996. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Lin H J, Myers L E, Yen-Lieberman B, Hollinger F B, Henrard D, Hooper C J, Kokka R, Kwok S, Rasheed S, Vahey M, Winters M A, McQuay L J, Nara P L, Reichelderfer P, Coombs R W, Jackson J B. Multicenter evaluation of quantification methods for plasma human immunodeficiency virus type 1 RNA. J Infect Dis. 1994;170:553–562. doi: 10.1093/infdis/170.3.553. [DOI] [PubMed] [Google Scholar]
- 5.Lin H J, Tanwandee T, Hollinger F B. Improved methods for quantification of human immunodeficiency virus type 1 (HIV-1) RNA and hepatitis C virus (HCV) RNA in blood using spin column technology and chemiluminescent assays of PCR products. J Med Virol. 1997;51:56–63. [PubMed] [Google Scholar]
- 6.Mellors J W, Rinaldo C R, Jr, Gupta P, White R M, Todd J A, Kingsley L A. Prognosis in HIV-1 infection predicted by quantity of virus in plasma. Science. 1996;272:1167–1170. doi: 10.1126/science.272.5265.1167. [DOI] [PubMed] [Google Scholar]
- 7.Mulder J, McKinney N, Christopherson C, Sninsky J, Greenfield L, Kwok S. Rapid and simple PCR assay for quantitation of human immunodeficiency virus type 1 RNA in plasma: application to acute retroviral infection. J Clin Microbiol. 1994;32:292–300. doi: 10.1128/jcm.32.2.292-300.1994. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Revets H, Marissens D, De Wit S, Lacor P, Clumeck N, Lauwers S, Zissis G. Comparative evaluation of NASBA HIV-1 RNA QT, AMPLICOR-HIV Monitor, and QUANTIPLEX HIV RNA assay, three methods for quantification of human immunodeficiency virus type 1 RNA in plasma. J Clin Microbiol. 1996;34:1058–1064. doi: 10.1128/jcm.34.5.1058-1064.1996. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Saag M S, Holodniy M, Kuritzkes D R, O’Brien W A, Coombs R, Poscher M E, Jacobsen D M, Shaw G M, Richman D D, Volderbing P A. HIV viral load markers in clinical practice. Nat Med. 1996;2:625–629. doi: 10.1038/nm0696-625. [DOI] [PubMed] [Google Scholar]
- 10.Schuurman R, Descamps D, van Weverling G, Kaye S, Tijnagel J, Williams I, van Leeuwen R, Tedder R, Boucher C A, Brun-Vezinet F, Loveday C. Multicenter comparison of three commercial methods for quantification of human immunodeficiency virus type 1 RNA in plasma. J Clin Microbiol. 1996;34:3016–3022. doi: 10.1128/jcm.34.12.3016-3022.1996. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.van Gemen B, Kievits T, Schukkink R, van Strijp D, Matek L T, Sooknanan R, Huisman H G, Lens P. Quantification of HIV-1 RNA in plasma using NASBA during HIV-1 primary infection. J Virol Methods. 1993;43:177–188. doi: 10.1016/0166-0934(93)90075-3. [DOI] [PubMed] [Google Scholar]
- 12.Yen-Lieberman B, Brambilla D, Jackson B, Bremer J, Coombs R, Cronin M, Herman S, Katzenstein D, Leung S, Lin H J, Palumbo P, Rasheed S, Todd J, Vahey M, Reichelderfer P. Evaluation of a quality assurance program for quantitation of human immunodeficiency virus type 1 RNA in plasma by the AIDS Clinical Trials Group Virology Laboratories. J Clin Microbiol. 1996;34:2695–2701. doi: 10.1128/jcm.34.11.2695-2701.1996. [DOI] [PMC free article] [PubMed] [Google Scholar]