Effect of Errors in the Sequence of Optical Densities from the Roche AMPLICOR HIV-1 MONITOR Assay on the Validity of Assay Results

Abstract

Specifications for the AMPLICOR HIV-1 MONITOR kit indicate that the results are invalid if the optical densities (ODs) from the PCR-amplified sample that are between 0.1 and 2.3 units are out of sequence. However, among 11,904 assays, results were biased only when ODs of 0.2 to 2.0 units were out of sequence, reducing the rate of invalid results from 3.2 to 0.59%.

Measurements of plasma human immunodeficiency virus (HIV) type 1 (HIV-1) RNA levels are used to determine patients' prognoses and to assess their responses to antiretroviral therapy (1, 2). The validity of each measurement is critically important, but given the expense of these assays, criteria for assay acceptance must be carefully defined to minimize the possibility of false rejection of valid results. This reports investigates one aspect of the validity of results from the AMPLICOR HIV-1 MONITOR assay.

Quantitative HIV-1 RNA estimates from the MONITOR assay are based on the optical densities (ODs) of six fivefold serial dilutions of the sample and two of the internal quantitation standard after both have been amplified by PCR. Quantitation is based on the greatest dilution of the sample amplicon for which the OD falls between 0.20 and 2.0 units. According to the instructions in the package insert for the kit (AMPLICOR HIV-1 MONITOR test, Roche Diagnostics Corp., Branchburg, N.J., 1999), the estimate of RNA concentration may be invalid if the ODs do not decrease over the six dilutions and the ODs that are out of sequence are between 0.10 and 2.30 units. Although not specifically stated in the instructions, this statement is interpreted to mean that at least one of the two values that are out of sequence falls in this range. No data are provided to support the claim that 0.10 to 2.30 OD units is the critical range for the identification of invalid assays. Furthermore, this range implies that invalid results may be obtained even if the ODs that are out of sequence do not fall within the range used to estimate RNA concentration. The six hypothetical examples of sequences of ODs in Table 1 illustrate the problem. Two ODs are out of sequence in each example. Examples 3 through 6 would be flagged as invalid under the manufacturer's kit instructions, but only examples 4 and 5 involve values that fall within the range used to estimate RNA concentration. An investigation was therefore conducted to assess the effects of various types of OD sequence errors on the validity of assay results.

TABLE 1.

Hypothetical sequences of ODs from the Roche HIV-1 MONITOR assay in which the six ODs do not form the expected declining sequence

Dilution	OD in examplea:
	1	2	3	4	5	6
Undiluted	2.5	2.9	2.2	1.8	2.9	2.9
1:5	2.9	2.5	2.9	2.9	2.5	2.5
1:25	1.5	1.5	1.5	1.5	1.5	1.5
1:125	0.6	0.6	0.6	0.6	0.08	0.08
1:625	0.08	0.05	0.08	0.08	0.25	0.12
1:3,125	0.05	0.08	0.05	0.05	0.05	0.05

^aODs that are out of sequence are shown in boldface type. 

Data were obtained from the HIV RNA Proficiency Testing Program of the National Institute of Allergy and Infectious Diseases, National Institutes of Health-sponsored Virology Quality Assessment (VQA) Program. Under the VQA Program, participating laboratories receive coded panels that consist of HIV-1 from a well-characterized HIV-1 subtype B concentrate spiked into plasma from healthy subjects and plasma from HIV-1-infected volunteers, both in fivefold serial dilutions (3). The data set for this analysis included assays of 8,187 HIV-spiked specimens and 1,744 samples from HIV-infected patients that were performed with the standard MONITOR assay and 1,973 assays of HIV-spiked specimens that were performed with the ultrasensitive MONITOR assay. Data from the standard assay were obtained in 30 rounds of proficiency testing in 69 laboratories between October 1995 and November 1999 (median, 7 rounds of testing per laboratory; range, 1 to 16 rounds). Data from the ultrasensitive assay were obtained in seven rounds of proficiency testing in 25 laboratories between May 1999 and January 2000 (median, four rounds of testing per laboratory; range, one to six rounds). Results from another 48 assays were rejected because of problems with the internal quantitation standard that invalidated the results (e.g., both ODs were outside the acceptable range for the assay). The nominal concentrations for the spiked samples tested with the standard assay were 1,500 to 750,000 copies/ml, while the median estimated HIV-1 RNA concentrations in the dilutions of samples from HIV-1-infected patients were 500 to 466,359 copies/ml. The nominal HIV-1 RNA concentrations in the spiked samples that were tested with the ultrasensitive MONITOR assay were 50 to 31,250 copies/ml. These samples spanned most of the linear ranges for both versions of the assay that are claimed in the package insert (standard MONITOR assay, 400 to 750,000 copies/ml; ultrasensitive MONITOR assay, 50 to 75,000 copies/ml).

By following the algorithm in the package insert, the OD from the greatest dilution of the amplicon, among those with ODs between 0.20 and 2.0 units, was used to calculate HIV-1 RNA concentration (e.g., the 1:125 dilution in example 1 and the 1:625 dilution in example 5 in Table 1). The effect of OD sequence errors was assessed by comparing log₁₀ recoveries in assays with OD sequence errors to the recoveries in assays without these errors. Recovery was calculated as the ratio of estimated HIV-1 RNA concentration to nominal HIV-1 RNA concentration (spiked samples) or median estimated concentration (plasma samples from HIV-1-infected patients). Log₁₀ recovery, rather than log₁₀ estimated RNA concentration, was used in the analysis so that the data could be combined across nominal and median concentrations.

Estimates of log₁₀ recovery were classified into six groups (Table 2). Group 1 (the reference group for this analysis) included assays for which the six ODs formed decreasing sequences and those that did not do so only because the ODs from the first two dilutions were at the maximum for the plate reader in use at the laboratory in question. The assays in groups 2, 3, and 4 had sequence errors that involved ODs of 0.10 to 2.30 units and that would therefore be flagged as invalid according to the criteria in the package insert. However, only the sequence errors in group 2 involved ODs that were in the range used to estimate the RNA concentration (0.20 to 2.0 units; examples 4 and 5 in Table 1). The sequence errors in groups 5 and 6 involved ODs that were outside the range specified in the package insert for the flagging of assays as invalid. Groups 2 to 6 did not include all of the theoretically possible OD sequence errors, but they did include all of the assays that were examined.

TABLE 2.

Descriptive statistics for log₁₀ recovery for six groups of assays, with the groups defined by the ODs that are out of sequence

Assay and group	ODs that are out of sequence	No. of samples	OD
			25th percentile	Median	75th percentile
Standard MONITOR assay
1	None	7,764	−0.14	−0.02	0.11
2	One between 0.2 and 2.0; any other value	68	−0.07	0.06	0.55
3	One between 0.10 and 0.20; one <0.20	243	−0.10	0.03	0.15
4	One between 2.00 and 2.30; one >2.0	17	−0.09	0.01	0.12
5	Both <0.10	1,405	−0.18	−0.05	0.08
6	Both >2.30	434	−0.15	−0.01	0.14
Ultrasensitive MONITOR assay
1	None	1,275	−0.12	−0.01	0.11
2	One between 0.2 and 2.0; any other value	2	−1.06		−0.38
3	One between 0.10 and 0.20; one <0.20	54	−0.24	−0.05	0.11
4	One between 2.00 and 2.30; one >2.0	2	−0.31		0.11
5	Both <0.10	608	−0.18	−0.03	0.11
6	Both >2.30	32	−0.16	−0.06	0.20

There were 163 results for the standard MONITOR assay and 90 for the ultrasensitive MONITOR assay, with two sequence errors each. In 194 (76.7%) of these, the two sequence errors resulted in the same classification. In 25 others, the first two ODs were at the maximum for the reader. These cases were classified on the basis of the second pair of ODs that were out of sequence. Seven cases, in which one sequence error involved a value between 0.20 and 2.0 units and the other did not, were placed in group 2. The remaining 27 cases were classified according to the first pair of ODs that were out of sequence in the dilution series. Reclassification of these results according to the second sequence error had no effect on the results reported here. In 16 cases, three pairs were out of sequence, and in 4 others, four pairs were out of sequence. Twelve of these 20 cases were placed in group 5 because all of the ODs involved in the errors were <0.10 unit. Five cases, in which one pair of ODs included a value between 0.10 and 0.20, while the others involved values that were <0.10, were placed in group 3. Two assays in which one pair of ODs included a value between 0.20 and 2.0, while the others involved values that were >2.3, were placed in group 2. One assay in which the first four ODs were at the maximum for the reader was placed in group 6.

Results from the standard and ultrasensitive MONITOR assays were very similar. They are therefore discussed in the aggregate here, although they are presented separately in Table 2. ODs were out of sequence in 2,865 (24.1%) assays, including 386 (3.2% of all assays) cases that fell in groups 2, 3, and 4 and that would be flagged as invalid according to the specifications in the package insert (Table 2). Comparisons of the descriptive statistics for log₁₀ recovery in groups 3 to 6 with those in group 1 indicate that OD sequence errors that placed assays in groups 3 to 6 did not bias estimates of the HIV-1 RNA concentrations. On the other hand, the 75th percentile of recovery for group 2 was substantially higher than those for the other groups, while the median was only slightly elevated, and the 25th percentile was similar to those for the other groups. This indicates that group 2 includes a subset of positively biased observations. The 75th percentile corresponds to a recovery of approximately 350%, indicating that a large proportion of the assays in this group was substantially biased.

The assays in group 2 were divided into three subgroups to determine if the subset of assays with biased results could be defined more precisely. Log₁₀ recovery was ≤−0.30 (recovery, ≤50%) in 3 of 16 assays in which the second OD that was out of sequence was >2.0 units. Log₁₀ recovery was either <−1.0 or >0.40 (recovery, <10 or >250%, respectively) in eight of nine assays in which the second OD that was out of sequence was <0.20 unit. Finally, log₁₀ recovery was either <−0.30 or >0.30 (recovery, <50 or >200%, respectively) in 14 (31%) of 45 assays in which both of the ODs that were out of sequence fell between 0.20 and 2.0 units. Thus, a substantial fraction of the assays in all three subgroups produced estimates of HIV-1 RNA concentrations that were biased by at least twofold.

These results indicate that the range of ODs specified by the manufacturer for the identification of OD sequence errors that invalidate assays is too stringent. Estimates of HIV-1 RNA concentrations from the standard and ultrasensitive MONITOR assays were biased only when the ODs that were out of sequence included at least one value between 0.2 and 2.0 units. Thus, only 0.59% of assays should be rejected because of OD sequence errors instead of the 3.2% that would be rejected under the criteria specified in the assay instructions. Given the high price of RNA quantitation, treatment of the results for the other 2.8% as valid will result in substantial cost savings without compromising accuracy.