Accuracy of the TRUGENE HIV-1 Genotyping Kit

Abstract

Drug resistance and poor virological responses are associated with well-characterized mutations in the viral reading frames that encode the proteins that are targeted by currently available antiretroviral drugs. An integrated system was developed that includes target gene amplification, DNA sequencing chemistry (TRUGENE HIV-1 Genotyping Kit), and hardware and interpretative software (the OpenGene DNA Sequencing System) for detection of mutations in the human immunodeficiency virus type 1 (HIV-1) protease and reverse transcriptase sequences. The integrated system incorporates reverse transcription-PCR from extracted HIV-1 RNA, a coupled amplification and sequencing step (CLIP), polyacrylamide gel electrophoresis, semiautomated analysis of data, and generation of an interpretative report. To assess the accuracy and robustness of the assay system, 270 coded plasma specimens derived from nine patients were sent to six laboratories for blinded analysis. All specimens contained HIV-1 subtype B viruses. Results of 270 independent assays were compared to “gold standard” consensus sequences of the virus populations determined by sequence analysis of 16 to 20 clones of viral DNA amplicons derived from two independent PCRs using primers not used in the kit. The accuracy of the integrated system for nucleotide base identification was 98.7%, and the accuracy for codon identification at 54 sites associated with drug resistance was 97.6%. In a separate analysis of plasma spiked with infectious molecular clones, the assay reproducibly detected all 72 different drug resistance mutations that were evaluated. There were no significant differences in accuracy between laboratories, between technologists, between kit lots, or between days. This integrated assay system for the detection of HIV-1 drug resistance mutations has a high degree of accuracy and reproducibility in several laboratories.

Combination antiretroviral therapy prolongs survival in human immunodeficiency virus type 1 (HIV-1)-infected persons (12), although virological failure occurs in 50% of persons in the first year of treatment (7). Virological failure occurs because of viral resistance to drugs (4) or insufficient exposure to drugs, as can occur due to poor adherence to prescribed regimens or decreased absorption or increased clearance of drugs (1). In retrospective studies, virological response to changes in antiretroviral therapy is correlated with baseline resistance to drugs, whether measured as viral genetic mutations (9, 20, 26) or drug susceptibility phenotype (8, 9, 21). Randomized prospective clinical trials have demonstrated that receiving results of genotypic drug resistance assays prior to changes in therapy improve virological responses (2, 10). Based on these studies of HIV-1 genotyping assays, the Department of Health and Human Services, the International AIDS Society-USA, and the European Guidelines group now recommend drug resistance assays for selection of new combination antiretroviral regimens after virologic drug failure while a patient is receiving drug therapy (14, 19; http://www.hivatis.org) and in pregnant women where optimal virological control is expected to help prevent vertical transmission to the infant (5). Recent reports of high prevalence of primary, or transmitted, drug resistance (3, 13, 18, 22, 23) and impaired virological responses in subjects acquiring drug-resistant HIV-1 when first infected (11, 17) suggest that resistance testing may also be warranted for drug-naïve persons starting therapy, as recommended by the European Guidelines group (19). The European guidelines also recommend resistance testing of the sources of persons presenting for postexposure prophylaxis and for children born to mothers with detectable viremia while receiving treatment (19).

The clinical trials demonstrating utility of drug resistance genotyping for the selection of antiretroviral therapy were supported by centralized reference laboratories with extensive experience with viral genotypic testing. In clinical practice, drug resistance genotypic testing is performed using a wide variety of methods whose performance characteristics have not been thoroughly established. Significant variations exist between laboratories in the extent of technical training, the reagents used, the amplification and sequencing protocols, and datum review and reporting procedures. This variation in methods may contribute to the variable quality of resistance assay results that have been observed in international evaluations (24).

We now describe a multicenter blinded study of the accuracy and reproducibility of results obtained with the TRUGENE HIV-1 Genotyping Kit and OpenGene DNA Sequencing System (Visible Genetics, Inc.). This is an integrated system that includes a training and certification program, positive and negative control specimens, reverse transcription (RT)-PCR amplification, dye primer labeled bi-directional sequencing, polyacrylamide gel electrophoresis, and sequence analysis including sequence contiguous element assembly, guided manual editing, and an interpretative HIV-1 resistance report based on algorithms developed and periodically revised by an international committee of experts. The study aimed to evaluate the accuracy of the assay system in the evaluation of plasma samples from HIV-1-infected persons and virus stocks derived from infectious molecular clones and to determine the variability in accuracy by laboratory site, by technician, by day, and by kit lot.

MATERIALS AND METHODS

Specimens.

EDTA-treated anticoagulated blood plasma was obtained by plasmapheresis from nine HIV-1-seropositive subjects having HIV-1 RNA levels in plasma ranging from 1,279 to 353,900 copies per ml (Roche Amplicor HIV-1 Monitor assay). Five of these specimens (VA-MH-001, -003, -004, -007, and -009) are also described elsewhere (15). All plasma specimens were stored at −70°C with no temperature fluctuation greater than 10% as determined by daily temperature monitoring. Specimens were aliquoted and labeled using specimen codes prior to distribution to study sites, where all investigators were blinded to the specimen identity and the results of other testing sites. In comparisons performed in the same laboratory, separate aliquots of specimens with distinct specimen identification codes were issued to assure blinded comparisons. Specimen collection was conducted using protocols approved by the Institutional Review Board at Stanford University and the Research and Development Committee at the VA Palo Alto Health Care System.

“Gold standard” sequence determination of plasma from HIV-1-infected persons.

Two aliquots of 1 ml each of plasma from each subject were centrifuged at 45,000 × g for 1 h at 4°C to concentrate viral particles, which were then resuspended in 150 μl of plasma. Viral RNA from the entire volume was mixed in lysis buffer then isolated using columns containing activated silica. The RNA was eluted in 50 μl according to the manufacturer's recommendations (QIAamp viral RNA kit; Qiagen Inc.). To allow broader sampling of the virus population, two aliquots of the extracted RNA (17 μl each) were independently amplified by RT-PCR using two distinct and different primer sets, neither of which are used in the TRUGENE HIV-1 Genotyping Kit. These extraction and amplification conditions are sufficient to consistently amplify specimens with viral loads as low as 60 copies per ml (R Lloyd, R. Schuurman, H. Stang, DeGroot, L. Hough, D. Burns, R. Mathis, and P. Feorino, 3rd Int. Workshop HIV Drug Resist. Treat. Strat., abstr. 52, 1999). PCR products were purified by agarose gel electrophoresis and extraction from the gel (QIA_QUICK gel extraction kit; Qiagen Inc.) and then ligated into pGEM-T vectors (Promega). The ligation products were introduced into Escherichia coli by transformation followed by drug selection. The presence of inserted patient-derived gene inserts was verified by restriction digestion of color-selected colonies. Purified plasmid DNA was sequenced using Cy5.0 and Cy5.5 labeled T7 and SP6 primers (5′-GTAATACGACTCACTATAGGG-3′ and 5′-ATTTAGGTGACACTATAGAATAC-3, respectively). Twenty cloned sequences from each specimen were derived by using the first 10 sequences from each of two independent PCR amplification reactions. The consensus sequence from aligned nucleotide sequences was used as the gold standard sequence from each specimen. In order to represent mixtures greater than or equal to 30% of the plasma HIV-1 population, bases reported in at least 6 of the 20 clones (30%) were included in the gold standard consensus sequence using standard ambiguity codes. Viral sequences in one of the nine plasma specimens (VA-MH-001) were difficult to clone using standard cloning techniques; therefore, only 16 sequences were used to establish the gold standard consensus in this instance.

Generation of virus stocks from infectious molecular clones.

Mutant HIV-1 viruses containing 72 different drug resistance mutations were prepared from infectious molecular clones of HIV-1. Mutant variants of HIV-1 were prepared by site-directed mutagenesis of an infectious molecular clone of HIV-1_LAI (pLAI.2) (kindly provided by L. Montagnier through the NIH AIDS Research and Reference Reagent Program). Mutations were introduced into the protease (PR)- or RT-coding region of pol using the Altered Sites II in vitro mutagenesis system (Promega, Madison, Wis.). The resulting plasmids were linearized by restriction enzyme digestion and electroporated into CEM-SS cells to generate infectious viruses, which were propagated to yield high-titer stocks. HIV-1 viral load was assayed (Roche Amplicor HIV-1 Monitor), and the virus stocks were diluted to the desired concentration in blood plasma from anti-HIV-1 seronegative donors and stored at −70°C with no temperature fluctuation greater than 10%.

Performance sites.

The laboratory analyses of specimens were performed using the TRUGENE HIV-1 Genotyping Kit assay in eight laboratories in the United States; six laboratories were involved in the analysis of plasma samples from infected persons and three laboratories were involved in the analysis of virus samples prepared from infectious molecular clones. Samples were distributed among the sites in a balanced manner to maximize the power to detect differences in assay performance between laboratory sites. All technologists participating in the study were trained and certified to perform the TRUGENE HIV-1 Genotyping Kit assay using standardized protocols. The training involved 1 week of practical and theoretical training at a reference laboratory (Visible Genetics, Inc., Atlanta, Ga.) followed by proficiency testing of a blinded practice panel and a blinded certification panel of four unknown specimens plus two controls. Technologists were required to achieve greater than 95% sequencing accuracy on the blinded panels.

TRUGENE HIV-1 Genotyping Kit performance.

All assay procedures were performed according to the manufacturer's recommendations (Visible Genetics, Inc.). Briefly, plasma viral RNA was extracted using activated silica in a column format (QIAamp viral RNA kit; Qiagen Inc.). For this study, 140 μl of unconcentrated plasma was mixed in a lysis buffer, placed over the column, washed, and then eluted in 60 μl. An aliquot of 17 μl of extracted RNA eluate was mixed with RT-PCR reagents containing RNase inhibitor, Taq polymerase, sequence specific primers, dATP, dCTP, dGTP, dTTP, magnesium, dithiothreitol, and pH buffers. The RT-PCR primers used in the kit amplify the entire protease and the first 250 codons of reverse transcriptase as a single amplicon. The reverse transcriptase reaction was performed at 90°C for 2 min, followed by incubation at 50°C for 5 min, at which point reverse transcriptase enzyme was added to the reaction mixture, which was then incubated for an additional 55 min before the reaction was terminated at 94°C for 2 min. DNA amplification was performed in the same tube using cycling conditions of 94°C for 30 s, 57°C for 30 s, and 68°C for 2 min for 20 cycles; this was followed by cycling conditions of 94°C for 30 s, 60°C for 30 s, and 68°C for 2.5 min for 17 cycles; and the cycling finished with one cycle of 68°C for 7 min and then a 4°C hold. Unpurified RT-PCR product (5 μl) from each specimen was used in each of 16 sequencing reactions, based on the CLIP principle (25), and employing four pairs of primers that sequence the protease reading frame (two pairs) and the beginning and middle of the reverse transcriptase reading frame (one pair each). For each segment, four sequencing reactions were used with four dideoxy terminators (ddATP, ddCTP, ddGTP, and ddTTP), Cy5-labeled forward primers, and Cy5.5-labeled reverse primers to generate CLIP sequencing products over 30 reaction cycles (94°C for 20 s, 56°C for 20 s, 70°C for 1.5 min) followed by a 5-min terminal extension at 70°C. Unpurified CLIP sequencing products were analyzed using 6% polyacrylamide-urea-Tris-borate-EDTA gel electrophoresis for 50 min at 2,000 V, with premanufactured Microcel cassettes, which were cast using Surefill 6% sequencing polyacrylamide gel cartridges (Visible Genetics, Inc.), containing premeasured amounts of acrylamide, urea, and Tris-borate-EDTA, which were polymerized using UV light.

Sequence data analysis.

Sequence data acquisition and analysis were performed using the Gene Objects software (Visible Genetics, Inc.). The software allows automated and real-time acquisition of sequence data from the DNA sequencers (Long Read Tower Sequencers). Sequence fragments are assembled automatically and electropherogram patterns are displayed for semiautomated review after alignment with a clade B HIV-1 reference sequence (HIV-1_LAI) and a mutated reference sequence containing codon changes typically observed in drug-resistant HIV-1. The use of both mutant and wild-type reference sequences serves to minimize biases toward the wild type during manual editing. The software can be set to skip automatically to positions where there are differences in base identification between the forward and reverse sequencing reactions or between the determined sequence and either of the reference strains or if results from only one sequencing direction is available. For this study, manual editing was limited to bases that were discordant between the two sequencing reactions. The edited sequences were exported from the software for analysis.

Statistical analysis.

All statistical analysis was performed by an independent contractor (HHI, Hunt Valley, Md.), who had no communication with any of the laboratory testing sites. The sequences derived from each assay were compared to the gold standard sequence to assess the extent of agreement. Agreement was compared for all nucleotide positions, for all codons, for codons that were mutated in the gold standard (relative to HIV-1_LAI), for codons that were wild type in the gold standard, and for codons that have been associated with decreased susceptibility or poor virological response to antiretroviral therapies. Only exact agreement was scored as an accurate result, defined as detection of all of the bases present in the gold standard sequences and no other bases. For example, if the gold standard sequence indicated that A was the predominant base present at a given site and G was present in 6 of 20 clones, the result would be scored as accurate only if A and G were detected and C and T were not detected. Detection of C would be considered inaccurate in the a priori data analysis plan even if C were detected in at least one of the gold standard sequences, but at a frequency of less than 30% of the clones. The mutation recognition rate was defined as the portion of mutated codons (relative to HIV-1_LAI) that were exactly detected by the assay. The wild-type recognition rate was defined as the proportion of nonmutant codons that were exactly detected by the assay. Codons associated with antiretroviral resistance were defined as codons that have been associated with virological drug failure in vivo, decreased drug susceptibility in vitro, or drug selection in tissue culture, as compiled in published tables (6, 14). Differences in accuracy between laboratories, technicians, kit lots, and days were evaluated for statistical significance using analysis of variance across all groups, followed by pairwise comparisons if there was evidence of differences. A secondary analysis, designed after the primary analysis, allowed results to be considered accurate if bases or codons present in the gold standard in fewer than 30% of clones were detected, provided that all bases present in more than 30% of the gold standard clones were also detected. The accuracy of replicate testing of virus stocks derived from infectious molecular clones was calculated for each of the 72 drug resistance mutations present in the panel.

Nucleotide sequence accession numbers. The consensus sequences from the gold standard sequencing procedures have been submitted to GenBank with accession numbers AF472535 to AF472552.

RESULTS

Gold standard sequences.

Tables 1 and 2 list the virus preparations that were evaluated in this study. Nine specimens were blood plasma derived from HIV-1-infected persons (Table 1), and 10 specimens were derived from stocks of infectious molecular clones spiked into blood plasma from seronegative persons (Table 2). Of the HIV-1-infected persons, seven were antiretroviral experienced and six were taking antiretroviral therapy at the time of specimen collection. The amino acid substitutions associated with drug resistance are listed, indicating mixtures whenever detected in more than 30% (e.g., >6 of 20) of the clones sequenced per specimen. The amino acid changes detected in the gold standard sequences are consistent with the prior and current antiretroviral therapy of these subjects. In addition to the mutations associated with drug resistance, there were silent and nonsilent mutations detected in each specimen derived from HIV-1-infected persons.

TABLE 1.

List of blood plasma specimens from HIV-1-infected persons

Specimen identification no.	Viral load (copies/ml)	Antiretroviral therapy		Gold standard resistance genotype
		Current	Other prior	PR	RT
VA-MH-001	353,900	None	Zidovudine, didanosine, zalcitabine, stavudine	L63P
VA-MH-002	1,722	Stavudine, didanosine, efavirenz	Zidovudine, lamivudine	L10I, L63P	M41L, K101E, K103N, M184M/I/V, T215Y/F, K219K/Q
VA-MH-003	2,618	Zidovudine, lamivudine	None		M184V
VA-MH-004	1,279	Stavudine, lamivudine, nevirapine	Zidovudine, saquinavir		D67N, K70R, K103N, M184V, K219Q
VA-MH-005	1,433	Zidovudine, lamivudine	None	K20R, L63P, V77I	M41L, M184V, T215Y
VA-MH-006	28,167	None	None	M36I
VA-MH-007	28,881	None	None	M36M/I
VA-MH-008	23,979	Stavudine, lamivudine, nelfinavir	Zidovudine, nevirapine, saquinavir,	L63P, A71T, V77I, L90M	M184V
VA-MH-009	43,525	Zidovudine, lamivudine, adefovir, abacavir	Didanosine, stavudine, nevirapine, indinavir, saquinavir, nelfinavir	L10I, L63P	K103N, M184V, T215F

TABLE 2.

Infectious molecular clones diluted in seronegative blood plasma

Specimen identification no.	Gold standard resistance genotype
	PR	RT
MIC-5	20R, 63P, 77I, 82T, 90M	62V, 74V, 75T, 77L, 115F, 116Y, 151M, 184V, 215F, 219E
MIC-9	10R, 30N, 71V, 90M	41L, 69D, 70R, 181C, 215Y
MIC-11	50V	188C, 190A, 210W, 219Q
MIC-13	None	181I, 184I
MIC-14	10V, 20M, 32I, 33F, 46L, 54L, 71T, 73A, 82F	98S, 103T
MIC-15	None	188H, 190S, 236L
MIC-16	10I, 24I, 36I, 73S, 82S, 88D	98G, 100I, 101E
MIC-17	None	179D, 188L, 190E
MIC-18	10F	50V, 69N, 70E, 75I
MIC-19	46I, 47V, 48V, 50V, 54V, 82A, 84V	65R, 67N, 103N, 106A, 108I

Testing process for plasma specimens from infected persons.

Overall, each of the nine plasma specimens was tested at least 18 times, sometimes 36 times, in a blinded fashion using the OpenGene system. All panels included at least one sample from each of the nine specimens and duplicate samples for an additional six specimens. The assays were performed by nine technicians in six laboratories using three kit lots. All of the technicians originally proposed to participate in the study passed a proficiency panel and a certification panel: seven of nine passed on the first attempt and two passed on the second attempt. Taken together, 324 assays were completed, representing 270 blinded specimens and 54 negative controls. Of the six laboratories, all had prior experience with DNA sequencing and were based at academic centers, one was licensed for clinical diagnostic testing, and none had any direct role in the development of Visible Genetics technology. Data submitted from these laboratories indicated that 194 (72%) of the test samples provided reportable results on the first assay attempt, while 76 (28%) of the test samples required some stage of the assay to be repeated, including 15 (5%) test samples repeated because of deviations from the study protocol (e.g., use of the incorrect kit lot). Among assays repeated due to inadequate sequence data, 38% required repeated gel electrophoresis, 24% required repeated CLIP reactions, 25% required repeat from RT-PCR, and 13% required repetition of the entire assay starting with RNA extraction. Two sequencing towers from one investigational site were replaced, one due to an unexplained error message and the other due to a failed internal heating plate temperature check. No other equipment replacements were required. A requirement for Food and Drug Administration submission was to report any injuries to those operating the system; there were none reported in this study.

Overall accuracy in clinical plasma samples.

The overall accuracy of the base detection was 98.7% counting only exact matches, defined as the detection of all of the bases present in at least 30% of the gold standard clones and no other bases (Table 3) (30% rule). Detection at sequence positions with pure wild-type codons in the gold standard clones was more accurate than detection at positions with some evidence of mutation, which frequently represented mixtures of bases (0.99 versus 0.82 for wild-type and mutant detection, respectively). A large portion of results deemed inaccurate using the 30% rule were due to detection of sequence variants that were present in less than 30% of the gold standard clones, which was the cutoff for detection established a priori for the data analysis plan. A secondary analysis was performed which allowed detection of bases or codons to be considered accurate provided that the base or codon was present in at least one of the gold standard clones (e.g., 1 of 20 or 5%), and all of the bases or codons present in at least 30% of the gold standard clones were detected in the assay (Table 3, 5% rule). This secondary analysis revealed substantially higher accuracy using the 5% rule to define the gold standard. For example, accuracy of mutation detection improved from 0.82 to 0.94 using the 5% rule. Wild-type mutation recognition agreement remained higher than the mutation recognition agreement rate, as expected, because wild-type codons by definition had no variation in the sequences used to define the gold standard, while mutant codons could represent any mixture of mutant and wild-type codons. There was no difference in assay accuracy at sequence positions that are known to be important for drug resistance and all other positions.

TABLE 3.

Accuracy of the TRUGENE HIV-1 genotyping kit

Comparison	30% rule		5% rule
	Agreement	SD	Agreement	SD
Base to base	0.987	0.01	0.993	0.01
All codons
Codon to codon	0.986	0.01	0.992	0.01
Mutation recognition agreement	0.82	0.17	0.94	0.07
Wild-type recognition agreement	0.99	0.01	1.00	0.01
Resistance codons
Codon to codon	0.976	0.03	0.985	0.02
Mutation recognition agreement	0.76	0.03	0.93	0.11
Wild-type recognition agreement	0.99	0.02	1.00	0.01

Accuracy in clinical plasma samples across laboratory sites, technicians, kit lots, and days.

There were no significant variations in exact base to base accuracy between the six study sites (accuracy range, 98.43 to 98.84%; P = 0.84), nor between two different technicians at each of three sites (accuracy range, 98.13 to 98.99%; P = 0.43), nor between two consecutive testing days at each of three sites (accuracy range, 98.03 to 98.85%; P = 0.52), nor between three kit lots evaluated at each of three sites (accuracy range, 98.65 to 98.91%; P = 0.94). Similarly, there were no differences in other measures of accuracy (codon to codon, mutation recognition, and wild-type recognition for all codons and drug resistance codons).

Accuracy in virus samples prepared from infectious molecular clones.

An analysis was performed to assess accuracy of detection of the entire spectrum of mutations associated with drug resistance in virus stocks derived from infectious molecular clones of HIV-1. Ten HIV-1 stocks derived from infectious molecular clones were spiked into plasma from HIV-1-seronegative donors and diluted to a concentration that just exceeded 10,000 HIV-1 RNA copies/ml. The infectious molecular clones were constructed to carry multiple antiretroviral resistance-associated mutations (Tables 1 and 2) and were sequenced 10 times each using the TRUGENE HIV-1 Genotyping Kit. Taken together, the panel contained 10 virus stocks containing 72 different drug resistance mutations. In a blinded fashion, the panel was tested six times in one laboratory and two times in each of two other laboratories. Fifty-four of 72 resistance mutations were correctly identified in all 10 replicate tests, and the remaining 18 mutated codons were correctly identified in >90% of test results. Fingerprint analysis of nucleotide sequences revealed that results from two aliquots of two specimens from one of the laboratory sites appeared to have been contaminated with sequences from another HIV-1 isolate, unrelated to the virus in the panel (data not shown). When results of these two contaminated sequences were excluded from the analysis, 31 of 31 (100%) resistance mutations in PR and 40 of 41 (97.6%) resistance mutations in RT were correctly identified in all of the remaining assay results; one RT mutation (K101E) was correctly identified in 9 of 10 (90%) of the test results.

Analysis of codon misidentifications in plasma specimens from infected persons.

All assay results (n = 90) that were used to compare assay performance between six laboratory sites were also used for a detailed analysis of the clinical implications of codon misidentifications that occurred in the course of this study. In this analysis, we considered 54 codon positions that have been associated with drug resistance, virological drug failure, or drug selection (6). Of the 4,860 drug resistance codon positions evaluated (i.e., 90 assays × 54 codon positions per assay), assay results did not exactly match the gold standard at 70 (1.4%) positions, using the 5% rule to define the gold standard (Table 4). Of the 70 misidentified codon positions, 3 (4.3%) proved to be due to mislabeling of one specimen aliquot prior to distribution to one of the laboratories. Among the remaining 67 misidentified codon positions, the assay correctly identified at least one (but not all) of the codons present in the gold standard. In 61% (41 of 67) of the codon positions, the assay identified a codon that was not present in the gold standard, in addition to the gold standard codon. In the other 39% (26 of 67) of codon positions, the assay detected only one of several codons that were present in the gold standard. The majority (n = 55 of 67; 82%) of the codon misidentifications occurred at five codon positions (PR 10 and 63 and RT 184, 210, and 215) in three specimens (Table 4). Test results for these codon positions were identical in the majority of independent assays, indicating a high degree of concordance between laboratories despite discordance with the gold standard. The remaining 12 codon sites that were not exactly matched with the gold standard were distributed across eight sites in three specimens.

TABLE 4.

Analysis of all incompletely identified codons at resistance positions

Specimen identification no.	Gold standard codon	Codon detected	No. (%) of assays with mismatch (n = 12)	Clinical relevance of codon mismatch
VA-MH-001	PR L63	PR L63L/P	12 (100)	None
VA-MH-001	PR V82	PR V82V/I	8 (67)	None
VA-MH-002	RT M41	M41M/L	1 (8)	None
VA-MH-002	RT L100	L100L/I	1 (8)	None
VA-MH-002	RT K101E	RT K101K/E	3 (25)	None
VA-MH-002	RT M184M/V/I	RT M184V	12 (100)	None
VA-MH-002	RT Y215T/Y/F	RT T215Y	12 (100)	None
VA-MH-002	RT K219K/Q	RT K219	2 (17)	None
VA-MH-004	RT K65	RT K65K/E	1 (8)	None
VA-MH-004	RT K101	RT K101K/Q	1 (8)	None
VA-MH-005	RT V179	RT V179V/I	1 (8)	None
VA-MH-005	RT L210L	RT L210L/W	11 (91.7)	nRTIa resistance overestimated
VA-MH-005	RT K219	RT K219K/E	1 (8)	None
VA-MH-008	PR L63P	PR L63L/P	1 (8)	Mislabeled specimen
VA-MH-008	PR L90M	PR L90	1 (8)	Mislabeled specimen
VA-MH-008	RT M184V	RT M184	1 (8)	Mislabeled specimen
Total			70

^anRTI, nucleoside reverse transcriptase inhibitor.

The clinical relevance of the misidentified codon sites was assessed by comparing effects on assay interpretation, taking into account the other mutations that were detected in the same test result. There were 11 tests of one specimen (VA-MH-005) that indicated the presence of the RT L210W mutation as a mixture with wild-type, while the RT L210W mutation was not detected in any of the clones used to define the gold standard. The RT L210W mutation contributes to nucleoside analogue resistance, which was also indicated in all tests of this specimen by the RT T215Y, M184V, and M41L mutations. The RT L210W mutation was detected in 11 of 12 (91%) independent assays, indicating a high degree of concordance between laboratories. According to the interpretation algorithm used during this study (GuideLines Rules version 4.0; Visible Genetics, Inc.), the detection of RT L210W in the presence of the other mutations would have increased the predicted level of resistance to stavudine, didanosine, and zalcitabine from “possible resistance” to “resistance.” No codon misidentification was associated with a change in interpretation from “no evidence of resistance” to any of the resistance categories, or the reverse. The remaining 56 codon misidentifications in this study would not have affected the assay interpretation. Thirty-four codon mismatches involved amino acids associated with drug resistance, but these would not have affected the assay interpretation, because other mutations were consistently detected that were sufficient to predict resistance patterns. For example, detection of only RT M184V in a specimen (VA-MH-002) is sufficient to indicate resistance to lamivudine, even though RT M184I and wild-type sequences were also present in the gold standard. Twelve codon mismatches involved detection of a proline in addition to leucine at PR codon 63 in one specimen (VA-MH-001), which has no clinical relevance because PR codon 63 is commonly polymorphic, the proline is insufficient to impair virological responses, and there was no other evidence of protease inhibitor resistance. Mismatches at other codons (n = 10) are expected to have no clinical relevance because the additional amino acid that was detected in the assay has not been associated with drug resistance (e.g., PR V82I, RT V179I, and RT K65E). Overall, codon misidentification at resistance sites occurred infrequently, and the majority of identification errors would have been insufficient to alter interpreted resistance patterns.

DISCUSSION

The TRUGENE HIV-1 Genotyping Kit proved to have a consistently high degree of accuracy and reproducibility during an extensive blinded evaluation in several laboratory sites using several technicians over several days and three kit lots. In all laboratories, the base-to-base accuracy was higher than 99% and mutation recognition rates were greater than 82%. Of the 270 assays performed in this study, 259 (96%) assays yielded drug resistance interpretations that exactly matched interpretations derived from the gold standard sequences, and the other 11 (4%) assays provided partially matching interpretations.

Several factors are likely to have contributed to the accuracy and reproducibility of this assay. First, the establishment of explicit training procedures and proficiency evaluation is important, especially for clinical laboratories where medical technologists may not have training or prior experience in molecular biology. Second, the detailed and explicit protocol for the assay contributes to consistent performance, which probably accounts for the improved performance of kit-based assays versus home brew genotyping assays in a recent international proficiency evaluation (W. Keulen, D. Brambilla, M. Buimer, J. Tijnagel, J. Bremer, S. Land, L. de Graaff, H. Versteeg, C. Boucher, and R. Schuurman, 5th Int. Workshop Drug Resist. Treat. Strat., abstr. 166, 2001; L. Shaker-Irwin, A. Scarsella, E. Rogolsky, J. Stryker, and S. Day, 5th Int. Workshop Drug Resist. Treat. Strat., abstr. 160, 2001). Third, the use of dye-labeled primers has been associated with more consistent detection of mixtures of sequences compared with dye-labeled terminators, which may have biased base incorporation rates depending on adjacent sequences. Fourth, the automated data acquisition, analysis, and reporting helps eliminate transcription errors which have accounted for a large portion of sequencing discrepancies in other studies (M. Hoover, D. Wentworth, J. Neaton, et al., 4th Int. Workshop HIV Drug Resist. Treat. Strat., abstr. 78, 2000). Fifth, strict adherence to good manufacturing practices during production of the of the kit, as well as hardware associated with the assay, may explain the lack of variation in accuracy from lot to lot. Finally, the TRUGENE HIV-1 Genotyping Kit has been simplified by elimination of DNA purification steps after the RT-PCR and the CLIP sequencing reactions, by use of the novel CLIP reaction and by use of preformed gel cassettes. These simplifications serve to make the assay more robust through minimization of assay steps.

The results of two assays of infectious molecular clones of HIV-1 were invalidated due to presence of a contaminating HIV-1 sequence. Following completion of data analysis, sequence contamination was investigated using the Genetic Fingerprint function of the OpenGene software. This software function compares the pattern of polymorphisms and mutations of a sequence to all other sequences in the software library's database. Sequences showing identity with another sequence in the database (other than those from the same patient) suggest the possibility of contamination and should be investigated prior to reporting the result. For this study of coded specimens, participating laboratories were instructed not to use the Fingerprint function, which could have biased accuracy results by revealing virus identities in the panel, thereby compromising blinding procedures. Use of the Fingerprint function is, however, recommended during routine laboratory use and was found to detect the contamination events that occurred in this study. Stringent precautions against contamination when performing PCR-based tests are needed (16), including architectural separation of laboratory areas used for PCR setup and DNA analysis, use of negative template controls, and sequence comparison with sequences determined previously in the same laboratory, as with fingerprinting. Uracil N-glycosylase is also used to decrease carry-over contamination in PCR-based assays, but was not included in the formulation of the TRUGENE assay which minimizes and detects contamination events by other methods. Further, uracil N-glycosylase treatment would not eliminate the risk of carryover contamination during the setup of CLIP reactions utilized in the TRUGENE assay.

Our study posed several real-life challenges to a drug resistance genotyping assay, including use of plasma from HIV-1-infected humans. Virus populations in vivo are diverse, which poses significant challenges to molecular diagnostic assays that involve oligonucleotide primers for amplification and sequence detection. These challenges are addressed in the TRUGENE HIV-1 Genotyping Kit through use of degenerate primers and ambiguity codes that allow information about sequence mixtures to be retained in the report. Nevertheless, all of the codon identification errors analyzed in Table 4 (other than those due to mislabeling of aliquots prior to distribution to testing sites) involved incomplete, but partially correct, identification of mixtures of codons present in either the gold standard consensus or the assay result. Even analysis of multiple viral sequence clones from two highly sensitive PCRs, as performed here to establish the gold standard sequence consensus, does not guarantee full representation of the diversity of clinically occurring HIV-1 populations. For example, the consistent detection of the RT L210W mutation (as a mixture with wild type) in 11 of 12 independent assays of specimen VA-MH-005 suggests that the mutation was present in the virus population at low frequency, although it was not detected in any of the 20 clones used to establish the gold standard. Failure to identify the mutation in the clones used for the gold standard could occur due to random sampling error of diverse populations. Ambiguity inherent in establishing gold standard measurements of naturally occurring virus populations is also indicated by the improvement in assay accuracy (Table 3) observed when using the 5% cutoff to define the gold standard, which allowed detection of codons present in less than 30% of the virus population to be considered correct. Mutation detection in homogeneous virus stocks derived from infectious molecular clones was also found to be highly accurate and reproducible, representing important measures of the assay performance that is not confounded by virus population diversity.

The challenges for training and quality control were also increased in our study because of the involvement of multiple laboratories (both academic and clinical), multiple testing personnel, and multiple kit lots. Use of large panels of specimens, unique labels for all aliquots, and an external data management and analysis agency eliminated any opportunities for partial unblinding of analysis or reporting in our study. Hence, we believe that the assay performance described here will reflect assay performance in a variety of clinical and research laboratories. Our estimates of assay performance cannot be directly compared with estimates based on less challenging evaluations that involve analysis of molecular clones of HIV-1, extensive use of reference laboratories, or proficiency panels that involve limited numbers of specimens and labeling that may allow laboratories to cross-validate results prior to reporting.

After the study was completed, the TRUGENE assay that is available to clinical laboratories was modified to include a centrifugation step to concentrate the virus from 1 ml of plasma prior to RNA extraction and provision of alternative PCR primers (version 1.5). The centrifugation step was found to decrease the minimum viral load required for genotypic evaluation to less than 100 copies/ml (R. Lloyd, R. Schuuman, H. Stang, DeGroot, L. Hough, D. Burns, R. Mathis, and P. Feorino, 3rd Int. Workshop HIV Drug Resist. Treat. Strat., abstr. 52, 1999). The alternative primers are designed to bind to diverse HIV-1 subtypes to permit testing of non-B HIV-1 subtypes using the TRUGENE system (L. Jagodzinksi, J. Cooley, S. Kelly, and N. Michael, 9th Conf. Retrovir. Opportun. Infect., abstr. 594-T, 2002). The version 1.5 primers are one of the primer sets used in this study to amplify the gold standard sequences. Both assay modifications aim to improve the assay success rate and did not affect the accuracy of sequence determinations in the clinical laboratory that participated in this study (data not shown).

The clinical utility of drug resistance genotyping assays has been consistently demonstrated in clinical trials where laboratory testing is performed in reference laboratories with extensive experience with molecular biology assays (2, 10). Translation of this experience from clinical trials to clinical practice will require the availability of assay technology that can be mastered and consistently controlled in clinical diagnostic laboratories. The results of this study indicate that the TRUGENE HIV-1 Genotyping Kit is an accurate and robust system suitable for clinical laboratory testing.