Genotyping External Quality Assurance in the World Health Organization HIV Drug Resistance Laboratory Network During 2007–2010

Abstract

The World Health Organization (WHO) has developed a global laboratory network to support human immunodeficiency virus drug resistance genotyping for public health surveillance in resource-limited countries. Blinded proficiency panels are an essential part of a genotyping quality-assurance program and are used to monitor the reliability of genotyping data in the WHO laboratory network. Laboratories in Europe, North America, Asia, Africa, and the Caribbean have tested panels annually since 2007; 103 of 131 submissions (79%) had >99% nucleotide sequence identity and resistance mutation concordance, compared with consensus. Most errors were associated with mixtures in the test specimen, leading to subjectivity in base-calling or amplification bias. Overall, genotyping assays used by the WHO laboratory network are reliable.

There is a risk of development and transmission of drug-resistant human immunodeficiency virus (HIV) strains in resource-limited countries as a consequence of the joint international effort to provide antiretroviral treatment (ART) to millions of persons living with HIV in these settings [1]. If ART is not effectively delivered, HIV drug resistance (HIVDR) could become widespread, leading to an increase in therapeutic failures, transmission of drug-resistant virus, and a decrease in therapeutic options, ART program effectiveness, and survival. The World Health Organization (WHO) global strategy for prevention and assessment of HIVDR [2] consists of a coordinated plan for HIVDR surveillance in countries where ART has been scaled up, to guide population-based selection of ART regimens. The WHO/HIVResNet Laboratory Network is a component of this strategy, supporting HIVDR surveillance by enabling the timely provision of accurate and reproducible genotyping results in a standardized format [3].

Although in resource-limited settings, many laboratories are experienced in genotyping, a lack of standardization limits the production of comparable and reliable results. Existing networks have made attempts to standardize practices and procedures, but there remains a need to develop a common approach for quality assurance. In resource-limited settings, the lack of adequate infrastructure and the cost associated with genotype testing limit the development of genotyping laboratories. Nevertheless, 20 of the 27 laboratories in the WHO/HIVResNet Laboratory Network are located in resource-limited settings (http://www.who.int/hiv/topics/drugresistance/laboratory/en/index.html).

Participation in external quality assurance (EQA) programs is a required component for laboratory accreditation in the WHO/HIVResNet Laboratory Network. EQA is intended to ensure the reliability and quality of genotyping results. Studies have been performed in experienced genotyping laboratories, showing that the quality of data can vary considerably [4–9]. Factors contributing to the quality of the results include the type of assay or kit used, the level of experience of the technician performing the analysis, and the viral subtype present in the clinical specimen. However, results from sequential rounds of proficiency testing demonstrate that, over time, the quality of genotyping results tends to improve [10].

This article reports the results from the annual WHO EQA program using blinded panels of specimens during a 4-year period from 2007 through 2010.

METHODS

Proficiency Panel Composition and Preparation

Four proficiency panels were developed in collaboration with the National Institutes of Health through the National Institute of Allergy and Infectious Diseases’ Virology Quality Assurance Program (VQA) [11] and sent to 32 network member or candidate laboratories in Europe, North America, Asia, Africa, and the Caribbean during 2007–2010 (1 panel each year). Each panel was composed of 5 specimens (Table 1), diluted to achieve final viral loads of 3500–57 000 copies/mL. Each panel contained at least 3 specimens with non-B subtypes, and the last 3 panels had at least 1 specimen with multiple resistance-associated mutations.

Table 1.

Proficiency Panel Composition, Nucleotide Alignment, and Drug Resistance Mutation Site Scores

						DRM Site Scores					Nucleotide Sequence Alignment Scores
Panel	Specimen	Subtype	Viral Load (Copies/mL)	Comments	No.a	N >99%	N >98%	Min	Max	Mean	N >99%	N >98%	Min	Max	Mean
1	1	D	18 547		26	18	23	86.9	100	98.7	8	24	97.3	99.0	98.8
	2	C	19 139	Specimen 2 and 3 are the same	26	26	26	100	100	100	26	26	99.6	100	100
	3	C	12 380		26	26	26	100	100	100	26	26	99.8	100	100
	4	C	48 955	Specimen 4 and 5 are the same	26	19	25	96.7	100	99.5	0	24	97.2	98.8	98.5
	5	C	16 663		26	18	21	96.7	100	99.2	8	25	97.6	99.1	98.8
	Overall				26	23	24	96.8	100	99.5	23	26	98.7	99.4	99.2
2	1	CRF01_AE	27 810		34	30	32	81.5	100	99.3	31	32	86.7	99.3	98.8
	2	C	14 282		34	25	25	86.4	100	98.6	28	32	86.5	99.9	98.9
	3	F	24 238		34	29	31	89.4	100	99.3	31	33	84.1	100	99.4
	4	B	3557	Multiple RAMs	32	19	27	95.5	100	99.0	19	32	98.1	99.5	99.1
	5	CRF02_AG	57 005		33	27	29	72.7	100	98.3	27	29	86.2	99.9	98.9
	Overall				35	23	30	84.8	100	98.6	28	31	86.5	99.7	98.7
3	1	F	13 053		33	31	31	55.9	100	98.6	31	31	71.2	100	99.0
	2	B	10 735		33	31	32	90.6	100	99.7	32	32	93.9	100	99.7
	3	C	7748	Major NRTI and NNRTI mutation	32	29	31	94.1	100	99.7	31	31	97.9	100	99.9
	4b	C	10 958	Multiple RAMs	23	18	22	64.2	100	98.2	22	22	92.6	100	99.6
	5	B	8398	Major NNRTI mutations	33	31	31	83.1	100	99.4	30	32	91.0	100	99.6
	Overall				33	31	31	80.8	100	99.1	31	32	88.5	100	99.5
4	1	F	15 350		36	34	36	98.5	100	99.9	32	34	96.4	100	99.5
	2	C	56 300	Major NRTI and NNRTI mutations	36	30	33	89.7	100	99.4	34	34	95.8	100	99.7
	3	B	8415	Major PI and NRTI mutations	37	20	33	95.5	100	99.1	35	36	97.2	100	99.6
	4	C	13 613	Major NRTI and NNRTI mutations	37	23	29	92.6	100	98.7	32	34	96.1	100	99.5
	5	B	5718	Major NNRTI mutation	37	35	37	98.3	100	99.9	33	36	97.7	100	99.6
	overall				37	30	35	97.2	100	99.4	33	36	97.1	100	99.6

Abbreviations: DRM, drug resistance mutation; NNRTI, nonnucleoside reverse transcriptase inhibitor; NRTI, nucleoside reverse transcriptase inhibitor; PI, protease inhibitor; RAM, resistance-associated mutation.

^aNumber of laboratory-assay combinations (some laboratories used >1 assay); amplification failures excluded.

^bIf specimen 4 was not amplified, only results from the other 4 specimens were considered (see Results).

All subtype B virus strains were obtained from the VQA donor pool. Donors were prescreened for viral load and genotype with use of both ViroSeq (Life Technologies) and TruGene (Siemens) assays. If results were acceptable, the donor returned 2 weeks later and donated a unit (∼450 mL) of blood. Whole blood was centrifuged (1500g for 15 minutes), and the plasma was recentrifuged (1500g for 10 minutes). Clarified plasma was aliquoted and frozen in bulk at −80°C until needed. Non-B subtype isolates were supplied to VQA as culture supernatant harvested from HIV peripheral blood mononuclear cell cocultures [12] from several sources worldwide. Subtype designation was verified by sequencing a large section of the HIV pol gene [13].

For proficiency panel production, isolates were diluted to a predetermined concentration in normal HIV-seronegative citrated plasma (panels 1–3) or normal HIV-seronegative ethylenediaminetetraacetic acid serum (panel 4). The diluted specimens were aliquoted (1.25 mL each), labeled, and frozen in liquid nitrogen until needed. Proficiency panels were retested for viral load with use of the COBAS Amplicor Monitor assay (Roche) and genotyped using both the ViroSeq and TruGene assays before shipping. All panels were shipped frozen on dry ice to the testing laboratories.

Genotyping

Specimens (1.25 mL each) were sent to testing laboratories as frozen plasma on dry ice and were tested using ≥1 methods in use at the time, including ViroSeq [14], TruGene [15, 16], or an in-house method (“home brew”). Nucleotide sequences were submitted to VQA for analysis within 4–8 weeks after proficiency panel receipt.

Details of the number of laboratories testing each panel and the assays used are as follows: Panel 1 (2007) included 21 laboratories that reported results from a total of 25 assays (8 ViroSeq, 6 TruGene, and 11 in-house); panel 2 (2008) included 28 laboratories that reported results from a total of 35 assays (11 ViroSeq, 7 TruGene, and 17 in-house); panel 3 (2009) included 23 laboratories that reported results from a total of 33 assays (13 ViroSeq, 6 TruGene, and 14 in-house; and panel 4 (2010) included 26 laboratories that reported results from a total of 37 assays (15 ViroSeq, 6 TruGene, and 16 in-house).

For panels 2, 3, and 4, data from at least 1 specimen from 1–3 laboratories were not included because of failure to amplify protease (PR), reverse transcriptase (RT), or both.

Data Analysis and Scoring

Analysis was limited to codons 10–99 in PR and 38–240 in RT. Consensus sequences for each specimen were generated on the basis of >80% concordance; individual test results were compared with that specimen's consensus sequence. The consensus base could be a mixture if 80% of the sequences considered it to have the same mixture. However, if 80% concordance was not achieved at a given position, an “N” was inserted in the consensus sequence and this position was ignored during scoring. Drug resistance mutation (DRM) codons containing the ambiguous base were also ignored.

An overall sequence identity score (percentage of matching nucleotides of the total number of unambiguous nucleotides in the consensus) and concordance at DRM codons (percentage of concordant DRM codons, as defined by the International Antiviral Society–USA [17], of the total number of DRM codons not containing an ambiguity in the consensus) were used to assess laboratory performance. Mixed vs unmixed bases were counted as discrepancies for nucleotide sequence alignment score. For the DRM site score, if a mutant codon was present in the consensus and included in the reported sequence, regardless of being mixed or not, it was not counted as a discrepancy. Frameshift errors caused by insertion or deletion of ≥1 nucleotides resulted in a sequence identity score penalty equivalent to 10 single nucleotide mismatches but did not affect the DRM site score (unless they occurred in a DRM codon).

RESULTS

In 2007 (panel 1), 21 of 26 submissions (81%) had overall DRM and alignment scores of at least 99% for the 5 specimens. In 2008 (panel 2), only 21 of 35 submissions (60%) had an overall DRM and alignment score of at least 99%. Three laboratories failed to reach this threshold because of inability to amplify PR or RT from ≥1 specimens. In 2009 (panel 3), 31 of 33 submissions (94%) had an overall DRM and alignment score of at least 99%. Because an unusually high number of laboratories were unable to amplify specimen 4 in this panel, it is likely that the viral load was lower than intended. Therefore, for this panel, if specimen 4 was not amplified, only results from the other 4 specimens were considered. Last, in 2010 (panel 4), 30 of 37 submissions (81%) had an overall DRM and alignment score of at least 99%.

Results from the 4 panels are summarized in Table 1, which shows the number of submissions reaching a 99% or 98% threshold for the DRM site score and nucleotide alignment scores and the minimum, maximum, and mean values. Overall, 103 of 131 submissions had >99% nucleotide sequence identity and DRM concordance, compared with consensus. Mean sequence identity and DRM site scores were 98.5%–100% and 98.2%–100%, respectively, over the 4 panels. In most panels, scores varied significantly between specimens. For example, in panel 1, all submissions reached 99% nucleotide identity with the consensus for specimens 2 and 3, but only 8 of 26 did so for specimens 1 or 5 and none for specimen 4; in panel 2, specimen 4 had the lowest scores. In panel 4, specimens 3 and 4 had the lowest DRM site scores. These observations indicate that ≥1 properties specific to certain specimens are related to difficulties in generating a reproducible sequence across laboratories.

To investigate the possible reasons underlying the generation of sequencing errors, discrepancies at DRM sites were analyzed. The frequencies of errors observed in at least 3 submissions were examined (data not shown). Frequently, the laboratory did not report a mixture present in the consensus sequence. Sometimes, the reported sequence did not result in a change in the amino acid (eg, codon 11 in PR of specimen 1, panel 1, which was GTY but many laboratories reported GTC). When the mixture resulted in both wild-type and a resistance-associated amino acid being encoded, in some cases the reported sequence included the base that corresponds to the resistant variant, whereas in others it included only the wild-type variant. Alternatively, discrepancies were a result of the presence of mixtures in the specimen that were not reported in >80% of the submissions and, thus, were not reflected by the consensus sequence. For example, 5 laboratories reported a mixture in the third position of codon 62 (GCY) in PR of specimen 5 of panel 3, where the sequence in the consensus was GCC. Together, these 2 types of discrepancies, involving a mixed base in either the consensus or the submitted sequence, accounted for 81% of the DRM site errors (Table 2). Other specific problems not involving mixtures that contributed to low DRM scores included low-quality sequence, missing sequence, polymerase chain reaction (PCR) contamination, and editing errors leading to frameshifts; these accounted for a total of 17% of the discrepancies.

Table 2.

Types of Base-Calling Errors

	Panel
Type of Error	1	2	3	4	Total
Mix in consensus not in test	21 (67.7 %)	13 (20.3 %)	20 (20.8 %)	99 (58.2 %)	153 (42.4 %)
Mix in test not in consensus	4 (12.9 %)	24 (37.5 %)	46 (47.9 %)	67 (39.4 %)	141 (39.1 %)
Other error involving mixtures	3 (9.7 %)	1 (1.6 %)	0 (0 %)	2 (1.2 %)	6 (1.7 %)
Outright discrepancy not involving mixtures	3 (9.7 %)	26 (40.6 %)	30 (31.3 %)	2 (1.2 %)	61 (16.9 %)

To further explore the relationship between reproducibility of sequence results and the presence of mixtures, the percentages of mixed bases in the consensus sequences (over the entire sequence or just at the DRM sites) were compared with the DRM site errors in the test sequences (Table 3). A general trend for increased error occurrence when more mixtures were present at the corresponding DRM site in the consensus was observed. In other words, after removing the DRM sites where an ambiguity was present in the consensus (because these sites were ignored in the test sequences for scoring purposes), the specimens with the most DRM site errors were also those with the most mixtures. In cases in which there were many errors in specimens with fewer mixtures, there was at least 1 sequence submitted that had poor quality or other problems (eg, 1 submission with missing sequence at the 3′ end of RT in specimen 5 of panel 2, and 1 submission of a sequence for specimen 4 in panel 3 that was a contaminant).

Table 3.

Distribution of Errors at Drug Resistance Mutation Sites, Compared With Sequence Heterogeneity

		Base Composition				Mixtures at DRM Sites			Errors at DRM Sites
Panel	Specimen	No. Mixed	No. Unmixed	% Mixed	% Mixed, no Nsa	PR	RT	Total	PR	RT	Total
1	1	9	870	1.02%	0.11%	1	0	1	10	1	11
	2	1	878	0.11%	0.11%	0	0	0	0	0	0
	3	1	878	0.11%	0.11%	0	0	0	0	0	0
	4	18	861	2.05%	0.92%	2	0	2	9	0	9
	5	17	883	1.89%	1.12%	3	0	3	11	0	11
2	1	6	873	0.68%	0.00%	0	0	0	1	4	5
	2	8	871	0.91%	0.80%	2	0	2	14	8	22
	3	0	879	0.00%	0.00%	0	0	0	0	4	4
	4	6	873	0.68%	0.23%	2	0	2	14	12	26
	5	1	878	0.11%	0.00%	0	0	0	3	4	7
3	1	0	879	0.00%	0.00%	0	0	0	0	2	2
	2	16	863	1.80%	0.90%	3	0	3	24	2	26
	3	1	878	0.10%	0.00%	0	0	0	2	12	14
	4	2	877	0.20%	0.00%	0	0	0	16	13	29b
	5	15	864	1.70%	0.80%	1	0	1	3	22	25
4	1	27	852	3.07%	1.50%	2	1	3	13	0	13
	2	0	879	0.00%	0.00%	0	0	0	9	9	18
	3	18	861	2.05%	0.81%	3	2	5	26	19	45
	4	19	860	2.16%	2.16%	0	5	5	8	49	57
	5	25	854	2.84%	0.81%	4	2	6	24	13	37

Abbreviations: DRM, drug resistance mutation; PR, protease; RT, reverse transcriptase.

^aNot counting Ns because these are ignored in scoring.

^bTwenty-four of these discrepancies were a result of the wrong sequence being submitted (possible contamination).

Across the 4 panels, a total of 58 submissions were derived from in-house assays, 47 from ViroSeq, and 25 from TruGene. The performance of the 3 different assay methodologies was compared using the nucleotide alignment scores from all panels combined or each panel individually. No differences were observed between the ViroSeq and TruGene assay, although for some panels, the low number of results by each method limited analysis (data not shown). Comparison of in-house methods with both commercial assays together revealed a small but consistent increase in scores observed when commercial assays were used; for example, when data from all panels were combined, the median scores were 99.7% for commercial assays and 99.5% for in-house methods (Mann–Whitney test, P = .0035) (Figure 1). The differences observed for panels 1, 2, and 4 were also statistically significant (Mann–Whitney test, P < .05). Although relatively small, a difference of 0.2% translates to 2 nucleotides per 1000 sequenced or approximately per specimen tested. However, DRM scores were not significantly different between the commercial assays (individually or combined), compared with in-house assays.

Figure 1.

Comparison of median nucleotide alignment scores for the different panels and assay methods. Error bars represent the interquartile range. White bars indicate in-house (IH) assays, gray bars indicate ViroSeq or TruGene commercial assay (CA). ^aMann–Whitney P values for IH versus CA (panel 1: P = .019; panel 2: P = .023; panel 3: P = .15; panel 4: P =.0029; all, P = .0035. Abbreviation: NT, nucleotide.

DISCUSSION

Sequence-based assays for assessment of HIVDR are widely used in high-income countries for individual patient management and are becoming more common in resource-limited settings. Because these assays involve multiple steps, are based on template amplification via PCR, and use specimens that might contain multiple related species of HIV (often referred to as quasispecies), it is challenging to maintain a high level of accuracy and reproducibility. Therefore, to maximize the reliability of genotyping data generated in multiple laboratories in different countries, a rigorous quality assurance system is required that includes annual proficiency testing with blinded specimens. This type of EQA is a well-accepted mechanism that helps achieve this goal.

Results generated by the laboratories participating in the WHO/HIVResNet Laboratory network and presented here are similar to those reported by other investigators involving laboratories in resource-rich countries [5, 11, 14–16, 18–20]. The use of commercial assays (ViroSeq and TruGene) for more than half of the submissions is likely to be at least partly responsible for this good performance, because these assays use consistently manufactured reagents that have been well tested during quality control before release. Indeed, we observed a small but significant increase in nucleotide alignment scores (but not DRM site scores) when either commercial assay was used, compared with in-house assays. However, use of commercial assays places a significant financial burden on country HIVDR survey budgets, which can be alleviated by the use of in-house assays. Most laboratories using in-house assays also performed well, although there was a tendency toward increased amplification failures, most likely because of higher minimum viral load requirements or the use of primers developed for certain subtypes that may be prevalent in the region in which the laboratory is located but that do not match all of the subtypes represented in the panel.

The vast majority of discrepancies between individual submitted sequences and the corresponding consensus sequence involved a mixed base in one or the other. Possible reasons for not detecting mixtures include overreliance on automated base-calling software that is not designed to recognize minority peaks at <50%, high background noise masking low-level peaks, manual editing procedures using a low threshold for minority peaks, and PCR amplification bias resulting in only 1 of 2 or more variants present in the specimen. Reporting of sequences with more mixtures than other laboratories could be related to suboptimal base-calling requirements (eg, not requiring that mixtures be detected in sequence generated from both strands), high background, and contamination. Because most discrepancies involve mixtures, specimens containing more mixtures will be more difficult for multiple laboratories to agree on or to reach a set minimum threshold of sequence identity. Therefore, the use of a single and stringent criterion (eg, >99%) for evaluation of sequence-based assays may be too stringent when using clinical specimens containing mixed bases. If set thresholds are applied, it is recommended that a flexible scoring system be used that makes allowances for mixtures (eg, at positions that do not result in amino acid changes) or that specimens with above-average proportion of mixed bases be avoided in EQA panels. Standardization of editing procedures might help to improve concordance rates for specimens containing above-average numbers of mixed bases.

Standardization of assay accuracy and reproducibility across laboratories is a key component of the overall quality assurance scheme inherent in the WHO HIVDR laboratory strategy. Continued participation and accreditation based on results from panels like these is an essential component of the strategy that gives researchers, epidemiologists, and public health officials the required confidence that results are as reliable as possible.