Identification of Performance Problems in a Commercial Human Immunodeficiency Virus Type 1 Enzyme Immunoassay by Multiuser External Quality Control Monitoring and Real-Time Data Analysis

Abstract

In June 2005, a pilot program was implemented in Canadian laboratories to monitor the performance of the Abbott human immunodeficiency virus types 1 and 2 (HIV-1/2) gO enzyme immunoassay (EIA). Two different external quality control (QC) reagents and a “real-time” software analysis program were evaluated. In November 2005, higher-than-expected calibrator rate values in these kits were first reported at the Ontario Ministry of Health (Etobicoke), followed by the Alberta Provincial Public Health Laboratory (Edmonton and Calgary) and others. These aberrations were easily and readily tracked in “real time” using the external QC reagents and the software program. These high calibrator values were confirmed in Delkenheim, Germany, by Abbott, and a manufacturing change was initiated beginning with lot 38299LU00, which was distributed to laboratories in Canada in April 2006. However, widespread reports of calibrator failure by laboratories outside Canada were made in March 2006. In April 2006, Abbott Diagnostics initiated a level III investigation to identify the root cause, which was prolonged storage, under uncontrolled storage conditions, of the raw material used in the manufacture of the matrix cells. To the best of our knowledge, this is the first example of a program in Canada for serological testing that combines a common external QC reagent and a “real-time” software program to allow laboratories to monitor kit performance. In this case, external QC monitoring helped identify and confirm performance problems in the Abbott HIV-1/2 gO EIA kit, further highlighting the benefit of implementing such a program in a national or multilaboratory setting for laboratories performing diagnostic and clinical monitoring testing.

Serological testing for human immunodeficiency virus (HIV)-specific antibodies by an enzyme immunoassay (EIA) remains the most practical method of screening for infection with HIV type 1 (HIV-1) and HIV-2. As an initial screening test, the HIV EIA remains an economical and convenient way of testing numerous samples at once. Significant improvements have been made in the HIV EIA so that the window period of the latest third-generation versions is quite sensitive (19 days [95% confidence interval, 15 to 23 days]) (1, 3, 5). While the window period for HIV can be shortened several days by newer, fourth-generation antibody-antigen assays and even further by nucleic acid testing, these technologies may not be practical or possible in resource-limited settings. Improvements to the HIV EIA have been the result of challenges imposed by the need to shorten the window period of detection due to seroconversion and ever-increasing genetic diversity. These challenges will always be present and serve as a reminder that quality in HIV testing needs to be constantly monitored.

One of the best methods that an individual laboratory can use to ensure high quality in HIV testing is to enroll in external quality assessment schemes (EQAS), which provide a measure of proficiency testing (PT) on well-characterized panels composed of several challenging samples (2). These panels are typically sent out several times a year by PT providers. EQAS challenge the whole facility in addition to the specific laboratory. In addition to providing a measure of competency, EQA programs are required for medical laboratories seeking and maintaining accreditation. As an example, international standard ISO 15189 for medical laboratories now has specific requirements for preexamination and postexamination procedures in addition to the examination phase itself (4). EQA programs are designed to help address all these areas. Furthermore, many standard-setting organizations, such as the International Standards Organization (ISO), require participation in third-party EQAS where possible (4). Problems identified by EQAS can then be addressed by process review, education, and/or other remedial actions. (For an excellent review of quality assurance practices in HIV testing, please refer to N. Constantine et al., Retroviral Testing and Quality Assurance: Essentials for Laboratory Diagnosis [2].)

One weakness of EQAS, however, is their inability to identify problems that may have already happened or are about to happen. Even though test performance is one of the components measured during a given EQAS-PT round, a problem may never be identified or may be identified too late due to the usual lapse between PT panel shipments and the receipt of the final report. A method that complements PT by EQAS and addresses these weaknesses is quality control (QC) monitoring, in which an external reagent is tested at the same time that routine testing is performed, and the results can be followed over time to monitor kit performance. QC monitoring thus can measure precision, or how well the assay reproduces the same test result under different operating conditions.

Here we report findings from a program implemented since March 2005 in Canada in which (i) two different but common external QC reagents and (ii) a real-time data analysis program (EDCNet [www.nrl.gov.au]) were provided to Canadian laboratories performing HIV testing using the Abbott AxSYM HIV-1/2 gO EIA (Chicago, IL). Within 6 months of implementation, the program clearly identified performance problems with this assay that led to unacceptably high calibrator values, and Abbott Diagnostics conducted a high-level investigation resulting in the implementation of QC procedures in its manufacturing processes. One consequence of higher-than-normal signal-to-cutoff (S/Co) ratios is increased and unnecessary confirmatory testing of samples falsely reactive by HIV antibody testing.

MATERIALS AND METHODS

Description of Canadian laboratories performing HIV testing.

Approximately 60 Canadian laboratory sites are enrolled in the PT program for HIV serological (antibody) testing of the National Laboratory for HIV Reference Services (NLHRS). Blood screening for transfusion programs is performed by the Canadian Blood Services and Hema Quebec. Outside of blood screening, the majority of HIV antibody testing for diagnostic purposes is performed by provincial laboratories/health ministries and university hospital-based laboratories. Many of these laboratories use the Abbott AxSYM HIV-1/2 gO EIA, the front-line EIA that is approved for use in Canada. This assay is a microparticle EIA that incorporates recombinantly derived antigens to four viral proteins (HIV-1 group M Env, HIV-1 group O Env, HIV-1 core, and HIV-2 Env) and two synthetic peptides corresponding to HIV-1 Env and HIV-2 Env. This HIV EIA represented the next version in Abbott's HIV EIA line to replace the AxSYM rDNA HIV-1/HIV-2 test. The HIV-1/2 gO EIA is not licensed by the FDA in the United States. The initial pilot study included five laboratories across Canada utilizing a combination of 10 different Abbott AxSYM instruments. These included the Alberta Provincial Health Laboratories (Edmonton and Calgary), the Saskatchewan Provincial Health Laboratory (Regina), and the Georges Dumont Hospital (Moncton, New Brunswick).

External QC reagents.

Two commercially available multimarker controls weakly reactive for HIV-1 and HIV-2 antibodies were provided to the Canadian laboratories: the PeliSpy Multi-Marker run control type 17 (AcroMetrix, Benicia, CA) and the Accurun 1 Multi-Marker positive control series 2600 (BBI Diagnostics [now SeraCare], Frederick, MD). Both markers were previously qualified for use specifically on the Abbott AxSYM HIV-1 gO EIA platform. The PeliSpy control was chosen so that laboratories could monitor their results with reference to those of other Abbott AxSYM users in Australia and other regions of the world where this external QC reagent was distributed. The Accurun control was specific only to the Canadian laboratories participating in this pilot. The PeliSpy control is composed of inactivated antibody/antigen-positive plasma diluted in normal citrated, defibrinated, and delipidized human plasma. The Accurun control is manufactured from human serum or plasma including materials reactive to HIV-1 and HIV-2. Both controls are designed to mimic patient samples and are ready to use, requiring no reconstitution or special handling. Laboratories decided the frequency of use of these controls according to their standard operating procedures. The results of the external control reagents and kit controls were entered into the real-time external QC software program.

External QC “real-time” software program.

The software used in this program was EDCNet, which was obtained from the Australian HIV Serology Laboratory (Melbourne) (www.nrl.gov.au). Briefly, EDCNet is an Internet-based application that allows an individual laboratory to enter results from the common external QC reagent and compare its results with those of other laboratories using the same QC sample and test kit. Data analysis was performed in “real time,” and the individual laboratory's performance was plotted in the form of standard Levey-Jennings plots. Standard Westgard rules were applied (7). A run failure was defined as a single data point outside 3 standard deviations (SD). A warning would be generated upon any of the following three findings: (i) 5 data points exceeding 2 SD, (ii) 2 of 3 successive data points exceeding 2 SD, or (iii) 10 data points on the same side of the mean. All data points exceeding 2 SD were closely monitored. The system allowed the performance of a laboratory to be measured in terms of intra- and/or interlaboratory comparison, and each registered instrument was tracked individually. In addition, the NLHRS was able to monitor all registered laboratories, which allowed identification of overall trends. Batch test results for each kit lot of the Abbott AxSYM HIV-1/2 gO EIA were also analyzed. Variances were tested using Bartlett's test of variance. The F test was computed for each pair of variances, and the overall alpha level was controlled using Bonferroni's correction for multiple comparisons. Means were tested using a two-sample t test for unequal variances. A t test was performed for each possible pair of means, and the overall alpha level was controlled using Bonferroni's correction for multiple comparisons.

RESULTS AND DISCUSSION

External QC monitoring and EDCNet.

Quality monitoring with an external control reagent, from a source other than the manufacturer of a test kit, can help detect and reduce deficiencies in a laboratory's internal testing process prior to the release of patient test results. This ultimately leads to improvement in the quality of results generated by the laboratory (6). In the case of multiple users using the same testing platform, the use of a common (lot-specific) external QC reagent allows an individual laboratory to further compare its performance with that of other users and can provide much better evaluation of specific kit lots over time. Figure 1 presents examples of tracking with the PeliSpy (Fig. 1a) and Accurun (Fig. 1b) external QC reagents over 2 months (12 April to 3 June 2005) in one laboratory (lab 389) using the Abbott AxSYM HIV-1/2 gO EIA. For this period in lab 389, the performance of the assay was normal, as indicated by the two different external QC reagents, with the data points distributed on both sides of the mean. The intralaboratory mean ± 2 SD is consistent with the mean ± 2 SD of the group as a whole. Comparison of QC data for the two different reagents has since been examined over 2 years. Although the performance of the Accurun control has paralleled that of the PeliSpy control, the users of the Accurun control were limited to Canadian laboratories; therefore, we have since discontinued its distribution and instead used only the PeliSpy (Acrometrix) control, due to its more widespread use and distribution.

FIG. 1.

External QC monitoring in lab 389 of the Abbott AxSYM HIV-1/2 gO EIA using the external control reagents PeliSpy (T17:s2068:PS050111) (a) and Accurun (2608-P 107084) (BBI) (b) for the period from 1 January 2005 to 7 June 2005. Blue solid lines represent the mean ± 2 SD for lab 389 only. Red dashed lines represent the mean ± 2 SD for all labs using a specific lot of external QC reagent. Lot numbers of the Abbott AxSYM kit are listed on the upper x axis.

Identification and monitoring of problems in the Abbott HIV-1/2 gO EIA by external QC monitoring.

In late November 2005, the Ontario Ministry of Health (Etobicoke) first reported high calibrator values on the AxSYM HIV-1/2 gO EIA. Unfortunately, this laboratory was not enrolled in the external QC program, so we were not able to track and monitor the performance of this assay in this laboratory. However, shortly after this initial report, other laboratories, which were enrolled in the program, began to report similar high calibrator values resulting in failed runs. The results of examination with the external QC reagent in these other laboratories were readily observed (Fig. 2). In lab 390 (Fig. 2a), the external QC reagent generated multiple data points above +2 SD on two different AxSYM instruments beginning 21 January 2006. This was specific to lot 33287LU00 of the AxSYM HIV-1/2 gO EIA kit, which the Ontario Ministry of Health had previously used in November 2005, when high calibrator values were initially reported. Similarly, lab 391 (Fig. 2b) also observed multiple data points for the external QC reagent above 2 SD on two different AxSYM instruments, beginning on 14 January 2006. Batch analysis of lot 33287LU00 and other lots from this kit was performed. Statistical analysis determined that batch 33287LU00 had the largest range of S/Co values and the largest coefficient of variance of all batches analyzed (Fig. 3) and that lots 28311LU01, 30254LU00, 35442LU00, and 36118LU01 were significantly different (P < 0.05). No significant difference was identified among other lots (P > 0.05). The largest mean S/Co ratio, not surprisingly, was noted for lot 33287LU00 (i.e., 5.49) and was significantly different (P < 0.05) from the mean S/Co ratios of lots 28311LU01, 30254LU00, and 33024LU00. It should be emphasized that statistical analysis was performed on group or summary data only and not on individual data points. Statistical analysis of the original raw data was not performed, and it is possible that such analysis would have revealed additional significant differences with additional lots.

FIG. 2.

Identification of a performance problem in lot 33287LU00 of the Abbott AxSYM HIV-1/2 gO EIA by external QC monitoring with the PeliSpy (T17:s2068:PS050111) QC reagent in lab 390 (a) and lab 391 (b) for the period from 13 May 2005 to 8 April 2006. Blue solid lines represent the mean ± 2 SD for each individual lab only. Red dashed lines represent the mean ± 2 SD for all labs. Lot numbers of the Abbott AxSYM kits are listed on the upper x axis. Each lab uses two AxSYM instruments, which are distinguished by differently colored dots.

FIG. 3.

Batch analysis of S/Co ranges of Abbott AxSYM HIV-1/2 gO kits. The PeliSpy (T17:s2068:PS050620) external control reagent was used in Australian and Canadian labs from 1 January 1999 to 19 April 2006. The mean and the variation around the mean for each lot are represented by the horizontal and vertical bars, respectively. The lot numbers of the Abbott AxSYM kits, including the affected lot (33287LU00), are given along the lower x axis.

Initial response by Abbott Diagnostics: changes to calibrator specification range and a manufacturing change do not lead to improvement.

Initially, in January 2006, Abbott Diagnostics was not able to reproduce the high calibrator readings in other lots outside of those that were first reported in December 2005. When the kits from the suspected lots 33287LU00 and now 34385LU00 were obtained from Canadian laboratories, the high calibrator results were readily reproduced by the scientists at Abbott. This confirmed that factors associated with Canadian laboratory conditions were not the reasons for the high calibrator readings. Abbott Diagnostics first attempted to address the issue in February 2006, when it modified the upper specification limit. This was reflected in a manufacturing change, and new lots, beginning with lot 38299LU00, were distributed to Canadian laboratories in April 2006. Modifications to the calibrator settings initially lowered the number of complaints; however, widespread calibrator failures were now being reported outside of Canada, with the number of reported complaints peaking in April 2006, most notably from Canada and France (Fig. 4). It is interesting that the majority of complaints were from France and followed the same pattern as those from Canada. Unfortunately, we were not able to determine the response of Abbott Diagnostics to the affected users in France; however, Abbott Diagnostics confirmed that their complaints were specific to high calibrator failures resulting in an increase in false-reactive samples.

FIG. 4.

Complaint overview for the AxSYM HIV-1/2 gO EIA. High calibrator values were first detected in reagent lot 35200LU00 in January 2006. Modification of the calibrator settings initially lowered the number of complaints (February), but complaints then increased, peaking in April 2006. ROW, rest of the world. Reprinted with the kind permission of Abbott Diagnostics.

Secondary response by Abbott Diagnostics: root cause analysis identifies matrix cells underlying high calibrator values and decreased specificity in the AxSYM HIV-1/2 gO EIA kit.

On April 13, 2006, Abbott Diagnostics launched a level III investigation. During this investigation, root cause analysis revealed that a small number of matrix cell lots were associated with problematic AxSYM HIV-1/2 gO EIA kits, and on 24 May 2006, customers were notified of these affected matrix cell lots. Initially, AxSYM matrix cells made from different raw matrices used in 2005 and 2006 were screened on AxSYM HIV-1/2 gO reagent lot 39437LU00. It was determined that the problematic matrix cells, associated with the increase in calibrator failures, derived from one single raw matrix lot (lot 5050366) (Fig. 5). Further root cause analysis determined that the affected lot (lot 5050366) of raw matrix had been stored for 90 days, which was longer than the storage time for unaffected lots (average, 22 days [data not shown]). Furthermore, the prolonged storage of raw matrix lot 5050366 occurred during the three summer months of May to August 2005, and according to Abbott Diagnostics, this material had been stored as a roll of raw matrix wrapped in plastic, but the temperature and humidity had not been controlled.

FIG. 5.

Matrix cells of all raw matrices used in 2005 and 2006 tested on AxSYM HIV-1/2 gO reagent lot 39437LU00. Certain lots of AxSYM matrix cells were associated with increased calibrator (Cal) rates. All affected AxSYM matrix cells were made from the single raw matrix lot 5050366. Reprinted with the kind permission of Abbott Diagnostics.

Interestingly, the effect of uncontrolled environmental conditions on the raw matrix and its role in high calibrator readings were confirmed by Abbott Diagnostics. An unaffected raw matrix lot, lot 5050365, was exposed to high heat and humidity (45°C and 100% humidity) for a period of 7 days prior to the manufacture of new matrix cells. As shown in Fig. 6, these experiments clearly showed an increase in the calibrator rates in the AxSYM HIV-1/2 gO EIA kit that used these affected matrix cells.

FIG. 6.

Replication of high calibrator (Cal) values after simulation of adverse environmental conditions. The raw matrix cell lot 5050365, previously unaffected, was exposed to high heat and humidity (45°C; 100% humidity), approximating the conditions observed for the affected raw matrix lot 5050366. Reprinted with the kind permission of Abbott Diagnostics.

On 10 August 2006, Abbott Diagnostics finalized the level III investigation with the statement that the root cause for the higher-than-expected calibrator values was faulty matrix cells produced from raw matrix lot 5050366, which had undergone prolonged storage at the vendor's facility under uncontrolled environmental conditions. As a result, Abbott Diagnostics announced two additional measures to be implemented in order to improve QC: (i) redefinition and implementation of specifications for the raw matrix component to avoid exposure to prolonged heat and humidity and (ii) pretesting of all new AxSYM matrix cell lots on the AxSYM HIV-1/2 gO EIA kit as an additional QC measure.

Conclusions.

The response by Abbott Diagnostics was thorough and comprehensive. Laboratory testing at Abbott Labs on many parameters of the test, including specificity and calibration, was extensive. Representatives from both Canada (Mississauga, Ontario) and Europe (Delkenheim, Germany) were involved. Within Canada, the HIV testing community is represented by the Canadian Association of HIV Clinical Laboratory Specialists (CAHCLS; www.cahcls.org). This organization, which includes among its members the provincial health ministries of British Columbia, Alberta, Saskatchewan, Manitoba, Ontario, Quebec, New Brunswick, Nova Scotia, Prince Edward Island, and Newfoundland, holds annual invitation-only meetings with provincial and regional laboratories involved in HIV testing, blood-screening agencies, kit regulators, and special-interest invitees to discuss issues related to HIV testing in Canada. From the initial reports of high calibrator readings to the resolution and favorable conclusion, extensive communication took place among the CAHCLS members, and several teleconferences between CAHCLS and Abbott Diagnostics were held to discuss the results of ongoing investigations conducted by both groups.

The importance of instituting QC practices in the laboratory setting cannot be emphasized enough. Even in highly industrialized regions of the world, such as North America, where some of the most technologically advanced testing can be found, errors due to poor QC procedures can still occur, as evidenced here with the Abbott AxSYM HIV-1/2 gO EIA. Although the HIV testing community in Canada is a cohesive group with frequent reporting of kit performance issues using e-mail and/or voice mail, this type of communication can be anecdotal. While individual laboratories may already implement internal QC programs for HIV testing, it may be difficult to have immediate knowledge of issues or problems with testing and lots arising in other laboratories. We have demonstrated that the implementation of a program utilizing a common external QC reagent and a software program that allows analysis in “real time” offers the ability for an individual laboratory to monitor performance problems easily and quickly and can be a model for any laboratory performing testing for HIV or any other infectious agent.