Concordant Proficiency in Measurement of T-Cell Immunity in Human Immunodeficiency Virus Vaccine Clinical Trials by Peripheral Blood Mononuclear Cell and Enzyme-Linked Immunospot Assays in Laboratories from Three Continents

Abstract

The gamma interferon (IFN-γ) enzyme-linked immunospot (ELISPOT) assay is used routinely to evaluate the potency of human immunodeficiency virus (HIV) vaccine candidates and other vaccine candidates. In order to compare candidates and pool data from multiple trial laboratories, validated standardized methods must be applied across laboratories. Proficiency panels are a key part of a comprehensive quality assurance program to monitor inter- and intralaboratory performance, as well as assay performance, over time. Seven International AIDS Vaccine Initiative-sponsored trial sites participated in the proficiency panels described in this study. At each laboratory, two operators independently processed identical sample sets consisting of frozen peripheral blood mononuclear cell (PBMC) samples from different donors by using four blind stimuli. PBMC recovery and viability after overnight resting and the IFN-γ ELISPOT assay performance were assessed. All sites demonstrated good performance in PBMC thawing and resting, with a median recovery of 78% and median viability of 95%. The laboratories were able to detect similar antigen-specific T-cell responses, ranging from 50 to >3,000 spot-forming cells per million PBMC. An approximate range of a half log in results from operators within or across sites was seen in comparisons of antigen-specific responses. Consistently low background responses were seen in all laboratories. The results of these proficiency panels demonstrate the ability of seven laboratories, located across three continents, to process PBMC samples and to rank volunteers with differential magnitudes of IFN-γ ELISPOT responses. These findings also illustrate the ability to standardize the IFN-γ ELISPOT assay across multiple laboratories when common training methods, reagents such as fetal calf serum, and standard operating procedures are adopted. These results are encouraging for laboratories that are using cell-based immunology assays to test HIV vaccines and other vaccines.

Most human immunodeficiency virus (HIV) vaccines currently in development aim to induce cellular immune responses, since these have been shown previously to temporally correlate with the containment of virus in infected individuals and, more significantly, to be crucial in the suppression of virus in the rhesus macaque model (2, 13, 15, 25). The ability to measure and quantitate cellular immune responses has been facilitated through the development of enzyme-linked immunospot (ELISPOT) and flow cytometry assays which determine the number of antigen-specific cells through surrogate markers of effector function, such as cytokine production or the degranulation of lytic granules (1, 8, 23, 29), and are more quantitative and less labor-intensive than traditional assays that detect T-cell responses, such as ⁵¹Cr release and lymphoproliferation assays (19). The gamma interferon (IFN-γ) ELISPOT assay is a primary assay employed to measure vaccine immunogenicity in HIV vaccine clinical trials, in addition to trials in the cancer, malaria, and tuberculosis vaccine fields (23, 30, 31). Although data on the performance of the IFN-γ ELISPOT assay across multiple laboratories both within and across continents are critical to the generation of standardized data on vaccine immunopotency (14), little published data exist. The IFN-γ ELISPOT assay results can demonstrate whether a vaccine is able to induce a range of immune responses in a particular population, therefore justifying further development. The value of standardized methods for determining vaccine immunopotency should not be diminished in spite of recent disappointing data from an HIV vaccine trial in which advancement to a phase IIb trial was based partly on IFN-γ ELISPOT data from phase I and II clinical trials (7, 26). Future modifications to the IFN-γ ELISPOT assay may increase its relevance to efficacy testing or allow it to correlate better with elaborate assays that yield critical effector functions such as the inhibition of viral replication (9, 24). The International AIDS Vaccine Initiative (IAVI), in collaboration with local partners, has developed good clinical laboratory practice (GCLP) guideline-compliant clinical trial laboratories at trial units across Europe, Africa, and India. These GCLP guideline-compliant laboratories can be used for the comparative assessment of HIV vaccine candidates developed by IAVI and other organizations and partners, for example, the Division of AIDS (NIH, Bethesda, MD) and biotechnology firms, to facilitate the development of an HIV vaccine (10, 22). As part of the ongoing assessment of laboratory performance and assay result comparability, IFN-γ ELISPOT proficiency panels are conducted regularly at the IAVI-sponsored laboratories. Such proficiency panels have also been conducted among laboratories from different organizations within the HIV vaccine field and have recently been implemented at laboratories working within the Cancer Vaccine Consortium (3, 4, 11). In contrast to published data, the findings of the present study demonstrate that when standardized training and validated assay methods are followed, the results of the IFN-γ ELISPOT assay and the associated handling of test material are notably and highly concordant among laboratories. These data hold promise for the HIV vaccine field as a whole and also for cancer, malaria, and tuberculosis cell-based vaccines. It is possible that comparable data can be obtained across multicenter trials and continents, facilitating concordant and, if warranted, accelerated vaccine development efforts.

MATERIALS AND METHODS

Participating laboratories.

The following laboratories are currently participating or have previously participated in IAVI-sponsored HIV vaccine trials: (i) the IAVI Core Laboratory London (hereinafter referred to as the IAVI Core Lab), Imperial College, London, United Kingdom, (ii) the Centre for Clinical Vaccinology and Tropical Medicine (hereinafter identified as Oxford), Oxford University, Oxford, United Kingdom, (iii) the Kenyan AIDS Vaccine Initiative (KAVI), University of Nairobi, Nairobi, Kenya, (iv) the Ugandan Virus Research Institute (UVRI), Entebbe, Uganda, (v) Contract Laboratory Services (CLS), Witwatersrand University, Johannesburg, South Africa, (vi) the National AIDS Research Institute (NARI), Pune, India, and (vii) the Tuberculosis Research Centre (TRC), Chennai, India.

Training of laboratory teams.

Prior to commencing studies, whether with an existing laboratory and staff or with a newly built laboratory and new staff, the laboratory team enters the IAVI Core training program. In brief, laboratory teams attend a 2-day basic training course on GCLPs, followed by up to 2 weeks of in-depth training in IAVI standard operating procedures (SOPs), which include the isolation, counting, and freezing of peripheral blood mononuclear cells (PBMC) and the actual ELISPOT assay procedures, among other things (28). A laboratory training manual is implemented, and after a review of the manual, each technician is required to successfully complete a written test. Further training is then provided at the on-site laboratory by an IAVI technician who covers the same procedures described in the manual, after which the site team is required to complete both a qualifying test for PBMC isolation and freezing and, separately, an IFN-γ ELISPOT qualifying test. Finally, successful laboratory teams receive a technical audit of laboratory assays every 6 months and are enrolled in an ongoing quality control program whereby proficiency in PBMC procedures is reviewed monthly and proficiency in ELISPOT assay procedures is reviewed using the negative and positive control data generated in ongoing clinical trials. The laboratories are also enrolled in a GCLP accreditation program (28).

Proficiency panel design.

Supplies for one to two proficiency panels are distributed every year. To date, sample sets for four proficiency panels have been submitted to a number of laboratories, and data from the first three panels have been evaluated (Table 1). In brief, the panel sample set consists of duplicate frozen PBMC samples that are thawed and exposed to blind stimuli consisting of a mock stimulus; a mixture of 32 influenza virus, Epstein-Barr virus (EBV), and cytomegalovirus (CMV) peptides (CEF peptides); a pool of CMV pp65 15-mer peptides; and phytohemagglutinin (PHA; Sigma, Poole, Dorset, United Kingdom). The CEF peptides are a panel of 32 8- to 10-amino-acid peptides encompassing epitopes from influenza virus, EBV, and CMV designed to cover diverse major histocompatibility complex class I genotypes; responses are detected in approximately 70% of healthy individuals in Africa, Europe, and the United States (5, 20; also data not shown). The CEF and CMV peptides were synthesized to 90% purity (Anaspec Inc., CA). Two operators each performed thawing and, for panels 1 and 2, repeat testing on two occasions, with results submitted to an independent statistician for evaluation. IAVI SOPs and proficiency panel work instructions were followed, and all procedures were performed under GCLP conditions, as described previously.

TABLE 1.

Design of proficiency panels 1 to 3

Panel	Participating laboratories	No. of PBMC samples	Plate type(s)	Counting instruments employed (no. of labs)a	Stimuli
1	Core, Oxford, CLS, KAVI, UVRI	6	1 Self-coated	Z1 Coulter Counter (4), hemocytometer (1)	Mock; HIV, CEF, and CMV peptides; PHA
2	Core, Oxford, CLS, KAVI, UVRI, NARI	3	1 Precoated, 1 self-coated	Z1 Coulter Counter (1), hemocytometer (1), Vi-CELL XR counter (3), Guava counter (1)	Mock, CEF and CMV peptides, PHA
3	Core, Oxford, CLS, KAVI, UVRI, NARI, TRC	8	1 Precoated, 1 self-coated	Z1 Coulter Counter (1), hemocytometer (1), Vi-CELL XR counter (4), Guava counter (1)	Mock, CEF and CMV peptides, PHA

^aTo obtain viable cell counts with the Z1 Coulter Counter, a hemocytometer and trypan blue staining were also used.

PBMC specimen handling.

PBMC were obtained from healthy HIV-seronegative donors through the United Kingdom National Blood Transfusion Service (Colindale) for panels 1 (six samples) and 2 (three samples) and from the South African Blood Transfusion Service under a local ethics body-approved blood-drawing protocol for panel 3 (eight samples). The 17 PBMC samples came from 17 different donors. All PBMC were isolated, counted, and frozen by following IAVI SOPs. The PBMC were isolated using Ficoll and density gradient centrifugation and counted with an automated cell counter. Three counters were used: the Z1 Coulter Counter (Beckman Coulter, United Kingdom), the Vi-CELL counter (Beckman Coulter, United Kingdom), and the Guava personal cell analysis system (Guava Technologies, Hayward, CA). Viability was assessed by using a hemocytometer and trypan blue for counts by the Z1 Coulter Counter. Samples were frozen at 10 million viable PBMC per vial in a rate-controlled freezer (Kryo model no. 560-16; Planer, United Kingdom). PBMC were stored in vapor-phase liquid nitrogen, and shipment to participating laboratories was performed using temperature-monitored cryogenic shippers (Taylor Wharton CX500; Jencons, United Kingdom). Following receipt, the PBMC continued to be stored in vapor-phase liquid nitrogen until use. Prior to use, the PBMC were thawed by being warmed in a water bath at 37°C until one small ice crystal remained and then washed in RPMI medium-20% fetal calf serum (FCS) and allowed to rest overnight in RPMI medium-20% FCS at 1.5 to 2 million PBMC/ml in an atmosphere of 5% CO₂ at 37°C. The following morning, viable cells were counted and placed onto the ELISPOT assay plates. All PBMC counts and recovery and viability results were recorded on batch records.

ELISPOT assay.

The IFN-γ ELISPOT assay was performed as described previously (22). In brief, 96-well Multiscreen HTS IP plates (MSIP4510; Millipore, United Kingdom) were incubated overnight with 10 μg/ml of clone 1-D1K mouse anti-human IFN-γ monoclonal antibody (Mabtech, Sweden). The next day, after being washed and blocked with RPMI medium-10% FCS, the PBMC were plated at 2 × 10⁵ viable PBMC per well and stimulated in quadruplicate according to the ELISPOT templates provided. Blind stimuli included a mock stimulus (RPMI medium-10% FCS with dimethyl sulfoxide [DMSO] to give a final concentration per well of 0.45% DMSO) to control for DMSO included in the peptide stimuli, CEF and CMV peptides at 1.5 μg/ml, and PHA (Sigma, Poole, Dorset, United Kingdom) at 10 μg/ml. Following overnight incubation at 37°C and 5% CO₂, the production of IFN-γ was assessed by the addition of 100 μl of 1-μg/ml filtered biotinylated clone 7-B6-1 mouse anti-human IFN-γ antibody (Mabtech, Sweden) for 2 to 4 h, the addition of ABC peroxidase-avidin-biotin complex (Vector Laboratories, Burlingame, CA) for 1 h, and development with filtered AEC (3-amino-9-ethylcarbazole) substrate solution (Vector Laboratories, Burlingame, CA) for 4 min. Plate results were read using an automated AID ELISPOT reader (AutoImmun Diagnostika, Germany). The ELISPOT data are expressed as the numbers of spot-forming cells (SFC) per million PBMC.

Statistical analysis.

Analyses of the recovery and viability results for thawed PBMC, the numbers of SFC per million PBMC, and the coefficients of variation (CV) of results were performed by the EMMES Corporation (Rockville, MD). The signed-rank test was used for paired observations (e.g., comparisons between operators), and the Kruskal-Wallis test was used for comparing multiple groups (e.g., samples within each panel). Measures of correlation are based on Spearman's correlation coefficient.

RESULTS

Recovery and viability of PBMC.

All PBMC were received at the participating laboratories at a temperature below −170°C. Two operators in each laboratory independently recorded the total number of viable cells following thawing and overnight resting, in addition to the percent viability and the calculated recovery percentage (Fig. 1). All recordings were received except those from one laboratory which did not provide viability data for panels 1 and 2. The median values (and ranges) for recovery were 81.2% (46 to 163%) in the first panel, 96.3% (60 to 155%) in the second panel, and 69.8% (35 to 170%) in the third panel.

FIG. 1.

Recovery and viability of thawed rested PBMC at participating laboratories in each proficiency panel. The recovery is indicated as the percentage of viable thawed cells recovered relative to the number of viable cells frozen. PBMC were cryopreserved in aliquots of 10 million; thus, the recovery of 6 million viable PBMC would be 60% recovery. The viability of the total PBMC fraction following thawing and resting is indicated. Each point represents a single thawed sample. Boxes represent the interquartile ranges, and horizontal lines within the boxes represent the medians. Vertical bars extend to the largest observed value within 1.5 times the interquartile range. Laboratories listed on the x axes are identified in Materials and Methods.

Comparisons between donors.

The Kruskal-Wallis test results showed variability in recovery rates between PBMC samples from different donors and were significant in panels 1 (P = 0.0005) and 3 (P = 0.0013) and borderline in panel 2 (P = 0.0657). This outcome may relate to natural variation in the propensity of cells for freezing and thawing or to the large volumes of blood handled (approximately 200 to 500 ml) in the processing blood bank samples, leading to inaccurate counts upon freezing. Data from the IAVI partner laboratory network revealed a median recovery of 70% (median viability of 92%) from 992 clinical trial samples frozen during 2006, of which the common blood draw volume was between 40 and 80 ml of blood (some of these data are shown in Table 2; also see below). No difference was seen between donors in the viability percentages (P, >0.23 for each of the three panels), which ranged from 80 to 100%, with medians of 95, 96, and 95% for panels 1, 2, and 3, respectively.

TABLE 2.

Comparison of recovery and viability results for PBMC thawed at proficiency panel sites for panel participation or at the IAVI Core Lab for assessment of immunological responses from clinical trial specimens

Specimen type and site	No. of samples	% Recovery					% Viability
		Mean	SD	Median percentile	5th Percentile	95th Percentile	Mean	SD	Median percentile	5th Percentile	95th Percentile
Proficiency panel specimens
CLS	34	77	23	75	50	120	91	3.9	91	82	95
TRC	16	65	25	64	35	121	95	2.1	96	90	98
Core lab	34	91	25	83	60	142	95	2.7	96	89	98
KAVI	34	81	28	73	50	136	91	4.4	92	81	98
NARI	22	91	21	89	58	127	98	1.6	98	96	100
Oxford	33	88	27	78	56	130	94	3.4	96	86	98
UVRI	34	76	26	69	46	123	96	1.5	96	93	99
Total	207	82	26	78	48	130	94	3.9	95	86	99
Clinical trial specimens
CLS	512	62	33	60	20	120	90	7.5	91	75	98
TRC	431	61	18	60	33	93	93	4.4	93	86	98
KAVI	426	74	21	70	40	110	92	4.4	93	85	97
NARI	233	77	30	70	40	130	91	6.1	93	77	97
Oxford	121	58	21	60	20	90	93	7.3	95	82	98
UVRI	369	65	17	60	40	90	92	6.0	93	82	97
Total	2,092	66	25	65	30	110	91	6.0	93	81	97

Comparisons between sites.

Significant differences between sites in the total recovery of viable cells (P values of 0.0005, 0.0079, and <0.0001 in panels 1, 2, and 3, respectively) and also the viability of PBMC (P values of 0.0050, 0.0003, and <0.0001, respectively), were noted (Fig. 1). This finding may relate to the difference in the counting methods employed at the sites (Table 1), since it has been reported previously that the levels of viability determined by automated counters are lower than that determined by manual counting (12).

Comparisons between operators.

To compare the observations of operators at each site, the paired differences in recovery and viability for each donor were tested using Wilcoxon's signed-rank test. Recovery data differed between operators at one lab in panel 1 (P = 0.031) and at three labs in panel 3 (P = 0.0156, 0.0078, and 0.0078). The viability percentages differed between operators at two labs in panel 1 (P = 0.031 for both) and one lab in panel 3 (P = 0.0156). With samples from only three volunteers, panel 2 had very low statistical power, and no differences between operators were observed.

Correlation with ELISPOT assay responses.

Overall, there was a statistically significant though not very strong correlation (20.5%; P = 0.0067) between the percentages of viability and the magnitudes of PHA responses in the ELISPOT assay. Among panels 1, 2, and 3, the correlations were inconsistent in magnitude and direction, being −19% (P = 0.201), 49% (P = 0.006), and 29% (P = 0.005), respectively. Similarly, there was a weak though statistically significant negative correlation (−24.4%; P = 0.0007) overall between recovery rates and magnitudes of CMV responses in the ELISPOT assay. Again, the correlations were inconsistent, being 0% (P = 1.0), −44% (P = 0.008), and −26% (P = 0.012), respectively, for panels 1, 2, and 3. No other correlations were observed.

ELISPOT assay performance.

Two operators per laboratory independently set up each ELISPOT assay by following the SOPs and template instructions for adding the blind stimuli. Responses for each donor sample in each laboratory to the different stimuli are expressed as the numbers of SFC per million PBMC and are shown in Fig. 2. Mock (i.e., background or medium) responses are the well counts, whereas CMV, CEF, and PHA responses are the well counts after the subtraction of mock response values. Only 3 (1%) of the 323 responses to the mock stimulus were above 55 SFC/10⁶ cells, indicating an excessive background count that would result in assay failure and subsequent retesting of the sample in present IAVI clinical trials. For the IAVI proficiency panels in the present study, the mean background level at the seven sites ranged from 2.3 to 13.6 SFC/10⁶ PBMC and was 7.7 ± 15.2 (standard deviation [SD]) overall. If the three specimens with >55 SFC/10⁶ PBMC are excluded, then the mean background level ± SD was 6.6 ± 8.0 SFC/10⁶ PBMC.

FIG. 2.

Laboratory ELISPOT spot-forming unit (SFU) counts for each donor PBMC sample in response to specific stimuli in panels 1 to 3. SFU counts shown for CEF and CMV peptides and PHA were determined by subtracting background values and are presented per 10⁶ PBMC. Each box plot represents all results for a single donor. A single observation represents the mean response from one lab and one operator.

The CEF and CMV stimuli allow the assessment of concordance in the magnitude of antigen-specific responses from donor PBMC across laboratories and, furthermore, for a given definition of a response classification (e.g., nonresponder or responder), the issue of whether responses from different laboratories would be classified equally.

The variation across laboratories in the responses of each donor sample to CEF peptides is shown in Fig. 2B. In general, the responses from the different labs are similar, with a range of about half a log for each sample. However, the figure also shows that any response classification (a horizontal line drawn from any point on the y axis) would result in at least one sample falling into more than one category. In IAVI clinical trials, the definition of CEF positivity is a response of >38 SFC/10⁶ cells from multiple samples evaluated over time. By this definition, the data across laboratories show 2 samples (no. 4 and 13) with only negative responses, 2 samples (no. 11 and 14) with mostly negative responses, one sample (no. 9) with mostly positive responses, and the remaining 12 samples with all positive responses. In panel 1, five of the six lowest CEF responses were analyzed by the same operator. Further investigation revealed that the operator had previously used only fresh PBMC for ELISPOT assays and had little experience with thawing PBMC. Revised instructions and training on the use of cryopreserved PBMC were provided for the subsequent panels.

The variation across laboratories in the responses of each donor sample to CMV pp65 is shown in Fig. 2C. Again, the range of responses is about half a log per sample (except for sample 4 from panel 1), showing the consistency in results across labs. If CMV-positive responses are arbitrarily defined as those with >50 SFC/10⁶ cells, then the CMV results would be categorized as six negative samples, two borderline responders (samples 4 and 9), and nine positive samples.

PHA responses are shown in Fig. 2D. The majority are >1,000 SFC/10⁶ cells, as expected, except for some panel 3 responses of ≥450 SFC/10⁶ cells and four panel 1 responses of <108 SFC/10⁶ cells. Three of the low panel 1 responses were measured by the same operator who obtained five of the six lowest CEF responses.

ELISPOT assay variation in each panel.

In IAVI trials, a typical analysis of ELISPOT data uses the mean count from replicate wells for each peptide on a plate. Since the number of replicates is generally small (usually three or four), the mean can easily be influenced by extreme values. Thus, as one of the criteria for defining positive responses, IAVI requires that the variation among the replicates be small relative to the mean. That is, the CV, defined as the SD divided by the mean, must not be greater than 70%. Since the three panels were conducted in sequence, with an interval of 6 to 9 months between each one, we wanted to investigate whether there was any change in the CV across quadruplicate wells. Typical examples are shown in Fig. 3. Clearly, as the mean spot counts increased, the CV decreased; it remained below 70% (in general ≤30%) for counts greater than 50 SFC per 10⁶ cells, and there was little variation among the three panels.

FIG. 3.

The CV between replicate wells for two of the laboratories that participated in all three panels are shown, in relation to grouped ELISPOT SFC counts for CEF and CMV peptides and PHA. The dotted line represents a CV of 70%. Counts were determined by subtracting background values and are presented per 10⁶ PBMC. Within each subgroup, there are up to three box plots, representing panels 1 to 3, respectively.

Concordance between HIV type 1 vaccine trial responses from two laboratories.

The conducting of ELISPOT assay proficiency panels is a critical tool for ensuring comparable laboratory performances both within and across networks and also for identifying and troubleshooting reasons for differences, if they exist. When dealing with actual volunteer samples during a clinical trial, laboratory personnel may be under increased pressure from the prioritization of work, the time at which samples are drawn, or late changes to scheduled visits. Table 2 shows the viability and recovery of PBMC thawed at the proficiency panel laboratories for panel participation (also see Fig. 1) and at the IAVI Core for the assessment of immunological responses from clinical trial specimens. Excellent recovery and viability of the PBMC shipped to the sites and of the PBMC cryopreserved on site and then shipped to the IAVI Core were seen. One vaccine trial in which the same PBMC samples were tested both fresh following blood drawing on site at KAVI in Nairobi and frozen after shipment to the IAVI Core in London provided ideal data with which to compare and assess performances in real time under real conditions (J. Bwayo et al., unpublished results). Trial donor PBMC samples assessed in the two laboratories exhibited concordant responses (Spearman's correlation coefficients ranged from 50 to 81% for six HIV peptide pools [P, <0.05 for all but one]), despite the use of either freshly isolated or thawed cryopreserved PBMC, providing further assurance that proficiency panel data are useful to indicate actual trial performance. These two laboratories also had concordant results in the proficiency panels (Fig. 2 and 3). Figure 4 shows typical examples of IFN-γ ELISPOT assay responses to CEF and HIV peptide pools used in the trial. There is a slight trend toward fresh samples scoring higher than frozen samples for vaccine-induced responses (those to HIV peptide pools), in contrast to CEF responses, which likely represent memory T-cell responses to previous CMV and EBV exposure. In addition, the CEF response should be entirely CD8 restricted (8- to 10-mer peptides) whereas the responses to Env, Pol, and Nef are mediated by both CD4 and CD8 T cells (15-mer peptides).

FIG. 4.

Spot-forming unit (SFU) counts, determined by subtracting background values, for samples isolated at all postvaccination time points and assessed for responses to HIV peptide pools during an HIV vaccine trial. Freshly isolated PBMC were used in Nairobi, and responses (y axes) were correlated to those of thawed frozen PBMC in the assay performed at the IAVI Core in London (x axes). Spearman's correlation coefficient is also shown and is statistically significant for all peptide pools (P is <0.05 for all except Pol B1, for which P is 0.0562). Regression lines were calculated only for positive responses (i.e., a response of >0 SFU after the subtraction of background values).

DISCUSSION

The IAVI Core and partner laboratories regularly participate in ELISPOT assay internal proficiency panels and external quality assurance (EQA) panels, with the aim of comparing their abilities to process PBMC, to evaluate the CMV and CEF ELISPOT assay responses of donor samples, and to identify and rectify any technical issues. Using standardized equipment and instructions and SOPs, with the only difference being the method of cell counting, the laboratory teams conducting the three panels analyzed to date have yielded remarkably concordant ELISPOT assay results. We have shown that IAVI partner laboratories are able, in the majority of cases, to successfully categorize samples across a range of low, medium, and high spot counts, to achieve low background values, and to correctly identify nonresponders. Cell viability and recovery results were much tighter and the minimum recoveries were much higher than those reported previously. These laboratories, with one exception, had never done ELISPOT testing prior to its implementation to support IAVI-funded clinical trials.

These types of results have not been achieved in previous proficiency panels among laboratories across organizations, either in the HIV vaccine field or in other fields such as cancer research (4, 11). The ability to determine whether a response is either positive or negative is critical for assessing vaccine immunopotency, i.e., the ability to induce an immune response (7, 8). When multiple laboratories are able to categorize samples in a consistent manner, comparative assessment and decision making for multiple vaccine candidates become easier both across and within networks. The use of multiple laboratories will accelerate the testing of vaccines and, hence, vaccine development and will furthermore provide robust ELISPOT data capable of distinguishing different response rates and magnitudes. The salient details that enabled concordant performances across seven laboratories based on three continents, an objective not achieved previously, were the standardized methods employed and the operators' familiarity with these methods (11). These included not only the methods and reagents used in the assay but also the ELISPOT assay reader model and settings, which are critical for counting spots with the same morphology (12). Methods of shipping, storage, thawing, and overnight resting of PBMC have been shown previously to affect measures of antigen sensitivity and assay performance (4, 6, 8, 17, 18, 27). Indeed, other panels have shown an improvement in sensitivity and general performance when some of these factors are standardized in successive panels (11).

In addition to the standardized methods used, another significant difference affecting performance in these panels versus other panels was the quality systems of participating laboratories. All IAVI partner laboratory personnel involved in the testing of IAVI- or other network-sponsored HIV vaccines undergo carefully integrated training, operate in a GCLP environment, and follow detailed SOPs that necessitate active interpretation of results and recording of incubation times. These aspects result in a highly controlled environment that may not be achieved in all laboratories. In support of this prospect, it is prudent to highlight that operator variation, a well-known factor in ELISPOT assay variability, was not of note in these panels (12), although consistent differences in ELISPOT assay counts from the two operators at one laboratory in panel 1 and another in panel 3 were obtained (data not shown). The background values observed in the panels described in this study were very low, with an overall mean of 6.6 SFC/10⁶ cells (determined by excluding 3 samples, of the total of 323, which had >55 SFC/10⁶ cells), whereas in other proficiency panels, numerous laboratories produced high background values that clearly affected the determination of positive responses (11). A possible explanation for high background levels may be the serum source. At IAVI, a standardized FCS is purchased in a large volume after prescreening to ensure that both low background and antigen-specific responses are supported.

Differences across laboratories with respect to viable cell counts were noted, even though the counts were obtained from the same donor PBMC isolated from the same blood draw. These differences were due most likely to the use of different automated counting equipment, some of which performed integrated viable cell counts and some of which did not. The number of laboratories per panel using each particular counter does not permit the statistical evaluation of this variable, though we note that the different viable cell counts did not in general correlate with the SFC values, indicating that perhaps differences in recovered cells were related to disparity in the numbers of cryopreserved PBMC per vial. In particular, cell counting should be standardized across laboratories and the use of automated counters should be encouraged. The cell-counting procedures for these automated counters can be validated and carried out under GCLP guidelines. We looked at the CV among replicate wells as a measure of the performances of those laboratories that participated in all three consecutive panels. No marked decrease or improvement in the CV was noted, probably due to the optimized methods in use since the first panel was conducted. This conclusion is supported by the values observed in the first panel, in which 36 (95%) of the 38 CEF and CMV ELISPOT assay counts in the range of 50 to 250 SFC/10⁶ cells had a CV below 50%, which is low for a biological assay of low magnitude (16, 21).

Upon the review of the panel 1 data, it was revealed that one operator was inexperienced at thawing frozen PBMC and encountered difficulties. As a result, improved instructions relating to these aspects, often considered routine in most laboratories, were provided. In the subsequent panels, no difficulties with thawing were observed.

Regular independent quality assurance testing is a key component of the quality systems required for any test being conducted by IAVI-sponsored GCLP guideline-compliant laboratories. Given that there is no independent EQA program such as the United Kingdom National External Quality Assessment Service CD4 program for ELISPOT assays at present, the proficiency panel provides a step toward such assurances within the IAVI program. In addition, frozen samples from all clinical trial sites are routinely shipped to the IAVI Core in London for independent testing. There remains a need across multiple programs for EQA panels.

The disappointing lack of efficacy of the Merck adenovirus-based HIV vaccine candidate led previously to a discussion concerning the utility of the IFN-γ ELISPOT assay (26). It is worth noting that the performance and robustness of this assay continue to make it a valid assay of T-cell vaccine immunopotency in early clinical development (7). This paper provides encouraging evidence that when applied using standardized methods, the ELISPOT assay is sensitive and discriminatory and that highly concordant results can be obtained across laboratories located on three different continents. This finding is encouraging for multicenter vaccine trials across disciplines and also for the possibility of obtaining comparable results in the detection and discernment of cellular immune responses of differential magnitudes.