Evaluation of the Interlaboratory Concordance in Quantification of Human Immunodeficiency Virus-Specific T Cells with a Gamma Interferon Enzyme-Linked Immunospot Assay

Abstract

The gamma interferon (IFN-γ) enzyme-linked immunospot (ELISPOT) assay is a reference method for the ex vivo monitoring of antigen-specific T cells and a primary tool for assessing clinical trials of human immunodeficiency virus (HIV) or cancer vaccines. Four experienced laboratories in Paris compared their results with this method by exchanging frozen blood samples from eight HIV-seronegative and eight HIV-seropositive subjects. Each laboratory measured the IFN-γ-producing cells specific for HIV, Epstein-Barr virus, cytomegalovirus, and influenza using the same set of peptides and the same ELISPOT reader but its own ELISPOT technique. The cutoff values for positive responses (50 or 100 spot-forming cells/10⁶ peripheral blood mononuclear cells over background) were consistent with the binomial statistic criterion. The global qualitative concordance, as assessed by the kappa index, ranged from 0.38 to 0.92, that is, moderate to excellent, and was better for non-HIV 9-mer peptide pools than for HIV 15-mer peptide pools. The interlaboratory coefficient of variation for the frequency of virus-specific T cells was 18.7% (data are expressed on a log scale). Clustering analysis of HIV-positive subjects showed qualitative agreement for ELISPOT results from all four laboratories. Overall, the good interlaboratory qualitative concordance of IFN-γ ELISPOT assays with only the peptide source and ELISPOT reader in common suggests that a qualitative comparison of interlaboratory findings is feasible. Nonetheless, a single set of standard operating procedures should be used in multicenter trials to improve standardization.

Efficient clinical trials to evaluate vaccination or immunity-based therapeutic strategies aimed at inducing human immunodeficiency virus (HIV)-specific T cells require rapid, accurate, and reliable methods for detecting and quantifying these cells. Significant progress in recent years has led to the development of two types of methods: one is based on the direct visualization of specific T cells with HLA-peptide tetrameric complexes (4, 6, 8, 15, 19, 21, 25, 29, 40, 42, 43, 57), and the other is based on ex vivo cytokine production upon specific antigen stimulation. Although multiparameter flow cytometric examination of phenotypic and functional profiles of antigen-specific cells is most likely necessary to characterize T-cell responses fully, it requires expensive equipment and substantial quantities of peripheral blood mononuclear cells (PBMC). The enzyme-linked immunospot (ELISPOT) assay (12) is a rapid, quantitative, and highly sensitive method that requires relatively few PBMC. It has been markedly improved by adding a computer-assisted microscope that simplifies readouts, it allows batch analysis of large series of samples, and it should facilitate standardization. It principally evaluates gamma interferon (IFN-γ) production, which occurs in large quantities and is considered a hallmark of the Th1 response. One recent study suggested that measurement of IFN-γ produced by T cells cannot alone define the immune correlates of protection in HIV infection (14), while others previously reported that interleukin-2 production and proliferative capacities may be important indicators of immune protection (20, 36, 56, 58). Nonetheless, IFN-γ production is currently used as the reference measurement, and ELISPOT assays for detecting IFN-γ-producing cells are widely used to quantify immune cells specific for HIV or other pathogens (1, 32, 33, 38, 41, 49, 51). ELISPOT detection of one or more cytokines (18) remains the recommended first step for screening in large clinical studies or vaccine trials. It is also widely used in pathogenesis studies, especially in cohorts (31, 39).

In such testing, most immunologists use pools of 15- to 20-mer overlapping peptides, which may stimulate both CD4 and CD8 T cells. This strategy obviates the need to define HLA alleles (5, 27, 38) and is now used to evaluate vaccine or immunity-based treatment trials of viruses (16, 23, 24) and cancers (54, 55). Alternatively, large panels of pools of optimal peptides (covering most HLA haplotypes and selected from the Los Alamos HIV molecular immunology database) can also be used; they provide results similar to those obtained with 15-mer peptide pools (A. Venet et al., unpublished data) (50). The latter approach, however, is limited to available epitopes and does not cover all HIV protein sequences or all populations of HLA molecules.

A further issue is that testing of new vaccines against HIV requires large international multicenter clinical trials and the participation of different hospitals and laboratories. These multicenter analyses mandate that participating laboratories compare and standardize their methods. Single-center studies have demonstrated that the IFN-γ ELISPOT assay is highly sensitive and reproducible (48), and a multicenter comparative study of healthy donors with influenza virus and HIV peptides found that the four laboratories involved obtained consistent IFN-γ ELISPOT results (47). More recently, 11 laboratories participating in international HIV type 1 vaccine trials underwent an external quality assurance audit to assess laboratory competence and comparability for ELISPOT IFN-γ assays that shared only their source of blood samples and their non-HIV peptides. The results showed remarkable qualitative concordance between laboratories, although the frequency of responding cells varied among laboratories (11). This study, however, did not check quantitative concordance for the specificity of the immune responses tested, since all individuals were prespecified as responders.

The French Agency for AIDS Research (ANRS), which sponsors many multicenter studies that require monitoring of HIV-specific T cells, set up a “standardization group” to analyze the interlaboratory variability of the ELISPOT IFN-γ method in four laboratories with experience participating in national or international cohort studies or clinical trials in the field of HIV or cancer (18, 22, 28, 35, 52). The aim of this study was to compare ELISPOT results from these laboratories. They used a common set of PBMC and antigens as well as the same automatic ELISPOT reader, but they all used their own ELISPOT techniques, which were developed according to similar principles. PBMC were collected from eight HIV-seropositive patients and eight HIV-seronegative donors, cryopreserved, and exchanged for testing in an ELISPOT IFN-γ assay against HIV (Gag, RT, and Nef), Epstein-Barr virus (EBV), cytomegalovirus (CMV), and influenza virus peptides. ELISPOT results were expressed as frequencies of antigen-specific cells for each subject in each laboratory, and concordance was compared between laboratories.

MATERIALS AND METHODS

Participants.

Four Paris laboratories (coded 1 to 4) participated in this study. All laboratories had much experience in cellular immunology and T-cell functional assays.

Study samples.

Unblinded blood samples were obtained from eight HIV-seropositive patients and eight HIV-seronegative donors recruited in the four participating centers (two of each from each center). All subjects gave informed consent in accordance with national laws on biomedical ethics. All HIV-positive patients had detectable plasma viral loads and CD4 counts above 400 cells/mm³ and were designated by the letters A to H. HIV-negative donors were designated by the letters I to P. PBMC were isolated in each laboratory from fresh blood by density gradient centrifugation and immediately cryopreserved locally; all laboratories used similar procedures: they froze PBMC in fetal calf serum medium containing 10% dimethyl sulfoxide and stored them in liquid nitrogen at −195°C until assayed. Frozen samples from each laboratory were then exchanged in dry ice. The transportation time was always less than 3 h. All four laboratories used the same thawing method, and all laboratories then incubated the thawed cells for 3 h at 37°C in 5% CO₂ and counted them with trypan blue exclusion to evaluate cell recovery and viability.

Peptides.

A set of 18 pools of 15-mer synthetic HIV peptides overlapping by 11 amino acids was used. Eleven pools covered the three HIV type 1 Gag proteins: three pools for p17 (positions 1 to 55, 45 to 99, and 89 to 143), five pools for p24 (positions 133 to 187, 177 to 231, 221 to 275, 265 to 319, and 309 to 363), three pools for the small Gag proteins (Smp) (positions 353 to 407, 397 to 451, and 441 to 512), four pools corresponding to polyepitopic RT regions (positions 293 to 352, 157 to 187, 177 to 216, and 4 to 52), and three pools corresponding to polyepitopic Nef regions (positions 181 to 206, 65 to 107, and 97 to 147). Pools were designated as follows: G1 to G11 for Gag, RT12 to RT15 for RT, and N16 to N18 for Nef. Peptides were synthesized by Neosystem (Strasbourg, France) and were kindly supplied by the ANRS. Also tested were three non-HIV pools of 9-mer peptides, covering the most common HLA-restricted epitopes from EBV (10 peptides), CMV (three peptides) (both synthesized by Epytop, Nîmes, France), and influenza virus (three peptides; synthesized by Neosystem) (Table 1).

TABLE 1.

EBV, CMV, and influenza A virus peptide sequences and HLA restriction elements

Virus	Protein: amino acid position	Peptide sequence	Allele(s)
EBV	EBNA 3 A: 379-387	RPPIFIRRL	B7
	EBNA 3 A: 603-611	RLRAEAQVK	A3
	EBNA 3 B: 416-424	IVTDFSVIK	A11
	EBNA 3 C: 163-171	EGGVGWRHW	B44
	EBNA 3 C: 881-891	QPRAPIRPI	B7
	EBNA 3 A: 325-333	FRLGRAYGL	B8
	EBNA 3 A: 458-466	YPLHEQHGM	B35
	LMP-2: 426-434	CLGGLLTMV	A2
	BMLF-1: 280-288	GLCTLVAML	A2
	BZLF-1: 190-197	RAKFKQLL	B8
CMV	pp65: 495-503	NLVPMVATV	A2
	GB: 619-628	FIAGNSAYEYV	A2
	IE1: 378-389	SDEEEAIVAYTL	B18
Influenza A virus	MP: 58-66	GILGFVFTL	A2
	NP: 265-273	ILRGSVAHK	A3, A28, A68
	NP: 338-346	FEDLRVLSF	B44

High-performance liquid chromatography profiles showed that all peptides were more than 80% pure. Lyophilized peptides were distributed to each laboratory, diluted to 0.5 mg/ml in water-10% dimethyl sulfoxide, and stored at −80°C according to the manufacturer's instructions.

IFN-γ ELISPOT assays.

Participants were asked to use their own laboratory IFN-γ ELISPOT protocols, which were all developed according to the same principle. Table 2 summarizes step-by-step details of the procedures. The 96-well plates (Millipore, France) were coated for 18 h at 4°C with capture mouse anti-human IFN-γ monoclonal antibody at 1 μg/ml. PBMC were added in triplicate wells at 1 × 10⁵ cells per well in the presence of pools of peptides (used at a final concentration of 2 μg/ml) or positive controls (Table 2) or in the presence of culture medium (six wells for the latter) as negative controls. Plates were incubated for 20 h at 37°C in 5% CO₂. After washing, the biotinylated mouse anti-IFN-γ monoclonal antibody was added, followed by conjugate and chromogen substrate and coloration development at room temperature. Frequencies of antigen-specific spot-forming cells (SFC) were measured in each laboratory with a similar automated microscope (Zeiss, Le Pecq, France). Results were expressed as SFC per million PBMC and were calculated for each pool of peptides as follows: 10 × (mean SFC/10⁵ cells from three antigen-stimulated wells − mean SFC/10⁵ cells from six unstimulated wells).

TABLE 2.

ELISPOT protocols^a

Step	Protocol for laboratory:
	1	2	3	4
1 (plate preparation)	Nitrocellulose plates ready to use	PVD bottom plates, incubate 10 min with EtOH 70% and 3 washings with 100 μl of sterile 1× PBS	Nitrocellulose plates ready to use	PVD bottom plates, 3 washings with 100 μl of sterile 1× PBS
2 (anti-IFN-γ coating)	Capture mouse anti-human IFN-γ monoclonal antibody (1-D1K; Mabtech, Sweden)	Capture mouse anti-human IFN-γ monoclonal antibody (B-B1; Diaclone, Besançon, France).	Capture mouse anti-human IFN-γ monoclonal antibody (1-D1K; Mabtech, Sweden)	Capture mouse anti-human IFN-γ monoclonal antibody (B-B1; Diaclone, Besançon, France).
3 (saturation)	3 washes with complete medium + 200 μl per well of complete medium for 2 h at 37°C	3 washes with PBS + 100 μl per well of 1× PBS + 2% dry milk for 2 h at 37°C	3 washes with complete medium + 200 μl per well of complete medium for 2 h at 37°C	3 washes with sterile PBS + 100 μl per well of complete medium for 30 min at 37°C
4 (distribution of cells and pools of peptides)	Cells + pools of peptides or positive control, PHA (0.5 μg/ml) (Abbott, Rungis, France)	Cells + pools of peptides or positive control, PMA (50 ng/ml; Sigma-Aldrich, France) + ionomycin (500 ng/ml; Sigma-Aldrich, France)	Cells + pools of peptides or positive controls, 1,000 cells per well + PMA (50 ng/ml; Sigma-Aldrich, France) + ionomycin (500 ng/ml; Sigma-Aldrich, France)	Cells + pools of peptides or positive control, PHA (0.5 μg/ml) (Abbott, Rungis, France)
5 (washings)	6 washings with 1× PBS + 0.05% Tween	6 washings with 1× PBS + 0.05% Tween	6 washings with 1× PBS + 0.05% Tween	9 washings with 1× PBS and PBS + 0.05% Tween
6 (detection)	1 μg/ml detection mouse biotinylated monoclonal anti-human IFN-γ (7-B6-1; Mabtech, Sweden) for 1 h 30 min at 37°C or overnight at 4°C	1 μg/ml detection mouse biotinylated monoclonal anti-human IFN-γ (B-G1; Diaclone, France) for 1 h 30 min at 37°C	1 μg/ml detection mouse biotinylated monoclonal anti-human IFN-γ (7-B6-1; Mabtech, Sweden) for 1 h 30 min at 37°C or overnight at 4°C	0.2 μg/ml detection mouse biotinylated monoclonal anti-human IFN-γ (B-G1; Diaclone, France) for 4 h at 37°C or overnight at 4°C
7 (washings)	6 washings with 1× PBS + 0.05% Tween	6 washings with 1× PBS + 0.05% Tween	6 washings with 1× PBS + 0.05% Tween	3 washes with 1× PBS
8 (visualization 1)	Alkaline phosphatase-labeled Extravidin (1/6,000; Sigma-Aldrich, France) for 1 h at room temp	Streptavidin-alkaline phosphatase conjugate (Diaclone, France) at a 1/1,000 dilution for 1 h at 37°C	Phosphatase alkaline E-2636 (1/5,000; Sigma-Aldrich, France) for 1 h at room temp	Streptavidin-alkaline phosphatase conjugate (RPN, 1234; Amersham, France) at a 1/1,000 dilution for 1 h at 37°C
9 (washings)	6 washings with 1× PBS + 0.05% Tween	3 washes with 1× PBS + 0.05% Tween	6 washes with 1× PBS + 0.05% Tween	3 washes with 1× PBS
10 (visualization 2)	100 μl/well of chromogenic alkaline phosphatase substrate (Bio-Rad, France)	100 μl/well ready-to-use NBT/BCIP (Diaclone, France)	100 μl/well ready-to-use NBT/BCIP (KPL, Gaithersburg, Md.)	50 μl/well ready-to-use NBT/BCIP (Sigma-Aldrich, St. Quentin, France)
11 (stop reaction)	After approx 10 min	After approx 10 min	After approx 30 min	Αfter approx 10 min
12 (washings)	When spots are clearly visible, plates are washed with distilled water and dried at room temp (°C)	When spots are clearly visible, plates are washed with distilled water and dried at room temp (°C)	When spots are clearly visible, plates are washed with distilled water and dried at room temp (°C)	When spots are clearly visible, plates are washed with distilled water and dried at room temp (°C)
13 (plate counting)	Plate's membranes are transferred onto a plastic support before counting	Plate's membranes are transferred onto a plastic support before count	Plate's membranes are transferred onto a plastic support before count	Plates are counted directly without transfer

^aPBS, phosphate-buffered saline; PVD, polyvinylidene difluoride; EtOH, dimethyl sulfoxide; NBT, Nitro Blue Tetrazolium; BCIP, 5-bromo-4-chloro-3-indolylphosphate.

Statistical methods.

Descriptive statistics including medians, interquartile ranges (IQRs), ranges, and proportions were calculated and plotted for negative control values (background) and for the HIV-seronegative donors' responses to HIV pools to determine a common cutoff value for positive responses. We used a binomial statistic criterion to confirm our definition of a positive response (45).

Kappa coefficients (range, 0 to 1) measured the concordance between laboratories for positive responses to different pools. This coefficient, a measurement of agreement for binary data, expresses the relative agreement beyond that expected by chance alone (kappa = 0). Poor agreement is usually defined as a kappa coefficient of <0.2, fair agreement is defined as a kappa coefficient of 0.2 to 0.4, moderate agreement is defined as a kappa coefficient of 0.4 to 0.6, substantial agreement is defined as a kappa coefficient of 0.6 to 0.8, and “almost perfect” is defined as a kappa coefficient of 0.8 to 1 (37). Comparative analysis of the different laboratories' values for individual pools used the Spearman correlation, plotted the difference between values and average values in two laboratories (after log₁₀ transformation), and calculated the mean difference plus or minus 2 standard deviations (SD) of the differences. These values defined the limits of agreement between two laboratories (7).

Inter- and intralaboratory coefficients of variation (CVs), evaluated on log₁₀-transformed SFC/10⁶ PBMC for data for individual pools, used statistical mixed models (SAS mixed-model software; SAS Institute Inc., Cary, NC). CVs were also calculated for classification of all individual pools into prespecified categories of response level (geometric mean of the values from the four laboratories equals 0 to 49, 50 to 99, 100 to 499 or ≥500 SFC/10⁶ PBMC).

Finally, we used hierarchical cluster analysis to group the quantified HIV-specific responses from the HIV-positive subjects. We used average linkage methods and Spearman correlation coefficients to measure the similarity of data for each subject from the different laboratories. SAS 8.1 software (SAS Institute, Cary, NC) was used for the statistical analysis.

RESULTS

To compare interlaboratory variability in the quantification of HIV-specific CD8⁺ T cells by an ELISPOT IFN-γ assay, 16 samples from eight HIV-positive and eight HIV-negative individuals were analyzed by four laboratories. Samples and peptide sources were the same, but each laboratory used its own ELISPOT technique. Tables 3 and 4 summarize the subjects, peptide pools, and assays (performed and interpreted).

TABLE 3.

Numbers of ELISPOT assays performed and interpreted for HIV-seropositive and HIV-seronegative subjects in all four laboratories for background and non-HIV and HIV pools^a

Lab	No. of ELISPOT assays
	HIV-positive subjects (n = 8)			HIV-negative subjects (n = 8)
	Background (n)	3 Non- HIV pools (n × 3)	18 HIV pools (n × 18)	Background (n)	3 Non- HIV pools (n × 3)	18 HIV pools (n × 18)
1	7	21	126	8	24	144
2	6	18	108	8	24	144
3	8	24	144	8	24	144
4	8	24	144	8	24	144
All labs	29	87	522	32	96	576

^aAssays for HIV-positive subject G in Lab 1 and HIV-positive subjects F and H in Lab 2 were not considered interpretable.

TABLE 4.

ELISPOT responses of HIV-negative subjects to 18 HIV pools

Lab	Median SFC/106 PBMC (IQR)	No. (%) of ELISPOT assays with indicated SFC/106 PBMC
		<50	50-100	>100
1	5 (0-15)	143 (99.3)	1 (0.7)	0 (0)
2	3 (0-13)	132 (92)	6 (4)	6 (4)
3	2 (0-8)	144 (100)	0 (0)	0 (0)
4	0 (0-3)	144 (100)	0 (0)	0 (0)
All labs	2 (0-10)	563 (98)	7 (1)	6 (1)

Overall results showed a mean of 90% viability, with an interlaboratory standard deviation of 5.5% and an interlaboratory CV of 6.1%.

Each laboratory's background ELISPOT results were assessed by measuring the number of SFC in the control wells, where cells were incubated with medium alone. We excluded the results of three analyses (three different HIV-positive subjects) that could not be interpreted from these comparisons: one had excessive background attributable to culture medium (2,200 SFC/10⁶ PBMC for subject H in laboratory 2 [Lab 2]), another had negative results with the positive control (30 SFC/10⁶ PBMC for subject F in Lab 2), and the third had a technical problem with the plate (subject G in Lab 1). All interpretable ELISPOT assays showed positive responses (>1,000 SFC/10⁶ PBMC to saturation levels) with phytohemagglutinin (PHA) or phorbol myristate acetate (PMA). The median background for the 61 interpretable values (29 from HIV-positive subjects and 32 from HIV-negative subjects) (Tables 3 and 4) was 7 SFC/10⁶ PBMC, with an IQR of 3 to 17 and a range of 0 to 90. Figure 1a depicts the distribution of background and reveals two different levels, one for Labs 1 and 3 and another for Labs 2 and 4 (P = 0.005; Friedman test).

FIG. 1.

Plots of HIV-positive and HIV-negative subjects across laboratories. (a) SFC/10⁶ PBMC for negative control values (background in medium alone). Horizontal bars indicate medians. (b) SFC/10⁶ PBMC (background deduced) for HIV-negative donors and HIV pools.

Cutoff values for positive responses.

PBMC from the eight HIV-negative donors against 18 HIV peptide pools were used to determine the cutoff values for positive responses to HIV. The overall median SFC/10⁶ PBMC was 2 over the background (IQR, 0 to 10). Figure 1b and Table 4 show their distribution across laboratories: all were below 50 SFC/10⁶ PBMC for Labs 3 and 4, and all except one (subject O for a Nef peptide pool) were below 100 in Lab 1. Lab 2, on the other hand, had six (4%) responses between 100 and 200 SFC/10⁶ PBMC (median, 124) and six more (4%) responses between 50 and 100 SFC/10⁶ PBMC, all concerning three donors (subject O, six Gag, three RT, and one Nef peptide pool; subject J, one Gag peptide pool; subject L, one Gag peptide pool). Therefore, we set the cutoff value for positive responses in this study at 50 SFC/10⁶ PBMC over background and found that only 13/576 responses to HIV peptide pools from HIV-negative subjects were positive, for an overall false-positive percentage of 2% (95% confidence interval, 1% to 4%) (Tables 3 and 4).

To test the validity of this experimental definition of a positive response to HIV peptides, we compared the cutoff method to the binomial criterion (45) for all responses to 18 HIV peptide pools, i.e., for 1,098 (522 for HIV-positive and 576 for HIV-negative subjects) individual pool results. Figure 2 presents the positive criterion curve for the binomial criterion, and positive responses to the HIV peptide pools appear above the curve. When a cutoff value of 50 SFC/10⁶ PBMC (background deducted) was used, the binomial criterion yielded the same result as the experimental criterion in 97% of cases. Discrepancies were defined as results that were negative with the binomial criterion but above the cutoff value of 50; they were due to a number of SFC/10⁶ PBMC that was very close to the threshold and included in the confidence interval (50 to 78%). HIV-negative subjects with experimental false-positive results also had false-positive responses with the statistical criterion. A cutoff value of 100 SFC yielded similar results in 98% of the 1,098 pools tested, with discrepancies in the interval of 70 to 98 SFC/10⁶ PBMC. The statistical criterion is thus consistent with the experimental cutoff method.

FIG. 2.

Statistical binomial criterion analysis of HIV-positive (HIV+) (n = 8) and HIV-negative (HIV−) (n = 8) subjects for 18 HIV pools. The line is the positive criterion curve with a false-positive rate of 5%, and a peptide pool is positive if the number of SFC (spot-forming cell raw values [background is not deducted]) attributable to the pool is above the curve (45).

Description of the virus-specific responses.

Figure 3 (a and b) presents the individual responses from each laboratory for non-HIV and HIV pools, respectively. There were 87 assays against non-HIV pools and 522 assays against HIV pools for HIV-positive samples and 96 assays against non-HIV pools for HIV-negative samples (Table 3). Overall, with a cutoff value of 50 SFC/10⁶ PBMC, we observed 30.1% (55/183) positive responses from all subjects (HIV positive and negative) against non-HIV pools (Fig. 3a) and 35.4% (185/522) positive response from HIV-positive subjects against HIV pools (Fig. 3b).

FIG. 3.

Plots of HIV-positive and HIV-negative subjects across laboratories. (a) SFC/10⁶ PBMC (background deduced) for non-HIV pools and all subjects. (b) SFC/10⁶ PBMC (background deduced) for HIV pools and HIV-positive subjects.

The HIV peptide pools most frequently recognized by HIV-positive subjects were Nef and Gag: N18 in 79% (23/29) of the assays, N17 in 59% (17/29) of the assays, G6 in 17/29 assays, and G4 in 16/29 assays. Overall, the median number of pools recognized per patient was 2 (IQR, 1 to 2) of 3 for Nef, 3 (IQR, 2 to 5) of 11 for Gag, and 1 (IQR, 0 to 2) of 4 for RT pools (Fig. 3b). The number of HIV peptide pools recognized per HIV-positive subject defined the breadth of response: the median was 6.5 (IQR, 3.5 to 8.5; range, 1 to 13) of the 18 pools tested.

The overall magnitude of the HIV-specific response for each HIV-positive subject was defined as the sum of the positive responses to individual pools (above 50 SFC/10⁶ PBMC, with the background deducted), with the results below 50 considered to be 0. As Fig. 4a shows, the median magnitude was 2,201 SFC/10⁶ PBMC or 3.3 log₁₀ SFC/10⁶ PBMC (IQR, 893 to 4,028; range, 206 to 10,076). Similarly, the magnitude of the response to the three non-HIV pools (sum of the positive responses to the CMV, EBV, and influenza virus pools) was 203 SFC/10⁶ PBMC or 2.3 log₁₀ SFC/10⁶ PBMC (range, 0 to 2,676) for the HIV-negative subjects and 405 SFC/10⁶ PBMC or 2.6 log₁₀ SFC/10⁶ PBMC (range, 0 to 1,946) for the HIV-positive subjects (Fig. 4b).

FIG. 4.

Magnitude of specific responses (log₁₀ total SFC) in each laboratory. (a) HIV pool responses by HIV-positive subjects. (b) Non-HIV pool responses by HIV-negative and HIV-positive subjects. For nonresponders, values are 0 on the vertical axis. FLU, influenza A virus.

Interlaboratory comparison. (i) Qualitative agreement.

We then assessed the qualitative agreement between laboratories with kappa coefficients, beginning by comparing the 39 responses of all subjects to the three non-HIV pools. All four laboratories agreed for 33/39 assays (85%). Pairwise comparisons found 85% to 97% agreement. Overall, the kappa coefficient for the four laboratories was 0.80 (P < 0.001), and the pairwise kappa values ranged from 0.69 to 0.92 (Table 5).

TABLE 5.

Concordance between laboratories for responses to individual peptide pools

Concordance	Kappa coefficienta						Spearman correlationb
	Common HIV-positive (n = 5) and HIV-negative (n = 8) samples against 3 non-HIV poolsc			Common HIV-positive (n = 5) samples against 18 HIV poolsd			Common HIV-positive (n = 5) and HIV-negative (n = 8) samples against 3 non-HIV poolsc			Common HIV-positive (n = 5) samples against 18 HIV poolsd
	Lab 1	Lab 2	Lab 3	Lab 1	Lab 2	Lab 3	Lab 1	Lab 2	Lab 3	Lab 1	Lab 2	Lab 3
Qualitative
Lab 4	0.87	0.80	0.77	0.59	0.54	0.54
Lab 3	0.76	0.69		0.38	0.38
Lab 2	0.92			0.68
Quantitative
Lab 4							0.67	0.63	0.83	0.70	0.63	0.83
Lab 3							0.52	0.72		0.76	0.67
Lab 2							0.53			0.52

^aKappa coefficients for positive/nonpositive responses between pairs of laboratories (P < 0.001 for all).

^bSpearman correlation coefficients between laboratories for quantitative ELISPOT responses to individual peptide pools (SFC/10⁶ PBMC) (P < 0.001 for all).

^cNumber of responses, 13 × 3 = 39.

^dNumber of responses, 5 × 18 = 90.

We then compared the 90 responses to the 18 HIV pools by PBMC from the five HIV-positive subjects tested by all four laboratories. They agreed completely in 51/90 assays (57%), with 17 all-positive, 34 all-negative, and 39 discordant results. Pairwise comparisons showed 67% to 87% concordance. Overall, as Table 5 shows, the kappa coefficient was 0.50 (P < 0.001), and the pairwise kappa values ranged from 0.38 to 0.68.

(ii) Quantitative agreement.

To assess the quantitative agreement between laboratories, we considered responses to all individual pools for all the subjects in common (that is, eight HIV negative and five HIV positive). All Spearman correlation coefficients were significant (P < 0.05 after Bonferroni adjustment). They ranged from 0.53 to 0.83 for non-HIV pools for all common subjects and from 0.52 to 0.83 for HIV pools for the five HIV-positive subjects in common (Table 5).

Figure 5 plots the differences between the SFC values and their means and shows the limits of agreement between the individual SFC values obtained in pairs of laboratories (mean difference of log₁₀ SFC/10⁶ PBMC ± 2 SD). The mean difference ranged from −0.47 (Lab 2 versus Lab 3) to +0.37 (Lab 3 versus Lab 4) on the log scale. As the figure shows, the differences were larger for values below the cutoff and were essentially stable for positive values over the measurement range. However, we could not calculate the limits of agreement when we used only the values above a cutoff of 100.

FIG. 5.

Bland-Altman plots of differences in SFC/10⁶ PBMC against the average of the two results in two laboratories. The solid lines represent the mean differences, and the dashed lines are 2 SD from the mean (95% limits of agreement).

We then evaluated the interlaboratory CV of the positive responses, again beginning with the individual responses to non-HIV pools (n = 3) for PBMC from the common subjects (eight HIV-negative and five HIV-positive subjects): the interlaboratory CV was 13.1%, with a geometric mean of 124 SFC/10⁶ PBMC (data not shown). The interlaboratory CV for the same analyses restricted to the HIV-negative subjects was 9.1%, with a geometric mean of 198 SFC/10⁶ PBMC (data not shown). The interlaboratory CV of the positive responses from HIV-positive subjects to each of the HIV pools was 18.7%, with a geometric mean of 150 SFC/10⁶ PBMC. When we analyzed the positive responses of HIV-positive subjects to all pools (HIV and non-HIV), the interlaboratory CV was 18.1%, and the geometric mean was 154 SFC/10⁶ PBMC (data not shown). We also considered the total magnitude of HIV-specific responses to the 18 HIV pools (sum of specific responses for each pool for each individual) in the HIV-positive subjects and found an interlaboratory CV of 11.5% and a geometric mean of 2,189 SFC/10⁶ PBMC (data not shown). Finally, we looked at the interlaboratory CVs according to the geometric mean for each HIV-positive subject and each HIV pool, as shown in Fig. 6: variability differed, depending on the level of response. The CVs estimated in each range of geometric means (0 to 49, 50 to 99, 100 to 499, and above 500 SFC/10⁶ PBMC) decreased as the response level rose, ranging from 77% for values below 50 SFC to 9.6% for values above 500 SFC.

FIG. 6.

Interlaboratory CV versus mean log₁₀ SFC across laboratories for each HIV pool and HIV-positive subject (n = 144) and estimated CVs (—) when geometric means are calculated as 0 to 49, 50 to 99, 100 to 499, and above 500 SFC/10⁶ PBMC.

(iii) Cluster analysis.

Although agreement between quantitative results is critical, the qualitative interpretations of IFN-γ ELISPOT results tested simultaneously in several laboratories must also be comparable. Accordingly, to examine the similarities between laboratories for assays of HIV-positive samples, we conducted a hierarchical cluster analysis of the data for HIV and non-HIV pools (SFC/10⁶ PBMC above background, 0 for nonpositive pools). The results are presented as a tree diagram in Fig. 7. The relative ranking of the results for a given patient and a given peptide pool did not differ between laboratories, and with one exception (one assay for subject H in Lab 1), the same subject was always sorted into the same cluster. Further evidence of the good qualitative agreement between laboratories is that clustering was essentially the same when we included SFC below the cutoff of 50 (data not shown).

FIG. 7.

Cluster analysis of HIV-positive subjects (Spearman correlation similarity with average linkage). Similar cases (subject by laboratory) are joined in the diagram. Five cases were excluded: subject F in Lab 1 without any positive pool and in Lab 2 (negative results with positive control [30 SFC/10⁶ PBMC]), subject B in Lab 2 for missing pools (influenza virus, G6, RT13, and RT14), subject H in Lab 2 (excessive background attributable to culture medium [2,200 SFC/10⁶ PBMC]), and subject G in Lab 1 (technical problem with the plate).

DISCUSSION

Our comparative study of ELISPOT IFN-γ assays found significant qualitative and quantitative agreement between four experienced Paris laboratories. The consistency of the cryopreservation and thawing procedures and the satisfactory cell viability analysis results for all samples, including those that were HIV positive, showed that experienced laboratories can exchange frozen PBMC. This information is helpful for planning logistics in multicenter trials. All median background values were below 50 SFC/10⁶ PBMC, and only one background value exceeded 50 SFC/10⁶ PBMC. These findings provide further evidence of methodological quality. The distribution of background levels across laboratories, however, revealed two distinct levels: one level in Labs 1 and 3 that was higher than that in Labs 2 and 4.

Thresholds for positivity are usually defined arbitrarily. Our analysis of HIV-specific T-cell frequencies from HIV-negative subjects helped us to define an experimental threshold for positivity. As expected, the frequency of false-positive responses decreased when the threshold increased, and the binomial criterion (45) helped us validate both experimentally defined cutoff values. The threshold of 50 SFC/10⁶ PBMC is consistent with the most common cutoff values used to determine a positive specific HIV or hepatitis C virus response in ELISPOT IFN-γ assays, usually 50 to 100 SFC/10⁶ PBMC (26, 34, 45). The participating laboratories now use this threshold (50 SFC/10⁶ PBMC) to monitor responses to HIV, hepatitis C virus, and vaccinia virus (2, 3, 9, 10, 13, 46, 50).

These results suggest that cutoff values may be affected by reagents and methods. A cutoff value of 100 SFC/10⁶ PBMC might thus be preferred in multicenter trials to limit qualitative variation when standard operating procedures (SOPs) are not used (28, 30). In this comparative interlaboratory study, the cutoff value we chose, 50 SFC/10⁶ PBMC over background, fit all the laboratories' values for positive responses.

The qualitative agreement between responses to non-HIV pools by PBMC from all HIV-positive and HIV-negative samples, assessed by the kappa coefficient, was classified as almost perfect (37), and quantitative analyses showed “moderate to substantial concordance.” Complete agreement between the four laboratories was more frequent for non-HIV than for HIV peptide pools, and the former provided better qualitative and quantitative concordance for both HIV-positive and HIV-negative samples. Concordance was only “moderate to substantial” for the pairwise comparisons of HIV pools and HIV-positive subjects. The cluster analysis nonetheless showed that agreement in our study was good: HIV-positive subjects were always sorted into the same cluster, even though quantitative individual responses for each of these subjects differed somewhat between laboratories.

Individual responses to both HIV and non-HIV pools in HIV-positive samples and to non-HIV pools in HIV-negative samples were strongly and significantly correlated in all laboratories. Again, the comparison between pairs of laboratories showed some differences. For instance, values from Lab 3 and Lab 4 were better correlated than those from Lab 2 and Lab 1. Note that the pairwise limits of agreement between individual SFC values were larger for values below the cutoff of 50 SFC/10⁶ PBMC than for values above this cutoff. This finding minimizes the consequences of variability.

Overall, the interlaboratory CV was better with HIV-positive and HIV-negative samples in non-HIV peptide pools (9-mer peptides) (13.1%) than with HIV-positive samples in HIV peptide pools (15-mer peptides) (18.7%). These CVs are nonetheless better than those found in a previous multicenter study of lymphocyte proliferation assays (20%) (17). These differences may be explained by the greater accuracy of 9-mer peptides than 15-mer peptides for testing of specific CD8 responses.

The differences observed between the four laboratories may be attributable to the technical differences listed in Table 2. We note, for example, that Labs 2 and 4 used polyvinylidene difluoride plates and the same antibody source, while both Labs 1 and 3 used nitrocellulose plates and the same antibody source, one different from that used by Labs 2 and 4. The ELISPOT plate material has been shown to cause some differences (44). Other variables that may influence the assay include the culture medium used for cell preparation, reagents (antibodies, buffer, and system), incubation time, and number of washings. The relative contribution of each of these characteristics should be further evaluated to improve the results. In contrast, differences are unlikely to be due to the reading procedure, since each laboratory obtained similar data when they exchanged plates. The automatic ELISPOT reader system was identical, but each laboratory used its own settings.

Evaluation of vaccination or immunity-based therapeutic strategies in multicenter clinical trials requires excellent quantitative concordance between participating laboratories. Our results make it clear that a single SOP should be used in such trials and that external quality assurance must precede immune monitoring. For example, one study reported that implementation of good clinical laboratory practices improved the CV for CD4 counts in different laboratories worldwide from 15% to 5% (53).

In conclusion, our results provide a solid basis for the standardization of ELISPOT assays of HIV-positive samples and show that consistent results can be obtained with the method in experienced laboratories. Despite overall good consistency, the variability still observed between pairs of laboratories points to the need to apply a uniform set of SOPs to provide even better quantitative agreement, which is essential for the evaluation of vaccines or immunity-based therapeutic strategies.