Short-Term Reproducibility of a Commercial Interferon Gamma Release Assay

Abstract

Interferon gamma release assays (IGRAs) have been shown to be sensitive and highly specific for the detection of immune memory against Mycobacterium tuberculosis. Little is known about the reproducibility and within-person variability of these assays. Various aspects of short-term reproducibility of a commercial IGRA, the QuantiFERON-TB Gold In-Tube (QFT-IT) assay, were assessed. The QFT-IT assay was performed twice within 3 days in 27 health care workers in Cape Town, South Africa. Two sets of tests were performed by different operators on day 1, and one set was performed on day 3. Aspects such as interoperator, intraoperator, day-to-day variability, and test-retest variability as well as different the storage methods of plasma were investigated. Seventeen of 27 (63%) of participants had at least one positive QFT-IT text; six had discordant results. The agreement of all aspects studied was high, with kappa values between 0.82 and 1.00 for dichotomous measures, and interclass correlations (ICC) of 0.809 to 0.965 were observed for continuous gamma interferon (IFN-γ) measures. The variability of the magnitude of response was highest comparing measures obtained from individuals on different days (ICC of 0.809). The magnitude of the IFN-γ responses between assays performed for individual participants was variable, with ranges from 0.03 to 11 IU/ml, resulting is discordant results for five participants. The results indicate that the QFT-IT assay is a robust and highly reproducible assay. Considerable intraindividual variability occurs in the magnitude of IFN-γ responses, which may influence the interpretation of serial measures.

Commercial T-cell-based interferon gamma release assays (IGRAs) have been shown to be sensitive and highly specific for the detection of Mycobacterium tuberculosis infection (19). IGRAs have recently been incorporated into international guidelines for tuberculosis (TB) screening and diagnosis in several countries including in the United States, Canada, United Kingdom, Germany, and France, either as a confirmatory test for a positive tuberculin skin test (TST) or as replacement for the TST (2, 4, 8, 13, 15). It has further been suggested that IGRAs could be used for the serial measurement of gamma interferon (IFN-γ) responses to detect M. tuberculosis infection in high-risk populations such as health care workers and as a tool to monitor the response to treatment in individuals with active TB disease (measured through a decline in IFN-γ responses) (1, 3, 6, 10, 13, 17).

Despite the increased use and availability of IGRAs, there are limited published data regarding the reproducibility of the two currently commercial assays, the QuantiFERON-TB Gold In-Tube (QFT-IT; Cellestis, Australia) and the T-SPOT.TB (Oxford Immunotec, United Kingdom) tests. In two recent publications, test-retest variability and within-person reproducibility of the QFT-IT assay were assessed over a period of 12 days and 3 months, respectively, focusing on test agreement, conversions, and reversions (20, 22). Little is known about the short-term within-person variation in T-cell IFN-γ responses. These could be nonspecific but may be important in the interpretation of serial measures and the definition of test conversion and reversion, especially if the risk of intercurrent M. tuberculosis exposure is low (14, 18).

In addition to the need for data guiding the interpretation of serial QFT-IT measures, there are additional aspects of the QFT-IT test that require investigation. Although testing of samples by enzyme-linked immunosorbent assay (ELISA) is traditionally performed in duplicate or triplicate, the manufacturers of the QFT-IT assay recommend testing of a single sample per stimulation condition, and limited data are provided regarding test-retest variability. The robustness of these test measures could also be influenced by additional laboratory factors including interoperator and intraoperator variability and storage practices. We conducted a study to investigate the short-term reproducibility of the QFT-IT assay for the detection of M. tuberculosis infection.

MATERIALS AND METHODS

This study was conducted among TB health care and laboratory workers in Cape Town, South Africa. A questionnaire documenting current symptoms suggestive of TB, previous antituberculosis therapy, and previous Mantoux TST results was administered. Human immunodeficiency virus infection status was not assessed. Phlebotomy was performed on all participants at two time points as follows: on day 1, 6 ml of venous blood was taken for two sets of QFT-IT tests; 2 days later, 3 ml was obtained for one set of QFT-IT (day 3) tests. A TST was performed using 2 tuberculin units of purified protein derivative RT-23 (Statens Serum Institute, Copenhagen, Denmark) in individuals with any recorded previous negative TST on day 3, following completion of phlebotomy. TST results were read after 48 h using the ball-point technique; an induration of ≥10 mm was classified as positive.

An overview of the study design is shown in Fig. 1. In brief, the following aspects that could influence variability were compared: the interoperator (two operators performing a test on the same day on aliquots of the same sample), intraoperator (one operator performing two tests on the same day on two different samples sets), day-to-day variability (two tests performed 2 days apart by the same operator), and test-retest variability (each test performed in duplicate) as well as test-specific characteristics for different storage methods of plasma.

FIG. 1.

Overview of study design, sample flow, and main comparisons for QFT-IT assay. Each QFT-IT set contains a nil control, mitogen, and antigens (ESAT-6, CFP-10, and TB 7.7).

All analyses were completed in an immunology research laboratory where staff were trained and accredited by the manufacturers. All QFT-IT test kits were from the same batch (lot 0594-50232). A standard protocol was followed to ensure standardization of all laboratory procedures. Blood was collected in QFT-IT test tubes and transported within 1 h to the laboratory and immediately processed. All tests were performed by two experienced and trained laboratory technicians.

The QFT-IT test was performed strictly according to the manufacturer's guidelines. In short, the test comprises a nil tube (i.e., a tube without antigens or mitogen), a mitogen tube (i.e., a tube coated with the mitogen phytohemagglutinin), and an antigen tube (i.e., a tube coated with ESAT-6, CFP-10, and TB7.7). Each tube was filled to contain between 0.8 ml and 1.2 ml. Tubes contained an undisclosed anticoagulant. As recommended, QFT-IT tubes were shaken 10 times directly after phlebotomy was performed and again before incubation. Incubation times for QFT-IT tubes were standardized to 20 h at 37°C and 5% CO₂. Plasma was harvested, split into aliquots, and loaded onto QuantiFERON-TB Gold ELISA plates. According to the protocol, QFT ELISAs were performed either immediately (fresh ELISA) or after storage for 4 weeks at either 4 or −80°C, following the manufacturer's protocol and using a QuantiFERON-TB Gold ELISA kit.

QFT-IT test interpretation.

For analysis of QFT-IT test results, the software provided by the manufacturer was used (QuantiFERON In-Tube, version 2.50). The software reported results automatically as positive (if the nil tube value is ≤8.0 IU/ml and the reaction to TB antigens minus the nil tube control is ≥0.35 IU/ml and >25% of the nil value), negative (if the nil tube value is ≤8.0 IU/ml and mitogen tube value minus nil tube value is ≥0.5 IU/ml and the reaction to TB antigens minus the nil tube control is <0.35 IU/ml or ≥0.35 IU/ml but <25% of the nil tube value), or indeterminate (if the nil tube value is >8.0 IU/ml or the mitogen tube value minus the nil tube value is <0.5 IU/ml). The magnitude of the IFN-γ response of the antigen minus the nil tube value was also reported in IU/ml (QuantiFERON-TB Gold In-Tube Method package insert, no. 05990301B; Cellestis, Victoria, Australia). For each ELISA plate three standard rows consisting of four points were performed for quality control.

QFT-IT sample storage.

To assess the influence of storage on test performance, plasma from the first set of QFT-IT tubes was divided into three aliquots (each in duplicate) following the 20-h overnight incubation. The first aliquot was processed immediately (fresh ELISA), the second aliquot was stored at 4°C for 4 weeks (as recommended by the manufacturer for short-term storage for up to 28 days), and the third was stored at −80°C for 4 weeks before the ELISA was completed. All other ELISAs were performed directly after incubation (fresh ELISA).

Statistical analysis.

Dichotomous and continuous measures for factors pertaining to QFT-IT test reproducibility were analyzed. Cohen's kappa coefficient was used for agreement between dichotomous measures; Fleiss' kappa was used for agreement of more than one variable. Continuous measures were compared using interclass correlations (ICC).

The study was approved by the Committee for Human Research at Stellenbosch University (project number N05/08/136). All participants gave written informed consent for participation.

RESULTS

Twenty-seven participants (median age, 38 years; range, 22 to 55 years) were enrolled. Two participants had previously been treated for active TB, 17 (63%) had had a previous TST induration of ≥10 mm; and 10 had a previous negative TST. The TST was completed in nine participants who reported previous negative TST results (<10 mm); three had TST indurations of ≥10 mm (20/27; 74% previous or current positive TST). None of the participants reported symptoms suggestive of active TB.

QFT-IT assay.

Three QFT-IT sets were obtained from each participant, two (sets 1 and 2) on day 1 and one on day 3 (set 3). Sets 1 and 2 were split into aliquots after incubation. Since all ELISAs were performed in duplicate, a total of six ELISA test pairs (duplicates 1 and 2) were performed per participant (Fig. 1).

Data were incomplete in 12 participants due to technical reasons (error in termination of color reaction or failed standard curve). Analysis excluded these results and was therefore performed on a total of 276 ELISA tests (138 samples and their duplicates).

Of all participants, 17/27 (63%) had at least one positive QFT-IT test result; in 11 (40.7%) all QFT-IT results were positive. Agreement between TST and QFT-IT results was 74% (20/27).

There were three (1.1%) indeterminate QFT-IT results; all occurred on the same ELISA plate. Two were duplicates of one sample; another was the duplicate of a positive sample (1.37 IU/ml).

Magnitude of response.

The magnitude of IFN-γ responses among various assays performed for individual participants was highly variable, with ranges from 0.03 to 11.11 IU/ml (Table 1). In five participants (19%), the minimum value was below 0.35 IU/ml, and the maximum was ≥0.35 IU/ml, thus resulting in discordant results. Mean as well as median values of participants with discordant results were below the cutoff in four of five participants, whereas these values were clearly above the cutoff in participants who had only positive test results.

TABLE 1.

Range in magnitude of response per participant in ascending order

Study no.	No. of valid tests	Magnitude of IFN-γ response (IU/ml)a								Overall response trend
		Minimum value	Maximum value	Rangeb	Mean	SD	Q25	Median	Q75
2	6	0.00000	0.03000	0.03000	0.01167	0.011690	0.00000	0.01000	0.02000	Negative
1	6	−0.01000	0.03000	0.04000	0.00500	0.013784	0.00000	0.00000	0.01000	Negative
5	6	−0.01000	0.03000	0.04000	0.00833	0.017224	0.00000	0.00000	0.03000	Negative
9	10	−0.03000	0.02000	0.05000	−0.00300	0.012517	−0.01000	0.00000	0.00000	Negative
11	10	−0.02000	0.04000	0.06000	0.00400	0.019551	−0.01000	0.00500	0.02000	Negative
14	12	−0.02000	0.07000	0.09000	0.00083	0.023143	−0.01000	−0.00500	0.00000	Negative
15	12	0.05000	0.27000	0.22000	0.16500	0.073175	0.10500	0.16000	0.23000	Negative
25	12	0.12000	0.38000	0.26000	0.20917	0.071409	0.16000	0.21000	0.23500	Discordant
27	12	−0.17000	0.14000	0.31000	0.08417	0.083389	0.08000	0.10500	0.13000	Negative
10	8	0.43000	0.82000	0.39000	0.61125	0.139021	0.51000	0.59000	0.72000	Positive
18	12	−0.69000	0.07000	0.76000	−0.04333	0.204732	0.00000	0.01500	0.02000	Negative
22	12	0.89000	1.84000	0.95000	1.19667	0.308849	0.99500	1.07500	1.24500	Positive
3	6	0.01000	1.02000	1.01000	0.18667	0.408444	0.01000	0.02000	0.04000	Discordant
13	12	0.05000	1.08000	1.03000	0.19000	0.285339	0.07000	0.08500	0.18500	Discordant
12	10	0.50000	1.59000	1.09000	1.06300	0.456315	0.56000	1.23500	1.46000	Positive
7	9	−0.01000	1.37000	1.38000	0.14889	0.457970	−0.01000	0.00000	0.00000	Discordant
4	6	0.15000	1.64000	1.49000	0.47500	0.575283	0.16000	0.29000	0.32000	Discordant
21	12	0.58000	2.24000	1.66000	1.47167	0.514955	1.23000	1.35000	1.98000	Positive
8	10	−1.90000	0.02000	1.92000	−0.18500	0.602629	0.00000	0.00000	0.01000	Negative
16	12	2.34000	4.53000	2.19000	3.20250	0.758840	2.68000	2.92000	3.81500	Positive
6	6	1.97000	5.19000	3.22000	3.33000	1.397555	2.13000	2.88500	4.92000	Positive
23	12	6.36000	10.26000	3.90000	7.91417	1.363881	6.55000	7.82500	8.74500	Positive
19	12	0.38000	5.41000	5.03000	1.58583	1.766071	0.51500	0.91500	1.47000	Positive
24	12	14.26000	22.39000	8.13000	16.67833	2.833455	14.88000	15.49500	17.59500	Positive
20	12	14.10000	23.05000	8.95000	19.45917	3.101611	16.42000	20.75500	21.45500	Positive
17	12	6.79000	16.77000	9.98000	9.40667	3.513047	7.33500	7.60500	9.98000	Positive
26	12	7.52000	18.63000	11.11000	9.84333	4.035752	7.82500	8.17000	8.89000	Positive

^aIndeterminate results are excluded from analysis. Q25 and Q75 represent the first and third quartiles, respectively.

^bDefined as the difference between the maximum and minimum values.

Test agreement.

Comparisons of potential sources of variability between fresh ELISAs as well as between different storage modalities prior to performance of QFT ELISA are shown in Table 2. The kappa value for the agreement between duplicates using dichotomous measures was 0.93; only 5 of 138 test pairs (3.6%) were discordant. The test-retest agreement was also high between continuous measures (ICC of 0.995; 95% confidence interval [CI] of 0.993 to 0.997) (Fig. 2).

TABLE 2.

Comparison of potential sources of variability of the QFT-IT assay

Comparison parametera	Description	No. of participantsb	Agreement
			Dichotomous measures (kappa [95% CI])	Continuous measures (ICC agreement [95% CI])
Test-retest	Duplicate 1 vs duplicate 2	27	0.930 (0.860-0.990)	0.995 (0.993-0.996)
Fresh ELISA	Overall, all fresh ELISAs	15	0.928 (0.732-1.000)c	0.886 (0.754-0.956)
Intraoperator	2 QFT-IT sets, same day, same operator	21	0.904 (0.696-1.000)	0.954 (0.892-0.981)
Interoperator	1 QFT-IT set, same day, different operators	21	0.826 (0.614-1.000)	0.965 (0.897-0.987)
Day-to-day	2 QFT-IT sets, different days, same operator	15	1.000 (1.000-1.000)	0.809 (0.406-0.938)
Storage	Overall, different storage of supernatant	21	0.936 (0.793-1.000)c	0.974 (0.948-0.989)
	Fresh ELISA vs storage at 4°C for 4 wks	21	0.904 (0.696-1.000)	0.956 (0.897-0.982)
	Fresh ELISA vs storage at −80°C for 4 wks	21	0.904 (0.674-1.000)	0.967 (0.92-0.986)

^aWith the exception of the test-retest comparison, results used were only from duplicate 1.

^bDue to failed ELISA runs, the total number of participants included in the analysis varies.

^cBootstrap confidence intervals.

FIG. 2.

QFT-IT test-retest reproducibility. Results for duplicate 1 versus duplicate 2 of the QFT-IT assay (ICC of 0.995; 95% CI, 0.993 to 0.996) are shown; indeterminate results were excluded.

For further analysis only results of duplicate 1 were used. In general, agreement was high, with kappa values ranging between 0.826 to 1.000 for dichotomous measures and ICC of 0.809 to 0.965 for continuous measures. The variability of the magnitude of response was higher comparing measures obtained on different days (ICC of 0.809; 95% CI, 0.406 to 0.938). There was good agreement between dichotomous measures obtained on the two different days (kappa, 1.000; 95% CI, 1.000 to 1.000).

DISCUSSION

In this study we assessed the short-term reproducibility of a widely used and recommended commercial IGRA for the detection of M. tuberculosis infection, the QFT-IT assay. Since there is a high risk of M. tuberculosis exposure among health care workers in this setting, we focused on the short-term reproducibility of the assay to minimize the influence of new M. tuberculosis exposure/infection; all tests were therefore performed within 3 days. In a rigorous approach we investigated different potential sources of variability that may influence test outcome and interpretation and showed that the test is highly reproducible. Previous studies of QFT reproducibility focused mainly on test conversions and reversions, whereas we addressed new aspects such as plasma storage mechanisms, test-retest variability (the comparison of duplicates performed at the same time), and the influence of different operators, which have not been assessed in previous studies (20, 22).

The QFT-IT was positive at least once in a high proportion of health care workers (63%), indicating the high risk of M. tuberculosis infection in this subpopulation; this is consistent with previous reports (11, 22). There was an encouragingly low level of indeterminate results in the present study, which may have been partly due to the rigorous standardization of sample collection and laboratory analysis.

Although individual responses were variable, we found the test to be robust, with limited variability and high kappa values (≥0.93) for a comparison of both dichotomous and continuous test readouts. Test-retest reproducibility was high for dichotomous and also continuous measures, showing that the assay itself is consistent and seems reliable even if performed as a single ELISA. The highest variability (ICC of 0.809) was found between the continuous readouts comparing IFN-γ responses on different days. The relatively wide CIs for some measures probably reflect the wide range of responses in individuals as well as the limited sample size.

Various factors in the laboratory which may contribute to test variability are addressed in the present study. According to the manufacturer, either the QFT ELISA can be performed directly after stimulation with the M. tuberculosis antigens, or the plasma can be stored following stimulation, either at 4 degrees for up to 4 weeks or at −80 degrees for a longer period of time. We verify in this study that storage does not influence test results to a high degree and that stored tests perform similarly compared to tests performed without prior storage. This is consistent with data from a previous study where two ELISAs were performed on samples stored at 4°C and where assays were completed 1 week apart (22). The advantage of sample storage is that tests can be more efficiently performed in larger batches, which may result in a reduction of workload as well as costs. At the same time, the option of receiving a test result within 1 day is an advantage of IGRAs over the TST if adequate capacity exists; storage of samples may, therefore, be more relevant to research than for routine daily clinical practice.

The QFT ELISA appeared to be robust even if performed by different operators for both dichotomous and continuous readouts. Despite the fact that the tests were conducted in a laboratory with extensive experience in conducting these tests and where formal training was completed, there were some invalid assays in the present study caused by technical errors. QFT-IT testing during the course of routine clinical care or in research therefore requires well-trained personnel, standard operating procedures, and regular quality control. In routine testing, sufficient plasma can be retrieved from the culture conditions to allow temporary storage to enable the repetition of failed ELISAs.

If one operator performs two tests in the same individual and on the same day, possible causes of test variability could include differences in blood volumes placed in the antigen/mitogen-coated collection tubes, differences in handling of the test tubes (including shaking of tubes), and performance of the ELISA, including pipetting errors and differences in ambient temperatures. We found only minor differences in our assessment of intraoperator variability, further confirming the robustness of the test. However, the QFT-IT assay is a whole-blood assay, and the number of lymphocytes in a test sample may vary, which may be a further cause of variability and a potential disadvantage compared to the T-SPOT.TB test, where adjustment for the cell count is made. This may be of special relevance in immune-compromised populations where low lymphocyte counts may affect test performance (12, 21).

The day-to day variability was assessed to gain information on short-term within-person changes in IFN-γ responses. Dichotomous test results did not change when tests were performed 3 days apart, but there were considerable changes in the magnitude of IFN-γ response, which may partly be explained by variation in the numbers of IFN-γ-producing T-cells in the peripheral blood. These findings, in conjunction with the wide range of IFN-γ responses between all tests performed in an individual, may therefore affect the interpretation of serial IGRA testing.

Several studies have investigated the dynamics of IGRA responses over time in individuals with M. tuberculosis exposure, latent TB infection (LTBI), or TB patients on treatment (1, 3, 6, 9, 10, 17, 20). Although interpretation of these results remains uncertain, serial testing has already been recommended in the United States (13, 18). A decline of IFN-γ response during and after treatment has been demonstrated in individuals with TB and in those with LTBI and has been interpreted as an indication of favorable treatment response (1, 6). However, declining IFN-γ responses have also been demonstrated in individuals with LTBI in the absence of treatment (6, 7). Although these variations in responses affected the magnitude of IFN-γ responses rather than actual test result interpretation, test conversions and reversions may also occur and do so often in those individuals with lower initial IFN-γ responses (6, 9). To what extent these reported changes can be attributed to within-person variations is unknown. In the present study, five individuals had discordant results for all tests performed within 3 days. Their median as well as mean IFN-γ responses were mainly below the cutoff value.

Questions remain: to what degree does a change in the magnitude of the IFN-γ response indicate a “real” change and, in individuals with results close to the cutoff value, what change indicates a real test conversion or reversion. Other salient questions include whether and to what degree changes in the magnitude of response may signal new infection or progression to disease in individuals with known M. tuberculosis infection and an initial positive IFN-γ response. These are beyond the scope of the present study.

Limited studies of healthy volunteers have been completed to gain information on IFN-γ variability in their responses to mycobacterial antigens (1, 20, 22). Among 63 individuals who had no documented risk of M. tuberculosis infection and who were investigated at two time points 3 months apart, participants with discordant QFT-IT test results had lower maximum IFN-γ levels than those with persistent, positive results. In the present study, the maximum value among the five participants with discordant QFT-IT test results was 1.64 IU/ml, whereas the maximum value among the 12 participants with consistently positive results was 23.1 IU/ml. The mean value among all participants with consistently positive results was higher than the mean value of participants with inconsistent results (6.7 IU/ml versus 0.2 IU/ml, respectively). All participants with discordant QFT-IT results had only one positive test among otherwise negative tests (one also had an indeterminate test). These findings suggest that individuals with lower IFN-γ responses are more likely to have inconsistent test results. However, these responses do not necessarily lie close to the cutoff of 0.35 IU/ml. Our findings are in agreement with another study among health care workers showing that only about 5% of all QFT-IT test results had values ranging between 0.25 and 0.45 IU/ml (16).

Of two published studies on QFT-IT reproducibility, one specifically addressed the question of how much change in IFN-γ response indicates a true change rather than nonspecific variability and calculated that increases in IFN-γ responses of more than 16% were statistically improbable for short-term variability but that variations of up to 30% might occur using a linear mixed-effects model (22). The authors concluded that an increase in the IFN-γ response of more than 16% over a short period indicates a change that may be attributable to a genuine change in M. tuberculosis infection status over a short period. In the present study, however, we showed a considerable range of individual IFN-γ responses among all assays performed in a participant. Both studies were performed with small sample sizes (n = 14 in the study of Veerapathran et al. and n = 27 in the present study). These findings should therefore be addressed in larger cohorts in the future.

We did not assess other factors that may influence reproducibility such as different incubation times and time to incubation; these have previously been shown to affect IFN-γ responses to mycobacterial antigens in whole-blood assays (5). Other immunological aspects such as cell-type-specific changes within individuals that could also be responsible for variability in IFN-γ responses should be addressed in future studies. The monitoring of IFN-γ production in intracellular cytokine assays in whole blood in parallel to IGRAs may be a valuable additional tool to measure non-T-cell IFN-γ production. In addition, controlling for the number of T cells and other IFN-γ-producing cells may also help to assess antimycobacterial immunity.

Conclusion.

We conclude that the QFT-IT test is a robust and highly reproducible assay but that rigorous laboratory technique is important for the conduct of IGRAs. Intraindividual variability in the magnitude of IFN-γ responses occurs even in the short-term assessment of the assay, partly affecting test results. This has to be taken into consideration in interpreting serial measures. Ongoing cohort studies will provide useful data on the clinical implications of IGRA conversions.