Bicentric Evaluation of Six Anti-Toxoplasma Immunoglobulin G (IgG) Automated Immunoassays and Comparison to the Toxo II IgG Western Blot

Abstract

A comparative study of the Toxoplasma IgG_I and IgG_II Access (Access I and II, respectively; Beckman Coulter Inc.), AxSYM Toxo IgG (AxSYM; Abbott Diagnostics), Vidas Toxo IgG (Vidas; bioMerieux, Marcy l'Etoile, France), Immulite Toxo IgG (Immulite; Siemens Healthcare Diagnostics Inc.), and Modular Toxo IgG (Modular; Roche Diagnostics, Basel, Switzerland) tests was done with 406 consecutive serum samples. The Toxo II IgG Western blot (LDBio, Lyon, France) was used as a reference technique in the case of intertechnique discordance. Of the 406 serum samples tested, the results for 35 were discordant by the different techniques. Using the 175 serum samples with positive results, we evaluated the standardization of the titrations obtained (in IU/ml); the medians (second quartiles) obtained were 9.1 IU/ml for the AxSYM test, 21 IU/ml for the Access I test, 25.7 IU/ml for the Access II test, 32 IU/ml for the Vidas test, 34.6 IU/ml for the Immulite test, and 248 IU/ml for the Modular test. For all the immunoassays tested, the following relative sensitivity and specificity values were found: 89.7 to 100% for the Access II test, 89.7 to 99.6% for the Immulite test, 90.2 to 99.6% for the AxSYM test, 91.4 to 99.6% for the Vidas test, 94.8 to 99.6% for the Access I test, and 98.3 to 98.7% for the Modular test. Among the 406 serum samples, we did not find any false-positive values by two different tests for the same serum sample. Except for the Modular test, which prioritized sensitivity, it appears that the positive cutoff values suggested by the pharmaceutical companies are very high (either for economical or for safety reasons). This led to imperfect sensitivity, a large number of unnecessary serological follow-ups of pregnant women, and difficulty in determining the serological status of immunosuppressed individuals.

Toxoplasmosis, caused by Toxoplasma gondii, is widespread in humans and warm-blooded animals. Although it is usually asymptomatic in immunocompetent humans, toxoplasmosis may cause severe disorders in pregnant women, because of the high risk of transplacental transmission and the occurrence of abortion or multiple congenital lesions in the fetus, and in immunocompromised individuals (5, 9).

Life-threatening reactivation of a previous infection is commonly observed in cases of severe immunodeficiency (human immunodeficiency virus-infected patients, organ and hematopoietic stem cell transplant patients). For these patients, the detection of Toxoplasma-specific antibodies showing serological reactivation or primary infection is therefore essential for the appropriate diagnosis and prevention of severe toxoplasmosis (2, 7).

The follow-up of patients with obstetric toxoplasmosis mainly depends on the detection of antitoxoplasma-specific immunoglobulin M (IgM) and IgG antibodies (14, 16, 18). The presence of toxoplasma-specific IgM at the time of the first blood test is a cause for concern. The presence of toxoplasma-specific IgG without IgM permits confirmation of the immunization of the patients and thus allows unnecessary and expensive follow-up to be avoided.

For both obstetric follow-up and diagnosis in immunocompromised patients, tests for IgG are crucial. Since the 1980s, toxoplasma-specific IgG assays have been standardized by different generations of World Health Organization (WHO) standards (15), and test results have been reported in international units per milliliter (IU/ml). The dye test (DT), first described by Sabin and Feldman 60 years ago, is still the reference method for the serodiagnosis of toxoplasmosis. However, this test uses live Toxoplasma gondii and is now used in only a few laboratories (13). A good alternative, the test Toxo II IgG Western blot (LDBio, Lyon, France) has been proposed to be a confirmatory technique by Franck et al. (6). The results of this test appear to be consistent with those of DT, with a specificity of 100% and a sensitivity of 99.2%. Thus, this immunoblotting technique can be used as a very reliable and easy confirmatory test in laboratories where DT cannot be implemented.

Despite the international standardization and the availability of a reference (or confirmatory) test, automated immunoassays frequently show discordance and moderate degrees of correlation (6, 12). A comparison of six random-access immunoassays (that report IgG levels in IU/ml) and the Toxo II IgG Western blot (LDBio) as a confirmatory technique was undertaken to review the analytical performance characteristics and the degree of standardization of the tests.

MATERIALS AND METHODS

Samples.

This study was done with 406 consecutive serum samples from adult patients tested in the laboratory of the CHU de Saint Etienne, Saint Etienne, France, for 20 days in December 2006. A total of 228 (56.2%) of these serum samples were from pregnant women who were followed up, 161 (39.6%) were from patients followed up for immunodepression, and 17 (4.2%) serum samples came from other departments. As most of the serum samples were from pregnant women, the male/female sex ratio was 1/3.06.

All the samples were given anonymous labels (as suggested by the ethical committee), decanted, and frozen within 24 h of receipt. They were then thawed and recentrifuged in series and tested in parallel in the first laboratory (Parasitology and Mycology Laboratory, University Hospital of Saint Etienne, Saint Etienne, France) on the same day with the Access Toxo IgG I and AccessToxo IgG II (Access I and Access II, respectively) tests (Beckman Coulter Inc.), the AxSYM Toxo IgG (AxSYM) test (Abbott Diagnostics), and the Vidas Toxo IgG (Vidas) test (bioMerieux, Marcy l'Etoile, France). The samples were again immediately frozen following testing, and another freezing and thawing was done for testing in the second laboratory (Laboratory Synerbio, Saint Etienne, France) by the Immulite Toxo IgG (Immulite) test (Siemens Healthcare Diagnostics Inc.) and the Modular Toxo IgG (Modular) test (Roche Diagnostics, Basel, Switzerland). In the case of qualitative discordance between techniques (negative, equivocal, or positive results), the Western blot was done as a confirmatory test.

Tests. (i) Automated immunoassays.

Six of the assays were automated and used random-access instruments that could perform a range of assays for infectious diseases and biochemical analysis. All tests were performed as instructed by the manufacturers. The manufacturers' cutoffs were applied to determine the reactivities of the sample. All assays report the test results in IU/ml. The random-access immunoassays used in the study were listed above. In Europe, the Access II test replaced the Access I test during the year 2008. The Access I (4, 8), AxSYM (4, 8, 11, 19), and Vidas (6, 8, 11, 17) tests use the WHO second international standard (IS), whereas the Access II, Immulite (10), and Modular (6) tests use the third WHO IS. It is to be noted that neither the second nor the third WHO IS was tested in parallel by the six automated techniques in the present study. The main features of each automated test are described in Table 1.

TABLE 1.

Characteristics of the six anti-toxoplasma IgG automated immunoassays

Immunoassay	Robot(s) used	Cutoff (IU/ml)		WHO IS used for calibration	Calibration curve
		Lower	Higher
Access I	Not used since 2009	4	6	Second	Multipoint
Access II	Access II, DXI 600, DXI 800	7.5	10.5	Third	Multipoint
Vidas	Mini Vidas, Vidas	4	8	Second	Two point
AxSYM	AxSYM	2	3	Second	Multipoint
Immulite	Immulite 2000	6.5	8	Third	Two point
Modular	Cobas e411, Cobas e601, Modular E170, Elecsys 2010	1	30	Third	Two point

(ii) Western blot test.

The Western blot test (6) is a qualitative immunoenzymatic test in which immunoblotting to nitrocellulose strips is used. After standardized incubation with sera and fixation of specific IgG on the band, the anti-toxoplasma IgG bound to the strip is detected with an alkaline phosphatase-conjugated antibody and a specific substrate (nitroblue tetrazolium and 5-bromo-4-chloro-3-indolylphosphate). The resulting bands on the patient's strip correspond to 30, 31, 33, 40, and 45 kDa. A positive result is defined by the presence of at least three matching bands on the patient's strip, including the specific band at 30 kDa.

A doubtful result is defined by the presence of the specific band at 30 kDa with less than three matching bands on the patient's strip.

Analysis. (i) Relative sensitivity, specificity, PPV, and NPV.

The relative sensitivities and specificities of each automated immunoassay were estimated by comparing the immunoassay's qualitative result (negative, equivocal, or positive) with the result of the Western blot. The relative sensitivity and the specificity of each immunoassay were calculated twice, first by interpreting an equivocal result as negative and second by interpreting an equivocal result as positive (Table 2).

TABLE 2.

Relative sensitivity, specificity, PPV, and NPV by testing of 175 positive and 231 negative nonselected samples for anti-toxoplasma IgG^a

Immunoassay and classification of equivocal result	Sensitivity (%)b	Specificity (%)b	PPV (%)c	NPV (%)c
Access I
Negative	94.8	99.6	99.4	96.2
Positive	98.3	99.1	98.8	98.7
Access II
Negative	89.7	100	100	92.8
Positive	94.3	99.6	99.4	95.8
Vidas
Negative	91.4	99.6	99.4	93.9
Positive	98.3	99.6	99.4	98.7
AxSYM
Negative	90.2	99.6	99.4	93.1
Positive	96	99.6	99.4	97.0
Immulite
Negative	89.7	99.6	99.4	92.7
Positive	94.3	99.6	99.4	95.8
Modular
Negative	98.3	98.7	98.3	98.7
Positive	99.4	94.3	93.5	99.6

^aThe criteria were determined with the high and low cutoff values specified by the manufacturers. The confirmatory test was the Toxo II IgG Western blot (LDBio).

^bPerformance values were calculated by using the Toxo II IgG test Western blot as the reference test.

^cPPVs and NPVs were calculated by using the seroprevalence of our sampling (43.1%), which is very close to the French seroprevalence (43.8%) (1).

The positive and negative predictive values (PPVs and NPVs, respectively) were calculated by using the seroprevalence for our sample (43.1%), which is very close to the seroprevalence in France (43.8%) (1).

(ii) Statistical analyses.

The results, expressed as IU/ml, were used in the statistical analyses. When an assay produced a result that was greater than the highest limit of the assay, the result was assigned a value of the limit level plus 1. When an exact titer was not determined, the range of the test's value was shown by the first, second, and third quartiles.

Statistical assessment of the differences in sensitivity, specificity, PPV, and NPV by the different tests was estimated by the chi-square test (a P value of <0.05 was considered significant).

Statistical assessment of the differences in the mean titers obtained by the different tests was estimated by analysis of variance (Fisher-Snedecor test; a P value of <0.05 was considered significant).

RESULTS

Global characteristics and concordance of the six automated immunoassays.

Of the 406 serum samples tested, 218 (53.7%) were negative by all six tests, 153 (37.7%) were positive by all six tests, and 35 (8.6%) were discordant. As a result, the overall concordance of all six techniques was 91.4% (371/406 serum samples). The results for all the discordant sera are shown in Table 3. Twenty-two of these serum samples were positive by the Western blot, with three, four, and five specific bands corresponding to doubtful or false-negative results by the majority of the automated tests. Of the remaining 13 serum samples, 3 had equivocal results by the Western blot (with two specific bands, one of which was p30) and 10 were negative by the Western blot. The latter 10 samples had false-positive results (compared to the Western blot result, which was negative) but only by the AxSYM test (1 serum sample) and the modular test (9 serum samples).

TABLE 3.

Qualitative results for the 35 discordant serum samples of the 406 samples tested by the anti-toxoplasma IgG automated immunoassays and comparison with the results of the Toxo II IgG Western blot^a

Western blotb	Total no. of samples (n = 35)	No. of samples with the indicated test result by the following automated immunoassays:
		Access I			Access II			AxSYM			Vidas			Modular			Immulite
		N	E	P	N	E	P	N	E	P	N	E	P	N	E	P	N	E	P
N	10	10			10			9		1	10			1	9		10
E	3	1	1	1	2	1		3			2		1			3	2		1
P	22	3	6	13	10	8	4	7	10	5	3	12	7	1	2	19	10	8	4

^aThe results of each automated immunoassay and the Toxo II IgG Western blot (LDbio) were obtained by using the cutoffs specified by the manufacturers.

^bFor the Western blot, the results were considered equivocal when there were only two bands, one of which was for p30. N, negative; E, equivocal; P, positive.

Standardization of titrations (IU/ml) obtained by the six automated immunoassays.

By using the 175 positive serum samples whose results were confirmed by the Western blot, we evaluated the standardization of the titrations obtained (in IU/ml) by comparing their distribution (first, second, and third quartiles). The median (second quartile) obtained from our sampling was statistically different from one test to the other, as follows: 9.1 IU/ml for the AxSYM test, 21 IU/ml for the Access I test, 25.7 IU/ml for the Access II test, 32 IU/ml for the Vidas test, 34.6 IU/ml for the Immulite test, and 248 IU/ml for the Modular test (Fig. 1). In the same manner, despite the bias caused by the different superior linearity limits of each technique, the means ± standard errors obtained were 22.97 ± 22.34, 54.14 ± 47.73, 57.50 ± 59.05, 62.65 ± 53.18, 64.77 ± 53.27, and 299.7 ± 169.91 for the AxSYM, Access II, Access I, Vidas, Immulite, and Modular tests, respectively.

FIG. 1.

Distribution of the 175/406 toxoplasma IgG-positive sera confirmed by the Toxo II IgG Western blot (LDBio) technique. The numbers are as follows, from bottom to top: the value at the bottom was confirmed by the presence of at least three bands (including the band for p30) by the Western blot technique; the first value in boldface corresponds to the first quartile, the second value in boldface corresponds to the median or the second quartile, the third value in boldface corresponds to the third quartile; and the value at the top corresponds to the limit of linearity of the technique without dilution.

Relative sensitivity, specificity, PPV, and NPV.

The reliability of each test was determined, and the characteristics are shown in Table 2. The only technique which showed a specificity of 100% was the Access II test. All the other tests except for the Modular test had a specificity of 99.6%; the Modular test showed a specificity of less than 98.7%. When the results obtained with the highest cutoff were evaluated, the sensitivities of the tests were found to be low: 89.7, 89.7, 90.2, 91.4, 94.8, and 98.3% for the Immulite, Access II, AxSYM, Vidas, Access I, and Modular tests, respectively. There were significant improvements in their sensitivities (P < 0.05, χ² test) when the lowest cutoff for each technique was used: 94.3, 94.3, 96, 98.3, 98.3, and 99.4% for the Immulite, Access II, AxSYM, Vidas, Access I, and Modular tests, respectively. With the exception of the Modular test, the sensitivities obtained by use of the lowest cutoff were associated with specificities that were always greater than 99%.

DISCUSSION

This study had two objectives: (i) to evaluate the standardization between the main automated tests and (ii) to define the main reliability criteria (specificity, sensitivity, PPV, NPV) for each of the automated tests.

Regarding the level of intertechnique standardization, the results were not as good as expected. This was demonstrated in this study from the medians of the results, which varied by a factor of 1 to 25, depending on the test used (Fig. 1). These intertechnique variations are particularly surprising since, in fact, an international reference exists (15). In comparison, a recent study (3) that evaluated eight anti-rubella virus IgG tests, of which six were automated, showed a much better standardization, with the variation factor ranging from 1 to 3 (average for the samples positive by the different tests). We can speculate that these variations could be related to the differences in the antigenic solutions used by each of these tests. Indeed, the study by Petithory et al. (12) showed variations related to the antigenic nature (complex antigenic structure with numerous membrane and cytosolic antigens), but these variations were at a much lower level. These different antigenic solutions induce intense and precocious responses in some cases and more delayed responses in other cases (4, 8). The major cause of intertechnique variation, however, is probably associated with the lack of cooperation of the companies providing the tests, thus limiting the harmonization and standardization of the titrations obtained and also the cutoffs proposed. The choice of the second or third WHO IS for standardization of the different techniques could also contribute to the variability of the quantitative results. However, in our comparative study, there was no significant difference in the results between those which used the second WHO IS and those which used the third WHO IS (Fig. 1). On the other hand, it is interesting to note that the standards of the third WHO IS contain IgG with a low avidity (13), which can, as a result, influence the reactivities of the different techniques.

In any case, all these factors induce considerable intertechnique variability, and these variations were especially noted for the AxSYM test and the Modular test when the results of those tests were compared to those of the other tests evaluated (Fig. 1), which makes any comparison of intertechnique titrations impossible. To date, one can compare two successive titrations only if the latter are done by the same technique and, preferably, in the same series.

It is interesting to note that despite the quantitative serologic variations, the number of qualitative discordant cases (negative, equivocal, positive) among the six automated techniques remained low. Thus, only 35/406 (8.62%) of the samples showed discordance (negative by one test and doubtful or positive by another), and only 17/35 (4.19%) had total discordance (i.e., negative by one test and positive by another). This high level of concordance is related in parallel to the variation of the cutoffs and the medians for the positive samples obtained by each of these techniques. For example, for the AxSYM test, the cutoff was 3 IU/ml and the median was 9.1 IU/ml, and for the Modular test, the cutoff was 30 IU/ml and the median was 248 IU/ml. These results, which showed good concordance, are in agreement with the results of previous studies (8, 11, 17) and confirm the reliability of all the automated techniques.

In order to determine the criteria for the reliability of the different techniques, we compared the sensitivities, specificities, PPVs, and NPVs of the six tests. When the doubtful (or equivocal) values were considered negative, the specificities were excellent: 100% for the Access II test; 99.6% for the Access I, AxSYM, Vidas, and Immulite tests; and 98.7% for the Modular test (P > 0.05, no significant difference, χ² test). It must be said that for the Modular test, the three samples with false-positive results had equivocal results by the Western blot, which does not allow confirmation of the presence of specific antibodies. Therefore, these sera were not necessarily false positive. On the other hand, the sensitivities were very close and low for all the commercial tests evaluated in the present study and varied from 89.7% to 91.4%, with the exception of that for the Modular test, which had a much higher sensitivity of 98.3%. When the doubtful (or equivocal) values were considered positive, the specificities remained excellent (>99%) for all techniques except the Modular test and their sensitivities increased significantly: they were 94.3%, 94.3%, 96%, 98.3%, and 98.3% for the Immulite, Access II, Vidas, Access I, and AxSYM tests, respectively (Table 2). When the reliability criteria were analyzed and also after analysis of the receiver operating characteristics (data not shown), we observed that almost all companies have prioritized the specificity by raising the cutoffs. On the contrary, the Roche Company (the Modular test) has given priority to sensitivity by choosing a lower cutoff. Paradoxically, the latter technique has the most elevated cutoff: 30 IU/ml. These surprising results, however, are consistent with those of Franck et al. (6), who showed sensitivity and specificity levels which were totally different between the Vidas technique and the Cobas test (the Cobas and Modular tests belong to the Cobas family of tests).

For all the automated techniques studied, the sensitivities and specificities obtained should be analyzed and interpreted in parallel with the quantitative results for the discordant and borderline sera (Table 3). These results confirm the idea that the cutoffs chosen by the companies (for all the tests except the Modular test) are too high and guarded. The lower cutoffs would achieve a better balance between sensitivity and specificity (Table 2). We have also noticed that it is possible to have false-positive cases, even though they occur at very low frequencies. An interesting point of our study is that there were not so many sera with false-positive or doubtful results by these automated techniques. As a result, in case of a primary doubtful result by one automated technique, it seems advisable to test the serum by another automated technique since cross false positivity between techniques has not been found. Thus, in the present study, all the sera with positive or doubtful results by the two automated techniques were confirmed to be either positive (20/22 cases) or indeterminate (2/22 cases) by the Western blot.

Evidently, the possibility of doing a confirmatory test by either the Western blot or DT would allow more confidence in the results and would permit the avoidance of unnecessary and expensive serological follow-up in up to 5% of cases.