Evaluation of Five Enzyme-Linked Immunosorbent Assays and an Agar Gel Immunodiffusion Test for Detection of Antibodies to Small Ruminant Lentiviruses

Abstract

In the framework of the Dutch control program for small ruminant lentiviral (SRLV) infections, too many drawbacks were encountered with respect to serological testing. To improve the quality of testing, five enzyme-linked immunosorbent assays (ELISAs) and an agar gel immunodiffusion test (AGIDT) were evaluated. The focus was on the sensitivity, specificity, and variances of the commercially available tests. Clear differences were found among the tests in analytical and diagnostic sensitivity and overall diagnostic performance, whereas no significant differences in specificity were found. For serodiagnosis of sheep with clinical symptoms of maedi-visna virus (MVV) (histopathologically confirmed), one ELISA was significantly more sensitive than the other ELISAs and than the AGIDT, while for asymptomatic sheep originating from infected flocks, three ELISAs and the AGIDT demonstrated similar performance. The diagnostic performance appeared to be related to animal species and virus infection (MVV or caprine arthritis encephalitis virus [CAEV]) as well as the phase of infection/progression of disease. Receiver operating characteristic analysis, demonstrating the diagnostic potential of tests irrespective of defined cutoffs, again revealed clear differences between tests with respect to diagnostic performance for detection of antibodies against CAEV or MVV. An indirect ELISA, of which the solid phase is sensitized with a combination of the core protein p27 of MVV produced in Escherichia coli and a peptide derived from the transmembrane protein gp46, appeared to be the test of choice for serodiagnosis of SRLV infections in sheep and goats.

Lentiviral infections in sheep and goats have considerable economic impact, especially on dairy goat farming (6, 13, 19), where the virus is efficiently transmitted via vertical and horizontal routes (4, 14). The sheep and goat lentiviruses cause different disease entities with comparatively long incubation times (8, 11). The ovine maedi-visna virus (MVV) and the caprine arthritis encephalitis virus (CAEV) have many similarities on the molecular and antigenic levels and therefore are considered to belong to one group, the small ruminant lentiviruses (SLRVs) (12).

Detection of the infection generally is performed by means of serological methods, such as indirect and blocking enzyme-linked immunosorbent assays (ELISAs), agar gel immunodiffusion testing (AGIDT), and immunoblotting (3, 7, 9, 10, 18, 20, 21, 22).

Apart from technical considerations with respect to the use of commercially available serological tests, problems may be encountered because of slow seroresponses and genetic heterogeneity expressed on the protein epitope level (5, 15, 23).

In The Netherlands, the Animal Health Service has been running an SRLV accreditation program for over 20 years based on serological testing of flocks using different ELISAs over time.

Problems experienced included incidental unexplainable positive results in a small percentage of accredited flocks and low levels of repeatability of such positive results in new serum samples. The lack of a gold-standard confirmation test in such situations and the uncertainty coming with it gave rise to the question of whether the tests’ sensitivities were sufficient for effective control of the infection. For monitoring of accredited flocks, the positive test results should be reliable, and therefore the specificity of the test also should be high.

In the Dutch dairy goat population, many farms are not yet accredited. CAEV control under the intensive management conditions of the modern goat dairy flock has proved to be difficult due to the relatively rapid horizontal spread of the infection. It also could be argued that the efficiency of detection of infected animals in these conditions is insufficient.

It is generally accepted that antibody detection tests, if not all tests, always are a compromise between the characteristics of sensitivity and specificity, because these factors basically oppose each other. This idea contends that tests with 100% sensitivity and specificity simply do not exist. Besides sensitivity and specificity, tests also are characterized by their repeatability, which associates with intraplate (between wells) as well as interplate (between plates) variations and the tests’ robustness.

Although a variety of tests are available for detection of antibodies against SRLV, the sensitivities and specificities of these assays have not been compared systematically (3).

To choose an SRLV antibody detection test to be used in our SRLV control program, we evaluated and validated six commercially available tests (five ELISAs and one AGIDT), focusing mainly on the test characteristics sensitivity, specificity, and within-batch variance.

MATERIALS AND METHODS

Flocks and samples.

A technical test panel was composed of well-defined sera. For the statistical test panel, samples were collected from the target populations. All sera were stored at −20°C.

Technical test panel.

The following panels of well-defined sera were available to determine technical specificity, technical sensitivity, detection limits, and variance: (i) negative sheep sera from an SRLV-free region (Iceland) (n = 99); (ii) negative goat sera (prechallenge) from infection experiments (n = 34); (iii) sheep sera from clinically and pathologically confirmed cases (n = 62); (iv) goat sera from clinically and pathologically confirmed cases (n = 18); (v) sequential sheep sera taken after experimental infection (n = 4); (vi) sequential goat sera taken after experimental infection (n = 30); (vii) samples from experimentally infected animals (three sheep and three goats) serially diluted (1/2 to 1/256) in negative sheep or negative goat serum and tested to determine detection limits; and (viii) one goat serum and one sheep serum, both confirmed positives, tested in two dilutions made in homologous negative serum in four wells of each test plate to determine within- and between-plate variance.

Statistical test panel.

The following panels of sera sampled from the target populations were available to determine statistical specificity and sensitivity: (i) sheep sera from MVV-free accredited flocks (n = 80); (ii) goat sera from CAEV-free accredited flocks (n = 55); (iii) sheep sera from infected flocks, in which clinical cases were diagnosed and histopathologically confirmed (n = 99); (iv) goat sera from infected flocks, in which clinical cases were diagnosed and histopathologically confirmed (n = 50); (v) sheep sera from nonaccredited flocks (status unknown) (n = 20); (vi) goat sera from nonaccredited flocks (status unknown) (n = 43).

Participating tests.

Five commercially available ELISA kits and one AGIDT were evaluated. The ELISAs included Chekit CAEV/MVV monophasic and Chekit CAEV/MVV biphasic (Dr. Bommeli AG, Bern, Switzerland), ELITEST-MVV (formerly INNOTEST-MVV; Hyphen BioMed, Neuville-sur-Oise, France), ELISA MAEDI VISNA/CAEV (Institut Pourquier, Montpellier, France), and Ceditest MVV (CediDiagnostics, Lelystad, The Netherlands). The AGIDT was MAEDITECT (Veterinary Laboratory Agency, Weybridge, United Kingdom).

Criteria.

Test kits were evaluated for their sensitivity, specificity, variance, and laboratory practicability (format of the kit, product information, incubation steps, ready-to-use reagents, incubation temperatures, sample dilutions, extra pipetting steps, suitability for automation, etc.).

Statistical analysis.

The agreement between ELISAs on one hand and AGIDT on the other hand was expressed as κ values, an index that compares the agreement against that which might be expected by chance.

Test agreement, sensitivity, and specificity were calculated using WinEpiscope 2.0 (http://www.clive.ed.ac.uk/winepiscope/). Significant differences (P < 0.05) in sensitivity and specificity were calculated with McNemar's test for correlated proportions. Correlation analysis was performed using Microsoft Excel 2000.

Receiver operating characteristic (ROC) curves were created, and the area under the curve (AUC) was estimated to quantify overall test performance. For this purpose, the test results of the sheep and goat sera described in this study were used, and the dilution series in negative serum for estimating detection limits were excluded. Significant differences in areas under the ROC curve were calculated using a Z test. Since tests were applied to the same animals, results cannot be considered independent. Therefore, Z values were calculated while taking into account the correlation between two tests (Pearson correlation coefficient).

Three test kits recognized a category of doubtful or ambiguous results. For samples with such results, the manufacturers advise that the samples be retested. If the same result was obtained, they advise that the animal be resampled and the sample tested again. In this study, equivocal results were considered to be evenly split, with half counted as positive and half counted as negative. Only limited numbers of doubtful results, however, were found for those tests with such a category.

RESULTS

Technical and statistical sensitivity.

Results for technical and statistical sensitivity are summarized in Table 1. For the serum panel from clinical sheep, test D was more sensitive than the AGIDT (P = 0.046 by McNemar test) and was significantly more sensitive than the other ELISAs (P < 0.001). The AGIDT was significantly more sensitive than test C (P = 0.002) and tests A, B, and E (P < 0.001). Tests C and E were significantly more sensitive than tests A and B (P < 0.001).

TABLE 1.

Summary of sensitivity data for the different tests^a

Serum panel (no. of animals)	% Positive (95% CI) by test:
	A	B	C	D	E	AGIDT
Clinical sheep (62)	30.6 (19-42)	30.6 (19-42)	63.9 (52-76)	98.4 (95-100)	58.0 (46-70)	86.9 (78-95)
Farms with clinical sheep (99)	56.6 (47-66)	59.6 (50-69)	87.9 (81-94)	90.9 (85-97)	84.8 (78-92)	86.9 (80-94)
Clinical goats (18)	94 (84-100)	94 (84-100)	56 (33-78)	94 (84-100)	78 (59-97)	67 (45-88)
Farms with clinical goats (50)	80 (69-91)	78 (67-89)	50 (36-64)	80 (69-91)	74 (62-86)	66 (53-79)
Sheep, noncertified farms (20)	20 (2-38)	20 (2-38)	40 (19-61)	50 (28-72)	45 (23-67)	25 (6-44)
Goats, noncertified farms (43)	48.8 (34-64)	48.8 (34-64)	48.8 (34-64)	58.1 (43-73)	58.1 (43-73)	44.2 (29-59)

^aEquivocal results were considered to be evenly split, with half counted as positive and half counted as negative. CI, confidence interval.

For the serum panel from infected sheep flocks (with clinical cases), no statistically significant differences were found between sensitivities of the AGIDT and tests D, C, and E, except for a small significant difference (P = 0.041) between tests D and E. However, these four tests all were significantly more sensitive (P < 0.001) than tests A and B.

For the serum panel originating from clinical goats, no statistical analysis was performed due to the small number of animals (n = 18).

For the serum panel originating from infected goat flocks (with clinical cases), a significant difference in sensitivity was observed only between tests D, A, and B on the one hand and test C on the other hand (P = 0.013), the latter one being less sensitive.

The category of sheep sera from nonaccredited flocks (n = 20) was not statistically analyzed because of the small number of animals and the unknown status of these flocks. Test D detected the highest seroprevalence in this group. The category of goat sera from nonaccredited flocks (n = 43) was not statistically analyzed because of the small number of animals and the unknown status of these flocks. Tests D and E detected the highest seroprevalence in this group.

Technical and statistical specificity.

No significant differences were observed with respect to both technical and statistical specificity for all tests (Table 2). However, the total number of samples tested was limited.

TABLE 2.

Summary of specificity data for different tests^a

Serum panel (no. of animals)	% Negative (95% CI) by test:
	A	B	C	D	E	AGIDT
Iceland sheep (99)	100	100	94.9 (91-99)	98 (95-100)	100	100
Accredited sheep flocks (80)	97.5 (94-100)	98.8 (96-100)	97.5 (94-100)	96.3 (92-100)	98.8 (96-100)	100
Accredited goat flocks (55)	100	100	100	100	98.1 (93-100)	100
Prechallenge goats (34)	100	100	100	100	100	100

^aEquivocal results were considered to be evenly split, with half counted as positive and half counted as negative. CI, confidence interval.

All sheep sera were completely negative only in the AGIDT, whereas for the samples from Iceland, 5/99 sera gave a false-positive result with test C, and none gave a false-positive result in tests A and B. All goat sera were negative in all tests, except for one sample in test E.

Detection limit.

Sheep samples (n = 3) collected at day 67 or day 81 and goat samples (n = 3) collected at day 55 after experimental infection were titrated in homologous negative serum (Table 3). Test D demonstrated the lowest (best) detection limit for sheep samples, while test E showed the best results for goat samples.

TABLE 3.

Detection limits for goat and sheep samples

Test	Antibody titer for samplea
	Sheep			Goat
	1	2	3	1	2	3
A	<2	<2	<2	8	2	8
C	4	2	<2	8	<2	4
D	128	8	16	16	2	16
E	<2	<2	<2	32	4	32

^aData are titers of positive samples from three sheep and three goats, twofold serially diluted in negative serum. The cutoff was chosen at the +/± level. (i.e., a positive or, if relevant, ambiguous level).

Sequential sera after experimental infection.

Sequential sera were tested from four sheep and 30 goats after experimental infection. The mean time of seroconversion is presented in Table 4. Using test D, seroresponses in sheep samples were detected consistently earlier than in the other ELISAs and the AGIDT. Tests A and B detected seroresponses relatively late or not at all in the time frame of the experiment, and the same result was found for the AGIDT.

TABLE 4.

Mean time to seroconversion after experimental infection of sheep and goats

Experimental group (no. of infected animals)	Mean time (days) to seroconversion as measured by test:
	A	B	C	D	E	AGIDT
Sheep (4)	>72.3	>71.5	66.5	41.0	64.5	>62.8
Goats (30)	34.5	33.4	>45.9	38.5	36.3	>88.5

For goat samples, tests A, B, D, and E demonstrated comparable seroconversion times, while test C scores took slightly longer. The mean time for seroconversion in goats using the AGIDT was considerably prolonged.

In the time frame of the experiment, the tests C, D, and E detected all of the sheep samples, while tests A, B, D, and E detected all of the goat samples (data not shown).

Intraplate and interplate variance.

For each test, a positive serum from an experimentally infected goat was diluted to obtain two concentrations, one concentration corresponding to a test result near the cutoff (RC1), and another concentration corresponding to a result of 50% of the ELISA signal range (RC2). Because variation coefficients are strongly influenced by the signal strength (optical density) of the sample, variation coefficients also were calculated by dividing the standard deviation by the cutoff as defined by the manufacturer. Intraplate variance (the percentage of the standard deviation/mean) ranged between 4.9% (test A) and 13.4% (test E). Interplate variance (the percentage of the standard deviation/mean) ranged between 10.6% (test E) and 13.6% (test C). Intraplate variance (the percentage of the standard deviation/cutoff) ranged between 2.5% (test A) and 15.3% (test E). Interplate variance (the percentage of the standard deviation/cutoff) ranged between 6.5% (test A) and 11.8% (test E).

Agreement of AGIDT with the ELISAs.

Generally, the AGIDT is considered highly specific and is often used as a reference test (gold standard), but its sensitivity is considered limited, which is supported by some of the results presented here. Therefore, the AGIDT cannot be used as a gold standard. However, to give an impression of the correlation between AGIDT and the ELISAs investigated, κ values and relative sensitivities of ELISAs using AGIDT as the gold standard are presented (Table 5). All Dutch field samples were combined for this comparison (sheep, n = 261; goats, n = 166).

TABLE 5.

Levels of agreement beyond the level of chance (κ values) and sensitivities of different ELISAs compared to the AGIDT

Animal group	Test	κ value	Sensitivity
Sheep	A	0.40	49.3
	B	0.42	53.6
	C	0.74	86.4
	D	0.79	98.6
	E	0.70	82.9
Goats	A	0.77	96.9
	B	0.76	96.9
	C	0.76	84.4
	D	0.78	100
	E	0.76	95.3

ROC analysis.

Results of all ELISAs were significantly correlated, and these correlations were taken into account when testing for statistically significant differences between AUCs. For sheep, the AUC of test D (0.93) was significantly higher (P < 0.001) than those of the other tests, while the AUCs of tests A (0.91) and E (0.90) were significantly higher (P < 0.001) than those of tests B and C (0.86 and 0.85, respectively). For goats, the AUC of test A (0.87) was significantly higher (P = 0.005) than those of the other tests, while the AUC of test D (0.84) was significantly higher (P = 0.008) than those of tests B, C, and E (0.83, 0.79, and 0.81, respectively). Finally, the AUC of test B was significantly higher (P = 0.044) than the AUC of test C.

Laboratory practicability.

Other features of the test kits also may be of importance. Some criteria are very important with respect to laboratory logistics and practicability, like the format of the kit, product information, incubation steps, ready-to-use reagents, incubation temperatures, sample dilutions, conjugate dilutions, extra pipetting steps, stopping time, interpretation, and suitability for automation. Although the evaluation of these criteria is partly subjective, for each criterion a score of 1 (lowest) to 3 (highest) was given. Total scores ranged between 37 and 45, with the best results being for tests C (45) and A (44) with respect to overall laboratory practicability.

DISCUSSION

ELISAs for the detection of SRLV infection can be broadly categorized into indirect ELISAs that use whole virus, recombinant proteins, and/or synthetic peptides as antigens and competitive ELISAs that use monoclonal antibodies (3). In this study, we compared a number of commercially available ELISAs representing these different formats. All serum samples also were tested by AGIDT, which was, until recently, the prescribed test for regulatory purposes as specified by the Office International des Epizooties.

For serological diagnosis of clinical sheep, test D was more sensitive than the AGIDT and was significantly more sensitive than the other ELISAs. This may be due to the simultaneous detection of antibodies directed against the proteins p25 and gp46 in this test (3, 17), whereas in tests C and E only antibodies against p25 are detected. Although antibodies against p25 arise relatively early after infection, the levels of these antibodies seem to decline once clinical signs appear. Antibodies against the envelope glycoproteins gp46 and gp135 arise later after infection but also seem to persist in the clinical phase (1, 3, 10).

The whole-virus indirect ELISAs A and B, however, showed a poor sensitivity for detection of clinical sheep samples but showed a much better sensitivity for detection of infected goat samples and for the panel originating from clinical cases, reflecting the choice of a CAEV strain for the coating of the test microplates (24).

In a small category of sheep sera from nonaccredited flocks, the highest seroprevalence was detected with test D. Lower detection limits and earlier seroconversions also sustain a higher analytical sensitivity for test D than for tests C and E, and this corroborates results presented by Boshoff et al. (1). These authors demonstrated a higher level of sensitivity for serodiagnosis of MVV infections for a combined recombinant p25/transmembrane protein gp46 indirect ELISA than for a viral core protein p25-based ELISA (test E).

In goat flocks for which clinical cases were diagnosed, the seroprevalence as measured by tests D, A, and B again was relatively high. The relatively poor sensitivity of test C may be explained by the observation that, for goats, a higher immunoreactivity for the transmembrane glycoprotein than for the core protein has been reported (2).

For sheep samples, test D showed the lowest detection limit, while test E showed better characteristics for goat samples. Test D also detected seroresponses after experimental infection in sheep consistently earlier than the other tests, while goat samples are effectively detected by all of the ELISAs, except for test C, and the AGIDT.

No significant differences between the tests with respect to specificity were found. Intra- and interplate variances were lowest for test A but did not show significant differences for the other ELISAs.

The most commonly used serological test for routine screening is the AGIDT (3). In the AGIDT for sheep, the test is read only for antibodies against gp135, since precipitation lines are judged for confluence with a monospecific anti-gp135 serum. The AGIDT is by no means a good gold standard for sheep and goat samples, since the results of many publications show a consistent pattern of lower sensitivity values (about 70% relative sensitivity) for the AGIDT than for the ELISAs. Furthermore, the test is quite laborious and consequently is expensive, and it is not suited for automation (3, 16, 18). The specificity of the AGIDT, however, is considered very high, and therefore the test often is used as a confirmatory test. More-sophisticated alternatives for confirmation, such as immunoblotting and PCR techniques, did not find wide acceptance up to now because of complexity of the methods, resulting in difficult standardization.

Overall, it was concluded that the sensitivities of the tests were correlated with the animal species under investigation and the category of infected animals, while all tests showed similar specificities. The indirect test D, of which the solid phase is sensitized with a combination of the core protein p27 of MVV produced in E. coli and a peptide derived from the transmembrane protein gp46, appeared to be the test of choice for serodiagnosis of SRLV infections in sheep and goats.