Validation of the Indirect MAP1-B Enzyme-Linked Immunosorbent Assay for Diagnosis of Experimental Cowdria ruminantium Infection in Small Ruminants

Abstract

The major antigenic protein 1 fragment B (MAP1-B) enzyme-linked immunosorbent assay (ELISA) for the diagnosis of Cowdria ruminantium infections was validated to determine cutoff values and evaluate its diagnostic performance with sheep and goat sera. Cowdria-infected populations consisted of 48 sheep and 44 goats, while the noninfected populations consisted of 64 sheep and 107 goats. Cutoff values were determined by two-graph receiver-operating characteristic (TG-ROC) curves. The cutoff value was set at 31 and 26.6% of the positive control reference samples for sheep and goat sera, respectively. The test’s diagnostic performance was evaluated with measurements of the area under the concentration-time curve (AUC) of the ROC curves and by the valid range proportion (VRP). The AUCs were 0.978 for sheep sera and 0.989 for goat sera. The VRP for both sheep and goat sera was approximately 1.0. The intermediate range (IR), which defines results that are neither positive nor negative, was 0 for goat sera and 2.81 for sheep sera. In an ideal test, the AUC and VRP would be 1.0 and the IR would be 0. In this study these parameters were close to those of an ideal test. It is concluded that the MAP1-B ELISA is a useful test for the diagnosis of C. ruminantium infection in small ruminants.

Cowdriosis (or heartwater) is a tick-borne disease of ruminants caused by the rickettsia Cowdria ruminantium and is transmitted by ticks of the genus Amblyomma. The disease is endemic in sub-Saharan Africa and the Caribbean and is a main obstacle to livestock development in the tropics (7, 30). Clinical signs and macroscopic postmortem changes are not pathognomonic for the disease, and diagnosis is based on the detection of rickettsial organisms in the cytoplasms of endothelial cells in brain capillaries. Antemortem tests for detecting C. ruminantium include animal subinoculation, cell culture isolation, serodiagnostic tests, DNA hybridization, and PCR. Serodiagnostic methods, such as the indirect fluorescent antibody test, immunoblotting, and enzyme-linked immunosorbent assays (ELISA), have been hampered by cross-reactions with Ehrlichia species (18, 22, 24). However, the use of recombinant major antigenic protein 1 (MAP1) of C. ruminantium has been recently introduced, and an indirect ELISA based on a specific fragment of this protein (fragment B, referred to herein as MAP1-B) has been developed (32). Cross-reactions were dramatically reduced, although sera from dogs infected with Ehrlichia canis and sera from human patients infected with Ehrlichia chaffeensis were also positive in this ELISA. Another recent study, using a monoclonal antibody-based ELISA for detecting MAP1, confirmed cross-reactions with E. canis, E. chaffeensis, and a newly discovered Ehrlichia-like organism from white-tailed deer (21). Furthermore, it has also been shown that E. chaffeensis can experimentally infect wild ruminants such as white-tailed deer (5). A preliminary validation of the MAP1-B ELISA was done by studying antibody profiles of C. ruminantium infections in domestic ruminants (25, 32).

Central to any serological assay is the determination of the diagnostic cutoff value. It is common practice to determine cutoff values for (i) reactions of a noninfected reference population with the addition of 2 or 3 standard deviations to the mean value or (ii) the doubling of the mean optical density readings of the negative reference sera on each ELISA plate (26). The first method is assumed to lead to a specificity of 97.5% (2); however, this assumption holds true only for normally distributed test variables (12), and the second method seems to have no statistical grounds. A cutoff value has to differentiate two subpopulations of infected and noninfected controls with defined operating characteristics (13). Recently a new approach to defining test cutoff values and performance had been proposed (13). The new approach utilizes the conventional receiver-operating characteristic (ROC) principle, modified in such a way that the test sensitivity and specificity can be read directly from these plots, unlike the conventional ROC plots. The modified ROC plot is known as a two-graph ROC (TG-ROC). TG-ROC was developed as a template within a standard spreadsheet computer program, and it provides a clear and comprehensible approach to the problems of selecting cutoff values and identifying intermediate results in ELISA tests (10). TG-ROC analysis also provides other indices, such as efficiency (9), Youden’s index (35), and likelihood ratio (LR) (28), for further cutoff value optimization. These indices are useful measures for minimizing the number of false positives and false negatives.

The aims of this study were (i) to calculate cutoff values for the MAP1-B ELISA for the diagnosis of cowdriosis with TG-ROC, (ii) to compare these values to those determined by conventional methods with sheep and goat serum samples, and (iii) to compare the performance of the MAP1-B ELISA for the diagnosis of cowdriosis in experimentally infected sheep and goats.

MATERIALS AND METHODS

C. ruminantium isolates.

The following C. ruminantium isolates (from the following locations) were used in this study: Senegal (Senegal), Lutale (Zambia) (19), Umpala (Mozambique) (1), Gardel (Guadeloupe) (31), and Crystal Springs (Zimbabwe) (4) and Ball 3 (14), Kümm (8), Kwanyanga (23), and Welgevonden (6) (all from South Africa).

Experimental animals.

Forty-eight adult female Tesselaar sheep, all nonpregnant and 12 to 18 months old, were used as the infected reference sheep population. The animals were challenged with different C. ruminantium isolates by needle infection 1 month after vaccination with an attenuated Cowdria isolate originating from Senegal (17, 20). Twenty-four sheep were challenged with the Senegal isolate, four with Welgevonden, and five sheep each with the Umpala, Lutale, Gardel, and Ball 3 isolates. The sera used in this study were collected between 4 and 8 weeks postchallenge. The infected reference population of Saanen goats was composed of 44 goats, of both sexes and 12 to 18 months old, experimentally infected by needle challenge with one of several isolates of C. ruminantium: Gardel (n = 1), Senegal (n = 16), Lutale (n = 1), Ball 3 (n = 5), Kwanyanga (n = 5), Kümm (n = 8), Crystal Springs (n = 3), and Welgevonden (n = 5). The infective dose of the different isolates was previously determined in experimental animals. All animals were tested serologically prior to infection and were shown to be negative. The animals had never been exposed to ticks and were born and bred in The Netherlands. The noninfected reference population of sheep consisted of 64 adult Tesselaar sheep, and the noninfected reference population of goats consisted of 107 Saanen goats. As with the infected reference population, the noninfected animals had never been exposed to ticks and were born and bred in The Netherlands.

Recombinant MAP1-B antigen.

The immunogenic region of the MAP1 protein (MAP1-B) was cloned and expressed in Escherichia coli with expression vector pQE9, as a fusion protein with six histidine residues added at the N terminus (32). Recombinant MAP1-B was purified with Ni²⁺-nitrilotriacetic acid agarose under denaturing conditions as described by the manufacturer (Qiagen Inc., Chatsworth, Calif.).

ELISA.

One hundred microliters per well was used in all the steps described below. MAP1-B antigen was diluted (1.4 μg/ml) in coating buffer (15 mM Na₂CO₃, 35 mM NaHCO₃ [pH 9.6]) and immobilized onto 96-well ELISA plates (Microlon Multibind immunoassay plates; Greiner Labortechnik, Alphen aan den Rijn, The Netherlands) by incubation for 1 h at 37°C and then stored overnight at 4°C. Plates were incubated for 15 min at 37°C with blocking buffer (phosphate-buffered saline [PBS], pH 7.3, supplemented with 0.1% Tween 20 and 1% nonfat dry milk [PBSTM]) (Protifar; Nutricia, Zoetermeer, The Netherlands). Plates were washed three times with PBS supplemented with 0.1% Tween 20 (PBST) and subsequently incubated with sera (diluted 1:200) in PBSTM for 1 h at 37°C. All samples were analyzed in duplicate on the same plate. Plates were washed three times with PBST and incubated for 1 h at 37°C with rabbit anti-goat or rabbit anti-sheep antibodies conjugated with horseradish peroxidase (RαG/IgG[H+L]/PO or RαSh/IgG[H+L]/PO; Nordic, Tilburg, The Netherlands) diluted in PBSTM (rabbit anti-goat antibodies, 1:1,500; rabbit anti-sheep antibodies, 1:1,750). ELISA plates were washed three times with PBST, and freshly prepared ABTS [2,2′-azinobis (3-ethylbenzthiazoline sulfonic acid)] substrate was added.

Color development was allowed for 30 min in the dark, and absorbance was measured at 405 nm with an ELISA reader (Ceres UV 900 C; Biotek Instruments BV, Abcoude, The Netherlands). Each plate contained one positive and one negative reference serum sample. The means of the duplicate measurements were calculated, and the optical density was expressed as a percentage positive (PP) value of the reference positive control.

ROC plots.

ROC plots were constructed by using the vector of 1-Sp_j, where Sp_j is the test specificity at a cutoff value (d_j) on the x axis and the vector of corresponding sensitivity (Se_j) values on the y axis.

The performance of the test was evaluated by calculating the area under the concentration-time curve (AUC) of the ROC plot. The AUC is given by the test statistic U of the Mann-Whitney U test (15) and is determined by the following equation: AUC = U/n_p · n_n, where U = n_p · n_n + [n_n · n_n + 1)/2] − R, R is the rank sum of the noninfected sample, and n_p and n_n are the numbers of infected and noninfected animals, respectively.

TG-ROC analysis.

TG-ROC analysis was carried out with the template described by Greiner et al. (13). For the construction of TG-ROC plots, the measurement range (MR), as observed for each pair of reference populations, was evenly divided into 250 intervals, with the resulting limits termed cutoff values (d_j). Se_j and Sp_j were calculated for each threshold d_j value obtained. The resulting matrix of d_j and the corresponding percentages of Se_j and Sp_j were plotted to represent the two observed parameters over the specified range of PP values. The intersection of the sensitivity curve with the specificity curve is at (d₀,θ₀) and is called the “point of equivalence,” whereby cutoff value d₀ yields equivalent test parameters (Se_j = Sp_j = θ₀). Two alternative cutoff values, which are the lower and upper limits of the intermediate range (IR), are defined at an accuracy level of 95%. In this study, the IR upper and lower limits were nonparametrically defined as the 95th and 5th percentile of the noninfected and infected reference populations, respectively. The valid range proportion (VRP) was determined as (MR− IR)/MR, where MR is calculated by PP_max− PP_min, with PP_max and PP_min being the highest and lowest ELISA PP values in the combined populations of infected and noninfected animals for each species.

The results for the noninfected reference populations of sheep and goats were used to calculate cutoff values (means ± 2 or 3 standard deviations). The negative reference sample on the ELISA test plates was used to determine cutoff values by the method of twice the negative.

Efficiency, Youden’s index, and LRs.

Three indices were calculated for further cutoff value optimization. Efficiency (at cutoff value d_j) was calculated as follows: Ef_j = P · S_e + (1 − P) · Sp_j, where P denotes the proportion of the reference infected sample. Youden’s index was determined by the following equation: J_j = (ad − bc)/(a + b)(c + d), where the sum of a and b is the number of infected animals a is the number of correctly diagnosed infected animals and b is the number of false negatives) and the sum of c and d is the number of noninfected animals (d is the number of correctly diagnosed animals and c is the number of false positives) at cutoff value d_j. Positive and negative LRs (LR+ and LR−, respectively) for each cutoff value (d_j) were calculated by the following equations: LR+ = Se_j/(1 − Sp_j) and LR− = (1 − Se_j)/Sp_j. The ratios were logarithmically transformed to give a symmetry, with a log(LR+) of 0 and a log(LR−) of 0 for a test yielding no information and a log(LR+) of ∞ and a log(LR−) of −∞ for an ideal test. The values of the indices were then plotted against the cutoff value.

RESULTS

PP values for the infected and noninfected reference sheep and goat populations were tested for normality and showed significant skewness (P < 0.05). Therefore, the nonparametric option of the TG-ROC analysis was used (13). Table 1 summarizes the results of the MAP1-B ELISA for the populations of sheep and goats. The cutoff values resulting in equal sensitivity and specificity, as well as two alternative cutoff values for definition of the IR, were read directly from the TG-ROC plot in Fig. 1 and are shown in Table 2. The VRP was approximately 1.0 for sheep as well as for goats. The IR for goat sera was zero, because at cutoff value d₀, sensitivity and specificity are both greater than 95%. The sensitivity and specificity measures of the test at cutoff value d₀ are shown in Table 3. Calculated cutoff values for sheep and goats varied considerably according to the different methods shown in Table 3. The cutoff values calculated by TG-ROC analysis for sheep (d₀ = 31.0) and for goats (d₀ = 26.6) were close to those calculated as the mean plus twice the standard deviation (assuming normal distribution of the data). The performance of the test as measured by the AUC of the ROC plots was very close to 1: 0.978 for sheep and 0.989 for goats (Fig. 2). The LRs in Table 3 were calculated from the TG-ROC analysis and are displayed in Fig. 3, which shows graphs of the LRs over the entire range of possible cutoff values within the measurement range. The efficiency of the test, measured by efficiency and Youden’s index, is shown in Fig. 4. The closer to 1 the indices are, the better the test’s performance at a given cutoff value, d_j.

TABLE 1.

Descriptive indices for the results of the MAP1-B ELISA for infected and noninfected reference populations of sheep and goats^a

Measurement	% of sera positive from:
	Infected		Noninfected
	Sheep (48)	Goats (44)	Sheep (64)	Goats (107)
Mean	77.1	83.4	18.8	11.5
Median	81.6	81.1	18.9	9.2
SD	21.2	35.6	8.8	7.7
Minimum	18.2	10.3	8.1	3.3
Maximum	115.2	148.5	54.5	37.6

^aResults are expressed as percentages of an internal positive control. The number of animals in each group is indicated in parentheses. 

FIG. 1.

TG-ROC analysis of MAP1-B ELISA results for sheep (a) and goat (b) sera. The IR is determined by using one cutoff value at 95% sensitivity (Se) and another at 95% specificity (Sp).

TABLE 2.

Results of TG-ROC analysis^a

Measurement	Result of TG-ROC analysis (CI) on sera from:
	Sheep	Goats
θ0	94.5	95.4
d0	31.0 (27–50)	26.6 (20–36)
IR	2.8 (0–36)	0
Upper limit	31.7 (27–54)	26.2 (20–36)
Lower limit	28.8 (18–45)	26.5 (19–45)
VRP	0.97 (0.662–1)	1

^aθ₀, point of equivalence where specificity is equal to sensitivity at cutoff value d₀; lower limit, 5th percentile of the percent positivity of the infected population; upper limit, 95th percentile of the percent positivity of the noninfected population. The 95% confidence internal (CI) is shown where appropriate. 

TABLE 3.

Cutoff values determined by different methods with their corresponding sensitivities, specificities, and LRs^a

Result	Source of sera	Mean + 2 SDs	Mean + 3 SDs	2× negative	TG-ROC
Cutoff	S	36.34	45.11	17.54	31.00
	G	26.90	31.26	13.48	26.60
Sensitivity (%)	S	94	87	100	94.50
	G	90	90	100	95.40
Specificity (%)	S	97	98	50	94.50
	G	95.40	98	80	95.40
LR+	S	4.48	6.05	1.22	2.27
	G	3.47	4.48	1.65	3.32
LR−	S	0.30	0.37	0.00	0.37
	G	0.29	0.37	0.00	0.37

^aS, sheep sera; G, goat sera; SD, standard deviation for the noninfected reference population; mean, mean value of the negative reference population; 2× negative, twice the value of the negative reference population; LR+, positive LR; LR−, negative LR. The sensitivities, specificities, and LRs were calculated by TG-ROC analysis. 

FIG. 2.

ROC plots of MAP1-B ELISA results for sheep (a) and goat (b) sera. The AUCs are 0.978 and 0.989, respectively (maximum AUC = 1.0).

FIG. 3.

Logarithm of negative (LR−) and positive (LR+) LRs plotted as a function of the selected cutoff value with sheep (a) and goat (b) sera.

FIG. 4.

Youden’s index (J) and efficiency (Ef) of the MAP1-B ELISA as a function of the selected cutoff value for sheep (a) and goat (b) sera.

DISCUSSION

The aims of this study were to determine cutoff values by TG-ROC analysis, to compare them to those obtained by previously used methods, and to evaluate the performance of the MAP1-B ELISA for the diagnosis of C. ruminantium infections in sheep and goats.

The establishment of a reliable cutoff value is essential for a serological test to be useful in differentiating infected and noninfected animals. The assembly of the reference population used for the calculation of a cutoff value is a critical procedure: the sample size has to be large enough to provide the desired statistical power, and moreover, the reference population has to be representative of the target population (12). In order to compensate for various factors that may influence the diagnostic sensitivity and specificity, R. Jacobson (16) suggested the use of at least 300 known positive and 1,000 known negative samples, which numbers are very difficult to obtain under experimental conditions. We conducted this study with the minimal number of positive samples required for a meaningful analysis. We did not have any further positive samples in stock (sera from 48 sheep and 44 goats), but a significant number of noninfected samples (sera from 64 sheep and 107 goats) were available. The latter precondition is very difficult to realize for ELISA tests designed for the screening of tropical infectious diseases: control sera from animals in regions where heartwater is not endemic are guaranteed to be disease free but might not be representative of the target population; on the other hand, negative sera from animals in regions where heartwater is endemic might not be guaranteed to be disease free (33, 34). Previously, determination of cutoff values for the MAP1-B ELISA has mainly been done by doubling the PP value of a reference noninfected sample included on each plate. As shown in Table 3, doubling the PP value of the reference noninfected serum results in a very low specificity and low positive LR for the test. Hence, a single reference sample can serve as an internal test control but can hardly be considered an adequate representation of a noninfected population. Mondry et al. (25) based their cutoff values for the MAP1-B ELISA on the frequency distribution of PP values for a noninfected population in the Caribbean. In their study, cutoff values were determined graphically on the basis of an acceptable number of false-positive results. The authors, however, did not explain how an acceptable number of false positives was defined. The values obtained were fixed at 50% positive for sheep and goats. An overall specificity of 99.4% was reported, but the effect of the cutoff value on the test sensitivity was not investigated in a large enough population of known infected animals.

TG-ROC plots are graphs that show the relationship between the sensitivity and specificity of a test wherein the definition of a positive test is modified over the entire range of obtained values. ROC curves make it possible to compare the quality of the tests with the quality of other quantitative tests and allow a systematic and objective choice of optimal cutoff values (29). Reporting only one value for sensitivity and specificity provides a possibly misleading and even hazardous oversimplification of accuracy. Similarly, calculating just a few sensitivity and specificity pairs provides only a glimpse of a test’s real diagnostic abilities (36). The TG-ROC method was originally tested on data obtained with an ELISA for the detection of antibodies to Borrelia burgdorferi (13) and also was used to evaluate another ELISA test for the diagnosis of maedi-visna virus (3), giving encouraging results.

Given the difficulties of obtaining animals not exposed to Cowdria or ticks in areas where cowdriosis is endemic, we recommend that the cutoff values for these areas be those given by the mean plus 2 standard deviations, if negative samples from regions of endemicity are available, or to use cutoff d₀ (31.0 and 26.6% positive for sheep and goats, respectively [Tables 2 and 3]), if no such sera are available. The method using the mean plus 2 standard deviations gave results that are quite similar to those obtained by the TG-ROC method. However, a cutoff value should be determined with defined diagnostic accuracy (13), which is not the case with the conventional methods. In this study, the values of d₀ correspond to the efficiency and Youden’s index’s highest values (Fig. 4). In TG-ROC plots (Fig. 1) an option is given such that the cutoff values can be chosen to suit the required level of accuracy and the effect of the selected cutoff value on the sensitivity and specificity of the test can be read directly from the plots. Likewise, the efficiency, Youden’s index, and LRs for the MAP1-B ELISA can be read directly from the plots in Fig. 3 and 4 for a selected cutoff value. Youden’s index has a value of 0 whenever a diagnostic test gives the same proportion of positives for both infected and noninfected groups (35).

The IR used to describe nonpositive and nonnegative test results was 2.81 for sheep and 0 for goats. The IR is 0 in cases where the θ₀ is greater than 95%, because the lower limit of the IR is greater than the upper limit. In this study, the IR for goats was 0; hence, over 95% of the goats were correctly diagnosed. The interpretation of intermediate test results depends on the specific diagnostic purpose of the test. Because of the ambiguity of borderline results, it is appropriate to consider only one cutoff value and indicate the test parameters (Se, Sp, and LR) for a given cutoff value selected for an epidemiological situation. In clinical diagnosis, the values that fall between the IR limits would require testing by a confirmatory assay or retesting for detection of seroconversion (16, 27).

The VRP and θ₀ are independent of any selected cutoff value and are, therefore, good measures for test comparison (13). In this study the VRP and θ₀ were reasonably high (Table 2) for both sheeps and goats, indicating the high performance of this ELISA in classifying the animals according to their true health status. It can be concluded that 95% of individual test results are valid, because the VRP was close to 1.0 for both species.

Another convenient way to quantify the diagnostic accuracy of a test is to express its performance by AUC measurements of ROC plots. This is a quantitative, descriptive expression of how close the ROC curve is to the perfect one (AUC = 1.0) (36). AUCs in ROC curves provide an index of accuracy by demonstrating the limits of a test’s ability to discriminate between the alternative state of health and the complete spectrum of operating conditions, unlike in TG-ROC plots, where the VRP is limited to 95% accuracy. The MAP1-B ELISA showed high performance because the index AUCs were 0.978 and 0.989 for sheep and goat sera, respectively. From these results (VRP, AUC, and IR), the test appears to have no differences in its diagnostic performance for sheep and goats.

Decisions regarding cutoff values for this ELISA should be reviewed as more data become available, since experimental infections sometimes produce an overoptimistic estimate of accuracy. Analysis similar to that done with sheep and goat sera needs to be done for bovine samples, and work on this has already been started in our laboratory. The effect of the cutoff values on the antibody profiles of ruminants should also be investigated for further cutoff value optimization.

Studies have shown that age is positively correlated with seropositivity but not with the detection of the parasite when a Trypanosoma antibody-detecting ELISA was used in an area in Uganda where trypansomiasis is endemic (11). In addition to age, many other factors (such as sex, breed, state of pregnancy, nutritional state, previous chemotherapy, passive immunization, and self-cured infections) may also influence the cutoff value. Further validation of the test precision needs to be done according to the ISO 5725-1986 international procedure, with interlaboratory comparisons of the ELISA results.

In this study we have attempted to calculate cutoff values by using known positive and negative experimental sera. It will be of great interest to repeat this study using samples from animals exposed to infected Amblyomma ticks under field conditions to check whether our cutoff values (determined with experimental animals) are also applicable to the situation in the field.