Interpretation of a Commercial Bovine Paratuberculosis Enzyme-Linked Immunosorbent Assay by Using Likelihood Ratios

Abstract

Evidence-based medicine encourages the use of quantitative diagnostic test results to estimate the probability of a particular diagnosis. Likelihood ratios (LRs) are among the best tools for maximizing the diagnostic information gained from diagnostic assays that provide results on a continuous scale. They provide the odds that an animal with a particular test result actually has the disease in question based on the magnitude of the test result. A commercial enzyme-linked immunosorbent assay (ELISA) was used to test sera from 143 dairy cattle infected with Mycobacterium paratuberculosis and 2,974 cattle free of this infection. This assay transforms ELISA reader optical density values into sample-to-positive (S/P) ratios. The LR was calculated for S/P results from 0.00 to 1.00 at 0.05-S/P unit intervals. LRs were directly but not linearly correlated with ELISA S/P ratios (r² = 0.94). The mathematical function describing the relationship between the ELISA S/P ratio and the LR was LR = 265 × (S/P value)^2.03. LRs were also directly related to the frequency of animals testing positive for paratuberculosis by fecal culture and other serologic tests. Based on these LRs, guidelines for interpretation and application of this ELISA for the diagnosis and control of paratuberculosis in dairy cattle herds are recommended.

The principal goal of a diagnostic test is to help practitioners increase the probability of a correct diagnosis. Predictive values are useful in this regard but require the estimation of disease prevalence (i.e., pre-test probability of disease) in the population (27) and only utilize categorical test results, positive or negative. Food-producing animals exist in numerous discrete populations (herds or flocks), and the prevalence of disease can differ greatly among them. Infection prevalence significantly affects predictive values, i.e., the positive predictive value of tests is low when prevalence is low, and the negative predictive value is low when disease prevalence is high, thus the predictive values of tests can vary significantly among herds.

Evidence-based medicine (EBM) is the scrupulous, explicit, and judicious use of the best evidence available in making decisions about the care of individual patients. The practice of EBM means integrating clinical expertise with the best available clinical evidence from systematic diagnostic research (19). Additional information on EBM is available at the Learning and Information Services website on EBM (http://www.herts.ac.uk/lis/subjects/health/ebm.htm). The likelihood ratio (LR) is one example of external clinical evidence and a powerful tool in EBM. LRs give the same information as predictive values but can be used independent of pre-test disease prevalence (5).

The purpose of the study was to generate an algorithm for the calculation of LRs for subclinical Mycobacterium paratuberculosis infection in dairy cattle (Johne's disease) based on an enzyme-linked immunosorbent assay (ELISA) for serum antibodies and to show its utility in the control of this economically important and potentially zoonotic infectious disease (1, 6, 10, 17).

MATERIALS AND METHODS

Serum samples.

Samples originated from two well-defined dairy cattle populations. Sera from 143 subclinically M. paratuberculosis-infected cows were part of a previously described specimen repository (23). The case definition for M. paratuberculosis-infected cattle was the isolation of M. paratuberculosis by fecal culture and/or histopathologic evidence of infection. Sera from uninfected cows included 760 samples from U.S. dairy cattle and 2,214 samples from Dutch dairy cattle. These cattle were from herds free of M. paratuberculosis infection as defined by a minimum of three negative annual whole-herd (all cattle, ≥2 years old) fecal cultures.

ELISA.

An M. paratuberculosis antibody test kit (IDEXX Laboratories, Inc., Westbrook, Maine) was used to test all 3,117 sera according to the manufacturer's instructions. With this kit, optical density (OD) values were transformed to S/P ratios based on the OD for the serum sample together with those for the negative and positive controls provided with the kit by using the following equation: S/P ratio = (OD of sample − OD of negative control)/(OD of positive control − OD of negative control). All assays were run in duplicate. Any assay with a between-well coefficient of variation of >10% was repeated, and the second result was used for data analysis.

Other tests for paratuberculosis.

ELISA results were compared to those of other tests for paratuberculosis run on samples collected at the same time from the same cattle. Fecal culture was done both by the BACTEC system (2) and by using conventional solid medium (Herrold's egg yolk agar) (29). Serum antibody measurements were done by the complement fixation test (12), agar gel immunodiffusion assay (20), and another commercial ELISA kit (Paracheck; Biocor Animal Health, Omaha, Nebr.) (3). These tests were performed independently and simultaneously at the time the original samples were collected. Results of these other diagnostic tests for paratuberculosis were reported previously and not done specifically for the present study (22-24).

Data analysis.

Frequency distributions for S/P values on the sera from infected and noninfected populations were tabulated in intervals of 0.05 S/P units. At each interval, the sensitivity (Se), specificity (Sp), and LR [Se/(1−Sp)] of the ELISA were calculated. Linear and nonlinear regression analysis was used to determine the equation describing the line best fitting the plot of the S/P cutoff value versus LR (Lotus Freelance Graphics release 9.6 for Windows; Lotus Development Corp.). The regression model providing the highest r² value was considered to best fit the data.

RESULTS

There was a small difference in ELISA false-positive rates between the U.S. and Dutch cattle free of M. paratuberculosis; maximum ELISA S/P values were 1.15 and 1.10 for the two groups, respectively, and the Sp values at the manufacturer's recommended S/P cutoff values of 0.25 were 98.7 and 96.3% for the U.S. and Dutch M. paratuberculosis-free cattle, respectively. Pooling the data for these two populations improved the precision of LR estimates and potentially made the findings more universally applicable across countries.

Estimated Se, Sp, and LR increased with increasing ELISA S/P cutoff values (Table 1). Regression analysis of ELISA S/P ratio category versus LR was attempted using linear, exponential, logarithmic, and power functions. The S/P-to-LR relationship that best fit the data was described by the power function LR = 265 × (S/P value)^2.03. This function fitted the data with an r² value of 0.94 (Fig. 1). Other regression functions had far lower r² values.

TABLE 1.

M. paratuberculosis serum antibody ELISA Se, Sp, and LR values by ELISA S/P value

ELISA S/P cutoff value	No. of sera above cutoff from:		% Spc	% Sed	LRe
	M. paratuberculosis-free cattlea	M. paratuberculosis-infected cattleb
0.00	1,904f	116f	36.0	88.6	1
0.05	995	94	66.5	71.8	2
0.10	545	77	81.7	58.8	3
0.15	331	71	88.9	54.2	5
0.20	198	70	93.3	53.4	8
0.25	91	63	96.9	48.1	16
0.30	71	59	97.6	45.0	19
0.35	54	57	98.2	43.5	24
0.40	46	55	98.5	42.0	27
0.45	36	51	98.8	38.9	32
0.50	27	51	99.1	38.9	43
0.55	21	50	99.3	38.2	54
0.60	19	50	99.4	38.2	60
0.65	15	47	99.5	35.9	71
0.70	14	46	99.5	35.1	75
0.75	10	46	99.7	35.1	104
0.80	8	44	99.7	33.6	125
0.85	6	44	99.8	33.6	166
0.90	5	42	99.8	32.1	191
0.95	5	42	99.8	32.1	191
≥1.00	3	41	99.9	31.3	310

^an = 2,974.

^bn = 143.

^cNumber below cutoff/total number of M. paratuberculosis-free cattle tested.

^dNumber above cutoff/total number of M. paratuberculosis-infected cattle tested.

^eLR, the odds that a cow with this ELISA S/P value is M. paratuberculosis infected were determined using the equation LR = Se/(1 − Sp).

^fSera from some tested cattle produced OD values below that of the negative control, thus giving S/P values less than zero; hence, this number did not equal the total number of sera tested in this population.

FIG. 1.

Nonlinear regression of ELISA S/P values and LRs (bold line) with 95% confidence intervals (thin lines).

When the 143 M. paratuberculosis-infected cattle were arbitrarily clustered into five groups according to ELISA S/P levels (levels found useful based on clinical experience), a relationship between the magnitude of ELISA S/P value and the rate at which cattle tested positive on other tests for paratuberculosis in historical data was apparent (Table 2).

TABLE 2.

Percentage of positive results by other diagnostic methods when M. paratuberculosis-infected cattle were grouped by ELISA S/P range

IDEXX ELISA S/P value	No. of cattle tested	% Positive results in fecal culture by:		% Serum antibody by:
		Modified BACTEC systema	Conventional methods on HEYb	AGIDc	CFd	Alternative commercial antibody ELISAe
<0.10	65	26	20	0	5	5
0.10-0.24	13	39	46	15	23	31
0.25-0.39	9	33f	22	22	44	44
0.40-0.99	15	67	33	20	7	67
≥1.00	41	93	80	76	68	95

^aModified BACTEC 12B media with filter-concentrated fecal specimens (2).

^bDecontamination with hexadecyl cetylpyridinium chloride, concentration by sedimentation, and inoculation of Herrold's egg yolk agar (HEY) as performed by the Wisconsin Veterinary Diagnostic Laboratory, Madison, Wis. (29).

^cAgar gel immunodiffusion assay (AGID) done by Rapid Johne's test (ImmuCell Corp., Portland, Maine) (20, 21).

^dComplement fixation test (CF) performed by the Wisconsin Veterinary Diagnostic Laboratory with reagents supplied by the National Veterinary Services Laboratory, Ames, Iowa (12).

^eParacheck (3).

^fConfidence intervals for the reported percentages were large, i.e., up to ±16% for ELISA categories with very few cattle.

DISCUSSION

Control of paratuberculosis in dairy cattle herds requires hygienic measures to limit opportunities for transmission of the infection from cows to calves in combination with the management of cattle most likely to be infectious (7, 8, 28). Infected cattle should not provide colostrum or milk to calves, and their manure should not be allowed to contaminate feed, water, or the environment. This is particularly true for the pens in which calves are born and the location on the farm where calves are raised. In addition, as many of the infected cows as possible should be culled from the herd when economically feasible. Because the majority of M. paratuberculosis-infected cattle are infectious (shedding the organism in their manure, colostrum, and milk) but clinically normal, laboratory diagnostics are needed to identify them.

Culture of feces to diagnose M. paratuberculosis infection with conventional culture media, such as Herrold's egg yolk agar, requires 8 to 16 weeks (29). Laboratories typically charge $12 to 25 per sample. A liquid-culture-based detection system such as the Trek ESP system and the BACTEC system are able to shorten the detection time to 4 to 8 weeks but are not less costly than conventional culture when the costs of isolate identification are considered (2, 9). Genetic M. paratuberculosis detection technology coupled with PCR methods theoretically should enhance detection Se and considerably shorten the time to detection. However, commercial tests have yet to attain the analytical Se of culture methods, are roughly twice as expensive, and are difficult to scale up for handling large sample numbers (30).

Serology provides a cost-effective alternative to organism detection-based diagnostic methods for bovine paratuberculosis. ELISA-based methods have the highest Se of serologic tests for paratuberculosis (24), plus they offer the kind of low-cost and high-throughput process (>1,000/day) needed to serve the dairy industry. A disadvantage of ELISAs for paratuberculosis is that assay Sp is less than that for fecal culture (considered to be 100%) (13, 25, 31) and the economic consequences of mistakenly culling a cow due to false-positive test results are high (roughly $1,300/cow based on the average price of replacement cattle in the United States in 2001 and the average salvage value of culled dairy cow).

Traditional ELISA interpretation is dichotomous (positive or negative) based on a single-assay cutoff value designed to optimize assay Se and Sp. Use of multilevel LRs capitalizes on the assay's ability to report results on a continuous scale, thereby enhancing the amount of diagnostic information gained. LRs quantify the probability of an accurate diagnosis. Diagnostic probabilities generated from LRs, with or without use of pre-test probabilities, can be integrated with the economic impact of actions taken based on test results such as culling. Thurmond et al. nicely demonstrated this by an ELISA for Neospora caninum infection in dairy cattle (26).

LRs are derived from Se and Sp estimates. These traditional measures of test accuracy for infectious diseases are influenced by the gold standards for the definition of infection and absence of infection in the tested populations and the spectrum of disease in the infected population (14). The standard for diagnosis of infection used in the present study was in the isolation of M. paratuberculosis from a fecal or tissue sample, a widely accepted standard. The standard for freedom of infection was the residence of the animal in a herd certified free of infection based on multiple independent (nonserologic) tests spanning several years. It is inappropriate to use test-negative cattle resident in M. paratuberculosis-infected herds for the estimation of diagnostic Sp because the chance of erroneously considering the animal noninfected is too great, i.e., such a “gold standard” is imperfect and inappropriate.

The spectrum of disease in the M. paratuberculosis-infected animals used in the present study was typical of that found in clinically normal adult Holstein cows raised in infected herds in Wisconsin, the type of herds in which ELISAs are performed. Clinical samples from these same cattle have been used in the evaluation of other diagnostic tests for paratuberculosis (22-24). Se and Sp estimates found using the single manufacturer's S/P cutoff of 0.25 (45 and 97%, respectively) are similar to those reported by other investigators, suggesting that the spectrum of disease was at least comparable to that of animals typically used for paratuberculosis serologic test evaluations (11, 16, 25, 32). Arguably, these cases of paratuberculosis may not be typical of all truly infected cattle and do not include M. paratuberculosis-infected cattle that test negative by all available diagnostic methods; however, they were selected without bias for any single diagnostic test. Calculated LRs would be somewhat lower or higher if the spectrum of disease in the infected population was biased toward early- or very-late-stage infections, respectively.

Diagnostic laboratory medicine for animal agriculture is driven more by economics than is companion animal or human diagnostic laboratory medicine. Containment of testing costs and consideration of the economic consequences of the actions resulting from the diagnostic results are critical considerations in deciding which laboratory services to offer. These are factors that veterinary practitioners must consider in deciding what test to use in which circumstances.

The ELISA evaluated in this study produced quantitative results that were directly related to the likelihood that a dairy cow was infected with M. paratuberculosis (Table 1) and to the rate of positive results by using other diagnostic tests for paratuberculosis detection (Table 2). While the use of LRs in clinical epidemiology has long been advocated (18), few clinicians actually use them to estimate diagnostic probability (15). Additionally, as Feinstein nicely points out in his recent review, “the most important roles of technological tests today are in non-diagnostic clinical decisions,” e.g., estimating prognosis (4). Acknowledging this, application of LR data has been simplified for practitioners by the creation of five categories of ELISA interpretation coupled to a recommended action scheme for paratuberculosis control in dairy herds (Table 3). These categories were derived somewhat arbitrarily but take into consideration the magnitude of the LRs and clinical experience with infected dairy herds. This scheme effectively lowers the cutoff for the identification of high-risk cattle to an S/P value of ≥0.10 (the manufacturer's S/P cutoff for a positive test is 0.25) and couples this with recommendations for low-cost interventions involving cows with such results to limit the spread of infection (segregated calving pens and discarding of colostrum). It also functionally raises the cutoff for cattle that are recommended to be culled from the herd to an S/P value of ≥1.00, whereas the ELISA Sp is ≥99.9% (Table 1), thereby limiting the rate of false-positive results and thus the economic impact on the herd owner caused by the mistaken sale and replacement of a noninfected cow. In this way, veterinary practitioners and dairy producers have a simple scheme for ELISA use in herds that is based on EBM and LRs without need of calculations or nomograms. Whether such a system will effectively work to control paratuberculosis in dairy herds is the subject of on going studies.

TABLE 3.

Application of LR to the interpretation and use of ELISA results in M. paratuberculosis-infected dairy herdsa,b

S/P ratioc	Interpretation	Explanation and recommendation
0-0.09	Negative	Antibodies to M. paratuberculosis were not detected. The animal is either not infected or in a very early, undetectable stage of infection. Retesting in 6 to 12 months will increase confidence that the animal is free of paratuberculosis.
0.10-0.24	Suspect	Evidence of serum antibodies above normal background levels. The cow may be in the early stages of infection. Cows with this level of ELISA are roughly 5 to 15 times more likely to be M. paratuberculosis infected than the ELISA-negative cows. Cows with this result should give birth to their calves in a separate pen, and their colostrum should not be fed to calves.
0.25-0.39	Weak positive	Low level of serum antibodies to M. paratuberculosis but above the standard cutoff for a positive test. Odds are 16:1 or higher that this cow is infected. However, cows with weak-positive ELISA results may remain in good health for another lactation and are of limited infection transmission risk to the herd, provided colostrum, milk, and manure from these, animals are kept away from calves.
0.40-0.99	Positive	Moderate level of serum antibodies to M. paratuberculosis. The odds that this cow is infected are at least 30:1. This cow is likely to be shedding the bacterium in its feces and milk and so should be culled from the herd soon and sold for slaughter only.
1.00-10.00	Strong positive	High level of serum antibodies to M. paratuberculosis. The odds that this cow is infected are over 200:1, and the animal is likely in the advanced stages of infection, shedding large numbers of the bacterium in feces and milk. It probably will soon develop clinical signs of the disease. The animal should be culled immediately and sold for slaughter only.

^aThis interpretation scheme only is valid for herds of dairy cattle proven to be M. paratuberculosis infected by having at least one culture-confirmed case in an animal born and raised in the herd (herd prevalence above zero).

^bAn earlier version of this table, based on preliminary data, appears on the Cornell website describing the New York State Cattle Health Assurance Program (http://nyschap.vet.cornell.edu/).

^cAn ELISA S/P ratio of 0 indicates an antibody level equal to that of the negative control provided with the diagnostic kit. ELISA S/P ratio of 1 indicates an antibody level equal to that of the positive control provided with the diagnostic kit.

Practitioners may wish to incorporate pre-test probabilities of infection (the estimated within-herd paratuberculosis prevalence) together with the magnitude of the M. paratuberculosis ELISA result, translated into an LR, for a more precise estimation of the post-test probability of M. paratuberculosis infection. Table 4 illustrates the interaction of pre-test probability and ELISA S/P value on the post-test probability of M. paratuberculosis infection diagnosis. Other serologic tests for veterinary use could and should be interpreted based on LRs.

TABLE 4.

Effect of pre-test probability and ELISA S/P value on the post-test probability^a of M. paratuberculosis infection

ELISA result	Sample midrange ELISA S/P value	Pre-test probability of M. paratuberculosis infection (estimated within-herd prevalence [%]) at:
		1%	5%	10%	20%	25%
Negative	0.05	1	3	6	13	17
Suspect	0.20	9	35	53	72	77
Low positive	0.30	19	55	72	85	89
Positive	0.70	57	87	93	97	98
High positive	2.00	92	98	99	100	100

^aPre-test odds × LR = post-test odds, converted to probability, for example, for a cow with an ELISA S/P value of 0.70 in a herd with a 10% infection rate (probability that any randomly selected cow is infected): Step 1, convert pre-test probability (P) to pre-test odds: [P/(1 − P)] or 1/(1 − 0.1) = 0.11. Step 2, calculate the LR from the ELISA S/P value by the regression function reported in this paper: LR = 265 × 0.7^2.03 = 128:1. Step #3, multiply the pre-test odds with the LR to find the post-test odds: 0.11 × 128 = 14.1. Step #4, convert odds to probability [odds/(odds + 1)], or 14.1/15.1 = 93%.