Proper Estimation of Sensitivity and Specificity

I write to express concern about the diagnostic accuracy statements of Eda et. al. concerning a new Johne's disease enzyme-linked immunosorbent assay (ELISA) (2).

The diagnostic sensitivity of this new ELISA for bovine paratuberculosis, reported to be 100%, was based on only 64 (according to the Materials and Methods) or 51 (according to the Results) Mycobacterium paratuberculosis fecal culture-positive cattle. Fecal culture-positive cattle with fewer than five colonies on culture were discarded from the dataset, which might explain the different numbers in the Materials and Methods and Results, but this is unclear in the paper. While the authors speculate that these very low fecal shedders of M. paratuberculosis “might arise from pass-through bacilli,” there is no legitimate basis for excluding these cattle from sensitivity analysis. No other reports on ELISA sensitivity for bovine paratuberculosis have done this. There is no way to ascertain if these cattle were truly infected or if such fecal culture results represented “pass-through” (i.e., the cows were not infected and M. paratuberculosis from the environment was simply in transit through the gastrointestinal tract).

Only 38 cows from a single herd in Japan were used to estimate ELISA specificity. Moreover, the authors state in the Materials and Methods that this herd was selected because it had been annually tested and found to be ELISA negative (assay source not specified) for 5 consecutive years. It is no surprise that these cattle were again found negative for serum antibodies to M. paratuberculosis by the new ELISA. The cattle population chosen for specificity estimation was biased, i.e., preselected for absence of serum antibody, too small, and all from a single herd. Studies with statistical validity use more than 400 animals from multiple herds for ELISA specificity estimation (1, 4, 5). Prior publications document herd-specific variation in M. paratuberculosis ELISA specificity (1). For these reasons, the claimed specificity of 97.4% is tentative at best and has a large 95% confidence interval (94.78 to 99.9%), a statistic that should have been reported.

Meaningful evaluation of immunoassays requires testing large numbers of subjects representative of target populations for diagnosis. For infectious diseases such as paratuberculosis, the infected and noninfected populations must have well-defined, objective, unbiased case definitions. Furthermore, receiver-operating characteristic curves and likelihood ratios should be used to complement sensitivity and specificity estimates (1, 3).

In summary, given the small and biased cattle populations tested and methods of data interpretation, the claims of a highly sensitive and subspecies-specific ELISA for bovine paratuberculosis by Eda et al. must be considered tentative at best.