Discrepant Analysis Is Still at Large

In the June 2000 issue of the Journal of Clinical Microbiology, a guest commentary entitled “Discrepant Analysis: How Do We Test a Test” (7) was published; in it, the author made a very strong case against using this method to evaluate new diagnostic tests. In light of this commentary, as well as the other published criticisms of discrepant analysis (2, 3, 5, 8), I was recently very surprised to discover that discrepant analysis is in active use as an approach to evaluate diagnostic tests for Chlamydia trachomatis. I discovered this while attending the recent meeting of the European Society for Chlamydia Research in Helsinki, Finland, this year.

It was in the field of chlamydia research that the controversy regarding discrepant analysis was first elucidated. A quick survey of the 341 abstracts published in the proceedings from this year's European Society for Chlamydia Research (9) reveals that 83 (22%) were in the diagnostics section. Of these 83, 19 (23%) specifically dealt with estimating test performance indices of diagnostic tests for Chlamydia trachomatis. Of these 19 abstracts, at least 9 (47%) describe versions of discrepant analysis.

Given that discrepant analysis has been the primary analytic method of test evaluation for nucleic acid amplification tests for Chlamydia trachomatis, I surmise that discrepant analysis was used in some of the remaining 10 studies. In fact, I viewed two posters which used discrepant analysis to evaluate the test performance indices, though no mention of this was given in the corresponding abstracts. Finally, an invited oral speaker who discussed the diagnosis of genital Chlamydia trachomatis infections referred to discrepant analysis as an analytic method to estimate test performance when there is no “gold standard.” No one in a room of over 100 chlamydia researchers (I am ashamed to say not even myself) brought up the many criticisms of discrepant analysis summarized in your recent guest commentary (7) or by others (2, 3, 5, 8).

It appears that the conclusions against the use of discrepant analysis by eminent scientists, including statisticians, have been disregarded in the very field where the controversy first arose and is best known. Why do researchers in this field continue to fail to rationally evaluate new diagnostic tests? In the past, ignorance and/or statistical naivete was one possible explanation for the widespread use of discrepant analysis, but what is the reason now? There are reasonable statistical methods to estimate test performance characteristics when there is no gold standard. I reviewed some of these approaches in my poster at the Helsinki meeting; these methods include latent class models (8), latent class models with random effects (4), use of a composite reference standard (1, 2), Bayesian methods (6), and the use of agreement claims. While all of these statistical methods have advantages and disadvantages, they are all scientifically sound approaches for the evaluation of diagnostic tests in the absence of a perfect gold standard, whereas any version of discrepant analysis is not.

While the science and discoveries in recent years regarding DNA technology have been exciting, let us not get caught up in the ardor of the new technology and form scientific conclusions before there is a valid analysis of the data. Spend the resources, use scientifically sound statistical methods, and give the nucleic acid amplification tests the opportunity to prove their worth by evaluating these diagnostic tests through the same rigorous scientific scrutiny as is required for new pharmacological agents.

In conclusion, it is disturbing that discrepant analysis continues to be widely applied in test evaluation. The Journal of Clinical Microbiology should insist that any paper submitted for publication which evaluates a diagnostic test explicitly describe a rational statistical method and refuse to publish the paper otherwise. This action would be consistent with the conclusions in the guest commentary of McAdam (7): “ … the bias that is inherent in discrepant analysis makes the statistical method unsatisfactory. If a newer, better test requires newer, harder methods of analysis, we are obliged to make the effort to accurately test the test.”