Unacceptable Performance and the Lack of Reproducibility Results in the Report of Colorimetric Methods for Early Detection of Vancomycin and Oxacillin Resistance in Staphylococcus aureus

In a recent paper by Coban and colleagues (6), two colorimetric methods for early detection of vancomycin and oxacillin resistance in Staphylococcus aureus were reported. The authors state that the results were compared by calculating essential agreement, absolute agreement, and minor, major, and very major errors, as recommended by the FDA (10). However, in the cited version of the FDA document, these terms were replaced by categorical agreement and minor, major, and very major discrepancy (10). The authors noted that all tests were performed in the duplicate (6, 10, 14). However, the authors did not report the results of duplicate testing that is necessary for the reproducibility evaluation. In fact, the cited FDA document describes a separate reproducibility (i.e., ≥95%) evaluation step with a triplicate testing (10). The aim of the FDA document is premarket evaluation of new methods which may not guarantee the validity of performance, and even the literature that assays the validity of performance should be read critically (13). In parallel to this view, the Journal of Clinical Microbiology's editorial policy advocates Standards for Reporting of Diagnostic Accuracy (STARD) guidance in the description of diagnostic microbiology assays and comparison of assay performance (1a, 4, 15). However, to our knowledge, a study design in compliance with STARD criteria in the field of microbiology does not exist in the literature to date, possibly because this evidence-based initiative is both new and not easy to comply with. Additionally, the authors report lower-than-90% essential agreement for both methods, except in the nitrate reductase assay for oxacillin testing, and state that results were in concordance with the standard method (6). Contrarily, this is an unacceptable performance according to the criteria described in the cited FDA document (10).

In conclusion, the report lacks the reproducibility evaluation, and whether this affects the final performance metrics cannot be known. In addition, the report does not show an acceptable performance of essential agreement, except in the nitrate reductase assay for oxacillin testing. Therefore, this research note should be considered a report of colorimetric methods with preliminary results that need reproducibility testing before acceptable validation through a study design as recommended by the FDA scheme or STARD initiative. It should also be noted that there are founding and preceding milestone articles that either contributed to the development of or directly used the oxidation-reduction color indicators and nitrate reductase assays in susceptibility testing of microorganisms, none of which are cited by the authors (1, 2, 3, 5, 7, 12, 16-18). It should be underlined that the use of the rapid colorimetric method for MIC testing is not new and is already available as FDA-approved and patented technology in certain nonautomated and automated commercial susceptibility systems (9, 16). The authors stated that such an automated system is expensive, which may refer to only the original capital investment for the initial technology acquisition purchase, but the reagent rental option usually also exists (8). In fact, it has been shown that the use of approved and standardized automated systems in routine rapid susceptibility testing can be cost-effective, not to mention the added values of computerized expert systems for data analysis and interpretation (9).