Data limitations hinder the development of AI-based decision support for the treatment of antibiotic-resistant bacteria

Our ability to treat and prevent bacterial infections with antibiotics is a cornerstone of modern healthcare. Diseases caused by antibiotic-resistant bacteria are, therefore, recognized as one of the most serious threats to human health worldwide. When treating an infected patient, time is often of the essence, as quickly administering an effective drug lowers the risk of transmission, prolonged illness, and death. Ideally, the selected drug should also precisely target the causative pathogen to minimize the selection for resistance to broad-spectrum antibiotics.

Diagnostics provide the treating physician with vital information for choosing the most suitable therapy. However, the urgency of life-threatening infections may require immediate action before complete diagnostic results are available. This practice, a form of “empirical treatment”, increases patient mortality and morbidity and leads to unnecessary antibiotic use [1]. The WHO has therefore highlighted that timely and accurate diagnosis is one of the pillars in tackling the resistance crisis [2].

Accurate diagnosis of bacterial infections can be difficult due to the high diversity of bacterial pathogens and their constant adaptation to strong antibiotic selection pressures. Conventional cultivation-based diagnostics, in particular antimicrobial susceptibility testing, are limited by the growth rate of the bacteria, and complete results for complex resistance profiles may, thus, take days to complete. Whole-genome sequencing and targeted genotypic assays (e.g., polymerase chain reaction) – which identify specific genetic changes that give rise to resistance – are used as complements, but they are highly specific and/or slow, and still too expensive to use at a large scale [3]. Combining available diagnostic results from various sources with patient information to select the most suitable treatment can, therefore, be challenging.

Recently, artificial intelligence (AI) has been recognized as a promising way to improve the diagnosis of infections caused by antibiotic-resistant bacteria [4]. AI enables the development of data-driven methods that do not rely on explicit assumptions about their internal mechanisms. In contrast to many conventional approaches, AI-based methods have a much larger number of parameters, which enables them to capture greater complexity and describe intricate phenomena, such as the often convoluted relationship between diagnostic results, patient symptoms, and clinical outcomes. In particular, AI-based methods can learn from thousands of prior infections and patient outcomes to provide evidence-based guidance for selecting the most appropriate therapy strategy. Therefore, adopting AI-based decision support systems in clinical settings holds promise for reducing patient morbidity and mortality caused by multiresistant bacteria and for promoting more sustainable use of antibiotics by facilitating the substitution of broad-spectrum antibiotics with more narrow-spectrum alternatives [5].

The potential of using AI-based methods to leverage clinically relevant data to improve the treatment of bacterial infections has recently been demonstrated. Kanjilal et al. [6] developed a decision algorithm that utilized electronic health records, particularly prior antibiotic exposure and infections with antibiotic-resistant bacteria, to guide the treatment of urinary tract infections. Based on a test cohort of 3,629 patients, they showed improved treatment of bacterial infections, leading to a 67% reduction in the use of second-line antibiotics compared to clinicians. Stracy et al. [7] predicted the emergence of treatment-induced resistance at the patient level. Using data from more than 140,000 patients, they demonstrated that AI guidance resulted in more than a 70% reduction of the risk for the emergence of resistance during treatment. AI-based methods have also improved the interpretation of diagnostic test results, including predicting resistance phenotypes from DNA sequencing data [8,9] and mass spectrometry data [10]. More recently, deep learning techniques, for example, transformers – the core component used in chatbots and AlphaFold – have offered new ways to integrate different types of clinical data [11].

These, and other, AI-based methods have shown great potential; however, few are yet in use in clinical settings. Indeed, before AI-based methods can be established in routine diagnostics, they must be deemed trustworthy by demonstrating acceptable performance and fairness, as outlined by the EU and WHO guidelines [12,13]. This is an intricate task, especially when used to guide the treatment of antibiotic-resistant bacteria.

Firstly, AI-based methods rely less on explicit assumptions and prior knowledge and must instead be trained and evaluated on data. This makes them highly dependent on datasets of sufficient quality and quantity, and too few data points or uncertain data can lead to inadequate performance. The required data volumes, however, depend on the type of AI method and the number of its parameters. Successful examples in the literature often use data from hundreds of thousands of infections [7,11], although there are examples demonstrating that smaller datasets can still yield relevant results [6]. Large datasets are often, fully or partially, extracted from routine diagnostics and aggregated over time and/or between laboratories. One major challenge here is the lack of standardization. In cultivation-based antimicrobial susceptibility testing, for example, zone diameters and minimum inhibitory concentrations are interpreted based on clinical breakpoints, which may vary between guidelines (e.g., EUCAST and CLSI) and evolve over time [14]. The testing strategy—i.e., which antibiotics to test for which pathogens – also varies between hospitals, further complicating data aggregation. Additionally, measurements from MALDI-TOF, which is commonly used to type the pathogens and detect specific resistance phenotypes, depend on the platform and the referenced database [15].

Furthermore, it is essential that AI-based methods are evaluated on data excluded from the training process. Performance is usually measured using various metrics, such as accuracy, sensitivity, and specificity. However, for evaluating antimicrobial resistance, the major error (ME) rate, i.e., the proportion of incorrectly identified sensitive isolates, and the very major error (VME) rate, i.e., the proportion of incorrectly identified resistant isolates, are often preferred. An important culprit when evaluating the performance of assessing resistance phenotypes is the often substantial class imbalance between antibiotics. Indeed, some antibiotics are commonly tested in routine diagnostics, such as first-line penicillins, while other antibiotics may only be tested under specific circumstances (e.g., colistin). The rate of resistance also varies significantly between patient categories and across different populations worldwide. Certain combinations of antibiotics, patient groups, and resistance phenotypes are rare, which can impair performance [16]. Therefore, AI-based methods for guiding the treatment of bacterial infections must be properly evaluated for each pathogen and resistance phenotype. It is, furthermore, necessary to critically assess performance across relevant patient groups to ensure it is overall satisfactory and fair.

Finally, the performance of AI-based methods must be evaluated using data that accurately reflects the patient population on which it will be applied. Data generation is often limited by practical constraints, which can lead to biases toward demographic factors and regional pathogen characteristics. Bias may also arise from patient enrollment strategies, measurement-specific parameters, and other technical factors. An AI-based method adapted to data from one hospital may, therefore, not be generalizable and, thus, perform poorly when applied in other healthcare settings. Furthermore, bacterial pathogens are constantly evolving, creating novel strains carrying new configurations of resistance determinants. It is, thus, important that the evaluation is performed on representative patient populations that include data on sufficiently recent infections. There are still only a few studies that explore the negative effects of the limited generalizability of AI-based methods for guiding the treatment of bacterial infections; however, highly significant effects have been reported for other diseases, such as cancer [17].

The development of AI-based methods to guide the treatment of antibiotic-resistant infections is hindered by the shortage of large, standardized diagnostic datasets. Efforts to compile new, comprehensive data sources are, therefore, urgently needed. Large volumes of clinical data are critical to ensure that AI-based methods can be appropriately trained and validated for use in routine diagnostics. Clinical information reflecting both patient and pathogen diversity globally is necessary to ensure that AI-based approaches are generalizable and, thus, applicable in different health care settings. Indeed, we argue that the availability of openly accessible high-quality data, preferably shared under FAIR principles, would catalyze the evaluation and, thus, the adaptation of AI-based decision support systems in clinical practice. This is necessary to efficiently meet the ever-increasing burden of antibiotic-resistant bacteria.

Funding Statement

This work was supported by the Swedish Research Council [2023-01685, 2024-06177], Joint Programming Initiative on Antimicrobial Resistance (JPIAMR) [2023-01685, 2023-01689], and the SciLifeLab & Wallenberg Data Driven Life Science Program [KAW2024.0159].

Author contributions

Anna Johnning: Conceptualization, Writing – original draft, Writing – review & editing.

Styrbjörn Käll: Conceptualization, Writing – review & editing.

Erik Kristiansson: Conceptualization, Writing – original draft, Writing – review & editing.

Disclosure statement

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties

Writing assistance

No writing assistance was utilized in the production of this manuscript

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.