A systematic review investigating the use of microbiology outcome measures in randomized controlled trials evaluating antimicrobial stewardship interventions published between 2011 and 2021

Abstract

Introduction

Antimicrobial stewardship interventions (ASIs) aim to reduce the emergence of antimicrobial resistance. We sought to systematically evaluate how microbiological outcomes have been handled and analysed in randomized controlled trials (RCTs) evaluating ASIs.

Methods

We searched PubMed and Embase from 2011–21. Studies were selected if they were RCTs evaluating ASIs. A narrative synthesis approach was taken, identifying whether the study reported any microbiological data (bacterial genus/species; bacterial colony counts; prevalence of bacterial, microbiologically defined infections; and antibiotic susceptibility, measured pre-randomization or post-randomization in one arm only) or outcomes (post-randomization data compared between arms). Studies with or without microbiological data/outcomes were summarized in terms of study characteristics, methods of reporting and analysis of these outcomes.

Results

We identified 117 studies, with 34 (29.1%) collecting microbiological data and 18 (15.4%) reporting microbiological outcomes. Most studies with microbiological outcomes were conducted in secondary care (12/18, 66.7%) and targeted adult populations (14/18, 77.8%), and the intervention involved biomarker-guided rapid diagnostic testing (7/18, 38.9%). The overall quality of reporting and analysing microbiological outcomes was low and inconsistent. The selected study population in analyses and methods of handling missing data were unclear.

Conclusions

This review demonstrates that the quality of handling and reporting microbiological outcomes in RCTs of ASIs was low. The lack of consistency and clarity made it difficult to compare the findings across studies, limiting policy- and clinical decision-making. Therefore, there is a clear need for the development of guidance for handling microbiological outcomes in RCTs and adopting appropriate methods to evaluate these data carefully.

Introduction

Antimicrobial resistance (AMR) is considered to be one of the major global public health threats of the 21st century.¹ The emergence of AMR has led to approximately 700 000 deaths per year,² treatment failures, serious illnesses, increases in healthcare costs, higher costs in second- or third-line drugs and difficult-to-control common infections in hospital.^1–5 AMR occurs naturally, but the emergence and spread of new resistance mechanisms may have been greatly accelerated by the overuse and misuse of antimicrobials in primary and secondary care settings as well as in agriculture.⁶ Although global actions have previously been taken to reduce the use of antibiotics and combat the emergence and spread of AMR,^7–11 AMR has remained high and the global antibiotic consumption has increased substantially.¹² Assuming no policy changes, global antibiotic consumption is projected to have a further 200% increase by 2030.¹² In the UK, antimicrobials in human medicine are one of the main drivers of AMR, with an estimated 8.8% to 23.0% of antibiotic prescriptions issued by general practices classified as inappropriate.¹³

The first use of the term ‘antimicrobial stewardship’ was published by McGowan and Gerding in 1996.¹⁴ This article highlighted the issues of increasing AMR and the urgent need to determine the impact of antimicrobial use. This article suggested ‘antimicrobial-use stewardship’ to determine the best methods to prevent and control AMR. Nowadays, antimicrobial stewardship interventions (ASIs) primarily measure the appropriate and judicious use of antimicrobials; other aims include to optimize clinical outcomes for the patient, to reduce adverse effects and to reduce the cost.^7,14–16 Some researchers have suggested that AMR should not be the primary goal of stewardship.¹⁵ However, the primary reason why ASIs are necessary is to control the growth of AMR.^16,17 Although some observational studies suggested that the reduction of antimicrobial use should lead to a reduction in AMR, there are no prospective high-quality randomized controlled trial (RCT) studies demonstrating that ASIs can reduce AMR.¹⁶ The evidence in this field remains sparse, and therefore being able to accurately and reliably report on AMR is important.^16,18,19

Measuring AMR involves the collection, processing and analysis of microbiological data. These data are complex and high dimensional;²⁰ each sample could contain multiple organisms, each having a different level of resistance to antibiotics. Common microbiological outcomes include the prevalence of certain pathogenic organisms at the sample site, bacterial load, number of different organisms found, organisms resistant to one or more different classes of antimicrobial (MDR) and the presence of specific AMR mechanisms (e.g. carbapenemase producers).²¹ However, the creation of composite microbiological outcome measures may not be reliable to measure the intervention impact or provide meaningful understanding, as the event rates are often low and no standardized definitions of terms such as MDR exist.^15,22

Methods of analysing microbiological outcomes in RCTs are varied and depend on the collected data and the estimand of interest.^23–25 An estimand is a precise description of the treatment effect to be estimated that reflects the clinical question posed by a given clinical trial objective.²⁵ As such, a carefully defined estimand should influence decision-making around study design, data collection and statistical modelling approaches (including selecting the correct study population and handling of missing data) to ensure that the study results are both interpretable and clinically meaningful. However, it is unclear which methods are appropriate for analysing the microbiological outcomes in ASIs of RCTs, and appropriate for assessing both the organism growth and account for changes in recovery due to organism growth.²⁶

To the best of our knowledge, no systematic review has studied how microbiological outcomes are handled and analysed in ASIs. The aim of this study was to systematically review and synthesize RCTs evaluating ASIs to investigate how these complex multivariate data have been used and analysed.

Methods

Protocol and registration

The review protocol was registered on the PROSPERO database (CRD42021226585).

Search strategy

We aimed to include all RCTs of ASIs published in the past 10 years. The databases used to identify the studies were PubMed and Embase. Studies were included if they were published between 1 February 2011 and 1 February 2021. The full search terms for PubMed and Embase are available in Tables S1 and S2 (available as Supplementary data at JAC-AMR Online). The searches were limited to published studies of ASI using an RCT design, written in the English language and involving human subjects. Studies were excluded if they were focused on HIV, were not RCTs or were narrative or systematic reviews.

Definitions

ASIs were interventions that aimed to reduce inappropriate antibiotic use and limit the spread of AMR. Interventions were defined as ASI if they included a component that aimed to improve antibiotic prescribing or reduce AMR (e.g. changing professional behaviour in antibiotic prescribing, effective or appropriate treatment, decision support tool, reduced duration).

Information was recorded on whether microbiological outcomes or other microbiological data were collected or used in the study. Data considered as microbiological data or outcomes included bacterial genus/species; bacterial colony counts; the prevalence of bacterial, microbiologically defined infections; and antibiotic susceptibility (e.g. MIC). A microbiological outcome was defined as any microbiological data collected post-randomization where a between trial group comparison of some summary of these data was planned. Other microbiological data were defined as auxiliary microbiological data (e.g. data collected pre-randomization only or post-randomization in one arm only and therefore precluding a comparison of trial groups). The microbiological data included both microbiological outcome and auxiliary microbiological data.

The economic status of a study’s contributing countries was categorized as low, lower-middle, upper-middle and high income, according to the World Bank classifications in 2020. Information on whether the study was conducted in primary, community or secondary care settings was collected. The study was classified as being conducted in primary care if implemented in general practices, pharmacies or dental practices; conducted in a community setting if implemented in university lectures, schools, community pharmacies, children’s day care services or playgroups; and conducted in secondary care if implemented in hospital clinics, ICUs or hospital wards. Due to the small number of studies in community settings, these were combined with primary care in the analysis.

Information on the infection or disease targeted by the study was also collected. These were categorized as abdominal infection, bacteraemia, respiratory illness or infection, sepsis, urinary tract infections (UTIs), no specific infection or diseases targeted, and others. Categorizations for the infection or disease were reviewed by two reviewers (T.M.M.L. and D.G.). Infections or diseases included in each category can be found in Table S3.

Studies were categorized according to the intervention types. Categorizations included audit and feedback, clinical decision support, delayed prescribing, education, optimal dosing, rapid diagnostic testing (i.e. microbiological rapid diagnostic testing and biomarker-guided rapid diagnostic testing), guideline implementation, restrictive and others.^18,27 Studies were double counted if the intervention was a multimodal or bundle intervention.

Pilot data extraction

Before performing the full review, a data extraction form was developed and tested through a pilot data extraction. The pilot extraction form was tested by all co-authors (D.G., R.D., K.H., M.W. and K.H.) on the first 100 articles extracted from PubMed and Embase databases. Additional suggestions and amendments to the search teams were made.

Study selection and data extraction

The primary reviewer (T.M.M.L.) screened titles and abstracts identified by the search and applied the inclusion criteria. Articles were excluded in the following order: (i) duplication; (ii) systematic review or meta-analysis; (iii) HIV focused; (iv) no abstract or conference abstract; (v) non-RCT design; and (vi) not ASI. For possibly relevant articles, the full article was obtained and carried forward to a full-text screening phase. Articles that met all inclusion criteria were progressed to the data extraction stage. The quality of included studies was appraised using the Cochrane risk of bias tool.²⁸ For completeness, we attempted to include subsequent analytic papers and published study protocols from the included studies using the method of forward (find all articles that cite back to a specific article) and backward (find all the cited references in a single article) citation searching using Google Scholar and Web of Science. A random 10% of the final included studies were assessed by an independent reviewer (D.G.). Disagreements were resolved at a meeting between reviewers.

Data extraction included study title, author, year of publication, location of the study, economic status, setting (primary/community setting or secondary care setting), study design (individual RCT or cluster RCT), study population, infection(s) being targeted, description of the intervention and sample size. To assess the use of microbiological outcome(s), we extracted data on the type of samples that were collected; timepoint(s) at which the samples were collected; number of samples available; laboratory methods used to analyse the samples; and statistical methods used to analyse the output from the laboratory analysis. To assess whether a microbiological outcome was mentioned in the study protocol as part of the research objective, we collected data from the published study protocol, protocol provided as supplementary material or summary protocol using the provided Clinical Trials Registration (CTR) number where appropriate.

Synthesis

Due to the nature of the research questions, a statistical meta-analysis was not suitable. A narrative synthesis guided by the methods described by Popay et al.¹² was used. This guide includes topics on developing a theoretical model and a preliminary synthesis for a narrative review, exploring the relationships in the data, and assessing the robustness of the synthesis product with the authorship team. As one component of this approach, the included studies were first divided into those that collected and did not collect microbiological data, with the former further divided based on whether or not the microbiological data included microbiological outcomes. Of the studies with microbiological outcomes, the studies were summarized in the following themes: (i) study design and setting; (ii) sample type; (iii) sample collections; (iv) type of microbiological outcomes; (v) laboratory work and definitions; (vi) the use of microbiological measures as the primary study outcome; and (vii) methods of statistical analysis.

Results

Characteristics of included ASIs

Our search strategy identified 8540 articles through PubMed and Embase, and an additional 22 articles were identified through reference list searches. After title and abstract screening, 233 articles were selected for full-text screening. Of those 233 articles, 97 were excluded and 136 articles (from 117 studies) were determined to meet the study content inclusion criteria. Thirty-three subsequent analytical articles were identified through forward and backward citation searching. A total of 169 articles concerning 117 studies were therefore included in the systematic review (Figure 1). Study characteristics and type of interventions of the final 117 studies can be found in Table S4.

Figure 1.

Inclusion and exclusion of studies in the systematic literature review. ^aIncluded secondary analytical papers from the 117 studies. ^bMicrobiological data collected at any point during the studies. ^cMicrobiological outcome collected post-randomization with comparisons made between trial groups.

Of the included studies, 34 (34/117, 29.1%) collected microbiological data and the remaining 83 studies (83/117, 70.9%) did not. The majority of the included studies targeted adults only (85/117, 72.6%), were conducted in Europe (61/117, 52.1%), were conducted in high-income countries (89/117, 76.1%) and targeted respiratory illness or infections (60/117, 51.3%). For studies with microbiological data, the majority of studies were individual RCTs (30/34, 88.2%), were conducted in a secondary care setting (27/34, 79.4%) and targeted specific infections or diseases (31/34, 91.2%). In studies without microbiological data, the majority of studies were conducted in a primary or community setting (52/83, 62.7%), targeted respiratory illness or infection (48/83, 57.8%) and had more than two-thirds of the studies conducted across multiple centres (61/83, 73.5%). Fifty-one studies were multimodal interventions (51/117, 43.6%). The most common types of intervention in studies with microbiological data included biomarker-guided rapid diagnostic testing (12/34, 35.3%), clinical decision support (11/34, 32.4%) and optimal dosing (6/34, 17.6%) (Figure 2). Among the included studies without microbiological data, the intervention frequently included education (40/83, 48.2%), audit and feedback (26/83, 31.3%) and clinical decision support (24/83, 28.9%). Published study protocols were found for 4 (4/34, 11.8%) of the studies with microbiological data and 21 (21/83, 25.3%) of the studies without microbiological data. For the remaining 93 studies, summary protocols were identified for 37 studies using the provided CTR number (https://clinicaltrials.gov/) (13/34 (38.2%) of the studies with microbiological data and 24/84 (28.6%) of the studies without microbiological data) (Table 1). There was no obvious link between the publication year and whether the study protocol detail was found (Figure S1).

Figure 2.

Number of included studies by intervention type and with or without microbiological data.

Table 1.

Characteristics of included ASIs by whether microbiological data were collected

Study characteristics	Without microbiological data (N = 83), n (%)	With microbiological data (N = 34), n (%)
Setting
Primary/community	52 (62.7)	7 (20.6)
Secondary	31 (37.3)	27 (79.4)
Centre
Single centre	22 (26.5)	13 (38.2)
Multicentre	61 (73.5)	21 (61.8)
Randomization
Individual	44 (53.0)	30 (88.2)
Cluster	39 (47.0)	4 (11.8)
Targeted population
Adults only (≥18 years)	57 (68.7)	28 (82.4)
Children only (<18 years)	18 (21.7)	2 (5.9)
Both	4 (4.8)	3 (8.8)
Unknown	4 (4.8)	1 (2.9)
Country
Multicounty	4 (4.8)	3 (8.8)
Single country	79 (95.3)	29 (85.3)
Europe	42 (50.6)	19 (55.9)
North America	12 (14.5)	6 (17.6)
South America	3 (3.6)	1 (2.9)
Asia	19 (22.9)	4 (11.8)
Eastern Africa	1 (1.2)	0 (0.0)
Australia	2 (2.4)	1 (2.9)
Economic status
High	61 (73.5)	28 (82.4)
Upper middle	15 (18.1)	5 (14.7)
Lower middle	4 (4.8)	1 (2.9)
Low	1 (1.2)	0 (0.0)
Mixture of economic statusa	2 (2.4)	0 (0.0)
Specific infection or diseases targetedb
No specific infection or diseases targeted	20 (23.8)	3 (8.8)
Abdominal infection	0 (0.0)	3 (8.8)
Bacteraemia	0 (0.0)	2 (5.9)
Respiratory illness or infection	48 (57.8)	12 (35.3)
Sepsis	2 (2.4)	3 (8.8)
UTIs	6 (7.2)	4 (11.8)
Other	8 (9.6)	10 (29.4)
ASIsc
Audit and feedback	26 (31.3)	4 (11.8)
Clinical decision support	24 (28.9)	11 (32.4)
Delayed prescribing	5 (6.0)	1 (2.9)
Education	40 (48.2)	3 (8.8)
Guideline implementation	3 (3.6)	4 (11.8)
Optimal dosing	4 (4.8)	6 (17.6)
Biomarker-guided rapid diagnostic testing	17 (20.5)	12 (35.3)
Microbiological rapid diagnostic testing	2 (2.4)	2 (5.9)
Restrictive	2 (2.4)	2 (5.9)
Other	3 (3.6)	2 (5.9)
Study protocol
With published study protocol	21 (25.3)	4 (11.8)
Without published study protocol	62 (74.7)	30 (88.2)
Unable to identify	32 (38.6)	12 (35.3)
Attached as supplementary material	6 (7.2)	5 (14.7)
Summary protocol using the CTR number	24 (28.9)	13 (38.2)
Year of the primary paper published
2009d	2 (2.4)	0 (0.0)
2011	5 (6.0)	2 (5.9)
2012	1 (1.2)	2 (5.9)
2013	12 (14.5)	3 (8.8)
2014	5 (6.0)	1 (2.9)
2015	8 (9.6)	6 (17.6)
2016	12 (14.5)	3 (8.8)
2017	9 (10.8)	3 (8.8)
2018	11 (13.3)	5 (14.7)
2019	9 (10.8)	3 (8.8)
2020	8 (9.6)	3 (8.8)
2021	1 (1.2)	3 (8.8)
Microbiological outcome
No	83 (100.0)	16 (47.1)
Yes	0 (0.0)	18 (52.9)

^aTwo studies were conducted in multiple countries from different economic strata (one study conducted in lower-middle, upper-middle and high countries; one study conducted in lower-middle and upper-middle countries).

^bTwo studies (one study without and one study with microbiological data) targeted multiple infections, one study targeted UTI, abdominal-biliary infection, pneumonia and non-purulent cellulitis, and one study targeted acute respiratory infection and UTI. These two studies were double coded in the relative categories.

^cFifty-one studies were multimodal interventions (36 studies without and 15 studies with microbiological data). These 51 studies were double coded in the relative categories.

^dWe have included two secondary analytic papers that were published between 2011 and 2021, but the primary trial paper was published in 2009.

The risk of bias for the majority of the included studies was assessed to be ‘some’ or ‘high’. Specifically, 66 studies (66/117, 56.4%) were classified as having a high risk of bias, 40 studies (40/117, 34.2%) were classified as having some risk of bias, and the remaining 11 studies (11/117, 9.4%) were classified as having a low risk of bias. The reasons for studies being classified as having high/some risk of bias included non-blinded study designs (randomization, personnel and outcome assessors), not reporting on the technique that used to implement the random allocation sequence, and some information not being provided to assess the bias (e.g. reasons for drop-out and other missing data).

ASIs with microbiological data

Among the 34 studies with microbiological data, the most common sample types collected were from blood (12/34, 35.2%), respiratory samples (10/34, 29.4%) and urine (8/34, 23.5%). The most common microbiological data collected were detection of microorganisms at baseline (21/34, 61.8%), prevalence of microbiologically defined infection (6/34, 17.6%). The study characteristics between studies with and without microbiological outcomes are outlined in Table S5. Microbiological data extracted for studies without collected microbiological outcomes are outlined in Table S6.

ASIs with microbiological outcome

Study design and setting

Of the 34 studies that collected microbiological data, 18 studies reported microbiological outcome measures (18/34, 52.9% of those with microbiological data, and 18/117, 15.4% of the total included studies). The majority of the studies were individual RCTs (14/18, 77.8%) with multiple centres involved (16/18, 88.9%), were conducted in a secondary care setting (12/18, 66.7%), were conducted in high-income countries (15/18, 83.3%), targeted adult populations (14/18, 82.4%), targeted a specific infection or diseases (17/18, 94.4%) and involved biomarker-guided rapid diagnostic testing (7/18, 38.9%). Half of the primary papers pertaining to studies that collected microbiological outcome were published between 2018 and 2021 (Figure 3).

Figure 3.

Percentage of the included studies with microbiological outcome by year of the primary paper published. *We have included two secondary analytic papers that were published between 2011 and 2021, where the primary trial paper was published in 2009.

Sample type

Of the 18 studies with microbiological outcomes, 10 studies ascertained the outcomes from the collected samples and the remaining 8 (8/18, 44.4%) ascertained the outcomes from hospital records, routine microbiological testing, national databases or care home records.

Of those studies with collected samples, five studies (5/10, 50.0%) collected samples from urine, four (4/10, 40.0%) from blood, four (4/10, 40.0%) from respiratory samples and two (2/10, 20.0%) from stool.

Sample collections

Of those 10 studies with collected samples, 6 studies (6/10, 60.0%) collected samples at fixed timepoints, 3 studies (3/10, 30.0%) collected samples when an infection developed and 1 study (1/10, 10.0%) included all samples collected during a pre-specified time window (e.g. between Days 8 and 45). Four studies (4/10, 40.0%) mentioned the number of samples obtained at each timepoint, and six studies (6/10, 60.0%) stated the number of samples included in the analyses. Of those studies that reported missing samples, the proportion of missing samples varied between timepoints and type of sample, ranging from 0% to 56%. Only one study detailed the reason for missing samples (insufficient for analysis and therefore not processed in the laboratory). One study compared the sample collections at each timepoint between trial arms, and one study mentioned there were no important measured differences between participants with and without the microbiological outcome. However, no study compared the demographics of the participants with and without a sample collected or detailed the reasons for missing samples (e.g. patient refused, withdrew consent, or study error).

For the remaining eight studies where the microbiological outcomes were extracted, data were extracted from the hospital records, routine microbiological testing during hospitalization, national databases or care home records. The data sources were unclearly described in some studies. The denominators of these microbiological outcomes were based on all randomized participants, and therefore it was unclear whether these data were available for all recruited participants and, if not, the proportion of participants with samples available.

Types of microbiological outcome

The microbiological outcomes evaluated in ASIs were varied. The most common outcomes were the presence of microbiologically defined infections (13/18, 72.2%) such as Clostridioides difficile infections (CDI) and UTI, which were investigated in seven (7/18, 38.9%) and three (3/18, 16.7%) studies, respectively. The other common outcomes included the presence of bacteria resistant to a certain type of antimicrobial, any antimicrobial or a list of pre-defined antimicrobials (4/18, 22.2%); MDR organisms (MDROs) (4/18, 22.2%); detection of bacteraemia (3/18, 16.7%); MRSA (3/18, 16.7%); and other resistant pathogens (3/18, 16.7%) (Table 2).

Table 2.

Summary of the 18 ASI studies with microbiological outcomes arranged by year

Ref.	Setting	ASI type	Infection	Type of sample and collection timepoints/Data extracted from	Micro-data at baseline	Micro-outcomes (% of randomized participants included in the analysis)	Reason for missing data	Analytic approaches
ASI studies with a sample collected
30	Secondary	Guideline implementation, biomarker-guided rapid diagnostic testing	Sepsis	Stool sample at baseline and Days 7, 28 and 180	Microbiological documentation (Escherichia coli, Klebsiella pneumoniae, Pseudomonas aeruginosa, Streptococcus pneumoniae, Haemophilus influenzae, Staphylococcus aureus, C. difficile) MDR pathogen (XDR, pandrug-resistant)	Infection-associated adverse events until Day 180a (96.2%) New infection by MDRO until Day 180 (96.2%) New infection by C. difficile until Day 180 (96.2%) Mortality associated with baseline infection by MDRO (96.2%) Faecal colonization by Days 7, 28 and 180 (CDI and MDRO) (96.2%)	All analyses excluded 10 participants who withdrew consent and requested data removal	Cox PH regression and logistic regressions
44	Primary	Biomarker-guided rapid diagnostic testing	Respiratory illness or infection	Sputum sample/throat swab at baseline and Week 4	Sputum analyses included colour, bacteria and antibiotic resistance	Percentage of tested antibiotics to which at least one cultured, potentially pathogenic bacterial species (from sputum) was resistant (18.7%) Percentage of total bacteria load from throat swabs that grew on the antibiotic selective plate (82.2%)	Sample data not available at Week 4	Logistic regression
31	Primary	Microbiological rapid diagnostic testing	UTI	Urine and stool samples at baseline and 2 weeks	Microbiologically confirmed UTI and UTI with causative organisms resistant to any first-line antibiotic (NIT, TMP or fosfomycin)	Concordant antibiotic usage (microbiologically confirmed UTI + antibiotic use + susceptibility to antibiotic)a (76%) Microbiologically confirmed UTI at 2 weeks (68.5%) Antibiotic resistance in urine and stool organisms at 2 weeks [CIP, third-generation cephalosporins (caused by ESBLs), gentamicin, carbapenems] (varied by each antibiotic) Recurrence of UTI within 3 months (unknown)	Urine, stool sample or the study-diary data not collected	Logistic regression
45	Secondary	Optimal dosing	Abdominal infection	Blood, pus or fluid collected from reoperation, or percutaneous drainage, surgical (reoperation or percutaneous drainage) and surveillance samples between 8 and 45 days	Most frequently isolated pathogens from surgical samples (Enterobacteriaceae, enterococci, anaerobes) Most frequently isolated MDR pathogens from surgical samples (ESBL-producing Enterobacteriaceae, AmpC-hyperproducing Enterobacteriaceae) A list of MDR bacteria cultured from surveillance samples or/and clinical samples The emergence of fungi	Microbiological failure between Day 8 and Day 45 (100%) The emergence of MDR bacteria in surveillance samples (89.4%) The emergence of MDR bacteria in clinical samples (89.8%) The emergence of MDR bacteria in surveillance samples and clinical samples (89.8%) Number of patients with fungi detected (90.3%) Number of recurrent infections (13.3%) Number of superinfections (30.5%) Bacteraemia between Day 8 and Day 45 (94.8%)	Analysis 2, 3, 4 and 5 excluded those without samples. Analysis 6 was based on those who underwent reoperation or additional drainage. Analysis 7 excluded those who left the hospital on the day of death. Reason unknown for analysis 8	Logistic regression
33	Primary	Microbiological rapid diagnostic testing	UTI	Urine sample at baseline and Day 14	Reference culture and susceptibility test [significant growth of uropathogens (UTI), resistance to TMP, sulfamethoxazole, NIT and mecillinam (pivmecillinam)]	Appropriate antibiotic treatment on the day after consultation (significant growth of uropathogens (UTI) in urine culture)a (90.7%) The microbiological cure rate (no significant growth in the control urine sample after 14 days) (unknown)	Consent withdrawn, did not fulfil inclusion criteria, other reasons, and missing the reference microbiological data	Logistic regression
46	Primary	Biomarker-guided rapid diagnostic testing	Respiratory illness or infection	Urine sample at Days 3, 4 or 5	No	Antimicrobial activity in urine samples on Days 3, 4 or 5 (bacterial assay to detect antibiotics in the urine) (82.4%)	No urine test on Day 5	Logistic regression
34	Secondary	Optimal dosing	Abdominal infection	Cultures from the abdomen, surgical samples, blood or fluid at baseline, and initial and subsequent infections	Culture detection (E. coli, Streptococcus species, Bacteroides species, Enterococcus species, Candida albicans)	Surgical-site infection or recurrent intra-abdominal infection with a resistant pathogena (100%) CDI (100%) Extra-abdominal infection with a resistant pathogen (100%)	N/A	χ2 test for association in a 2 × 2 table
29	Primary	Educational	Bacteraemia	Surveillance samples (nares, oropharynx, enteral feeding tube insertion site, suprapubic catheter site, groin, perianal area and wounds) at baseline and Day 15, and then monthly for up to 1 year	No	MDRO (all MDRO, MRSA, VAN-resistant enterococci, CAZ-resistant Gram-negative bacilli, CIP-resistant Gram-negative bacilli) overall and by all indwelling devices, urinary catheters, feeding tubesa (76.3%) New MRSA acquisition (26.3%) New VAN-resistant enterococci acquisition (11.5%) First new CIP-resistant or CAZ-resistant Gram-negative bacilli acquisition (18.4%) Number of MDRO-positive samples (unknown) Mean number of MDRO isolated from each anatomic site (unknown)	Analysis 1 excluded those with baseline only. Analysis 2, 3 and 4 excluded those with one visit and colonized with MRSA, VAN-resistant enterococci and CIP- or CAZ-resistant Gram-negative bacilli at baseline, respectively. Analysis 5 based on positive sample/number of samples collected. Reason unknown for analysis 6	Poisson regression model, Cox proportional hazards model, 2 × 2 χ2 test and non-parametric statistics (specific test unspecified)
47	Secondary	Clinical decision support, biomarker-guided rapid diagnostic testing	Other	Sputum, blood and urine samples for culture were performed routinely for microbiological examinations on the patients when infection, bacteriemia or sepsis was suspected	No	Patient with infections (86.8%) With wound infection (97.6%) With pneumonia (97.6%) With sepsis, multiple organ dysfunction syndrome (95.1%) With urinary infections (95.1%)	Reason unknown	Logistic regression
48	Secondary	Clinical decision support, biomarker-guided rapid diagnostic testing	Other	Blood, urine, airway and intra-abdominal samples at baseline, three times per week and whenever infection was suspected	No	Mean time to appropriate antimicrobials microbiologically verified infections (unknown) Gram-negative rods other than WT E. coli/Klebsiella (more resistant strains) (unknown) ICU survivors with infection clinically judged at the time of discharge (100%)	Analysis based on the number of samples collected	Student’s t-tests, Mann–Whitney U-tests and 2 × 2 χ2or Fisher’s exact test for small numbers
ASI studies without samples collected and with the microbiological outcome extracted from elsewhere
49	Secondary	Clinical decision support, guideline implementation	UTI, abdominal infection, respiratory illness or infection, other	Hospital data at pre-randomization and any time during inpatient stay	Number of participants with MDRO, UTI, community-acquired pneumonia, cellulitis and pneumonia	CDI within 180 days (100%) New MDRO within 180 days (100%)	N/A	Logistic regression
50	Secondary	Guideline implementation	Respiratory illness or infection	Routine microbiology testing during inpatient stay	No	Number of pathogens (100%) Number of antibiotic-resistant pathogens to hospital discharge, death or 56 days (94.3%) Presence of CDI and MRSA infection up to hospital discharge, death or 56 days (94.3%)	Unclear	Poisson, linear and logistic regression
51	Primary	Clinical decision support, educational	UTI	CDI was defined according to each facility’s (nursing home) established protocol	Rate of CDI	Rate of CDI (100%)	N/A	Poisson regression
32	Secondary	Optimal dosing	Bacteraemia	Microbiological documentation during inpatient stay and supplemented by access national or regional healthcare databases	Bacteria type (E. coli, K. pneumoniae, other Enterobacteriaceae, Acinetobacter spp., P. aeruginosa, other) and MDR Gram-negative bacteraemia	All-cause mortality: included relapse of the bacteraemia, local suppurative complications or distant complications, and readmission or extended hospital staya (100%) Relapse of bacteraemia (100%) New clinically or microbiologically documented infection (100%) Resistance development (100%) CDI (adverse events) (100%)	N/A	2 × 2 χ2 or Fisher’s exact test
52	Secondary	Audit and feedback	Other	Patients’ medical records	Gram-negative bacteria (E. coli, K. pneumoniae, Acinetobacter baumannii, carbapenem-resistant strains, P. aeruginosa, carbapenem-resistant strains, ESBL-producing strains) and Gram-positive bacteria (Enterococcus spp., S. aureus)	Favourable microbiological response (absence of the original baseline pathogen) (100%)	N/A	2 × 2 χ2 test
53	Secondary	Audit and feedback, educational, restrictive	No	The Intermountain Healthcare enterprise data warehouse	No	Incidence of CDI in the study hospitals (100%)	N/A	Analysis not performed due to low event prevalence
54	Secondary	Clinical decision support, biomarker-guided rapid diagnostic testing	Sepsis	Standard laboratory tests (microbial cultures repeated on Day 3, Day 5 and at ICU discharge)	No	MRSA (91.4%) Third-generation cephalosporin resistant (ESBL) (81.0%) Enterobacteria or Klebsiella species (82.8%)	Not mentioned	2 × 2 χ2 test
55	Secondary	Clinical decision support, biomarker-guided rapid diagnostic testing	Other	Hospital records (microbiological examinations)	No	Characteristics of infectious episodes, type of infection, severity, clinician confidence, bacteriologically documented, confirmed by infectious disease specialist (100%)	N/A	2 × 2 χ2 test and Kruskal–Wallis test

^aPrimary outcome.

CAZ, ceftazidime; CIP, ciprofloxacin; NIT, nitrofurantoin; Ref. reference; TMP, trimethoprim; VAN, vancomycin.

Laboratory work and definitions

For the eight studies with microbiological outcomes extracted from elsewhere (i.e. not collected directly from participants as part of study processes), laboratory guidelines that were used to define the outcomes were unclear. Only two studies (2/8, 25.0%) reported the guidelines that the laboratory followed to define microbiological outcomes and one study (1/8, 12.5%) reported the guidelines that the laboratory followed to process the samples.

For the seven studies where a sample was collected and infection outcomes reported, six studies (6/7, 85.7%) detailed or provided a reference for the methods that the laboratory used to identify an infection. For the three studies (2/3, 66.7%) that collected a sample and reported MDROs, two studies provided two distinct definitions of MDRO. For the three studies that collected a sample and antibiotic resistance outcome, two studies (2/3, 66.7%) provided methods or breakpoints they followed to define a resistant organism (e.g. EUCAST). Laboratory procedures used to process the samples were mentioned in six (6/10, 60.0%) of these studies where a sample was collected.

The use of microbiological measures as the primary study outcome

Four studies had a published study protocol, and three studies provided the study protocol as supplementary material. For the remaining 11 studies, a summary protocol was found for 7 studies using the provided CTR number. We were unable to find a study protocol for the remaining four studies. Using information extracted from the study protocol and the study papers, six studies’ primary outcomes comprised some element of microbiological data. Of those, five studies collected samples (5/6, 83.3%) and three studies (3/6, 50.0%) were conducted in a primary care setting, and the intervention types included microbiological rapid diagnostic testing (n = 2), optimal dosing (n = 2), guideline implementation (n = 1), biomarker-guided rapid diagnostic testing (n = 1) and educational (n = 1). Details of the primary objectives, sample size calculations, statistical methods, sample collections, primary outcome results and reasons for missing data are available in Table 3.

Table 3.

Summary for the six studies that included a microbiological outcome in the primary study objective

Primary aim of the study	Setting	ASI type	Infection	Primary outcome	Study sample size calculations	Statistical methods	Sample collections	Primary outcome results	Reason for missing data
To investigate if PCT-guided early discontinuation of antimicrobials could reduce the rate of infection-associated adverse events in sepsis30	Secondary	Guideline implementation, biomarker-guided rapid diagnostic testing	Sepsis	The rate of infection-associated adverse events composed of any new case of CDI, MDRO and death that was associated with MDROs or CDI at baseline. CDI and MDROs were detected from stool samples collected at baseline and on follow-up Days 7, 28 and 180	266 patients with 80% power at the 5% level demonstrate a 30% decrease in the standard of care arm to a 15% decrease in the PCT arm	Cox regression model	Unclear the number of samples that were collected and analysed	256/266 (96.2%) randomized patients were included in the analysis. The infection-associated adverse event was detected in 29/256 (11.3%) patients	10 patients withdrew consent and requested data removal
To compare 7 versus 14 days of antibiotic therapy for uncomplicated Gram-negative bacteraemia32	Secondary	Optimal dosing	Bacteraemia	All-cause mortality including relapse of bacteraemia, hospital readmission, extended hospitalization beyond 14 days, distant complications and suppurative complications	600 patients with 80% power a 10% α-risk to exclude the non-inferiority of short-course to long-course antibiotic therapy with a 10% non-inferiority margin	2 × 2 χ2 test	Surveillance sampling was not conducted	All 604 randomized patients were included in the analysis. The primary all-cause mortality was found in 284/604 (47.0%) patients, and relapse of bacteraemia was found in 16/604, (2.6%) patients	Not applicable
To compare the concordant antibiotic use on Day 3 between the POCT and standard care arms31	Primary	Microbiological rapid diagnostic testing	UTI	Concordant antibiotic use defined by antibiotic prescriptions and the susceptibility of UTI pathogens isolated in the laboratory. Urine samples at baseline and 2 weeks	614 female adults, with 90% power at the 5% level is to detect a difference of 15 percentage points difference and allowed for a 25% loss to follow up	Multi-level logistic regression	Baseline urine: 96% collected; 95.6% analysed. Follow-up urine: 69.4% collected; 68.5% analysed.	497/644 (77.1%) randomized females were included in the analysis. 207/497 (41.6%) females categorized as concordant antibiotic use	Missing 2 week diary or urinalysis data
The effect of adding POC susceptibility testing to POC culture on the appropriate use of antibiotics in general practice33	Primary	Microbiological rapid diagnostic testing	UTI	Appropriate antibiotic prescribing defined by antibiotic prescriptions and urinalysis. Urine samples were collected at baseline consultation and 14 days	750 elderly women to detect a 10 percentage-point difference between the two groups with 80% probability at 5% level, assuming an intra-class correlation of 0.2 between patients in the same practice, and account for possible dropouts and sub-analyses	Logistic regression model	Unknown for baseline urine, 38.3% with a follow-up urine sample	341/376 (90.7%) randomized women were included in the analysis. 241/363 (66.4%) women categorized as an appropriate choice of treatment	Consent withdrawal, did not fulfil inclusion criteria, others and missing the reference microbiological data
To compare the prevalence of failure conditions between a fixed-duration and a longer course of antibiotic therapy34	Secondary	Optimal dosing	Abdominal infection	Surgical-site infection (definitions included organisms isolated from cultures), recurrent intra-abdominal infection or death within 30 days	The study assumed a 5% level to detect a 10% difference in complication rate between groups and four interim analyses. These parameters suggested 505 patients per group, and a total of 1120 patients is needed to allow for 10% dropouts and withdrawals	2 × 2 χ2test	Unclear the number of samples that were collected and analysed	517/518 (99.8%) randomized participants were included in the analysis. 114/517 (22.1%) participants with failure conditions	Withdrawal of consent after receiving antibiotic therapy
If a targeted infection programme can reduce the prevalence of MDROs and incident device-related infections29	Primary	Educational	Bacteraemia	Prevalence density rate of MDROs, defined as the total number of MDROs isolated per visit averaged over the duration of a resident’s participation. Surveillance samples were obtained at baseline and Day 15, and then monthly for up to 1 year	A total of 12 nursing homes, with a mean of 137 beds each, were planned to enrol to detect a 30% reduction (rate ratio, 0.70) in MDRO prevalence with 80% power at a 5% level	Mixed-effects multilevel Poisson regression model	The study detailed the number of active surveillance swabs collected for each follow-up	316/418 (75.6%) randomized residents were included in the analysis. 3031 positive MDROs (29.6%) were detected from all indwelling devices	The resident with baseline visits only and no follow-up

Among the studies that included a microbiological outcome as the primary study objective, outcomes included relapse of bacteraemia (n = 1), surgical-site infection (organisms isolated from cultures, n = 1), MDROs (n = 1), infection-associated adverse events (included CDI and MDROs, n = 1) and the use of inappropriate antibiotics by performing urinalysis to test whether the uropathogen was susceptible to the prescribed antibiotic (n = 2). All primary outcomes were reduced to a binary variable. In terms of exclusion, two studies excluded participants without a sample, two studies excluded participants who withdrew consent and one study excluded participants without follow-up visits.

Methods of statistical analysis

Almost all studies (15/18, 83.3%) with at least one microbiological outcome reduced to a binary variable. These outcomes typically involved dividing the number of positive samples (e.g. positive for a particular bacterial species, or a panel of organisms considered to be pathogenic, or resistant to one or more antimicrobials) by either the number of all recruited participants or the number of samples collected. Three studies (3/18, 16.7%) presented outcomes as rate ratios (e.g. ratios of the CDI rate of 0.04 per 1000 days). Three studies (3/18, 16.7%) presented outcomes as quantitative variables (e.g. the number of antibiotic-resistant pathogens). However, the median value of these quantitative variables was less than one due to a small number of events. It is unclear whether these outcomes were clinically meaningful.

Seven studies (7/18, 38.9%) evaluated the difference between trial arms using univariable analyses such as Pearson’s χ² test. Eleven studies (11/18, 61.1%) performed regression analysis to estimate the effectiveness of the ASI where logistic regression was the most common statistical method (9/18, 50.0%). Of those studies where regression analysis was performed, four (4/11, 36.4%) detailed the covariates that were adjusted for in the regression analyses; the remaining studies did not provide any or enough information on the covariates.

Discussion

We have undertaken the first narrative systematic review focusing on the use of microbiological outcomes in ASIs to investigate how these outcomes have been handled and summarized. This review has provided a summary of the quality of current reporting of microbiological outcome data in this field.

Of the 117 included ASIs, 15.4% reported microbiological outcomes, 5.1% had a microbiological outcome as their primary outcome and 3.4% had outcomes that were resistance related.^29–34

Laboratory work and definitions

We found that laboratory procedures for sample processing and guidelines used by the laboratory to define an infection and resistance were inadequately detailed. For both bacterial identification and particularly for antimicrobial susceptibility testing, the accuracy of results is dependent on whether the most up-to-date laboratory procedures are employed. A detailed description of the laboratory guidelines, standard operating procedures and definitions is essential to for both replicating the research but also for comparing results across different studies. Without reference to a standard, it is easy to imply the quality of the testing results.

Reporting on sample collections

Very few studies described the collection of samples. Of those where this was reported, we found a large variation in the proportion of samples that were collected and analysed. Furthermore, no study compared the demographics of the population in whom samples were obtained or detailed the reasons for missing samples. These missing samples were potentially unexpected and were not considered as part of the trial design. Eventually, this could have introduced the possibility of differential selection bias by trial arm.^35,36

Analysis of microbiological outcome

We found that the denominators used in microbiological outcome analysis were frequently based on all randomized participants, without restriction on whether the sample data were obtained. This makes an implicit assumption that, in the absence of a sample, the event of interest was absent. The estimand framework provides a guide for appropriately considering what it is a research team wants to estimate, and from there aids in considering the appropriate target population and how to handle missing data and other intercurrent events.²⁵ As such, additional sensitivity analyses should be considered where the denominator of the outcome is restricted to participants with a sample collected, and the mechanism underpinning any missing samples is investigated and attempts are made to account for this (as these mechanisms could introduce selection bias in the estimate of the intervention effect). For studies with no samples collected and where microbiological data were extracted from elsewhere, the data sources were mostly unclearly described. It was unclear whether the data were available for all recruited participants and the quality of the data was also unknown.

Microbiological data are complex, quantitative and multi-dimensional, as organisms vary in volume and growth, and their resistance to an antimicrobial will depend on the antimicrobial susceptibility breakpoints, which themselves change over time. However, we found that the creation of composite measures represented as dichotomous variables was the most common method used to summarize microbiological outcomes, such as the presence or absence of infection (e.g. microbiologically defined CDI) or the presence of resistance. This method comes with several disadvantages, and the paper published by Altman and Royston³⁷ has described these disadvantages in detail, which include information loss, reduced statistical power and underestimated variation in outcomes between groups. Moreover, it is questionable how clinically meaningful these dichotomous variables are due to the complexity of the microbiological data. In microbiology observational research, more flexible methods, e.g. based on machine learning algorithms, have been widely applied to address classification and interaction problems.^20,38 Although none of the included studies used machine learning methods, the feasibility of doing so for the analysis of ASIs in RCTs may be worth exploring.

Strengths and limitations

This systematic review has several strengths. First, our narrative search approach proved a unique and detailed overview for the use of microbiological outcomes in ASIs. Second, we restricted the ASI studies to RCT design, which means the estimand should be carefully selected and pre-defined, and information on these outcomes (including missing data handling) should be more precise.^39,40 Finally, this review followed the preferred reporting items for systematic reviews and meta-analyses statement.⁴¹ The limitations of our review were, firstly, we only searched PubMed and Embase and excluded non-English studies and unpublished studies. It is possible that some ASIs published elsewhere and in other languages were missed. Secondly, we limited the ASI studies using an RCT design, therefore our findings might not apply to non-randomized research designs. And lastly, only 10% of the final included studies were double coded by an independent reviewer; although the agreement was high (94.2%) and the discrepancies were minor, it is possible that a small proportion of studies have been wrongly coded and excluded. Our findings were consistent with two previously conducted reviews that highlighted a small number of ASIs in this field, dominated by low-quality research.^18,19 The issue of unclear laboratory procedures and definitions has also been highlighted in a literature review.⁴²

Summary and recommendations

The primary reason for conducting ASIs is to control AMR,^16,43 and hence microbiological outcome data should be collected with elevated importance. However, we found that only a minority of RCTs evaluating ASI have collected microbiological (and in particular AMR) outcomes. Furthermore, when microbiological outcomes have been considered, the impact of ASIs on these is unclear since most studies considered these outcomes as secondary, with consequentially no consideration of sampling, selection bias and statistical power. Without proper consideration of these, the whole process could be argued a waste of time and money, and of patients’ effort to provide samples. Results from studies in this field need to be robust to provide a basis for clinical decision-making and policymakers, therefore guidance development is needed for reporting and analysing microbiological outcomes in RCTs of ASI, as summarized in the following recommendations:

1. AMR should be considered an important key secondary outcome even if not a primary outcome.

2. Sampling, selection bias, and statistical power should be considered during study design.

3. Laboratory procedures for sample processing and guidelines used by the laboratory should be reported.

4. Reasons for missing samples should be reported in detail and the consequences for selection bias carefully examined.

5. Sensitivity analyses, e.g. where the denominator for the microbiological outcome is restricted to participants with a collected sample, should be considered.

6. Appropriate estimands of interest and methods of analysis should be explored due to the complexity of the microbiological data.

Transparency declarations

None to declare.

Supplementary data

Tables S1 to S6 and Figure S1 are available as Supplementary data at JAC-AMR Online.