The impact of digital interventions on antimicrobial stewardship in hospitals: a qualitative synthesis of systematic reviews

Abstract

Background

Antimicrobial stewardship (AMS) programmes in hospitals support optimal antimicrobial use by utilizing strategies such as restriction policies and education. Several systematic reviews on digital interventions supporting AMS have been conducted but they have focused on specific interventions and outcomes.

Objectives

To provide a systematic overview and synthesis of evidence on the effectiveness of digital interventions to improve antimicrobial prescribing and monitoring in hospitals.

Methods

Multiple databases were searched from 2010 onwards. Review papers were eligible if they included studies that examined the effectiveness of AMS digital interventions in an inpatient hospital setting. Papers were excluded if they were not systematic reviews, were limited to a paediatric setting, or were not in English.

Results

Eight systematic reviews were included for data extraction. A large number of digital interventions were evaluated, with a strong focus on clinical decision support. Due to the heterogeneity of the interventions and outcome measures, a meta-analysis could not be performed. The majority of reviews reported that digital interventions reduced antimicrobial use and improved antimicrobial appropriateness. The impact of digital interventions on clinical outcomes was inconsistent.

Conclusions

Digital interventions reduce antimicrobial use and improve antimicrobial appropriateness in hospitals, but no firm conclusions can be drawn about the degree to which different types of digital interventions achieve these outcomes. Evaluation of sociotechnical aspects of digital intervention implementation is limited, despite the critical role that user acceptance, uptake and feasibility play in ensuring improvements in AMS are achieved with digital health.

Introduction

Antimicrobial stewardship (AMS) is a term used to describe programmes aimed at optimizing antimicrobial use, ultimately improving patient outcomes, and reducing adverse consequences including antimicrobial resistance, in a cost-effective way.¹ AMS programmes are typically hospital-wide and multifaceted, including strategies such as prescribing guidelines, formularies and approval systems, post-prescription audit and feedback, and point-of-care interventions.¹ A plethora of studies have been published evaluating the impact of different AMS programmes and interventions on antimicrobial prescribing and patient outcomes.^2,3 A range of outcome measures including antimicrobial use, cost, resistance patterns, secondary infections, patient outcomes (e.g. mortality and length of stay) and readmission rates have been used by studies to assess the impact of AMS in hospitals.⁴ AMS programmes have been shown to reduce inappropriate antimicrobial use, hospital length of stay, costs, rates of resistance and hospital-acquired infections.^3,5–7

The use of digital interventions within AMS programmes is recommended by international guidelines.^7–9 AMS processes that were traditionally paper-based are gradually becoming computerized with the increasing implementation of information technology in hospitals. Many AMS programme barriers previously cited in the literature can potentially be addressed with the use of digital interventions. For example, difficulty identifying patients receiving suboptimal therapy and difficulty communicating recommendations to prescribers were barriers to undertaking audit and feedback.¹⁰ These can be overcome with computerized provider order entry (CPOE) systems or electronic medical records (eMRs), which allow an AMS team to easily identify patients on an antimicrobial and enter notes into the system to document and communicate recommendations to prescribers. Integrating restriction policies and implementing guidelines can also be done more efficiently with the use of CPOE^10–12 and clinical decision support (CDS). For example, an alert signalling de-escalation to narrower-spectrum agents can assist prescribers in making appropriate antimicrobial choices at the point of prescribing.¹³ Despite these benefits, research has also identified a range of challenges and risks associated with the use of AMS digital interventions. For example, implementing too many CDS alerts can lead to alert fatigue,¹⁴ many stand-alone systems do not interface with existing clinical information systems, and digital interventions are not always user-friendly.¹⁵ These issues can hamper uptake of systems and so reduce effectiveness of digital interventions in practice.¹⁵

Hospital-based AMS interventions^1–3,16,17 and digital interventions supporting AMS^18–20 have been the focus of previous reviews. These reviews have focused on specific types of interventions, like CDS, or specific outcome measures. It would therefore be challenging for those selecting or purchasing AMS digital interventions, or those embarking on an evaluation of digital interventions, to synthesize this available information and evidence. We set out to perform a review of reviews. The objective of the current study was to provide a systematic overview and synthesis of the evidence on the effectiveness of digital interventions to improve antimicrobial prescribing and monitoring in hospitals. Specifically, we aimed to answer three questions: (1) What digital interventions are used in hospitals to improve antimicrobial prescribing and monitoring? (2) What outcome measures are used to evaluate the effectiveness of digital interventions designed to improve antimicrobial prescribing and monitoring? and (3) How effective are digital interventions in improving antimicrobial prescribing and monitoring in hospitals?

Methods

Eligibility criteria

Systematic review papers were eligible if they included studies examining the effectiveness of digital interventions related to antimicrobial prescribing and monitoring of antimicrobial use in an inpatient hospital setting. Papers were excluded if they did not include a clearly defined search strategy (i.e. were not systematic) or they were not in English. Reviews limited to paediatric settings were also excluded due to the unique challenges associated with prescribing and monitoring antimicrobials in this context.²¹

Information sources and search strategy

The following databases were searched: MEDLINE, Embase, Scopus, CINAHL and the Cochrane Database of Systematic Reviews. The search was limited to reviews from 2010 onwards as recent systematic reviews would have included papers from older systematic reviews and contained the most up-to-date information. Keywords and subject headings relating to ‘information technology’ and ‘antimicrobials’ were defined for each database. The database search terms are provided in Table S1, available as Supplementary data at JAC Online. The search was conducted on 12 June 2020 and updated on 14 October 2021.

Study selection

The study selection process is depicted in Figure 1. Papers were imported into Endnote and duplicates were removed. Two researchers independently screened the remaining titles and abstracts based on the eligibility criteria. Researchers met to discuss discrepancies until consensus was reached. Any titles or abstracts that the researchers were unsure of were included for full-text review. Full text of the remaining 45 potentially relevant papers were retrieved and reviewed independently by two researchers. Discrepancies were discussed and consensus was reached on included systematic reviews.

Figure 1.

PRISMA flow diagram. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.

Data extraction

Conducting a meta-analysis was not possible due to heterogeneity of interventions and outcomes. We therefore undertook a meta-summary, which is viewed to be an appropriate method when integrating variable quantitative and qualitative evidence.²²

Data were extracted into an Excel spreadsheet by one researcher and reviewed independently by a second researcher. A third researcher with a pharmacy background also reviewed the data for clarity and accuracy of categorization. General information about the reviews was extracted, such as setting, review objective, sources and number of included papers. For reviews that included data summary tables for their included papers, digital interventions and outcome measures were extracted from these tables. When tables were not available, digital interventions and outcome measures were extracted from the results section in each review. Data on the effectiveness of the digital interventions were also extracted from the results section of the reviews.

Quality assessment

The Assessment of Multiple Systematic Reviews 2 (AMSTAR 2)²³ was used to assess the quality of the systematic reviews. AMSTAR 2 is a 16-item checklist that allows for the critical appraisal of systematic reviews specifically related to healthcare interventions. It is a validated tool and was chosen as it can be used for systematic reviews and/or meta-analyses that include both randomized controlled trials and non-randomized studies of interventions.^23–25 Two researchers independently applied the AMSTAR 2 checklist to the reviews and met to discuss discrepancies until a consensus was reached on the quality of the reviews. As AMSTAR 2 does not have a numerical scoring system, the overall quality of papers was determined by the number of critical and non-critical flaws (see Table S2). Overall confidence in the results of reviews was rated as high, moderate, low or critically low, as suggested by Shea et al.²³

Results

Review selection

The database search returned 454 papers. Titles and abstracts of records were screened and excluded based on exclusion criteria, from which we retrieved 45 papers for full-text review. After screening, eight papers met the inclusion criteria and data were extracted for analysis (see Figure 1).

Review characteristics

The characteristics of the included reviews are presented in Table 1. Review papers reported results from between 34 and 143 studies. The majority of reviews (n = 5) were specific to antibiotics and all were published in the last 5 years.

Table 1.

Characteristics of included studies and results of quality assessment

Author (year)	Setting/population	Objective	Sources	Number of included papers	Antimicrobial type	Quality of included papersa	Funding source reported?
Baysari et al. (2016)26	Hospital, inpatient	To review evidence of the effectiveness of information technology interventions to improve antimicrobial prescribing in hospitals.	MEDLINE, Embase, PubMed, reference lists	45	Antimicrobials	Majority were low quality	Yes—Public
Carracedo Martinez et al. (2019)18	Primary care and hospital	To examine whether the use of a CDSS is associated with improved antibiotic prescribing, and the secondary objective was to determine whether CDSSs are associated with lower morbidity and mortality.	MEDLINE, Embase	34	Antibiotics	Majority were low quality	Yes—Public
Cresswell et al. (2017)27	Hospital	To identify and describe existing and emerging approaches to promoting the appropriate use of antibiotics through hospital ePrescribing systems.	MEDLINE, Embase, CDSR, Clinicaltrials.gov, ISRCTN Registry, NHS EED, PROSPERO, Google Scholar	143	Antibiotics	NR	Yes—Public
Curtis et al. (2017)19	Hospital, inpatient	To evaluate the evidence for CDS in improving quantitative and qualitative measures of antibiotic prescribing in inpatient hospital settings.	MEDLINE, Embase, PubMed, Web of Science, CINAHL, Cochrane library, HMIC, PsycInfo	81	Antibiotics	Majority were low quality	Yes—Private
Helou et al. (2020)28	Hospital, inpatient	To systematically review AMS apps and their impact on prescribing by physicians treating in-hospital patients.	MEDLINE, Embase, Cochrane Central, Web of Science, Google Scholar	13	Antimicrobials	Majority were low to medium quality	Yes—No funding
Laka et al. (2020)20	Primary care and hospitals	To assess the effectiveness of CDSSs at reducing unnecessary and suboptimal antibiotic prescribing within different healthcare settings.	MEDLINE, Embase, PubMed, CENTRAL, Scopus, CINAHL, PsycInfo, Web of Science, reference lists	57	Antibiotics	Majority were low quality	Yes—Public
Rawson et al. (2017)29	Primary care and hospitals	To understand the current scope of CDSSs for antimicrobial management and analyse existing methods used to evaluate and report such systems.	MEDLINE, Embase, HMIC, Global Health	58	Antimicrobials	Majority were medium to low quality	Yes—Public
Rittman et al. (2019)30	Hospital	To review the current status of an interactive, patient-centred CDSS on antibiotic use.	PubMed	45	Antibiotics	Majority were low to medium quality	Not reported

CDSR, Cochrane database of systematic reviews; CENTRAL, Cochrane central register of controlled trials; CINAHL, Cumulative Index to Nursing and Allied Health Literature; HMIC, Healthcare Management Information Consortium; ISRCTN, International Standard Randomised Controlled Trial Number; NHS EED, National Health Service Economic Evaluation Database; PROSPERO, international prospective register of systematic reviews.

^aQuality of included papers as determined by authors of the review.

Results of the AMSTAR 2 quality assessment are presented in Table S2. Three reviews were rated as moderate, two low and three critically low quality. All reviews reported potential sources of conflict; most performed the search study selection in duplicate (n = 7) and provided a discussion of the heterogeneity observed in the review results (n = 6). All reviews failed to provide a list of excluded studies and justified exclusions. The majority of reviews (n = 7) also failed to explain their selection of the study designs for inclusion. Of the four reviews that included a meta-analysis, all used appropriate statistical methods and assessed the potential impact of risk of bias in individual studies on the results of the meta-analysis.

What digital interventions are used in hospitals to improve antimicrobial prescribing and monitoring?

The digital interventions investigated in the papers contained in the reviews are collated in Table 2 and mapped to the medication management pathway cycle.³¹ The majority of digital interventions supported the decision to prescribe and the review of medicine orders, but many of the tools supported multiple processes. For example, CDS tools were used to support the decision to prescribe, the recording of medicine orders, reviewing medicine orders and monitoring responses. None of the included reviews compared digital interventions to other interventions.

Table 2.

Digital interventions evaluated in papers, mapped to the medicine management pathway cycle³¹

Digital interventions	Medicine management pathway cycle stepsa
	Decision to prescribe	Record medicine order	Review medicine order	Issue medicine	Distribution and storage of medicine	Administration	Monitor response
Alerts in eMR	✓	✓	✓				✓
Applications providing local resistance maps and preliminary microbiological reports with therapeutic recommendation	✓		✓
Automated dispensing system					✓	✓
Automated microscopy testing	✓
Calculator (smartphone application)		✓
CDS (within CPOE/eMR, stand-alone, web-based)	✓	✓	✓				✓
Checklist in eMR	✓		✓
Clinical dashboard			✓
Closed-loop order-processing system		✓	✓	✓	✓	✓
Computerized antimicrobial approval system	✓		✓
Computerized system that links pharmacy, chemistry, microbiology and patient management data	✓		✓
Data warehouse and monitoring system			✓				✓
Dosing calculator in CPOE		✓
Electronic microbiology reporting	✓		✓
Electronic overview linked to electronic health record			✓
Electronic screening tool to predict likelihood of developing C. difficile infection	✓
Guidelines (in eMR, web-based, on smartphone application)	✓		✓
Mobile technology for pharmacists to verify medication orders			✓
Predictive models for treatment recommendations	✓		✓
Stand-alone dose-prediction software	✓
Surveillance system			✓
Susceptibility results (smartphone application)	✓		✓

^aMedication management pathway cycle steps are not included in the table if they were not supported by digital interventions, as reported in papers.

What outcome measures are used to evaluate the effectiveness of digital interventions designed to improve antimicrobial prescribing and monitoring?

We identified a large number of outcome measures in the papers included in the eight reviews (Table 3). Most outcome measures could be categorized into antimicrobial use, antimicrobial appropriateness, efficiency, clinical, microbiological and economic measures; however, there was substantial variability in how these outcomes were defined and measured.

Table 3.

Outcome measures reported by studies in the eight systematic reviews

	Outcome measures
Antimicrobial use	DDD per 100 or 1000 occupied bed-days; ICU bed-days; patient bed-days
	Number of antimicrobials: per patient; per hospitalization; per department
	Number of antimicrobials: orders; requests; courses prescribed; drugs dispensed
	Number of: patients prescribed antimicrobial; patients receiving excessive dosages; regimen changes; interventions; sepsis interventions
	Number of doses per antimicrobial course
	Prescription rate
	Consumption of antimicrobials: oral; broad spectrum; restricted
	Antimicrobial-free days; days of excessive use; days on antimicrobial; days patient treated for infections
	Failure of antibiotic re-dosing
	Duration of exposure; duration of therapy
	Patients on IV >72 h; IV-to-oral switch; discharges on oral antimicrobials
	Change in third/fourth-generation cephalosporin use
Appropriateness	Antimicrobial susceptibility mismatches; organism predication; adherence to preliminary microbiological reports; adherence to local resistance map recommendations
	Adherence to: guidelines; recommendations; dosage targets
	Appropriate: dose; dosing intervals; empirical therapy; prescribing; initial levels ordered; antimicrobial coverage
	Monitoring-based appropriateness: initial trough concentrations; plasma concentrations within therapeutic range; appropriate TDM
	Proportion of: days with adherence to guidelines; appropriate courses; correct prescriptions; medication errors
	Antimicrobial escalation and de-escalation
	Antimicrobial prescribed to those with an allergy
	Discontinued: in appropriate time frame; within 48 h of surgery (prophylaxis)
	Errors; prescribing and omission error rate per order
	Number of orders with appropriate dosing intervals
	Pharmacy interventions; rejections from ID team
	Prophylaxis improvement
	Risk-appropriate antimicrobial
	Sensitivity, specificity, positive predictive value, negative predictive value
	Timely discontinuation of prophylaxis; % of surgeries where prophylaxis discontinued after surgery
	Usage in low-risk β-lactam allergy
Efficiency	Delays in antimicrobial therapy
	Time from culture to appropriate antimicrobial
	Time to: administration; dosing; prescription; prophylaxis
	Time savings; time spent by antimicrobial managing team; time spent managing antimicrobial utilization; time to make decision
Clinical	Survival; 180 day survival rate
	Mortality; 30 day all-cause mortality; hospital mortality; ICU mortality
	Length of stay; length of stay in ICU
	Hospital-acquired infections; hospital infection rate; patients at risk of C. difficile infection; rates of surgical site infections; detection of sepsis; ROC-predicated bloodstream infections
	Rates of ADEs; incidence of toxicity; rates of nephrotoxicity
	Fevers; severity of illness; patient complexity; response rate; catheter days
	Hospital readmission; 30 day readmission; transfers to ICU; ED visits within 72 h
	Patient disposition from ED
Microbiological	Incidence of nosocomial infections with C. difficile, MRSA and MDR organisms
	Antimicrobial resistance; drug-resistant pathogen emergence; resistance patterns; resistance rate
	Monthly urinalysis ordered; urine cultures ordered after urinalysis; quantity of radiological and microbiological diagnostics
Economic	Case mix index
	Cost of: antimicrobials; hospitalization; surveillance of ADEs; pharmacy
	Resource intensity weight of each hospitalization

ADE, adverse drug event; ED, emergency department; ID, infectious disease; ROC, receiver operating characteristic; TDM, therapeutic drug monitoring.

In evaluating antimicrobial use, outcomes predominantly related to the quantity of antimicrobials prescribed or administered (e.g. number of antimicrobials ordered per patient³²) or time (e.g. duration of therapy^33,34). Outcomes related to appropriateness varied, with some classifying an antimicrobial as appropriate if the order was compliant with guidelines^28,35–37 and others classifying an antimicrobial as appropriate if the microbe cultured from the patient was susceptible to the antimicrobial.^38,39 There were also differences in what components of antimicrobial orders were assessed for appropriateness (e.g. appropriate dose,⁴⁰ proportion of appropriate courses⁴¹).

Economic outcomes were assessed in all eight reviews but few studies in each review focused on these. The most common economic outcome evaluated was cost of antimicrobials.^28,42,43 Similarly, some studies included in reviews reported outcome measures related to use and acceptance of digital interventions, such as proportion of accepted CDS system (CDSS) recommendations,⁴⁴ or use of mobile apps,²⁸ but these were reported to a lesser degree than the outcomes in Table 3.

How effective are digital interventions in improving antimicrobial prescribing and monitoring in hospitals?

The impact of digital interventions on antimicrobial use, appropriateness and clinical outcomes, as reported in the eight review papers, is presented in Table 4. Four of the included reviews^18–20,26 conducted a meta-analysis and the findings from these analyses are presented in Table 4. The findings from the remaining papers were extracted from their results and conclusion sections. Seven of the eight reviews that assessed the impact of digital interventions on antimicrobial use found that implementation of an intervention was associated with a decrease in use. The majority of papers also found an increase in antimicrobial appropriateness following implementation of digital interventions. Impact on clinical outcomes was less clear. Two meta-analyses^19,20 revealed that digital interventions resulted in a significant reduction in mortality and one also found a reduction in length of stay.²⁰

Table 4.

Impact of digital interventions on antimicrobial use, appropriateness and clinical outcomes

Author	Antimicrobial use	Appropriateness	Clinical outcomes
Baysari et al.26	Decreased	Increased.a Appropriate use of antimicrobials: pooled RR 1.49; 95% CI 1.23–1.81; P < 0.0001.	No significant effect on mortalitya or length of stay.a Mortality: pooled RR 0.91; 95% CI 0.82–1.00; P = 0.06. Length of stay: pooled mean difference −0.84; 95% CI −2.43 to 0.76; P = 0.30.
Carracedo Martinez et al.18	Decreaseda	Increased.a Accuracy of empirical treatment: random OR 1.93; 95% CI 1.21–3.08; P < 0.0711. Compliance with guidelines: random OR 4.22; 95% CI 1.89–9.42; P < 0.0001.	No significant effect on mortalitya or length of stay.a Mortality: random OR 0.98; 95% CI 0.84–1.14; P = 0.2321. Length of stay: SMD −0.24; 95% CI −0.49 to 0.02; P < 0.0001.
Cresswell et al.27	Decreased	Increased	Inconsistent
Curtis et al.19	Decreased	Increased.a Adequacy of antibiotic coverage: OR 2.11; 95% CI 1.67–2.66; P = 0.00001.	Reduction in mortality.a Mortality: OR 0.85, 95% CI 0.75–0.96; P = 0.01.
Helou et al.28	Decreased	Inconsistent	NR
Laka et al.20	Decreased	Increased.a Appropriateness of antimicrobial therapy: pooled OR 2.28; 95% CI 1.82–2.86; P = 0.000.	Reduction in mortality.a Mortality: pooled OR 0.82; 95% CI 0.73–0.91; P = 0.045.
Rawson et al.29	NR	Inconsistent	Inconsistent
Rittman et al.30	Decreased	Increased	Inconsistent

NR, not reported; RR, risk ratio; SMD, standardized mean difference.

^aResults are based on meta-analysis.

Helou et al.,²⁸ Baysari et al.²⁶ and Laka et al.²⁰ were the only reviews to examine one specific type of digital intervention or present results based on subgroups of digital interventions. Helou et al.²⁸ only presented results specific to smartphone applications, but did not conduct a meta-analysis due to large variations in study designs and outcomes. Baysari et al.²⁶ categorized digital interventions into stand-alone CDSSs, CDSSs embedded in CPOE, approval systems and surveillance systems. Baysari et al.²⁶ did not conduct a subgroup analysis as part of their meta-analysis but in their description of findings they reported that antimicrobial use decreased following implementation of all these systems except surveillance systems, and appropriateness increased with stand-alone CDSSs and CDSSs embedded in CPOE. CDSSs embedded in CPOE and surveillance systems resulted in a decreased length of stay and surveillance systems also resulted in a decrease in adverse drug events. Laka et al.²⁰ conducted a subgroup analysis of CDS types and found that all CDS types were associated with an increase in antimicrobial appropriateness.

All reviews examined the economic impact of the digital interventions. Four reviews^20,27–29 found that digital interventions were associated with a reduction in cost of antimicrobials, with the remaining reviews reporting mixed or insignificant results. No reviews examined cost-effectiveness of digital interventions.

Discussion

In this review of systematic reviews, we have summarized and synthesized the evidence for digital interventions supporting AMS. Eight reviews that examined the effectiveness of digital interventions related to antimicrobial prescribing and monitoring in hospitals were identified, most of moderate quality. Information on the digital interventions and outcome measures was collated, highlighting the inconsistency across studies in what AMS component was evaluated, and in the approaches used for evaluation. Despite this variability, reviews were consistent in the conclusions drawn. Digital interventions appear to reduce antimicrobial use and improve antimicrobial appropriateness, with the majority of studies showing interventions had inconsistent findings or no significant impact on clinical outcomes.

There was a large degree of variability in the digital interventions examined, although there was a strong focus on CDS interventions. However, we identified no review or study that compared CDS interventions with one another. This result shows that there is limited information available to guide hospitals on intervention selection. Further research comparing specific digital interventions for AMS, including different CDS types, is needed to identify optimal choices for hospitals.

Outcome measures reported in papers also differed across studies but were predominantly related to antimicrobial use and appropriateness, consistent with previous reviews of AMS interventions in general.^2,3 However, there was substantial variability in how these concepts were defined and measured. For example, antimicrobial consumption can be measured and expressed in various ways including duration of therapy, number of prescriptions, or number of antimicrobials dispensed, all of which were measures found in this review. This variation precluded us from conducting a meta-analysis and also makes comparison of antimicrobial consumption levels between sites or studies difficult. The WHO’s Collaborating Centre for Drug Statistics Methodology promotes the use of defined daily dose (DDD) as a drug consumption measure.⁴⁵ IDSA and the Society for Healthcare Epidemiology of America advocate DDD per 1000 patient-days as an outcome measure for AMS programmes.⁴⁶ However, there are some flaws to using this metric and a change in DDD is not solely suitable for drawing conclusions on an AMS intervention’s success.⁴⁷ For example, DDD biases against combination therapy even if that therapy is more appropriate, and it underestimates exposure when doses are reduced for renal dysfunction.^47,48 Days of therapy (DOT) or exposure days are often used and thought to be more clinically relevant to prescribers; however, they share the same limitation in biasing against combination therapy.⁴⁸ Using a combination of metrics may be a more appropriate approach to measure success of an AMS intervention, as it can provide a more comprehensive evaluation.⁴⁹ For example, a novel method proposed by Evans et al.⁵⁰ uses multiple metrics combining clinical outcomes, safety outcomes and duration of therapy to provide an overall probability of a better outcome occurring for patients with the intervention.

In this review, a core question we set out to answer was how effective digital interventions are in improving antimicrobial prescribing and monitoring in hospitals. Unfortunately, the variability in interventions and outcome measures used in review papers prevented us from answering this. Our results indicated that digital interventions, on the whole, decrease antimicrobial use and increase antimicrobial appropriateness. Reviews that undertook meta-analyses showed that appropriateness increased by 1.9- to 4.2-fold, and one review showed that antimicrobial use declined by 70%. However, these ranges should be interpreted with caution, as variable measures were used.

The impact of digital tools on clinical outcomes is less clear. Clinical outcome measures such as mortality and length of stay are difficult to attribute directly to AMS interventions due to confounding variables such as infection control, resistance patterns, quality improvement initiatives and severity of illness.⁵¹ A modified Delphi study with an expert panel to select relevant outcome measures for AMS interventions in acute-care hospitals recommended that programmes focus on antimicrobial use and process measures.⁵² The reluctance to include clinical outcomes was reportedly due to the difficulty in detecting changes in clinical outcomes and attributing them to the AMS intervention.⁵² The Delphi study also highlighted that there are no clear standardized definitions for AMS-specific clinical outcomes such as infection-related mortality or readmission related to infectious diagnoses, a finding which also emerged from our review. It has been recommended that clinical measures be used to demonstrate that interventions do not cause harm rather than to evaluate effectiveness.⁵² When interventions result in a reduction in antimicrobial use, it is important to assess clinical outcomes such as mortality to ensure the intervention does not increase health risks.^20,48 From the reviews in our synthesis that conducted a meta-analysis, two found a reduction in mortality^19,20 and two found no significant effect on mortality or length of stay.^18,26 Although a limited number of reviews, these findings are reassuring as they suggest that digital interventions did not cause harm.

It is interesting to note that we found limited assessment and reporting of sociotechnical aspects associated with implementation of AMS digital interventions. A small number of reviews briefly examined use and acceptance of digital interventions, but these were in the minority. When evaluating digital interventions, clinical outcomes are viewed to be just one aspect of effectiveness, with additional aspects recommended to be examined, such as organizational, human and social, ethical and legal, technological, clinical, cost and economical.⁵³ Our review identified a clear evidence gap in understanding and evaluating these other components of AMS digital intervention effectiveness.

This review was limited by the lack of high-quality reviews that met our inclusion criteria, and by the quality of studies included in the eight reviews. Our results and conclusions are significantly limited by the quality of information contained in systematic reviews. Similarly, a Cochrane review of interventions to improve antibiotic prescribing found substantial design heterogeneity and all non-randomized trials and controlled before–after trials had a high risk of bias.³ This may be due to the complexity of evaluating AMS interventions as they are often implemented in ‘real-world’ settings and are consequently prone to bias, confounding and random time effects.⁵⁴ More high-quality studies are needed to strengthen the evidence base on the impact of digital interventions in AMS programmes on prescribing and monitoring of antimicrobials, and in evaluating sociotechnical aspects of digital intervention implementation.

To conclude, the results of this review indicate that digital interventions reduce antimicrobial use and improve antimicrobial appropriateness in hospitals, but no firm conclusions can be drawn about the degree to which different types of digital interventions achieve these outcomes. The impact of digital interventions on clinical outcomes is less clear and high-quality evaluation, including the use of recommended outcome measures,⁵² is needed to assess the impact on patients and organizations. Evaluation of sociotechnical aspects of digital intervention implementation is limited, despite the critical role that user acceptance, uptake and feasibility play in ensuring improvements in AMS are achieved with digital health.

Funding

This work was supported by the Australian Government Research Training Program (RTP) Scholarship to B.A.V.D.

Transparency declarations

None to declare.

Author contributions

All authors were involved in the study conception. B.A.V.D. and M.T.B. reviewed search results and completed data extraction. B.A.V.D. drafted the manuscript and M.T.B., J.P. and A.R. critically revised it. All authors have given final approval of the published version and agree to be accountable for all aspects of the work.

Supplementary data

Tables S1 and S2 are available as Supplementary data at JAC Online.