Evaluating antibiotic stewardship and healthcare-associated infections surveillance assisted by computer: protocol for an interrupted time series study

Abstract

Introduction

Antibiotic resistance is one of the most pressing health threats that mankind faces now and in the coming decades. Antibiotic resistance leads to longer hospital stays, higher medical costs and increased mortality. In order to tackle antibiotic resistance, we will implement in our tertiary care university hospital a computerised-decision support system (CDSS) facilitating antibiotic stewardship and an electronic surveillance software (ESS) facilitating infection prevention and control activities. We describe the protocol to evaluate the impact of the CDSS/ESS combination in adult inpatients.

Methods and analysis

We conduct a pragmatic, prospective, single-centre, before–after uncontrolled study with an interrupted time-series analysis 12 months before and 12 months after the introduction of the CDSS for antibiotic stewardship (APSS) and ESS for infection surveillance (ZINC). APSS and ZINC will assist, respectively, the antibiotic stewardship and the infection prevention and control teams of Nancy University Hospital (France). We will evaluate the impact of the CDSS/ESS on the antibiotic use in adult (≥18 years) inpatients (hospitalised ≥48 hours). The primary outcome is the prescription rate by all healthcare professionals from the hospital of all systemic antibiotics expressed in defined daily doses/1000 patients/month. Concurrently, we will assess the safety of the intervention, its impact on the appropriateness of antibiotic prescriptions and on additional precautions (isolation precautions) as recommended in guidelines, and on bacterial epidemiology (multidrug-resistant bacteria and Clostridioides difficile infections) in the hospital. Finally, we will evaluate the users’ satisfaction and the cost of this intervention from the hospital perspective.

Ethics and dissemination

The protocol has been approved by the Ethics Committee of Nancy University Hospital and registered on the ClinicalTrials platform. Results will be disseminated through conferences’ presentations and publications in peer-reviewed journals.

Trial registration number

NCT04976829.

Strengths and limitations of this study.

• A large data set with monthly data over a 12-month before and 12-month after period will provide a precise estimation of the effects of the intervention.

• A multidisciplinary collaboration approach—with infectious disease physicians, microbiologists, pharmacists, infection prevention and control practitioners and computer scientists—will be reinforced throughout the intervention.

• This would be a first of its kind evaluation of antibiotic stewardship and healthcare-associated infection computerised-decision support system in France, evaluating its effectiveness, safety, acceptability and costs.

• The study is limited to only one hospital in France and there is no control group in the interrupted time series quasi-experimental analysis.

Introduction

Background and rationale

Antibiotics are drugs that can prevent and treat bacterial infections. Unfortunately, in response to their use—and accelerated by their misuse—bacteria adapt by developing antibiotic resistance. Without urgent action to change the way healthcare professionals prescribe antibiotics, common infections and minor injuries could once again kill. If we do not find proactive solutions now to slow down the rise of antimicrobial resistance, the impact of drug-resistant infections would lead to 10 million people dying every year by 2050 and would cost up to US$100 trillion across the world.¹

Several studies have estimated the morbidity and mortality of infections due to the most common multidrug-resistant bacteria (MDRB) from the European Antimicrobial Resistance Surveillance Network data. In Europe, burden assessments of infections caused in humans by eight MDRB revealed the increase of antibiotic resistance over the last years. Between 2007 and 2015, the estimated attributable yearly deaths to these infections increased from 25 000 to 33 110.^{2 3} For example, percentage of Escherichia coli from invasive isolates with resistance to third-generation cephalosporins increased significatively from 14.6% in 2015 to 15.1% in 2018, and 58.3% of E. coli isolated in 2018 were resistant to at least one of the antibiotic groups.⁴ In France, the number of infections and deaths due to MDRB was estimated at respectively 124 806 and 5543 in 2015.³ Antibiotic resistance has also increased for other bacteria than E. coli. For example, percentage of Klebsiella pneumoniae from invasive isolates with resistance to third-generation cephalosporins increased from 4.1% to 28.8% between 2005 and 2017.⁵ The development of antibiotic resistance is correlated with antibiotic use, so it is crucial that antibiotics are only used when indicated and that the most appropriate antibiotic regimen is used.⁶ In 2018, France had the fourth-highest human consumption of antibacterials in Europe: 25.3 defined daily doses (DDD) per 1000 inhabitants per day were consumed in the community and hospital sector.⁷

The antimicrobial resistance global action plan of WHO has five strategic objectives including the strengthening of surveillance and research, the reduction of infection incidence through hygiene and infection prevention and control (IPC) measures and the optimisation of the use of antimicrobial medicines.⁸ Antimicrobial stewardship (AMS) programmes are defined as ‘a coherent set of actions which promote using antimicrobials responsibly’. They are encouraged to improve and measure the appropriate use of antibiotics by promoting the selection of the optimal drug regimen including dosing, duration of therapy and routes of administration.⁹ In the hospital setting, the emergence of electronic medical records and other electronic patient data has facilitated the use of computerised decision support systems (CDSS) for AMS¹⁰ and electronic surveillance software (ESS) for healthcare-associated infections (HCAI) surveillance.¹¹ CDSS and ESS match electronic patient data—such as clinical, microbiological, pharmaceutical and administrative patient records–with computerised knowledge base (rules based on expert opinion and clinical guidelines) for the purpose of organising and presenting the appropriate and updated information to users.^{10 11} Healthcare workers who use CDSS may then make clinical decisions with reduced error and increased accuracy.¹⁰ CDSS targeting prescribers are recommended¹²: they appear beneficial for improving the appropriateness of prescriptions and for reducing the average hospital length of stay and the antimicrobial spending.^{10 13} Implementation of IPC measures is also essential, and has been associated with a decrease in MDRB.¹⁴ ESS for HCAI are automated methods which enable continuous and hospital-wide surveillance for the purpose of presenting the appropriate information to IPC team who may then focus less on infection detection and more on infection prevention.^{11 15} ESS maximises the effectiveness of IPC activities¹⁵ and reduces the amount of time that IPC teams spend on surveillance activities.¹⁶ The implementation, in 2022, of a CDSS supporting AMS with an ESS supporting HCAI surveillance (CDSS/ESS) in our hospital is an opportunity to evaluate the CDSS/ESS impact. On the one hand, several studies had evaluated the effectiveness of CDSS in decision-making of physicians on the prescription of antibiotics.¹⁷ On the other hand, the studies evaluating ESS were mostly either retrospective and prospective cohorts.¹⁶ But, to the best of our knowledge, no study evaluated simultaneously the CDSS/ESS effectiveness, safety, acceptability and impact on costs using a quasi-experimental design study. Quasi-experimental studies are recommended to evaluate AMS interventions.^{18 19}

Objectives

The aim of this study is to evaluate the impact of the CDSS/ESS on the overall use of antibiotics (DDDs/1000 patients/month) prescribed to adult inpatients.

The secondary objectives are to assess: (I) the safety of the CDSS/ESS use on (i) all-cause intrahospital mortality, (ii) length of stay and (iii) the incidence of nosocomial epidemies and of the most frequent HCAI (E. coli, Staphylococcus aureus, Enterococcus faecalis, Pseudomonas aeruginosa and K. pneumoniae); (II) the impact of the CDSS/ESS on (i) the use of specific antibiotics and therapeutic classes with potential for high selective pressure, (ii) the appropriateness of prescriptions of antibiotics and of additional precautions as recommended in guidelines and (iii) the bacterial epidemiology (MDRB and Clostridioides difficile infections); (III) the costs associated with the implementation and the use of the CDSS/ESS, as well as cost savings due to potential antibiotic use decrease and length of stay reduction, from the hospital perspective; and (IV) the users’ satisfaction and acceptability of the CDSS/ESS.

Methods and analysis

Study design

This quasi-experimental study is a pragmatic, single centre, interventional, before–after uncontrolled study using interrupted time series (ITS) analyses. The study will cover a retrospective period of 12 months before and a prospective period of 12 months after the CDSS/ESS implementation.

Setting and participants

Nancy University Hospital (France) is a 1497-bed tertiary-care hospital including two main geographical sites. In 2019, it counted 9600 employees (including 1514 full-time equivalents (FTE) for healthcare professionals and interns) and 162 624 hospital stays.²⁰ An AMS team is present—including infectious disease physicians (1.2 FTE), clinical pharmacists (1.2 FTE) and clinical microbiologists (0.1 FTE)—as well as an IPC team including IPC physicians and pharmacists (2.0 FTE), and nurses (4.5 FTE). The University Hospital of Nancy meets all the seven Core Elements of Hospital Antibiotic Stewardship Programs developed by the US Centers for Disease Control and Prevention²¹ and meets the essential healthcare facility core elements for AMS programmes developed by the WHO.²² It owns an advanced IPC level according to the IPC assessment framework²³ which support the WHO guidelines on core components of IPC programmes.²⁴ In 2017, the point prevalence on one single day of patients treated with antibiotics was 24.4% in the University Hospital of Nancy versus 21.5% (18.9%–24.4%) in comparable French University Hospitals. In the same nationwide point-prevalence survey, the prevalence of HCAI was 7.8% in the University Hospital of Nancy versus 7.0% (6.0%–7.9%) in comparable French University Hospitals. In this nationwide point-prevalence survey, 25 University and Regional Hospitals were randomly select in France (including the University Hospital of Nancy) and constituted a representative sample.²⁵ Total antibiotic consumption was 591 DDD/1000 hospital days in 2017 in the French University Hospital of Nancy and the median consumption was 518 DDD/1000 patients in comparable French University Hospitals.²⁶ The five microorganisms responsible for over 55% of HCAI are: E. coli, S. aureus, E. faecalis, P. aeruginosa and K. pneumoniae.²⁵

The study includes the hospital departments of medicine (intensive care, cardiology, hepato-gastroenterology, haematology, pneumology, infectious diseases, neurology, …), surgery, long-term care and rehabilitation care. The paediatric and gynaeco-obstetric activities are excluded.

To assess the impact on antibiotic use and intrahospital mortality, length of stay, incidence of HCAI, all inpatients ≥18 years and hospitalised ≥48 hours in Nancy University Hospital will be considered in the study. To assess the impact of the CDSS/ESS on the appropriateness of antibiotic prescriptions and of additional IPC precautions, inpatients will be randomly selected from the study adult population receiving respectively antibiotic and additional precautions prescriptions before and after the CDSS/ESS implementation (due to the wide number of patients receiving antibiotic and additional precautions prescriptions, a random sample will be used to allow the analysis of the patient files by AMS and IPC teams). To assess the impact of the CDSS/ESS on bacterial epidemiology, patients will be randomly selected from identification of bacterial isolates in the study population. To assess the hospital costs due to antibiotic use, all the study adult population treated by antibiotics will be considered. Finally, to assess the users’ satisfaction and acceptability, AMS and IPC teams will be interviewed.

Study organisation and patient involvement

This study is promoted and coordinated by the University Hospital of Nancy. A scientific committee is in charge of supervising all scientific aspects and organisational issues occurring during the study period. This committee is multidisciplinary, including one pharmacist (BD), two infectious diseases specialists (AC and CP), two clinical microbiologists (AL and NAi), two IPC practitioners (one MD (AF) and one PharmD (JL)) and two epidemiologists and public health specialists (NAg and NT). This evaluation is coordinated by both a PhD student (AB) and a project manager. The committee will meet regularly: at least one meeting before the study starts to define the protocol, at least two meetings during the study period to discuss potential organisational issues and at least one meeting after the end of the study period to present and discuss the results.

Patients were not involved in the design, or conduct, or reporting, or dissemination plans of our research.

Intervention

The intervention consists of the implementation of a CDSS supporting AMS and an ESS supporting HCAI surveillance, in 2022, to help respectively AMS and IPC teams. These teams will continue their usual practices but with the assistance of the CDSS/ESS. All inpatients will potentially benefit from the intervention, there is no random allocation nor control group.

AMS team activities and implementation of the CDSS supporting AMS

The AMS team aims at reducing inappropriate and unnecessary antibiotic prescribing. The AMS programme in the University Hospital of Nancy was described in 2011 by Bevilacqua et al²⁷ and evaluated in two studies.^{28 29} Currently, the AMS team reviews antibiotic prescriptions without a specific software: hospital pharmacists analyse all electronic antibiotic prescriptions one by one using the electronic patient’s medical record. Given the number of patients hospitalised each day, this prescription analysis is time consuming for pharmacists, and only a limited number of antibiotic prescriptions are analysed at D0, D3 and D7 of therapy with an infectious disease physician (mainly last-line and broad-spectrum antibiotics)³⁰. Since 2017, two infectious disease physicians of the AMS team conduct regularly (at least weekly) ward rounds in several targeted units in the hospital; they also call physicians managing the patients to discuss difficult cases with them and advise on a therapeutic strategy and regular follow-up. The microbiologists send in real-time emails to the infectious disease physicians of the AMS team to inform them of positive bacterial blood cultures. Infectious disease physicians also meet daily with microbiologists in the microbiological department—except during the weekends—to discuss bacterial identification and antibiotic susceptibility tests of positive blood cultures and other relevant samples.

The implementation of the CDSS supporting AMS—named APSS (Lumed, Sherbrooke, Québec, Canada)—will help the AMS team in its daily activities. APSS is an antimicrobial stewardship system which monitors the relevant clinical information of inpatients to facilitate the evaluation of the appropriateness of antibiotic prescriptions by AMS team members (age, gender, respiratory rate, temperature, body mass index, systolic and diastolic blood pressures, cardiac frequency, absolute neutrophil count and white blood cell count as indicators of immune system status, creatinine clearance as an indicator of renal function, …) and to assist them in the post prescription review process. As new information becomes available, it automatically checks in real time if the ongoing treatment seems appropriate or not according to predefined rules in its knowledge base based on the local ‘Antibioguide’ guidelines.³¹ The knowledge base contains rules of contraindications related to drug–drug interactions, drug bug or drug-laboratory mismatches, cheaper alternatives, maximum daily dose, maximum and minimum dose and frequency, maximum duration and route of administration. APSS generate automatic alerts sent to the AMS team when a prescription seems inappropriate. These alerts are prioritised by APSS using rules which generate a score to target patients who are most in need of post prescription review. The approach used by APSS to review antimicrobial prescriptions was previously described by Beaudoin et al.³² The AMS team will screen these alerts daily (08:30 to 18:30)—except during the weekends, where the alerts will be treated the next Monday—according to their prioritised levels. If the number of alerts is too hight, the lowest levels of alert will not be reviewed. The AMS team will accept or refuse the alerts based on clinical relevance, after a review of the electronic medical record if needed. For each alert, the AMS team will indicate in APSS whether the alert is over-ridden or considered clinically relevant or irrelevant. If the alert is found to be relevant, the AMS team will contact prescribers to discuss their prescriptions and advise them on potential improvements. Prescribers will remain free to change their prescriptions or not.

IPC team activities and implementation of the ESS supporting HCAI

The IPC team is a team including mainly hygiene-specialised physicians, pharmacists and nurses which aim at improving the standard and additional IPC precautions, and to reduce the HCAI rate, including MDRB infections. Currently, the IPC team achieves its mission without a specific software by receiving email alerts concerning MDRB and epidemic micro-organisms identified by the microbiological department. Then, the IPC team checks the electronic medical record of these patients and looks for contact patients (ie, patients managed in the same hospital ward and during the same period of time as the infected patient) using a software developed by the IPC team. The IPC team searches for the location of each contact patient—one-by-one—by reviewing their electronic medical records. Finally, the IPC team informs healthcare professionals managing the patients to discuss the implementation of hygiene precautions and screening tests.

The ESS supporting HCAI surveillance—named ZINC (Lumed, Sherbrooke, Québec, Canada)—will be used by the IPC team. ZINC is an ESS which monitors the microbiological information of inpatients and facilitates the surveillance of HCAI and diseases with epidemic potential (all MDRB including extensively drug-resistant and pandrug-resistant bacteria, and other bacteria and viruses with epidemic potential like C. difficile and measles for example). All relevant clinical and microbiological electronic hospital data are automatically sent to ZINC. ZINC allows for real-time identification and follow-up of patients suspect of having a HCAI. Moreover, it easily identifies the list of contact patients based on a definition of contact time with the infected patient. Finally, it recognises all MDRB from the results of the antibiotic susceptibility testing results.³³ Alerts will be generated by ZINC for all identification of micro-organisms requiring additional precautions. The IPC team will screen the alerts daily (08:30 to 18:30)—except during the weekends, where the alerts will be treated the next Monday. The IPC team will contact the physician in charge of positive or contact patients to inform him/her and double-check the implementation of hygiene precautions and/or the screening tests according to French guidelines.^34–36

Outcomes measures, data sources

The primary outcome is the prescription rate by all healthcare professionals from the Nancy University Hospital of all systemic antibiotics (J01 code according to the Anatomical Therapeutic Chemical—2017 classification) expressed in DDDs/1000 patients/month, calculated monthly during the 12-month before and 12-month after period. These data are routinely collected through the hospital pharmacy database.

The secondary outcomes—presented in table 1—will be collected monthly during the whole 24-month study period, except the proportion of prescriptions compliant with guidelines (evaluated twice during the study period), the users’ satisfaction and the costs of the implementation of the CDSS/ESS which will be evaluated respectively after 6 and 12 months of CDSS/ESS implementation. Concerning the assessment of the users’ satisfaction, a qualitative study based on individual and semi-directive interviews will be performed by an investigator (AB) with all the members of AMS and IPC teams who will use the software.

Table 1.

Description of secondary outcomes

Outcome	Outcome measures	Measured by	Population	Data source
Safety	All-cause intrahospital mortality rate.	All causes of death/1000 inpatients/month.	All adult inpatients.	Medical records department.
	Average length of stay.	Total length of stay for each inpatient/number of stays/month.	All adult inpatients.	Medical records department.
	Incidence of nosocomial epidemies and of the most frequent HCAI.*	New nosocomial epidemies and new cases of HCAI/number of inpatients/month.	All adult inpatients.	IPC team.
Impact	Use of overall antibiotics and by therapeutic classes.	DDDs/1000 patients/month.	All adult inpatients.	Hospital pharmacy.
	Proportion of the antibiotic prescriptions compliant with guidelines.	Number of antibiotic prescriptions compliant with guidelines/number of antibiotic prescriptions evaluated by AMS team.	Sample of adult inpatients receiving antibiotics.	Medical files sampled by the medical records department. Medical files reviewed by the members of the AMS team.
	Proportion of the additional precaution prescriptions compliant with hospital recommendations.	Number of additional precaution prescriptions compliant with hospital recommendations/number of additional precaution prescriptions evaluated by IPC team.	Sample of adult inpatients required additional precautions.	Medical files sampled by the medical records department. Medical files examined by the members of the IPC team.
	Proportion of Clostridioides difficile.	Number of stays with C. difficile infections/number of stays/month.	All adult inpatients.	ZINC (microbiological data).
	Proportion of MDRB.	Number of stays with MDRB identification by bacterial sample done after 48 hours of hospitalisation/number of stays with identification of bacterial isolates/month.	All adult inpatients with identification of bacterial isolates.	ZINC (microbiological data).
Hospital costs	Costs of antibiotics.	Costs (in €) of all oral and intravenous antibiotics delivered by hospital pharmacy/month.	All adult inpatients treated by antibiotics.	Hospital pharmacy.
	Average cost of hospital stays.	Costs (in €) of hospital stays for each inpatient/number of stays/month.	All adult inpatients treated by antibiotics.	Financial department.
	Costs of the implementation of the CDSS/ESS and of the purchased equipment.	Costs (in €) of installation and maintenance of the CDSS/ESS after 12 months of implementation.	/	Information technology service.
Users’ satisfaction and acceptability	Satisfaction.	Qualitative study after 6 months of use.	All members of AMS and IPC teams.	Individual semi-structured interviews.
	Proportion of APSS’ alerts accepted by the AMS team.	Number of APSS’ alerts accepted by the AMS team/number of alerts generated by APSS/month.	All alerts generated by APSS.	APSS.
	Proportion of AMS team’s recommendations accepted by prescribers.	Number of AMS team’s recommendations accepted by prescribers/number of recommendations given by the AMS team to prescribers/month.	All recommendations made by the AMS team.	APSS.

*The most frequent healthcare associated infections (HCAI) are infections acquired by patients more than 48 hours after their admission to the hospital (and not present or incubated at admission) and caused by Escherichia coli, Staphylococcus aureus, Enterococcus faecalis, Pseudomonas aeruginosa and Klebsiella pneumoniae.

AMS, antimicrobial stewardship; CDSS, computerised decision support systems; DDD, defined daily dose; ESS, electronic surveillance software; IPC, infection prevention and control; MDRB, multidrug-resistant bacteria.

The intervention is planned for 2022. Data for the before period will be collected retrospectively, while data for the after period may be collected prospectively.

Confounding factors, such as other antibiotic stewardship or IPC interventions during the study period will be collected and taken into account in the statistical analysis. Changes in the configuration of the CDSS/ESS, their reasons and dates of occurrence will also be collected.

The number of hospitalisation episodes and of inpatients receiving antibiotics will be collected—with inpatients’ gender and age— to describe the studied population. To describe the AMS team activity, the number of reviewed antibiotic prescriptions, notably concerning last-line and broad-spectrum antibiotics will be collected. The number of recommendations made by AMS and IPC teams following an APSS or ZINC alert will be reported, as well as the type of AMS’s intervention (dosing adjustment, switch from intravenous to oral therapy, immediate discontinuation of treatment, planned end of treatment, adjustment based on antimicrobial serum levels or adjustment according to microbiology results for example).

Statistical analysis

Power and sample size calculations

Based on the study conducted by Nault et al,¹³ we estimate that the effect size (expected intervention effect over its SE) of the CDSS supporting AMS intervention on our primary outcome will be ≥2 for both level and trend changes. According to the simulation-based power calculation for designing ITS of Zhang et al³⁷—with both level and trend changes assuming effect size ≥2, equal pre-intervention and post-intervention time periods and statistical significance level=0.05—a 24-time periods study is sufficient to obtain a power >0.9. The time period of 1 month is currently used in ITS.³⁸ A 12-month before and 12-month after intervention period allows the adequate evaluation of seasonal variations.³⁹

Due to potential COVID-19 impact on several outcomes (it had notably affected the use of some antibiotics in several French hospitals⁴⁰), a preliminary study regarding COVID-19 impact on the antibiotic use will be performed in our hospital to take into account the possible variation during the different waves of COVID-19. The results of this preliminary study will be used to improve the statistical analysis and to discuss the local results.

To increase the power study and to assess the long-term sustainability, an extended study which will cover a 24-month before and 24-month after period will be performed at a later stage using the same outcomes.

Descriptive analysis

Inpatient characteristics, prescriptions and other descriptive data will be described as numbers, percentages and CIs for categorical variables and as means and SD or as medians and IQRs—depending on the distribution—for continuous variables.

Qualitative analysis

The interviews will be fully transcribed. Then a duplicate analysis of the data will be performed by two investigators using an inductive method derived from the grounded theory (a qualitative research method focused on the identification of concepts that emerge from study interviews or observation⁴¹). NVivo software (QSR International, Melbourne, Australia) will be used to facilitate the coding process of the transcripts.

Interrupted time series analysis

Assessment of the CDSS/ESS impact will be realised with ITS analysis. ITS analysis is the best approach for quasi-experimental studies assessing longitudinal effects of interventions introduced at a specific time point where a randomised trial is infeasible or unethical.^{39 42 43} Before–after quasi-experimental studies using ITS analysis is the most common study design used for evaluation of antibiotic stewardship interventions for hospital inpatients.⁴⁴ ITS analysis is also recommended to evaluate the effects of IPC interventions according to Effective Practice and Organisation of Care and ORION guidelines (guidelines for transparent reporting of Outbreak Reports and Intervention studies Of Nosocomial infection).^{45 46} In this study, we comply with ITS guidance^{38 47} and recommendations to evaluate the impact of AMS interventions with two limits: the study is limited to a single centre and there is no control group.¹⁸ This study will not include an internal control group for practical and ethical reasons. Indeed, the implementation of the CDSS will need a change in practice to support routine work of the AMS team who will not continue its current way of operation in a control group for efficiency reasons. Moreover, the implementation of the ESS will include all hospital wards for ethical reasons because it should allow a better prevention and management of the HCAI which can result in serious illness, prolong hospital stay, drain healthcare resources and may result in loss of life. Moreover, the use of an external control group (like another hospital) seems not appropriated because, in ITS, the control group should be exposed to any such co-interventions or events that might also affect the intervention group.⁴⁸ This is not possible with another hospital as control.

The segmented regression analysis will be used to evaluate the longitudinal impact of the CDSS/ESS implementation on the outcomes of interest. Segmented regression is used to measure statistically the changes in level (immediate change in the rate of the outcome) and slope (changes in the trend) during the before period as compared with the after period.^{38 39 42} Segmented regression analysis is a powerful statistical method for estimating intervention effects in ITS studies.^{39 49} Before this analysis, we will graph the mean/percentage and SD/CI scores for all outcomes presented as monthly data to determine the structure and trend of the data. We will test for the presence of the autocorrelation (extent to which the data are dependent on each other) using the Durbin-Watson statistic^{50 51} and, if necessary, we will adjust for autocorrelation and seasonality. The transition period—during which the CDSS/ESS will be implemented, and the healthcare users will be trained—will not be included in the analysis. Such a lag period is currently applied in ITS study.⁵² Statistical analysis will be conducted using RStudio (RStudio, Boston, Massachusetts, USA)—an integrated development environment developed for R.

Ethics statements

Patient consent for publication

Not applicable.