Implementation evaluation of nurse-initiated care in 29 Australian emergency departments: a RE-AIM multi-method evaluation

Abstract

Background

Nurse-initiated care improves timely treatment access, but adoption varies. The Ministry of Health introduced 73 Emergency Care Assessment and Treatment (ECAT) protocols for public emergency departments (EDs) to standardise nurse-initiated care. A behaviour change strategy was used to implement ECAT in 29 EDs as part of a trial, using implementation strategies like education, clinical champions, videos, and audit and feedback. The aim of this study was to evaluate the implementation of the ECAT protocol and the strategies used for reach, effectiveness, adoption, quality, and maintenance (REAIM).

Methods

This multi-method implementation evaluation used the RE-AIM framework. Methods comprised: i) emergency nurse surveys (Intervention and Control groups); ii) scoring by site implementation nurses of fidelity, adaptation and effectiveness of implementation strategies out of 5; iii) audits determining appropriateness of ECAT protocol use at 6–12 weeks and iv) implementation tracking logs during all study periods. Descriptive statistics were used for quantitative data, and content analysis for qualitative data (surveys/logs).

Results

Data from surveys (n = 787), audits (1375 audits, 509 nurses), fidelity scores (n = 11), and implementation tracking logs (>600 entries) were used to evaluate implementation of ECAT protocols in the 29 EDs. Emergency nurses reported high use of ECAT protocols in their daily practice (median (IQR) 9.0 (8.0, 10.0), and audits demonstrated 92.9% of eligible nurses used at least one ECAT protocol in their shift (509/548). High effectiveness was demonstrated through 93.7% selection of the fully appropriate ECAT protocol (1288/1375). Implementation strategy fidelity was high (4.6/5) and effective (3.9/5), particularly the Education program (885 nurses, 97 courses), Audit and Feedback (1375 audits, 509 nurses), stakeholder engagement (>450 meetings) and electronic medical record update (all sites). The strategy with the lowest fidelity and effectiveness was Clinical Champions. Maintenance was demonstrated by training, use and monitoring embedded in practice at 12 months.

Conclusion

Behaviour-change informed implementation strategies successfully enabled large-scale successful adoption of nurse-initiated care at 12 months. Relying on clinical champions already operating at full capacity in an overcrowded ED to assist with implementation was impractical. Ongoing investment in facilitation, education, and adaptive support structures is essential to maintain gains and realise the full potential of nurse-initiated emergency care.

Trial Registration

Australian and New Zealand Clinical Trial, ACTRN12622001480774p. Registered prospectively on 27 October 2022. Link: https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=384899%26isClinicalTrial=False

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-026-01524-0.

Contributions to the literature.

• A multi-faceted behaviour-change strategy achieved consistent, long-term adoption of nurse-initiated care across diverse emergency settings despite political and unexpected challenges.

• Clinical champions as a strategy in acute healthcare is becoming less sustainable because it adds extra responsibilities to clinicians who are already working at full capacity.

• This study provides an example of a multi-method evaluation for implementation in the emergency context and offers a model of how implementation strategies can be applied and evaluated for implementation projects.

Background

Global demand for emergency care has grown over the past decade and continues to exceed resource supply [1, 2], posing serious risks to patient safety [3]. Emergency department (ED) visits are at an all-time high in the United Kingdom [4], and the United States of America (29.4 and 42.7 per 100 people, respectively) [5]. Aging populations and poor primary health care access contribute to ED overcrowding and access block [6, 7]. Overcrowding and access block result in delays in treatment and longer ED length of stay [8], associated with suboptimal care, and negative patient experiences [9].

Likewise, Australia’s 300 public hospital EDs are plagued by overcrowding, access block and treatment delays affecting the nine million annual episodes of care [10]. These free, primary emergency care services are overwhelmed by rising presentations—up from 31.0 to 33.3 per 100 in a decade [11]. Australia’s EDs are unable to keep up with demand, with only 63% of “urgent” patients (42% of total) attended to within 30 minutes, risking severe consequences from delayed assessment and treatment [12, 13]. Australians of all ages seek emergency care with varying urgency, requiring prompt assessment and treatment [14].

Nurse-initiated care has been suggested as a potential solution to overcrowding and ED demand issues by empowering experienced emergency nurses to commence investigations and care. Nurse-initiated care includes clinical treatments, such as analgesia, or investigations such as imaging or pathology, for eligible patients under protocol, based on a nurse assessment and prior to physician review, beyond their usual scope of practice [15]. EDs use a team-based care model; however, emergency nurses are the first clinicians a patient encounters. After triage, nurses are solely responsible for patients until medical review. In regional areas, emergency nurses may be the only clinicians patients encounter [16]. Nurse-initiated care has been shown to reduce ED length of stay [17], and reduce time to analgesia [18–21], imaging [17], treatment [22], in both adult and paediatric populations [23–25].

Despite the evidence supporting nurse-initiated care, its implementation and use have lacked consistency and standardisation. In New South Wales (NSW), Australia, a large, geographically diverse state, nurse-initiated care was used variably at emergency sites and nurses needed medical sign-off. To reduce unwarranted clinical variation in the 180 state, public EDs, the NSW Agency for Clinical Innovation (ACI) released 73 standardised Emergency Care Assessment and Treatment (ECAT) protocols for nurse-initiated care for common and time-critical adult and paediatric emergency presentations [26]. The ECAT protocols provided nurses with expanded authority and guidance to assess patients and initiate investigations and treatments without medical sign-off, supported by structured, condition-specific clinical guidance. The ECAT protocols were implemented with support from NSW Health for policy and online education modules. Alongside the state implementation, the ECAT protocols were implemented in 29 sites as part of the EPIC-START Type II Hybrid implementation effectiveness trial [27]. The implementation at the trial sites was supported by a targeted strategy informed by implementation science and behaviour-change theories [28].

Evaluation of the implementation strategy for ECAT protocols was essential to understand the success or failure, including the effectiveness of strategies used to support it [29]. In the ED, implementation of evidence-based practice is often negatively impacted by high workflow, patient turnover, noise levels [30] and staff attrition [31], with frequent examples in the literature of unsuccessful implementation [32, 33]. Implementation fidelity is the extent to which an intervention is delivered per the intended implementation plan [34]. In this study, an evaluation of a comprehensive behaviour change strategy was needed to assess implementation fidelity and inform the validity and reliability of the overall trial’s results, and to better understand which strategies were effective in implementation. The aim of this study was to evaluate the implementation of the ECAT protocols and the strategies used to achieve reach, effectiveness, adoption, quality, and maintenance at the 29 study sites [27].

Methods

The Intervention: ECAT protocols

The ECAT protocols were developed in a statewide, co-designed project by NSW Health based on previous work by study authors. The initial work at one local jurisdiction included 56 stakeholder meetings with 60+ senior emergency nurses, doctors and pharmacists, resulting in 54 guidelines for nurse-initiated care. NSW Health’s 2022–2023 consultation built on this initial work with previous nurse-initiated care models, evidence checks and extensive consultations with committees of 20–40 stakeholders from medical, nursing, allied health, radiology, pathology, and pharmacy.

The final endorsed set included 73 ECAT protocols − 41 for Adult and 32 for Paediatrics, covering a wide spectrum of presentations such as wounds and burns, mild head injuries and traumatic injuries, allergic reactions, shortness of breath, chest pain, and suspected sepsis. Emergency nurses who have completed the required training take a patient history and perform an assessment, select and use the ECAT protocols, and then initiate investigations and treatments if the patient meets specific criteria, without physician sign-off. It is important to note, ECAT protocols guide care initiation but do not include diagnosis or definitive treatment. Once a healthcare provider assumes care, nurses should stop using the ECAT protocol or follow standard escalation procedures if the patient is too unwell (https://aci.health.nsw.gov.au/ecat/).

Implementation strategies

A multi-faceted plan for the ECAT protocol implementation was developed based on behavioural diagnostic analysis from 847 nursing and medical staff surveys, using the Behaviour Change Wheel and Theoretical Domains Framework and refined with implementation experts and stakeholders [28]. The resulting plan included six implementation strategies consisting of 18 behaviour change techniques [35], with delivery through face-to-face Educational Programs, local clinical champions, individualised audit and feedback, medical staff engagement, environmental strategies, steering committees, leadership and stakeholder outreach [28] (Box 1).

Each site had a tailored delivery of the strategies to meet their needs. The structured education program supported staff attending during work hours, and included an Instructor Course (train-the-trainer design) and two Provider Courses tailored for experienced emergency nurses with recognised prior learning of nurse-initiated care previously (EPIC-START CREDIT) and less experienced emergency nurses without recognised prior learning (EPIC-START CIN) (see Box 1). All permanent and casual nurses were eligible and were allocated in consultation with the site managers and educators to the appropriate course. The Education Program aligned with the statewide ECAT policy, which contained 12 policies (456 pages) that linked to a further 58 policies (1964 pages) to comply with clinical governance, and was grounded in the pedagogical principles of scaffolded learning, constructive alignment [36] and backwards design [37], which have been successfully used in other large-scale trials in emergency nursing [38].

Clinical champions included senior clinical nurses who expressed interest in contributing to supporting the change. In addition, ‘implementation nurses’ were employed for the duration of the study to support implementation processes and data collection. Individualised audit and feedback were provided by the implementation nurses to nurses using the ECAT protocols. Medical staff were engaged through consultation at meetings by the implementation nurses. Environmental strategies included statewide policy and modifications to the electronic medical record system to streamline access to ECAT resources. The implementation process was managed with locally led steering committees that included local leaders.

Context and setting

The 29 study sites were from NSW, a large geographical and populous state in Australia, with metropolitan, rural and remote settings represented and combined presentation numbers of over 650,000 annually (Table 1). The included sites were emergency services or departments, defined as hospital-based facilities that are equipped to provide 24-hour urgent care for life-threatening injuries or illnesses within a health system, and included triage service [39]. Sites within the same jurisdictions, ‘Local Health Districts’, had the same key stakeholders, executives, and policies that were essential to the implementation strategy and were therefore grouped together as four clusters.

Table 1.

Context and setting details for the clusters

Cluster/LHD	LHD information	ED presentations per year	ED nurses employed
Cluster 1 (11 sites)	Regional - rural Area: 44,534 km2 Residents: 211,000+	107,073	180
Cluster 2 (12 sites)	Regional – rural Area: 20,732 km2 Residents: 300,000+	198,380	430
Cluster 3 (3 sites)	Major cities Area: 126 km2 Residents: 722,000+	163,839	402
Cluster 4 (4 sites)	Major cities Area: 780 km2 Residents: 1.1 million+	197,279	280
		666,571	1,292

At the time of implementation, the sites employed approximately 1292 nurses. Sites varied in size and staffing; for example, the larger metropolitan sites had up to 200 staff, while the smaller sites had fewer than ten permanent staff. Sites had varied resources; some of the smaller sites, particularly in rural areas, had limited or no radiology, pathology, and specialist services on-site (Supplementary file 1).

Study design

This pre-post observational implementation evaluation used surveys, audits, fidelity scoring and implementation tracking log data. The methods were guided by the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation (fidelity/adaptation) and Maintenance/Sustainability) [40]. Quantitative data were collected to measure proportions and determine the level of achievement for each component of RE-AIM. Qualitative data were used to understand the “how” and “why” of quantitative results, and any deviations or adaptations to the proposed implementation strategy. The REAIM definitions, related study outcomes and sources are presented in Table 2. Reach was measured by the proportion of eligible nurses using the ECAT protocols regularly, aiming for over 80% [41]. Effectiveness was assessed based on increased nurse confidence scores after implementation and appropriate ECAT protocol use. Adoption was measured by the proportion of eligible sites that implemented ECAT protocols, including if this was by the required date and as intended. Implementation quality was determined by the consistency with which individual components and the overarching implementation strategy were delivered, in terms of fidelity, adaptations and quality. Maintenance was assessed through demonstrating the integration of ECAT protocols into routine operations at 80% of sites 12 months post-implementation. (Table 2)

Table 2.

RE-AIM domains and definitions* with related study intended outcomes, data sources^ and findings

RE-AIM domain and definition*	Intended outcome	Data source(s)^	Findings
Reach: The number, proportion, and representativeness of individuals receiving intervention/strategy	Nursing staff report using ECATs regularly	Nurse Surveys (post)	Nurses reported using ECATs in their practice (9.0 (8.0, 10.0)
	Proportion of eligible staff using protocols	Audits	92.9% (n = 509) of eligible staff (n = 548) used at least one ECAT
Effectiveness: The effects of the intervention on intended outcomes, as well as potential negative effects, heterogeneity, quality of life, and economic outcomes, and the reasons why.	ECATs used appropriately	Audits	93.7% appropriate selection of ECAT
	Increased confidence scores for Intervention compared to control group for nurse-initiated care	Nurse Surveys (pre vs post)	Increased nurse confidence intervention vs control (p < 0.05)
Adoption: The number of sites eligible and willing to initiate a program, and why	% of eligible sites that implement ECAT protocols using the implementation strategy within time (100%)	Site implementation tracking	100% clusters implemented Sites committed to implement with leadership support
Implementation quality: Consistency of implementation across settings and within planned time frames to the intervention strategy	Overall high implementation quality ie (high fidelity, low adaptations, high effectiveness) for the strategies	Implementation fidelity scoring Nurse Surveys (evaluation) Audits Site implementation tracking	High implementation scores (mean out of 5) High fidelity = 4.5 Low adaptations = 1.3 High effectiveness = 4.0 77.8% behaviour change techniques fully (14/18) 22% partially implemented (4/18) Low effectiveness for champions, medical engagement
Maintenance/Sustainability: The maintenance dimension assesses both individual-level behaviour change and organisational-level intervention sustainability	>80% sites embedded business as usual 12 months post-implementation Proportion of sites with education program embedded in new nurse orientation program	Site implementation tracking	All sites have a plan for ongoing education and use

*Definitions from Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health, 89(9), 1322–1327. 10.2105/ajph.89.9.1322 Abbreviation: ECAT – Emergency care and treatment protocols, ED – emergency department. ^Data Sources: 1: Nurse Surveys, 2: Audits of emergency nurse ECAT use (Audits), 3: Implementation nurse fidelity scoring, 4: Site implementation tracking

The study was approved by and conducted in accordance with Sydney Local Health District Human Research Ethics Committee (2022/ETH01940) requirements. The 2025 National Health and Medical Research Council Australian Code for the Responsible Conduct of Research standards [42] were followed and reporting guided by Standards for Reporting Implementation Studies (StaRI) Statement [43]. The study was prospectively registered with Australian New Zealand Clinical Trials Registry (ACTRN12622001480774p), with variations reported in Supplementary file 2.

Data sources and participants

Four data sources were used in this multi-method evaluation. Each source had different participants and methods, explained below, and linked to the RE-AIM framework (Table 2).

Data source 1: nurse surveys

Population and process: All nursing staff employed at the 29 study sites were invited to participate in an electronic survey before (control) and after (intervention) the implementation of ECAT protocols. Surveys were administered through REDCap® (Research Electronic Data Capture) [44], a secure web-based data management and survey delivery application. Invitations were sent via email by nurse managers or study implementation nurses every two weeks, and advertisements featuring QR codes were placed in staff meal rooms to encourage their participation. Participants received a $20 AUD voucher as a token of appreciation for their time. Consent was obtained electronically through REDCap®, and responses were anonymous.

Survey Instruments: Surveys were adapted from similar questionnaires developed using the Theoretical Domains Framework (TDF) [45] and other validated change management tools [46] (Supplementary file 3). The surveys included closed and open-ended questions, enabling elaboration and capture of potentially overlooked perspectives. The surveys comprised six sections and were conducted to evaluate different aspects of implementation of ECAT protocols:

• i.Participant characteristics (Intervention and Control groups): position, years worked as a nurse and in the ED, highest qualification
• ii.Confidence in ECAT protocol use (Intervention and Control groups): nurses indicated their confidence in ECAT protocol use (e.g. protocol selection, ordering of pathology, radiology, medications) on a scale of 0 (no confidence) to 10 (complete confidence) for adult and paediatric presentations.
• iii.ECAT protocol usability (Intervention group): statements related to usability and feasibility of protocols scored on a scale of 0 (strongly disagree) to 10 (strongly agree).
• iv.Utility of implementation strategies (Intervention group) scored on scale of 0 (strongly disagree) to 10 (strongly agree)
• v.Education program evaluation (Intervention group)
• vi.Free-text questions for elaboration and general comments (Intervention and Control groups)

An advisory group of emergency nursing and medical researchers and clinicians with content expertise assessed both surveys for face and content validity. Items were phrased both positively and negatively to counter response set bias. The advisory group completed the survey on desktop computers and mobile devices, providing feedback on the functionality of the survey, completion time, typographical errors, questions irrelevant to their allocated role, and the clarity of the instructions. Comments were collated and only minor editorial changes were needed.

Data source 2: audits of emergency nurse ECAT use

Population and process: Site implementation nurses conducted up to three audits for each eligible emergency nurse 6–12 weeks after implementation. Emergency nurses included in the audit had participated in EPIC-START Education Programs and were approved to use ECAT protocol in practice. Lists of eligible staff were uploaded into a purpose-built audit tool in REDCap® [44], with pre-entered nurse names, sites, and attended courses. Implementation nurses randomly selected patient presentations in which an ECAT was used, this was the only criteria for patient selection.

Instrument: The audit tool was purpose-built to assess ECAT use (quantity) and appropriateness of use (quality). The audit tool was developed by investigators based on the ECAT protocol requirements and structure: History, Identify Red flags, Assessment, Interventions, Diagnostics, communication and reassessment [47]. The audit tool was reviewed and tested by implementation nurses, who gave feedback on usability and accuracy. Their knowledge of ECAT protocols and ED practices was vital in development. Implementation nurses received training and a quick reference guide. Investigators regularly met with auditors to address inconsistencies. Since auditors were also implementation nurses involved from the start, they understood ECAT protocol requirements. They were experienced emergency nurses familiar with assessment and documentation. The audit tool also served as an implementation strategy, allowing nurses to record and provide feedback.

In the REDCAP audit tool, auditors recorded the total number of ECAT protocols used by each nurse. If ECAT protocols were used fewer than five times, a reason or a follow-up note was documented. Nurses who had not used ECAT at all for reasons such as retired or on long leave were not counted in the proportions for reach. Then, audits were conducted using available clinical information from the electronic medical record, ECAT protocols and relevant guidelines. ECAT use was audited for the appropriateness of nurse-initiated protocol selection, specifically whether the nurse selected the correct ECAT, and the appropriateness of radiology, pathology, and/or medication selection and use. Items were scored as yes (appropriate), no (not appropriate), not applicable (e.g. not indicated or not available on site), or not sure/partial (insufficient information to decide). Auditors were subsequently able to select the reason for their choice and add comments to explain their choice as needed. The quality of nurse documentation for patient history, red flags, assessment, and reassessment was also audited, as these are essential elements of emergency nursing care and would affect nurse ECAT selection. These items were scored as Yes, No (no documentation available), Partial, or Not Applicable.

Data source 3: implementation nurse fidelity scoring

Population and process: Participants included a convenience sample of site implementation nurses, principal investigators, and individuals who were significantly involved in the implementation process at each cluster. Participants were asked to score the implementation strategies in an evaluation scoring sheet during an online meeting with an author (SK) and two to three participants from each cluster. Participants were asked questions such as “Do you think the education programs were delivered as intended?”, “Do you think the education programs were effective as a strategy at your sites?”, and “Were changes made to the education program? What changes and why?”. Participants were asked to score each strategy and then discuss as a group to reach consensus on their scores. After the meeting, the final decisions were shared with participants for feedback. Final scores were collated and compared across clusters.

Instrument: The implementation score sheet was developed similarly to previous evaluations, with scoring for fidelity, adaptations and effectiveness of ECAT implementation, scoring strategies on a 0–5 scale [48]. Each strategy was scored on an implementation fidelity scale and an adaptation scale (Supplementary file 3). The implementation fidelity score was defined as the extent to which the strategy was implemented as intended in the initial plan, scored from 0 (not implemented) to 5 (fully implemented). The adaptation score indicated the level of adaptations made to implement the plan, scored from 0 (no adaptations) to 5 (many changes). The effectiveness score was how effective the item was as a strategy, scored from 0 (not effective) to 5 (very effective). Results are presented with a mean score out of five and range. Site implementation nurses provided comments on the reasons for their choice.

Data source 4: site implementation tracking logs

Detailed implementation tracking logs were kept by site implementation nurses in an Excel^TM template. Implementation nurses were asked to record education events, issues, actions, and meetings related to the implementation of ECAT. Site implementation tracking logs were collated. Personal identifiers were removed, and data synthesised and summarised into categories (site visits, meetings, risks, education events) by author (RR) and checked by second author (SK). A summary of events with counts is presented (Supplementary file 1).

Data analysis

Where quantitative data were used, they were analysed by statisticians in the investigator team (LL, NRA) using the Statistical Analysis System (SAS 9.4), SAS Institute Inc, North Carolina, USA. Participant characteristics were summarised using descriptive statistics. Categorical variables were presented as numbers and percentages [n (%)]. Mean and standard deviation (SD) were presented for normally distributed numerical variables, and median and interquartile range (IQR) were presented for not normally distributed numerical variables. If any measures were missing, the percentages for categorical variables and summary statistics for numerical variables were calculated based on the available values. Comparisons of Intervention and Control group confidence scores from the nurse surveys were conducted using Wilcoxon rank test and the Kruskal-Wallis test for comparison between clusters.

Qualitative data from free text responses were imported into NVivo™ v14 (QSR International, Burlington, MA) and analysed collectively using the Graneheim, Lindgren [49] method of inductive content analysis. The text was read several times to obtain a sense of the whole; the content was divided into meaningful units (codes) for analysis. The codes were abstracted to sub-categories and categories at the highest level. Abstraction and interpretation of the latent content were kept low as the content was brief. This process was completed by one author (SK); increasing the reliability of the interpretation by limiting the number of coders. Analysis was checked by a second author (KC). Any queries were resolved through discussion among authors (SK, KC). Quotes have been edited for simple spelling and grammatical errors.

Results

Data sources and their characteristics

Surveys were completed by 787 nursing staff whose characteristics are presented in Supplementary file 1. Fidelity scores were completed by 11 implementation nurse participants representing each of the four clusters. Audit data was collected by six implementation nurses for 628 staff who used 1288 ECAT protocols. Implementation tracking logs contained over 900 entries from all sites. The findings are presented per the RE-AIM domains and summarised in Table 2.

Reach

ECAT protocol uptake

ECAT uptake was high. Nursing respondents reported high usage of ECAT in their practice (median (IQR) 9.0 (8.0, 10.0)) (0’ indicates never. ‘10’ indicates all the time) (Table 3). Of eligible nurses audited, 92.9% (n = 509) used at least one ECAT. Those who hadn’t were mostly on leave, retired, or worked in areas like management and were unable to use ECAT.

Table 3.

Usability of nurse-initiated protocols (ECATs) and implementation evaluation results from nursing survey in intervention period (trained staff only) medians and interquartile ranges (IQR)

Item	All (n = 201)	Cluster 1 (n = 30)	Cluster 2 (n = 70)	Cluster 3 (n = 50)	Cluster 4 (n = 70)	Cluster comparisons p-value*
ECAT usability
I use ECATs in my practice	9.0 (8.0, 10.0)	9.0 (8.0, 10.0)	8.0 (6.0, 9.0)	9.0 (8.0, 10.0)	9.0 (8.0, 10.0)	<0.001
The doctors see the patients before I can use an ECAT	6.0 (3.0, 8.0)	5.0 (3.0, 7.0)	8.0 (5.0, 9.0)	7.0 (5.0, 8.0)	5.0 (3.0, 7.0)	<0.001
The department is usually too busy for me to use an ECAT	5.0 (3.0, 7.0)	4.0 (2.0, 6.0)	6.0 (4.0, 8.0)	5.0 (3.0, 7.0)	5.0 (2.0, 7.0)	0.015
It takes me too long to order using the online ECAT	4.0 (2.0, 7.0)	2.0 (1.0, 3.0)	4.0 (2.0, 7.0)	6.0 (3.0, 8.0)	4.0 (2.0, 7.0)	0.001
My workload is too high to use ECATs	5.0 (2.0, 7.0)	2.5 (2.0, 4.0)	5.0 (3.0, 7.0)	5.0 (3.0, 7.0)	4.0 (2.0, 6.0)	0.002
I would prefer to wait for the doctor to review the patient	2.0 (1.0, 5.0)	2.0 (0.0, 3.0)	2.0 (1.0, 5.0)	3.0 (1.0, 5.0)	2.0 (1.0, 3.5)	0.043
I do not feel confident in using ECATs	1.0 (0.0, 3.0)	1.0 (0.0, 2.0)	2.0 (1.0, 4.0)	1.0 (0.0, 3.0)	1.0 (0.0, 3.5)	0.237
I do not think its my job to have to use ECATs	1.0 (0.0, 3.0)	1.0 (0.0, 2.0)	1.0 (0.0, 3.0)	1.0 (0.0, 3.0)	1.0 (0.0, 4.5)	0.481
Why or why not I used ECAT
The ECAT protocols are easy to follow	8.0 (7.0, 9.0)	9.5 (8.0, 10.0)	8.0 (7.0, 9.0)	8.0 (7.0, 9.0)	8.0 (7.0, 9.0)	0.021
I use the ECAT protocols to guide my assessment	8.0 (6.0, 9.0)	9.0 (8.0, 10.0)	8.0 (6.0, 9.0)	8.0 (5.5, 9.0)	8.0 (7.0, 9.0)	0.011
There are enough protocols to treat most presenting problems	7.0 (5.0, 8.0)	8.0 (7.0, 9.0)	7.0 (5.0, 9.0)	6.0 (3.0, 8.0)	7.0 (5.0, 8.0)	0.001
Protocols are easy to locate electronically	9.0 (7.0, 10.0)	9.5 (9.0, 10.0)	9.0 (6.0, 10.0)	9.0 (7.0, 10.0)	9.0 (8.0, 10.0)	0.033
The process to order a medication is quick	8.0 (5.0, 9.0)	8.5 (8.0, 10.0)	7.0 (4.0, 8.0)	8.0 (6.0, 9.0)	8.0 (6.0, 9.0)	<0.001
The process of ordering an x-ray is quick	7.0 (5.0, 9.0)	8.0 (6.0, 10.0)	6.0 (4.0, 8.0)	8.0 (6.0, 9.0)	7.0 (5.0, 9.0)	<0.001
The process to record that I have used an ECAT is quick	8.0 (5.0, 9.0)	9.0 (8.0, 10.0)	7.0 (5.0, 9.0)	8.0 (5.0, 9.0)	8.0 (5.0, 9.0)	0.005
Implementation evaluation	All	Cluster 1	Cluster 2	Cluster 3	Cluster 4	p-value*
Supportive of use:
Hospital executive	7.0 (5.0, 9.0)	8.0 (6.0, 10.0)	8.0 (4.5, 9.0)	6.0 (4.0, 8.0)	6.5 (5.0, 8.5)	0.113
ED nursing leadership	9.0 (8.0, 10.0)	10.0 (9.0, 10.0)	9.0 (8.0, 10.0)	9.5 (8.0, 10.0)	9.0 (8.0, 10.0)	0.154
ED medical staff	7.0 (5.0, 9.0)	7.0 (5.0, 9.5)	8.0 (6.0, 9.0)	7.0 (5.0, 9.0)	6.5 (5.0, 8.0)	0.359
Radiology	7.0 (5.0, 9.0)	8.0 (7.0, 10.0)	7.0 (5.0, 8.0)	7.0 (5.0, 9.0)	7.0 (4.5, 8.0)	0.065
Pharmacy	7.0 (5.0, 9.0)	8.0 (5.5, 10.0)	7.5 (5.0, 9.0)	6.0 (4.0, 9.0)	6.0 (5.0, 8.0)	0.363
Pathology	8.0 (5.0, 9.0)	9.0 (6.0, 10.0)	8.0 (5.0, 9.0)	8.0 (5.0, 9.0)	8.0 (5.0, 9.0)	0.286
The eMR link to the ECAT protocols helps make them easier to use	8.0 (7.0, 10.0)	9.0 (8.0, 10.0)	8.0 (6.0, 9.0)	8.0 (7.0, 10.0)	8.5 (7.0, 9.0)	0.081
Sufficient time was allocated to education	8.0 (6.0, 10.0)	9.0 (7.0, 10.0)	8.0 (6.0, 9.0)	8.0 (6.0, 10.0)	8.0 (6.0, 10.0)	0.597
The ECAT Policy Directive clearly explains my scope of practice	8.0 (7.0, 10.0)	9.0 (8.0, 10.0)	8.0 (6.0, 9.0)	9.0 (7.0, 10.0)	9.0 (7.0, 10.0)	0.017
My scope of practice was clear to me when using ECAT	9.0 (7.0, 10.0)	9.0 (9.0, 10.0)	8.0 (7.0, 9.0)	9.0 (8.0, 10.0)	9.0 (7.0, 10.0)	0.017
Support from the EPIC-START project team has been integral to successful implementation of ECATs	8.0 (6.0, 10.0)	9.0 (7.0, 10.0)	8.0 (5.0, 9.0)	8.0 (6.0, 10.0)	8.0 (7.0, 10.0)	0.115
I had enough clinical support for the transition	8.0 (6.0, 10.0)	9.0 (7.0, 10.0)	8.0 (5.0, 9.0)	8.0 (6.0, 9.0)	8.0 (7.0, 10.0)	0.120
The HETI mandatory ECAT eLearning modules helped prepare me to use ECATs in clinical practice	8.0 (5.0, 9.0)	8.0 (7.0, 10.0)	7.0 (5.0, 9.0)	8.0 (6.0, 10.0)	8.0 (6.0, 9.0)	0.039
The face-to-face education program helped prepare me to use ECATs in clinical practice	9.0 (8.0, 10.0)	10.0 (9.0, 10.0)	9.0 (7.0, 10.0)	9.0 (7.0, 10.0)	9.0 (8.0, 10.0)	0.060
Participant manuals helped prepare me to use ECATs in clinical practice	8.0 (5.0, 9.0)	9.0 (8.0, 10.0)	7.0 (5.0, 9.0)	8.0 (5.0, 9.0)	7.5 (5.0, 9.0)	0.018
Individual feedback on my use of ECAT helped me use	8.0 (5.0, 9.0)	9.0 (7.0, 10.0)	7.0 (4.0, 9.0)	8.0 (5.0, 10.0)	8.0 (6.0, 9.0)	0.138
Departmental feedback on the use of ECAT was valuable to adjust my practice (e.g. feedback during huddles or via newsletter)	8.0 (5.0, 9.0)	8.0 (6.0, 9.0)	8.0 (5.0, 9.0)	8.0 (5.0, 9.0)	7.0 (5.0, 9.0)	0.122
Follow-up in-services helped me to use ECATs in clinical practice	8.0 (5.0, 9.0)	7.0 (6.0, 9.0)	5.0 (3.0, 8.0)	8.0 (6.0, 9.0)	8.0 (6.0, 9.0)	0.002
Clinical champions available on floor helped me to use ECATs in clinical practice	6.0 (3.0, 9.0)	7.5 (5.5, 9.0)	5.0 (2.0, 8.0)	7.0 (4.0, 9.0)	7.0 (4.0, 9.0)	0.024
Opportunity to ask questions helped me to use ECATs in clinical practice	8.0 (6.0, 10.0)	9.0 (7.0, 10.0)	7.0 (5.0, 9.0)	9.0 (6.0, 10.0)	9.0 (6.0, 10.0)	0.028

*Comparisons between clusters, The Kruskal-Wallis test. ECAT - Emergency Care Assessment and Treatment, ED – Emergency Department, eMR – electronic medical record, EPIC-START - Emergency nurse Protocol Initiating Care—Sydney Triage to Admission Risk Tool, HETI – Health Education and Training (department)

Effectiveness

Appropriate use of ECAT protocols

ECAT protocols were used appropriately across all clusters (Table 4) and the appropriateness of ECAT selection was very high, at 93.7% fully appropriate (1288/1375). There were 46 audits with inappropriate protocol selection (3.3%) (42 unique staff) and 41 not rated as unsure/partial appropriate (3.0%) [(isolated limb injury (n = 17), shortness of breath (n = 11) and abdominal pain (n = 10)]. Errors resulted from a mismatch with the selected ECAT and patient clinical information (36.4%), presenting history (24.1%), or past medical history (16.1%) (Supplementary file 1). Where clinical investigations and treatments were required, pathology (78.5%) and medication (78.3%) were ordered per protocol. Appropriate radiology use was lower than these (63.0%), mostly related to insufficient documentation (69.7%) rather than incorrect radiology test selection (8.4%).

Table 4.

Appropriateness of use of ECATs, results from audit data

		All	Cluster 1	Cluster 2	Cluster 3	Cluster 4
Appropriate protocol	Yes	1288 (93.7%)	335 (97.7%)	208 (89.7%)	357 (93.5%)	388 (92.8%)
	Not sure/Partial	41 (3.0%)	4 (1.2%)	10 (4.3%)	16 (4.2%)	11 (2.6%)
	No	46 (3.3%)	4 (1.2%)	14 (6.0%)	9 (2.4%)	19 (4.5%)
History identified and documented	Yes	949 (88.9%)	333 (97.1%)	160 (69.0%)	349 (91.4%)	107 (97.3%)
	Not sure/Partial	55 (5.2%)	2 (0.6%)	25 (10.8%)	26 (6.8%)	2 (1.8%)
	No	63 (5.9%)	8 (2.3%)	47 (20.3%)	7 (1.8%)	1 (0.9%)
Red flags identified and documented	Yes	857 (78.7%)	246 (71.9%)	138 (59.7%)	341 (89.3%)	132 (98.5%)
	Not sure/Partial	72 (6.6%)	3 (0.9%)	36 (15.6%)	31 (8.1%)	2 (1.5%)
	No	160 (14.7%)	93 (27.2%)	57 (24.7%)	10 (2.6%)
Assessment identified and documented	Yes	881 (64.2%)	293 (85.7%)	124 (53.7%)	335 (87.7%)	129 (30.9%)
	Not sure/Partial	159 (11.6%)	8 (2.3%)	49 (21.2%)	40 (10.5%)	62 (14.8%)
	No	333 (24.3%)	41 (12.0%)	58 (25.1%)	7 (1.8%)	227 (54.3%)
Radiology ordered per ECAT	Yes	225 (63.0%)	71 (79.8%)	23 (38.3%)	93 (66.0%)	38 (56.7%)
	Not sure/Partial	25 (7.0%)	1 (1.1%)		11 (7.8%)	13 (19.4%)
	No	107 (30.0%)	17 (19.1%)	37 (61.7%)	37 (26.2%)	16 (23.9%)
Pathology ordered per ECAT	Yes	424 (78.5%)	46 (63.9%)	52 (70.3%)	192 (95.0%)	134 (69.8%)
	Not sure/Partial	30 (5.6%)	1 (1.4%)		5 (2.5%)	24 (12.5%)
	No	86 (15.9%)	25 (34.7%)	22 (29.7%)	5 (2.5%)	34 (17.7%)
Medication ordered per ECAT	Yes	702 (78.3%)	203 (76.9%)	110 (72.8%)	215 (91.5%)	174 (70.7%)
	Not sure/Partial	44 (4.9%)	3 (1.1%)		9 (3.8%)	32 (13.0%)
	No	150 (16.7%)	58 (22.0%)	41 (27.2%)	11 (4.7%)	40 (16.3%)
Repeat assessment done?	Yes	925 (83.2%)	238 (86.5%)	170 (73.6%)	252 (75.0%)	265 (98.1%)
	Not sure/Partial	2 (0.2%)			1 (0.3%)	1 (0.4%)
	No	185 (16.6%)	37 (13.5%)	61 (26.4%)	83 (24.7%)	4 (1.5%)

ECAT - Emergency Care Assessment and Treatment

Nurse confidence

Nurse confidence significantly increased in the Intervention compared to the Control group in all aspects of nurse-initiated care. In particular, confidence increased for identifying the need to initiate nurse-led care, as well as clinical treatments, such as pathology, radiology, opioids, simple analgesia, bronchodilators, antiemetics, and IV therapy (Table 5). In subgroup analysis comparing years of emergency nursing experience, a significant increase was still observed in most groups, except for simpler clinical treatments (e.g., simple analgesia, pathology) that had already reached maximum confidence scores (Supplementary file 1). In comments, nurses expressed they felt confident to use ECAT protocols but were less confident with more complex or uncommon clinical treatments. They noted that opportunities to practice were variable and sometimes unavailable, such as when doctors started treating patients before the nurse. One participant described challenges with confidence due to reduced exposure or opportunity:

With less common presentations- there is reduced confidence initiating medications due to uncertainty of treatment pathway. With more severe presentations- there is reduced opportunity for nurse-initiated care as they are often reviewed relatively quickly by a medical officer. Therefore, reduced confidence initiating medications in relation to those presentations; which in turn reduces frequency and confidence. RN3156

Table 5.

Nursing confidence in using ECATs sites combined, intervention vs control periods; medians and interquartile ranges (IQR)

	All staff				Nursing confidence Control vs Intervention (trained)
	Control	Intervention	Intervention (trained)	Intervention (untrained)	Wilcoxon Statistic	p value^	Control vs Intervention (all) p value^
	n = 582	n = 469	n = 205	n = 264
Identify the need to start an ECAT protocol	9.0 (7.0, 10.0)	9.0 (8.0, 10.0)	9.0 (8.0, 10.0)	8.0 (7.0, 10.0)	91585	<0.001	0.043
Identify the correct ECAT protocol		8.0 (7.0, 9.0)	9.0 (8.0, 10.0)	8.0 (7.0, 9.0)
Pathology	8.0 (6.0, 10.0)	9.0 (7.0, 10.0)	9.0 (8.0, 10.0)	8.0 (6.0, 10.0)	94572	<0.001	<0.001
Radiology	6.0 (3.0, 9.0)	8.0 (5.0, 9.0)	9.0 (7.0, 10.0)	6.0 (2.0, 9.0)	98934	<0.001	<0.001
Opioid analgesia	8.0 (5.0, 10.0)	8.0 (6.0, 10.0)	9.0 (8.0, 10.0)	7.0 (3.0, 9.0)	94476	<0.001	0.006
Simple analgesia (e.g. paracetamol)	10.0 (8.0, 10.0)	10.0 (9.0, 10.0)	10.0 (9.0, 10.0)	9.0 (8.0, 10.0)	83148	0.002	0.940
Bronchodilators	8.0 (6.0, 10.0)	9.0 (7.0, 10.0)	9.0 (8.0, 10.0)	8.0 (6.0, 9.0)	92452	<0.001	0.005
Thrombolysis	4.0 (0.0, 7.0)	4.0 (1.0, 7.0)	5.0 (1.0, 8.0)	3.0 (0.0, 6.0)	71662	0.012	0.750
Antiemetics	8.5 (6.0, 10.0)	9.0 (8.0, 10.0)	10.0 (9.0, 10.0)	9.0 (7.0, 10.0)	94680	<0.001	<0.001
IV therapy	8.0 (5.0, 10.0)	9.0 (6.0, 10.0)	9.0 (8.0, 10.0)	7.0 (5.0, 9.0)	97614	<0.001	<0.001
IV antibiotics	5.0 (1.0, 8.0)	7.0 (3.0, 9.0)	8.0 (5.0, 9.0)	6.0 (2.0, 8.0)	97713	<0.001	<0.001
Steroids	5.0 (1.0, 8.0)	7.0 (4.0, 9.0)	8.0 (6.0, 10.0)	6.0 (2.0, 8.0)	100733	<0.001	<0.001

^ Wilcoxon rank test, ECAT - Emergency Care Assessment and Treatment

Overall, respondents reported they were able to use ECAT protocols because the protocols were easy to follow and use. Nurses reported that feeling confident takes practice and time, for example:

I find that my confidence in using the [ECATs] grow with practice as well RN3114

Adoption

Overall implementation at study sites

All 29 sites implemented ECAT protocols (100%) using our implementation strategy. However, this was not without challenges. Once the ECAT protocols and policies were released, and to ensure the education program aligned with policy, we needed to train over 800 nurses in 5 months. This involved non-clinical preparation time, travel over large geographical distances of up to 600 km, and rostering considerations for attendance at a one- or two-day course (Fig. 1).

Fig. 1.

Timelines of ECAT nurse-initiated care protocol implementation as part of trial and planned implementation (PI). *Implementation go-live date for cluster, PI- planned implementation showing stepped approach

Implementation quality (implementation strategies)

Overall, implementation quality was good, with high combined mean implementation fidelity scores (4.6/5), low adaptations (1.3/5), and high strategy effectiveness (3.9/5) (Table 6). Four of six of the implementation strategies had implementation scores over 4/5, except Clinical Champions and Medical Engagement. Of the 18 behaviour change techniques in the implementation plan, 14 (77.8%) were fully and four (22.2%) partially implemented (Supplementary File 1). At three months post-implementation, 885 nurses had attended one of the three education programs (65 Instructor/820 for EPIC-START CIN/CREDIT) at 97 courses (Table 7).

Table 6.

Implementation scoring by research site nurses for implementation fidelity, adaptation and effectiveness scores (0–5)

	Implementation score	Adaptation	Effectiveness	Comment
TOTALS	4.5	1.3	4.0
(average (range))
Strategies
EPIC-START Education Programs - overall	4.7 (4–5)	1.5 (1–2.5)	4.0 (3.5–5)
EPIC-START Instructor	5 (all 5)	0 (0–1)	3.83 (3–5)
EPIC-START CIN*	4.7 (4–5)	2.3 (2–3)	4.2 (2.5–5)	Cluster 4 lower effectiveness at 2.5
EPIC-START CREDIT*	5 (all 5)	0 (0–1)	5 (all 5)
Environmental change e.g. electronic medical record change	5 (all 5)	1.7 (1–5)	4.3 (4–5)	Cluster 4 lots of adaptation
Leadership and stakeholders	5 (5–5)	1.3 (0–2)	4.7 (4–5)
Audit and feedback	4.3 (4–5)	1.0 (0–2)	5 (5–5)
Clinical champions	3.7 (2–4)	2.8 (1–3.5)	2.5 (2–3)
Medical engagement	3.5 (3–4.5)	1.3 (0–2)	3.0 (2–4)
Scores per cluster (average (range))
Cluster 1	4.4 (3–5)	1.5 (0–3.5)	4.1 (2.5–5)
Cluster 2	4.6 (4–5)	1.3 (0–3)	3.3 (2–5)
Cluster 3	4.6 (3–5)	1.1 (0–2)	4.6 (3–5)
Cluster 4	4.6 (2–5)	1.8 (0–5)	3.7 (2–5)

Table 7.

Education tracking at go-live and at three months post-implementation for EPIC-START Instructor, EPIC-START CREDIT and EPIC-START CIN courses, n (%)

	Cluster 1		Cluster 2		Cluster 3		Cluster 4		Total
	Go-live	3 month	Go-live	3 month	Go-live	3 month	Go-live	3 month	3 month	6 months
Individual attendance
EPIC-START Instructor	13	13	18	20	17	17	12	15	62	65
EPIC-START CREDIT	57 (92%)	57 (92%)	106 (97%)	106 (97%)	126 (100%)	126 (100%)	94 (89%)	106 (100%)	383(95%)	395 (98%)
EPIC-START CIN	42 (58%)	63 (86%)	197 (76%)	202 (78%)	10 (9%)	41 (38%)	22 (8%)	119 (45%)	271(38%)	425 (60%)
										885 total, 820 CIN/CREDIT (74%)

EPIC-START education programs

The EPIC-START Education Program was well received, implementation nurses expressed that the EPIC-START CREDIT and EPIC-START CIN Courses were effective. Nurses identified that sufficient time was allocated to education (median (IQR) 8.0 (6.0, 10.0)), and that face-to-face learning was especially beneficial (median (IQR) 9.0 (8.0, 10.0)). Although, qualitative data indicated some nurses felt a need for more education and training to support implementation.

EPIC-START Instructor course: The EPIC-START Instructor Courses (n = 9) were delivered with high reach (100%), 63 instructors attended across four clusters. Instructor Courses were implemented as planned with high fidelity (5.0/5), and no adaptations (0.0/5) and moderate effectiveness (3.8/5). Effectiveness scoring was consistent with instructor evaluation survey findings (31 instructors, 57.1% response rate). Instructors indicated they had a clear understanding of their role (median (IQR) 8.5 (7.0, 10.0)) and were engaged with the course (median (IQR) 9.0 (7.0, 10.0)). Instructors reported that the in-person skill and scenario stations provided a better understanding of the application of ECAT (median (IQR) 9.0 (6.0, 10.0)) and 8.0 (6.0, 10.0), respectively). They expressed confidence in delivering education sessions (median (IQR) 8.0 (7.0 to 10.0)), but were not as prepared as they would have liked:

I really enjoyed the day but when I came to instructing on the days I didn’t feel prepared. – RN 2079

EPIC-START CIN/CREDIT Courses: 88 EPIC-START courses were delivered (38 CREDIT and 50 CIN) over the implementation period (Table 7). By go-live, all sites had achieved EPIC-START CREDIT attendee completion rates of over 90%, meeting the 80% reach target. Participants from both courses reported that they found the course to be engaging (median (IQR) 8.0 (7.0, 9.0)) and prepared them to use ECAT protocols in clinical practice (median (IQR) 8.0 (7.0, 9.0)). Face-to-face skills and scenario stations were seen to be effective (median (IQR) 8.0 (7.0, 9.0)), but the participant manual and recommended pre-attendance videos were less effective (median (IQR) 7.0 (5.0, 9.0)). Implementation nurses scored both courses with high fidelity (5.0/5, 4.7/5) and reported some adaptations for EPIC-START CIN (2.3/5), such as changing the timetabling, but they did not change content.

Clinical champions/Site implementation nurses

The clinical champions’ strategy, which included self-nominated nurses providing on-site implementation support, rated lower than most other strategies for fidelity (3.7/5) and effectiveness (2.5/5), despite moderate adaptations to improve their use (2.8/5). Likewise, nurses reported that clinical champions were less helpful (median (IQR) 6.0 (3.0, 9.0)) compared to other strategies, but they felt they had sufficient clinical support (median (IQR) 8.0 (6.0, 10.0)). This is likely related to the presence of implementation nurses who were very helpful (median (IQR) 8.0 (6.0, 10.0)). There was positive feedback on the impact of site implementation nurses:

[Site Implementation nurses- (names redacted)] have been exceptional in the promotion, implementation and transition to ECAT RN1054

Audit and feedback

Implementation quality for the Audit and Feedback was high in fidelity (4.3/5), with low adaptations (1.0/5) and high effectiveness (5.0/5). This was supported by audit completion numbers, with a total 1375 audits with at least one completion for 509 nurses and all three for 419 of those staff (73%) (Supplementary file 1). Feedback was recommended for 674 (45.1%) of audits and 361 (57.5%) of eligible staff. In the nurse survey, participants responded that individual and departmental feedback was helpful (Table 4).

Medical staff engagement

The medical engagement strategy had moderate fidelity (3.8/5) and was partly effective (2.5/5). Six medical-specific briefings were recorded, as well as monthly steering committee meetings. However, despite the invitation to these meetings, medical staff attendance was poor. Nurses reported they felt doctors were mostly supportive (median (IQR) 7.0 (5.0, 9.0)).

Environmental strategies - policies and electronic medical record (eMR)

Environmental strategies, specifically changes to the eMR for an easier ordering process, scored highly for fidelity (5.0/5), with some adaptations primarily in one cluster (Cluster 4), and demonstrated high effectiveness (4.3/5). All sites used the statewide policy to guide practice, and one site initiated a local policy to further inform local practice. All clusters used the state-recommended eMR build, with some local adaptations. Nurses reported that the enablers to use ECAT protocols were that they were easy to locate electronically, easy to follow and record, and quick to order medication (Table 3). However, some nurses referred to challenges with access or complexity of eMR use (n = 20), such as a lengthy process:

too many clicks to get to where you need to initiate orders RN 2015

Steering committees, leadership and stakeholder outreach

Fidelity to leadership and stakeholder strategies was high (5.0/5) with low adaptations (1.0/5) and moderate effectiveness (3.8/5). Over 450 site meetings were conducted with stakeholders, leadership, and allied health professionals. A total of 187 site visits were recorded by implementation nurses, with the majority from regional sites, which have more locations across a large geographical area for implementation nurses to cover. Site visit numbers were lower at the metropolitan sites because implementation nurses had fewer sites to visit and were stationed at one of the larger sites within their cluster. As a result, even if they were on-site, these visits were not recorded as such. Nursing leadership was reported as highly supportive in the nursing survey (median (IQR), 9.0 (8.0, 10.0)). However, the following nursing survey comment highlighted some challenges:

Some of our ED NUMs [nurse unit managers] don’t want us to commence ECAT protocols at all, preferring to wait until Dr [doctor] sees them. RN 2133

Maintenance

Site implementation tracking logs confirmed that all sites have embedded the ECAT protocols as ‘business as usual’ twelve months post-implementation. This was achieved through the inclusion of ECAT training in onboarding processes, staff training, and integration into policy. Planning for maintenance commenced pre-implementation. Each site prepared and endorsed its own maintenance plan, involving its steering committees and key stakeholders. At the time of writing, audit and feedback have been integrated into site systems through the development of eMR reports and embedding into staff roles, and education programs have been integrated into existing career development pathways with site adaptations.

Several risks, unplanned events, and disruptions occurred and were recorded over the 12-month study period. These included industrial action (nursing and medical), where nurses were instructed not to use ECAT by local union representatives, eMR downtime (which required staff to access paper versions of protocols), and significant natural weather events. One cluster was impacted by a Category 4 tropical cyclone and flooding, which impacted the ability of staff to visit sites, and impacted staff’s use of ECAT on-site. Mitigation strategies were established through consultation and pre-arranged mechanisms.

Discussion

This study demonstrates the effective implementation of ECAT protocols through a behaviour change-informed strategy, as evidenced by high reach, effectiveness, adoption, quality, and maintenance across 29 emergency sites in NSW, Australia. Implementation of ECAT protocols was successful, with high appropriateness of use, high nurse confidence and high patient reach evident in quantitative and qualitative measures. In this study, the implementation strategies were used with high quality, fidelity, and effectiveness, with minimal adaptation of strategy components.

Effective implementation of ECAT protocols

The implementation of ECAT protocols was effective using the strategies. The reach for ECAT protocol use was over 90%, which is relatively high for implementation studies in EDs [50]. Further, nurses used ECAT protocols appropriately, with 94% correctly using ECAT and ordering the most appropriate medication or pathology test. This supports research demonstrating that nurses, when enabled, can select and use ECAT protocols without incident [18, 19].

Increased confidence may have contributed to the effectiveness of the implementation, as confidence is a mediator of barriers to implementation [51]. Nurse confidence in initiating care increased significantly across nearly all care deliverables within the ECAT protocols, independent of years of emergency experience. The greatest gains in confidence were observed among less-experienced nurses and for more complex clinical treatments, suggesting our bespoke education program reduced professional hesitancy and provided a safety net for the use of ECAT by addressing staff concerns and learning needs [52]. The Education program implementation strategy was developed and delivered by highly experienced emergency nurses and educators using robust educational principles and intended learning outcomes, and providing protected face-to-face learning as requested by staff during pre-implementation consultation and surveys [28].

Challenges of ECAT implementation

Contextual and organisational factors resulted in variation in uptake [53]. Some sites had medical-led care models or medical hierarchy, which constrained ECAT use, despite initial organisational readiness and enthusiasm. This affected the nurse’s capacity to begin care, as some medical officers assigned themselves to patients too early, before they were ready. This meant ECAT protocols could not be used, and patient care was delayed.

Other challenges arose from unplanned disruptions, which posed real risks to implementation continuity, such as industrial action, government delays in releasing the protocols, government mandates to implement earlier than sites had originally intended, and catastrophic weather events. The ability of sites to adapt and sustain ECAT use in the face of these challenges attests to the resilience of the implementation design, similar to our previous work [54], though such events should be systematically considered in future scalability planning.

Inappropriate ECAT selection was rare but occurred, especially with a mismatched medical history or clinical assessment. Nurses should conduct a thorough history and assessment before choosing ECAT, in line with NSW ECAT policy [55]. It is strongly recommended that nurses perform a comprehensive assessment of their patients prior to ECAT selection to aid ECAT selection through use of the evidence-based and validated HIRAID® emergency nursing framework [56].

The majority of the implementation strategies were used successfully. However, unexpectedly, and in contrast to other literature [57, 58], clinical champions scored low on fidelity and effectiveness in this study. This was despite the use of supports known to improve champion effectiveness [59]. Champions self-nominated to the role and received resources and support from the implementation nurses. We hypothesise that, because most champions hold clinical positions with considerable workloads, they were unable to dedicate the necessary time. Relying on clinical champions as an implementation strategy in acute healthcare is becoming increasingly impractical and challenging. Adding to clinicians’ workload only exacerbates existing pressures, negatively impacting their well-being. The implementation nurses, employed by the research team solely to support implementation, were relied upon as the default champions. This prompts a discussion regarding the role of clinical nurses, their participation in the translation of evidence-based practice, and the need to remunerate champions [60]. Another issue may have been that the champions were not yet confident and needed time to develop their skills before assisting others. Further research is required to elucidate the mechanisms underlying the success or failure of these strategies.

Research and practical implications

The application of the RE-AIM proved invaluable in structuring evaluation, and frameworks should be considered a standard in implementation research. However, a lack of consistency remains in evaluation assessments across the literature [61], which complicates both internal and cross-study comparisons. Further, deeper exploration by implementation researchers into the mechanisms underpinning effective facilitation and champions, the role of interprofessional collaboration, and the contextual factors that shape fidelity and sustainability is warranted.

The challenges highlight the importance of pragmatism, flexibility, and adaptation in implementing change and conducting health services research, where investigators have limited control over highly influential variables. Policy makers and health system leaders should recognise the need for both top-down mandates and local adaptability. Although feasible and often related to political cycles, accelerated rollouts require a significant investment in training, support, and change management to avoid superficial compliance and achieve meaningful clinical change. It is essential to support implementation with ongoing practice opportunities and adaptable supports to sustain confidence and competence in the specific knowledge and skills for patient assessment and critical thinking [62].

Strengths and limitations

A key strength of this research is the use of the RE-AIM framework, which is useful in evaluating pragmatic trials as it emphasises external validity and generalisability across diverse, real-world settings [63]. Other strengths lie in the use of a multi-faceted and rigorous research approach to the evaluation.

There are limitations to this study. Firstly, while the evaluation benefits from the application of the RE-AIM framework, the reliance on self-reported fidelity and reach metrics may introduce reporting bias or overestimate true levels of adoption [40], multi-methods were used to negate these concerns. Additionally, the possibility of response bias exists, as nursing staff who used the ECAT protocols may have been more likely to participate in the survey, potentially skewing the results [64]. The clinical impact of the ECAT protocols, including patient impact, safety and satisfaction, ED length of stay, and the proportion of eligible patients who received ECAT protocols will be addressed in a separate study, which is essential for a complete understanding of overall ECAT protocol impact on patient and health service outcomes. Secondly, we do not address the potential impact of concurrent initiatives or external policy changes that may have influenced the results beyond our control. To our knowledge, there were no major policy changes. Thirdly, implementation tracking logs relied on the implementation nurses entering data regularly and accurately, and the audits relied on the nurses documenting accurately, which may have resulted in inaccuracies due to missing data. Unfortunately, because the implementation timing was very close between sites, analysis to compare clusters could not be conducted, and there was no access to data from sites not part of the research to compare findings. Finally, comparisons were not presented in this study because they were beyond the scope.

Conclusion

This study reinforces the potential for behaviour-change informed implementation strategies to drive large-scale adoption of evidence-based ECAT protocols with high fidelity and reach, in diverse and resource-variable ED contexts. Relying on clinical champions as an implementation strategy in acute healthcare is becoming increasingly untenable because it imposes additional roles on clinicians already operating at full capacity. Ongoing investment in facilitation, education, and adaptive support structures is essential to maintain gains and realise the full potential of nurse-initiated emergency care.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Abbreviations

ACI

Agency for Clinical Innovation

ED

Emergency Department

eMR

Electronic medical record

ECAT

Emergency care assessment and treatment

EPIC-START

Emergency nurse protocol initiating care—sydney triage to admission risk tool

EPIC-START CIN Course

Emergency nurse protocol initiating care—sydney triage to admission risk tool clinical initiatives nurse course

EPIC-START CREDIT Course

Emergency nurse protocol initiating care—sydney triage to admission risk tool credit course

HIRAID®

History, Infection risk, Identify Red flags, Assessment, Interventions, Diagnostics, communication and reassessment

LHD

Local Health District

MEWS

Modified Early Warning Score

NSW

New South Wales

NUM

Nurse Unit Manager (might also see NM)

START

Sydney Triage to Admission Risk Tool

SW-cRCT

Stepped wedge cluster randomised controlled trial

Author contributions

Conceptualisation - KC, SK, MD, RZS, AS, MF, JC, TS conceived and designed the study. LL, KC, SK, NRA oversaw the statistical design. Other authors contributed to further refinements to the design. Data curation – SK, LL, KC, NRA, RR. Formal analysis - SK, LL, KC, NRA, RR. Funding acquisition – KC, MD, RZS, AS, MF, JC, SK, CA, TS. Investigation – SK, LL, MM, SBR, NRA, RR, SBR, LC. Methodology – KC, SK, MD, RZS, AS, JC, LL, TS. Project administration – SK, RVS, CA, MM, RR. Supervision – KC, RZS, JC, LL, MD Validation - SK, LL, KC, NRA, RR. Visualisation – SK, KC, NRA, LL, RR. Writing – original draft - SK completed the first draft of the paper with significant input from KC. Writing – review & editing – All authors have contributed substantially to the conception and design of the work and drafted or substantively revised the work. Each author has approved the submitted version and agreed both to be personally accountable for their contribution and ensure that questions related to the accuracy or integrity of any part of the work.

Funding

National Health and Medical Research Council, Medical Research Future Fund (MRFF). Australian Department of Health (MRFF), Grant number 2017845, https://www.health.gov.au/our-work/medical-research-future-fund. The funder had no role in study design, data collection, management, analysis, and interpretation, report writing, or decision to submit the report for publication.

Data availability

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

Declarations

Ethics approval and consent to participate

The Sydney Local Health District Research Ethics Committee (Reference Number 2022/ETH01940) provided ethical approval on 14 December 2022. Informed consent was obtained for survey participants, audit and implementation tracking log data did not require consent.

Consent for publication

Not applicable

Competing interests

No competing interests to declare