Formative evaluation of an emergency department clinical decision support system for agitation symptoms: a study protocol

Abstract

Introduction

The burden of mental health-related visits to emergency departments (EDs) is growing, and agitation episodes are prevalent with such visits. Best practice guidance from experts recommends early assessment of at-risk populations and pre-emptive intervention using de-escalation techniques to prevent agitation. Time pressure, fluctuating work demands, and other systems-related factors pose challenges to efficient decision-making and adoption of best practice recommendations during an unfolding behavioural crisis. As such, we propose to design, develop and evaluate a computerised clinical decision support (CDS) system, Early Detection and Treatment to Reduce Events with Agitation Tool (ED-TREAT). We aim to identify patients at risk of agitation and guide ED clinicians through appropriate risk assessment and timely interventions to prevent agitation with a goal of minimising restraint use and improving patient experience and outcomes.

Methods and analysis

This study describes the formative evaluation of the health record embedded CDS tool. Under aim 1, the study will collect qualitative data to design and develop ED-TREAT using a contextual design approach and an iterative user-centred design process. Participants will include potential CDS users, that is, ED physicians, nurses, technicians, as well as patients with lived experience of restraint use for behavioural crisis management during an ED visit. We will use purposive sampling to ensure the full spectrum of perspectives until we reach thematic saturation. Next, under aim 2, the study will conduct a pilot, randomised controlled trial of ED-TREAT at two adult ED sites in a regional health system in the Northeast USA to evaluate the feasibility, fidelity and bedside acceptability of ED-TREAT. We aim to recruit a total of at least 26 eligible subjects under the pilot trial.

Ethics and dissemination

Ethical approval by the Yale University Human Investigation Committee was obtained in 2021 (HIC# 2000030893 and 2000030906). All participants will provide informed verbal consent prior to being enrolled in the study. Results will be disseminated through publications in open-access, peer-reviewed journals, via scientific presentations or through direct email notifications.

Trial registration number

NCT04959279; Pre-results.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• Given limited prior evidence on real-life implementation of best practice recommendations for agitation symptoms, this study aims to identify patient and user-centred strategies to develop a clinical decision support (CDS) system that facilitates management of agitation in an acute care setting.

• Our tool will include pragmatic strategies to implement best practice recommendations for risk assessment and timely de-escalation techniques in agitation management prior to definitive psychiatric treatment.

• The CDS design process will follow an iterative, user-centred approach with feedback from end-users at every step to refine and develop an electronic health record embedded, fully functional prototype.

• Our risk assessment data, qualitative design, and pilot trial will arise from the same geo-political area and health system, which may limit generalisability.

Introduction

Behavioural health-related visits to emergency departments (EDs) are growing.^1–3 Agitation, defined as excessive psychomotor activity leading to aggressive and violent behaviour,⁴ is a frequent symptom of such visits. An estimated 1.7 million agitation episodes occur annually in EDs across the USA alone.^{5 6} When an individual becomes agitated, they may cause harm to themselves, hospital staff, and property.^7–9 Rapid management of agitation is imperative and use of physical restraint may be necessary to facilitate patient assessment and prevent injury.⁵ Although physical restraints are routinely used in the ED,^{10 11} they are associated with up to 37% risk of injury in patients, including blunt chest trauma, asphyxiation, respiratory depression, and sudden death.^12–17 To address these challenges, the American Association for Emergency Psychiatry sponsored Project BETA (Best Practices in Evaluation and Treatment of Agitation).¹⁸ Project BETA was a pioneer effort to create a comprehensive list of five sets of best practices for preventing and managing agitation through multidisciplinary consensus panels. Key strategies within Project BETA included use of structured risk assessment⁴ to help clinicians screen patients at risk of developing agitation and pre-emptive intervention using behavioural techniques,¹⁹ environmental modification,²⁰ and consensual use of medication therapy¹⁸ to obviate use of restraints.

Despite these established best practice recommendations, multiple systems-level barriers challenge their practical implementation.^21–23 Care delivery in the ED occurs in a uniquely complex environment. Clinical decisions are made under time pressure, using limited information and amidst multiple and frequent interruptions and other unpredictable factors due to the dynamic course of acute, undifferentiated conditions.²⁴ As burden on the emergency care system rises in the USA,^25–28 these systems-level challenges are particularly relevant for patients at risk for agitation because behavioural and de-escalation techniques require investment in time and effort to build a strong rapport and trusting therapeutic relationship with the patient. Given that clinicians may have difficulty accurately identifying patients at risk for agitation and access to expert psychiatric evaluation in such settings may be limited,^29–32 there is a significant mismatch between resources available and application of those resources to individuals who would most benefit from early risk assessment and intervention. A recent prospective study observing 100 at-risk patients in the ED found that over 60% of individuals develop agitation more than 30 min into their visit,^33–35 presenting opportunities to prevent agitation earlier in the course.

Clinical decision support (CDS) tools can help address systems-based challenges, facilitate risk assessment, and guide clinicians to use best practices strategies recommended by Project BETA in the ED. CDS tools show increasing promise in the emergency setting to help clinical staff identify high-risk patients and provide more efficient and higher quality of care,³⁶ including individuals requiring use of high-cost imaging³⁷ and older adults.^38–41 A CDS system encompasses any on-screen tool designed to improve healthcare delivery by enhancing medical decisions with targeted clinical knowledge, patient data and other health-related information.⁴² Use of a CDS tool to assist in assessment and management of potentially agitated patients may be an effective strategy in the ED.^{40 43 44}

Rationale and aims

This study is one of the first that aims to prevent agitation and improve outcomes for ED patients with agitation. We will achieve this by (figure 1):

Figure 1.

Overview and steps for each phase of ED-TREAT design and pilot implementation study. CDS, clinical decision support; ED-TREAT, Early Detection and Treatment to Reduce Events with Agitation Tool.

1. Designing and developing an electronic health record (EHR)-embedded, user-centred, CDS system, Early Detection and Treatment to Reduce Events with Agitation Tool (ED-TREAT), using a contextual design approach to obtain input from key stakeholders and iterative user-centred design process.

2. Conducting a pilot study to evaluate the feasibility, fidelity and bedside acceptability of ED-TREAT. ED-TREAT aims to help ED staff and clinicians to identify patients at high risk for developing agitation, guide them through appropriate risk assessment and efficient decision-making to implement best practice recommendations for preventing development of agitation, and minimise use of restraints.

We hypothesise that, with the use of ED-TREAT, ED staff will be able to identify at-risk individuals, conduct appropriate risk assessment, implement interventions to minimise use of restraints and improve patient experience and outcomes related to agitation management in the ED. If this study is successful, a planned subsequent clinical effectiveness trial will compare effectiveness of ED-TREAT to usual care across multiple ED sites in the future. The long-term goal of this CDS tool is to increase fidelity with best practice recommendations for prevention of agitation through early use of behavioural techniques prior to onset of agitation.

Methods and analyses

Patient and public involvement

We invited patients to help us design, develop and test the intervention so that it is designed to improve the public good and help individuals with lived experience of mental illness and behavioural crises. In addition, we will solicit patient feedback and guidance on dissemination of study results to participants and the local community through networks at our affiliated community-based organisations that have had sustained engagement with our team for several years to implement a research agenda in agitation care that is patient centred and recovery oriented.

Aim 1: developing and refining ED-TREAT

With the goal of developing a final prototype that maximises usability, staff self-efficacy, satisfaction and patient-centred care, ED-TREAT will be designed, developed and refined in three phases (P1–P3) (figure 2). During phase 1 (P1), we will conduct a needs assessment to collect input from key stakeholders including ED physicians, nurses, patient care technicians, behavioural health experts, informatics experts and patients with lived experience of being restrained in the ED. We will combine needs assessment findings with Project BETA recommendations for agitation management in the ED⁵ and risk factors for agitation from the literature to design an initial prototype. Next, in phase 2 (P2), we will conduct formative usability testing, which will consist of ‘think aloud’ protocols, a standard usability procedure for CDS design,⁴⁵ with clinician users and standardised patients in a controlled simulation setting to guide further modifications of the tool in an iterative fashion. Finally, in phase 3 (P3), we will conduct field testing in the ED through observational workflow analyses to identify and address barriers in the real-world clinical environment.

Figure 2.

Overview and steps for aim 1: ED-TREAT user-centred design and prototype development. ED-TREAT, Early Detection and Treatment to Reduce Events with Agitation Tool.

Participant recruitment

We will design ED-TREAT for use by staff members who work mostly closely with agitated patients in the ED.7 35 46–48 These consist of ED physicians, nurses and patient care technicians. We plan to recruit these staff participants via email and biweekly staff meetings. In addition, we will also recruit patients with prior lived experience of being restrained in the ED to solicit their input and ensure that patient-centred practices are considered during the design process. We will recruit these patients from the pool of peer support workers from local community-based recovery organisations via email and presentations at monthly staff meetings. These peer support workers have a history of mental health and substance use disorders and have received training to become employed as patient advocates on community-based treatment teams. Finally, our design team will interview informatics and behavioural health experts to solicit ideas and relevant CDS design strategies to improve decision-making during agitation management.^{49 50}

Phase 1: needs assessment and initial design of ED-TREAT

We will conduct focus groups and observations starting June 2023 via a contextual inquiry³⁶ approach. This approach seeks to engage prospective users and participants described above to understand clinical workflow, roles of different members of the patient care team and thought processes involved in managing a behavioural health crisis in the ED. We will first conduct in-depth qualitative interviews with staff and patient participants using a semistructured interview guide that will include open-ended questions to cover topics (online supplemental table 1) related to application of Project BETA recommendations in preventing agitation and user-centred design.³⁷ Each focus group will consist of 2–5 participants from the same stakeholder group and be approximately 60 min in duration. Sessions will be audio recorded after obtaining participant verbal consent (online supplemental file 2). In addition, members of the design team will conduct observation sessions in the ED to observe ED staff during real-life agitation management events. These sessions will provide information about environmental and systems contexts, interactions among members of patient care team during an active agitation episode, and logistics of integrating CDS tool into EHR workflow to facilitate clinical decision-making.

After identifying crucial user requirements in P1, we will develop an initial low-fidelity prototype for ED-TREAT. This will occur in collaboration with EHR analysts and informatics experts by incorporating Project BETA recommendations for preventing and managing agitation⁵ and potential risk factors for agitation to identify potential at-risk patients in the ED. These risk factors (online supplemental table 2) will be based on existing literature on risk factors for agitation and workplace violence in the healthcare setting,^{4 42 51} our team’s prior work on agitation management in the ED,⁴⁸ patient perspectives of ED visits resulting in restraint use,⁵² characterisation of physical restraint use among adults presenting to the ED with agitation,¹¹ and attributes and levels of agitation impacting thresholds for restraint use in the ED.^{34 35}

Preliminary work by our team has shown that identifying risk factors for agitation and implementing EHR-based interventions for agitation are feasible in the emergency setting.⁴⁶ The CDS will extract patient-specific data on variables of interest from existing patient chart data via EPIC’s Cogito analytics performance suite.⁵³ Based on current expert recommendations from Project BETA,⁴⁶ we anticipate that ED-TREAT will likely stratify patients into three risk groups (table 1) and recommend increasing levels of resource utilisation and pre-emptive management as the risk level increases based on best practices for preventing agitation from Project BETA. Depending on results from the needs assessment, ED-TREAT’s recommendations may include automated order sets for medication therapy, staff instructions and communication orders and templated clinical documentation. Our team will collaborate with EHR analysts to create an initial interactive prototype within the EHR testing environment, ‘Epic Playground’, a non-production, functional, simulated EHR replica used for developing and validating new workflows. We anticipate that our final product will be an EHR-integrated web application through a dedicated graphical Application Programming Interface.⁵⁴

Table 1.

Sample elements of initial ED-TREAT prototype

Risk level	Project BETA5 guidelines	Recommended tasks
Negligible	Medical evaluation and triage13	Collect medical history and physical exam Obtain early vital signs and fingerstick glucose Address psychosocial needs and establish rapport Standardise assessment to avoid structural biases/inequities
Mild to moderate	Psychiatric evaluation and risk assessment76 Psychopharmacology18	Tasks in negligible risk level PLUS: Measure Behavioural Activity Rating Scale75 Offer voluntary oral medication (with allergies cross-checked)
High	Environmental modification (avoidance of restraint and seclusion)20 Verbal de-escalation19	Tasks in mild to moderate risk-level PLUS: Alert clinician early (from triage/nursing assessment) Perform pre-emptive verbal de-escalation (ten domains) Move to quiet, low activity/volume area

BEAT, Best Practices in Evaluation and Treatment of Agitation; ED-TREAT, Early Detection and Treatment to Reduce Events with Agitation Tool.

Phase 2: usability testing of ED-TREAT

Heuristic evaluation

We will first perform heuristic evaluation⁵⁵ of CDS interface usability before testing with clinician participants. This will consist of an expert team of three evaluators with experience in emergency medicine as well as CDS design who will navigate through different aspects of the CDS and judge the compliance and usability of the tool in agitation management triangulated between their expertise and usability standards. Each evaluator will inspect the interface and assess the guidelines and recommendations provided by the CDS for various levels of agitation. After individual assessment is completed, we will debrief as a group and aggregate the results of each evaluator to examine deficiencies in the prototype design. After addressing concerns and refining the prototype CDS tool, we will conduct further usability testing with clinicians in simulated training protocols.

‘Think Aloud’ protocol

Next, our team will perform one-on-one ‘think aloud’ protocol⁴⁵ sessions with clinician users in a quiet office at the EHR training classroom. We will ask participants to perform designated tasks with ED-TREAT using mock patient charts in Epic Playground and ‘think aloud’ how they would use ED-TREAT to test whether the user understands and is using the CDS as intended. We will develop a facilitator guide that focuses on domains related to usability of the prototype and design of the CDS interface (online supplemental table 3). We will videorecord each usability testing session, incorporating user screen captures and field notes⁵⁶ taken during the session. At the end of the session, each participant will complete the System Usability Scale (SUS)⁵⁷ to measure perceived usability of and satisfaction with a health informatics tool. The SUS is a widely used and effective survey composed of ten statements assessed on a 5-point Likert scale, with inter-item correlations of 0.69–0.75 and a reliability coefficient α of 0.91.⁵⁸ Each session is expected to take 30 min.

Simulation sessions

Additionally, we will observe the clinical team in a simulation session with a live actor or mannequin simulating being an agitated patient. Participants will be briefed on how the CDS tool works and will be encouraged to use it at various points in the care process. We will similarly videorecord each session and distribute the SUS to gather participant feedback. We will make iterative refinements to the prototype until the team reaches consensus that it has reached a threshold level of usability.

Phase 3: field testing of ED-TREAT

We will recruit staff participants working in the ED for field testing of the ED-TREAT prototype through observational workflow analysis of ED visits with mild-to-moderate or high risk of agitation (table 1). We will develop an observational guide based on sample topics of usability testing (online supplemental table 3) that will detail both the workflow of managing an at-risk patient and barriers to adopting ED-TREAT in the clinical environment. Field notes will detail events, actions and their time and duration,⁵⁶ while maintaining an open-ended format to describe and follow variations or workarounds in workflow. Either the PI or a trained research associate will complete the observations as an unobtrusive non-participant observer through a patient’s visit from ED arrival to patient disposition, similar to procedures we developed for prior observations of agitation.^{34 35} We will enter field notes using a portable electronic tablet into a word document for free text and a spreadsheet for data elements.

Data analysis

In P1 (design and development), our team will use the iterative and phased approach of building an affinity diagram, a commonly used organisational tool that allows large numbers of ideas stemming from brainstorming and qualitative data to be sorted into groups, based on their natural relationships, for review and analysis.^{59 60} Following completion of each focus group and contextual inquiry session, we will conduct an interpretation session to review the user-provided keynotes from the inquiry session and capture them as affinity notes.⁶¹ To help identify common issues, work patterns and needs, we will arrange the affinity notes into hierarchical categories (‘must-have’, ‘good to have’ and ‘nice to have’) based on common themes in the data to create an affinity diagram.⁶¹ Building of the affinity diagram will occur online using Miro software (RealtimeBoard, San Francisco, California, USA).⁶² We will mock up a low fidelity prototype of ED-TREAT based on current best practices and iteratively refine it based on user data from P1.

For P2 (usability testing) and P3 (field testing), field notes will be analysed using a deductive coding method to conduct directed content analysis⁶³ based on predetermined usability requirements and recommended tasks from ED-TREAT (online supplemental table 3).⁶⁴ We will use Dedoose (SocioCultural Research Consultants, Manhattan Beach, California, USA),⁶⁵ a collaborative and cloud-based qualitative software package, for thematic analysis and data organisation of transcripts. Codes identifying suboptimal or deficient performance of the prototype will uncover critical system factors impacting adoption and usability that need optimisation and adjustment. Two trained reviewers will perform independent coding and we will calculate inter-rater reliability assessments with kappa scores. For the SUS,⁵⁷ participants’ scores from each question of are added together and then multiplied by 2.5 to convert the original scores to continuous data from 0 to 100. Scores will be described using mean and SD, and >85 will be indicative of excellent usability.⁶⁶ We will use the results generated from this analysis process at each round of revisions to make appropriate adjustments to the ED-TREAT prototype in close collaboration with the EHR analyst team until we derive a final deliverable prototype that will be ready for the pilot trial.

Sample size

We will use purposive sampling⁶⁷ to ensure the full spectrum of perspectives for clinicians who will engage with ED-TREAT and peer support workers who have had experience as patients in the ED. We will conduct data collection until reaching thematic saturation,⁶⁸ when new concepts no longer emerge from iterative analysis of the data.⁶⁹ For initial design (P1), we anticipate that this will occur after 5–6 focus groups with 6 participants (staff and patients) in each focus group.⁷⁰ As enrolment of 10–12 subjects can identify up to 90% of usability problems,⁷¹ we will perform usability testing (P2) for approximately 5 participants in each round of refinement and expect about three rounds of refinement (15 participants total) as per our prior published work.⁷² For field testing (P3), we plan to observe eight patient encounters to detect any usability problems when deployed in the ED.

Aim 2: pilot trial and feasibility testing

We will conduct a pilot randomised controlled trial (RCT) for ED-TREAT to compare the intervention to usual care. This will allow us to evaluate acceptability of the intervention to its end-users (ED staff), fidelity of its intended outcomes to identify at-risk individuals and prevent agitation, feasibility of randomisation, ease of subject enrolment and measurement of other outcomes of interest. This will be a mixed-methods study, wherein we will quantitatively measure usability and efficiency of clinical decision-making via the SUS⁵⁷ and specific patient outcomes, as well as qualitatively assess the effect of ED-TREAT on clinical workflow and patient care. In addition, this pilot trial will (1) test the integrity of the study protocol in preparation for a future comparative effectiveness clinical trial, (2) evaluate randomisation protocols, (3) estimate rates of recruitment and retention of trial subjects and (4) estimate effect size for sample size calculation in the subsequent trial.⁷³ Pilot trials⁷⁴ are not designed to test the efficacy of the intervention but will help establish acceptability and feasibility in preparation for a future multicentre RCT. We hypothesise that it will be feasible to implement the tool, measure identified outcomes, be acceptable to its end-users and work as intended.

Study setting, participants and randomisation

We will conduct the pilot trial at two adult ED campuses that belong to a large regional healthcare system in the Northeast USA, with a planned trial start date in the Fall of 2024. Prior to initiation of the pilot trial, all emergency physicians, ED nurses and ED patient care technicians at both campuses will receive an email introduction and a link to a brief training regarding the use of ED-TREAT. Eligibility criteria for patients and recruitment will include ability to provide verbal consent for the study and a score of ‘4’ (quiet and awake; normal level of activity) or less on the Behavioural Activity Rating Scale (BARS),⁷⁵ an accepted seven-point scale to assess levels of agitation in acute care settings. We will first perform screening for eligibility via an ED-TREAT administrative interface that performs risk assessment for each ED patient on arrival. Inclusion criteria for ED patients include adult (age ≥18) patients, presenting to the ED during the pilot trial period, deemed to have a mild-moderate or high risk of agitation as determined by ED-TREAT, do not require physical restraint orders within <30 min of arrival, with a score of ‘4’ (quiet and awake; normal level of activity) on the BARS, have comfort with conversational English, and able to provide verbal consent. Exclusion criteria include presence of a restraint order <30 min of arrival and presence of a non-violent physical restraint order where indications are not due to agitation (eg, for protecting intubation or life-preserving equipment). A research associate will then approach eligible patients and their designated clinician team members for enrolment after confirming ability to consent and assessing patient BARS scores as close to the beginning of the visit as feasible. Study procedures, risks/benefits of participating, and the purpose of ED-TREAT will be described, and verbal consent will be obtained for patients and staff participants. Since we plan to enrol patients prior to onset of agitation, we anticipate that most patients should be able to engage in decisions and provide verbal consent. Our prior work found that >70% of ED patients with subsequent agitation arrived with a normal mental status and BARS scores ≤4.^{34 35} We will perform 2:1 randomisation at the patient level and also recruit a higher proportion of high-risk patients in each arm, as a primary aim of the pilot trial is to test the acceptability of ED-TREAT and we anticipate that our intervention will recommend more tasks for high-risk patients. Randomisation will occur using sequentially numbered, opaque, sealed envelopes that will only be opened by the research team member after enrolment of each patient. In the intervention group, ED-TREAT will automatically launch as part of the clinical team’s workflow in the EHR after randomisation. We anticipate that critical steps in ED-TREAT will occur at four stages of a visit (figure 3): (1) at initial arrival with automated risk stratification using predetermined criteria set during the CDS design process; (2) at triage assessment; (3) at initial nurse interaction and (4) at initial clinician interaction. In the control group, ED-TREAT will notify the research team regarding the patient’s risk group but will not launch for the clinical team’s interfaces.

Figure 3.

Anticipated clinical steps for the intervention arm of ED-TREAT. ED-TREAT, Early Detection and Treatment to Reduce Events with Agitation Tool.

Data collection

Our anticipated data collection strategy is summarised in table 2. In addition to visit characteristics (system factors, relevant clinical data) collected through the EHR during the visit, we will collect acceptability and fidelity measures, feasibility assessment and potential outcomes of interest for each visit.

Table 2.

Anticipated data collection for ED-TREAT pilot trial

Measure	Tool or strategy	Timing of measurement
Visit characteristics
System factors	EHR (eg, staff traits, National ED Overcrowding Scale)77	During and end of visit
Clinical data	EHR/ED-TREAT (eg, risk category)	During and end of visit
Acceptability, fidelity
Clinician acceptability of ED-TREAT	System Usability Scale57 (satisfied, useful)	End of visit
Fidelity of ED-TREAT	Observational workflow checklist (perform as intended)	During visit
Effect on patient experience	Qualitative interviews with patients	End of visit or <72 hour after visit
Potential bias or differential treatment	Implicit Association Test (for clinicians), patient interviews	End of visit or <72 hour after visit
Feasibility
Available subjects	# of eligible visits	Every 3 months
Subject identification	% eligible visits approached	Every 3 months
Enrolment	% visits with consent to enrol from patient/clinical staff	Every 3 months
Retention	% visits with completed measures	Every 3 months
Effect on clinical workflow	Qualitative interviews with clinicians	Every 3 months
Outcomes
Physical restraint order	EHR	During and end of visit
Intramuscular chemical sedative order	EHR	During and end of visit
Level of agitation	Behavioural Activity Rating Scale75	Highest level during visit
Disposition	EHR	End of visit
Length of stay	EHR	End of visit

ED, emergency department; ED-TREAT, Early Detection and Treatment to Reduce Events with Agitation Tool; EHR, electronic health record.

For all clinicians caring for patients in the intervention group, we will administer the SUS⁵⁷ either in person at the end of the ED visit or within 72 hours by email. In addition, we will perform observation workflow analyses as described earlier using a task checklist to determine if clinicians were using ED-TREAT as intended, and if any barriers or unintended consequences occurred because of the intervention. We will perform brief, semistructured interviews with patients either at the end of a visit or within 1 week after disposition to evaluate the impact of ED-TREAT on their experiences.

Feasibility

To assess the feasibility of a comparative effectiveness trial, we will evaluate the following at 3-month intervals: (1) available number of potential subjects (# of eligible patient visits), (2) subject identification (% of eligible patients/staff approached), (3) enrolment (% of patients/staff with consent to enrol) and (4) retention (% of visits with completed outcome measures). We will also conduct brief, semistructured interviews with staff participants to evaluate their experiences with ED-TREAT and effect on clinical workflow.

Outcome measures

The anticipated primary outcome of the comparative effectiveness trial will be the presence of a physical restraint order during the ED visit (>30 min after arrival). Additional secondary outcomes include the presence of an intramuscular chemical sedative order, the highest level of agitation on the BARS⁷⁵ during visit, disposition and length of stay.

Sample size and data analysis

As this pilot trial is not designed to test the efficacy of ED-TREAT, a power calculation is not appropriate.⁷³ To determine the sample size for this pilot randomised trial, we will use the outcome of fidelity as measured by the proportion of visits in the intervention arm that are adherent to >80% of the observational workflow checklist. To estimate the proportion achieving this level of fidelity with a reasonable precision (95% CI with a width of ±20%), a total of at least 26 eligible subjects will be enrolled in the pilot trial. We will determine ratings from the SUS⁵⁷ and calculate proportions of each clinician group with scores of >85, indicating excellent usability. We will consider ED-TREAT to be acceptable if ≥90% of each clinician group give ratings >85. For feasibility, we will measure the proportion of potentially eligible patient visits with successful enrolment and collection of all outcomes of interest. Based on our group’s anecdotal experience with pilot studies, we will consider a comparative effectiveness trial feasible if ≥30% of visits assessed for eligibility are enrolled and ≥90% of all outcome measures are collected. Qualitative data obtained from interviews will be analysed with Dedoose using the analytic strategy mentioned earlier for iterative refinement of the study protocol in preparation for a comparative effectiveness trial.

Ethics and dissemination

We plan to conduct our study in accordance with the Yale Institutional Review Board (IRB). We have obtained the necessary regulatory and human subjects protection approvals for each aspect or phase of our protocol. Ethical approval by the Yale University Human Investigation Committee was obtained in 2021 (HIC# 2000030893 and 2000030906). After careful review, Yale IRB has approved Aim 1 of this study to be eligible for exemption of full IRB review under 45 CFR 46.104(d)(4), since any information collected by the investigator will be in such a manner that the identity of the human subjects cannot readily be ascertained directly or through identifiers linked to the subjects. Aim 2 of the study has been approved as a full protocol for a clinical trial and is undergoing annual continuing review. As we work with structured EHR patient data, we will maintain deidentification where necessary and keep access to datasets secure. Additionally, all clinicians and patients participating in focus groups, feasibility testing or the pilot trial will be informed of their rights as subjects. Clinicians will retain the right to retain control of their practice and patients will retain the right to not participate and request termination of participation at any point in the study. All staff, participants and patients will provide verbal consent prior to involvement with the study. Sample consent forms for patients and staff are included as online supplemental materials. The pilot trial is registered at ClinicalTrials.gov (clinical trial registration number: NCT04959279).

Monitoring for data integrity and safety will be the responsibility of the principal investigator (AHW) and the Yale Human Investigation Committee, and a data safety monitoring board (DSMB). DSMB members will be composed of experts in care disparities and health equity for vulnerable and disadvantaged populations, clinical trials for mental illness and substance use disorders, measurement and risk stratification for disinhibited behaviours, and an expert in statistical analysis of clinical trials in emergency medicine. Twice annually, the DSMB will review the progress of the study and frequency of serious adverse events. All adverse events, as well as any unanticipated problems that arise, will be reported within 48 hours to the Human Investigation Committee. A full report will be provided annually or on request to the IRB and the sponsor’s Programme Official. The effect of adverse events on the risk/benefit ratio of the study will be re-evaluated by the investigators with each event, with appropriate adjustments made to the protocol or consent forms if needed. Given the minimal risk of the study and intervention, the investigators do not anticipate the occurrence of any serious adverse events.

Results and outcomes of the study will be disseminated through peer-reviewed journals and presentations at relevant scientific meetings throughout the study timeline. A successful pilot trial will aid in a future full RCT to fully measure the effectiveness of the ED-TREAT CDS tool in agitation management in ED settings. At the time of publication of any manuscripts that arise from this research, the deidentified data for that manuscript will be made available to share for scholarly activities. Sharing of the data will require a data use agreement to be established between the requesting and host institutions. Data will be shared through secure file transfer.

Ethics statements

Patient consent for publication

Not applicable.