Promise and Caution: Mapping Opportunities for AI Decision Support in Emergency Medical Services

Abstract

Designing AI-enabled decision support for fast-paced medical teams remains underexplored. We use Emergency Medical Services (EMS) as a critical use case to examine where AI can, and should not, support time-critical decision-making. We conducted participatory design workshops and formative user evaluations with EMS providers to elicit, prototype, and critique candidate AI application areas for EMS work. Providers identified several promising uses of AI: (1) AI-enabled information retrieval to accelerate access to protocols and medication references; (2) speech-based documentation support to reduce charting burden and generate draft records during care; (3) AI-generated patient “snapshots” that summarize relevant history from prior encounters; and (4) AI-based medication recognition to identify home medications and surface key safety information. In contrast, participants were cautious about clinical reasoning support that could be interpreted as diagnostic assertions, ambient “always-on” monitoring to flag errors or workflow deviations, and voice-based exchanges with AI in front of patients. Across various concepts, providers emphasized key factors critical for AI adoption in EMS, including seamless workflow fit, robustness to noisy environments, transparency, appropriate trust calibration, and the preservation of professional autonomy. We conclude with design implications for responsibly advancing the use of AI technology for EMS decision support.

I. Introduction

Clinical decision support systems (CDSS) have long been proposed to enhance clinical decision-making, reduce errors, and improve patient outcomes [1]. Recent advances in artificial intelligence (AI) further expand the capabilities of CDSS, from summarizing and organizing patient information to assisting with diagnosis and forecasting patient risk (e.g., mortality) [2], [3]. Yet adoption and sustained use of AI-enabled CDSS (often referred to as AI-CDSS) remain limited [4], [5]. A key reason is that many AI-CDSS tools are developed without sufficient attention to clinical workflows, clinicians’ needs, and sociotechnical constraints, which can erode trust and lead to underuse or abandonment [4], [5]. These limitations are especially consequential in time-critical medical environments, where interruptions are costly and every second matters.

Emergency Medical Services (EMS) is a prototypical time-critical medical setting in which prehospital care unfolds in unpredictable environments. Providers must make high-stakes judgments with incomplete information and high cognitive load while delivering hands-on treatment and coordinating with multiple parties (e.g., dispatch, medical control, receiving hospitals). This combination of time pressure and multitasking makes EMS a compelling context for examining what AI support can realistically contribute and what it might disrupt [6], [7]. However, most CDSS research has been conducted in comparatively stable, hospital-based settings (e.g., inpatient wards, radiology, primary care) [8], [9], with few studies looking into how AI-enabled support should be designed for time-critical EMS care [10], [11], [6], [12]. As a result, we still lack a clear understanding of which forms of AI support EMS providers would welcome in practice, which they would reject, and what factors shape adoption in the field.

In this paper, we use EMS as a critical use case to examine where AI concepts could—and should not—support decision making in time- and safety-critical care. Through a user-centered process, we elicited providers’ needs, generated candidate AI use cases, and assessed their fit with EMS constraints [13]. Overall, providers viewed AI as most valuable when it serves as a supportive, secondary aid that reduces cognitive and clerical burden without displacing clinical judgment. In particular, participants welcomed AI-assisted documentation to streamline note-taking and patient record charting, AI-enabled information retrieval to accelerate access to protocols and medication dosing information, AI-generated patient “snapshots” that summarize relevant history from prior encounters, and AI–based medication recognition to quickly identify drugs and surface key safety information. However, they also expressed clear concerns about several AI uses that felt risky or misaligned with EMS practice, especially AI-driven clinical reasoning support that could be interpreted as diagnostic assertions, ambient AI monitoring to flag medical errors or workflow deviations, and voice-based exchanges with AI in front of patients. Finally, providers articulated critical factors for AI adoption in EMS settings, including seamless workflow fit, robustness to noisy environments, transparency, appropriate trust calibration, and the preservation of professional autonomy, highlighting that premature or overly assertive AI integration into EMS workflow can introduce risk without commensurate benefit.

This work contributes (1) empirical insights into what AI can—and should not—do to support EMS decision making; and (2) design implications for responsibly advancing AI support in prehospital emergency care.

II. Related work

Traditionally, CDSS relied on knowledge-based approaches that encode explicit rules derived from clinical guidelines and expert consensus [14], [15]. Recent advances in AI have broadened this paradigm toward data-driven approaches that use predictive analytics and machine learning to generate adaptive support in near real time [16], [17], [18]. For example, models trained on large-scale electronic health record (EHR) data can produce patient-specific diagnoses and risk predictions that extend beyond what static, rule-based CDSS can provide [19]. Despite this promise, real-world use of AI-enabled decision support remains constrained by challenges of interpretability, clinician trust, and workflow integration [13], [20], [21]. Clinicians may hesitate to act on recommendations produced by “black-box” models with unclear rationale [5], [22], and concerns about algorithmic bias, uncertain reliability, and limited explainability can further erode confidence and appropriate reliance [8], [23], [24], [25]. These barriers are likely amplified in time-critical settings, where decisions must be made within seconds and cognitive resources are already stretched thin [26], [27], [28], [29], [30]. In such scenarios, even technically strong AI may provide little benefit, or be ignored entirely, if its outputs are not aligned with clinicians’ needs and domain-specific workflows [31], [32], [33].

As a prototypical time-critical care setting, EMS offers a compelling domain for examining how AI might support clinical decision-making under extreme constraints, including intense time pressure, hands-on and multitasking work, unpredictable and mobile environments, and limited access to data, diagnostic tools, and medical resources. Despite the clear need for timely support—and growing interest in AI for emergency care—the literature has not sufficiently explored and addressed what AI can do, and what it should not do, to support decision making in EMS [11], [34], [35], [36]. In particular, empirical evidence remains limited on how EMS providers want AI support to be scoped, which uses they view as helpful versus risky, and what factors are critical for AI adoption in prehospital care. Our work aims to address these knowledge gaps.

III. Methods

A. Study Design and Participants

This study is part of a larger project focused on designing and developing workflow-compliant AI-enabled decision support for EMS providers. As an initial step, we used a user-centered approach to elicit EMS providers’ perspectives on where AI could—and should not—support decision making in the field [37]. We conducted participatory design (PD) workshops to surface challenges and generate early ideas for AI-enabled support. Based on these insights, we iteratively developed design concepts and high-fidelity wireframes, which we then used as design probes in follow-up formative user evaluations to gather feedback on feasibility, workflow fit, and perceived benefits and risks. The timeline and sequence of our studies are shown in Figure 1.

Fig. 1.

Study Design: The design process followed a three-stage approach: (1) Participatory design (PD) workshops conducted in Nov–Dec 2024 with 10 EMS providers to gather initial needs and ideas; (2) Rapid prototyping and wireframe development using Figma from Jan–Mar 2025; and (3) Formative, individual user evaluations in Apr–May 2025 with 16 EMS providers to elicit feedback and evaluate different design concepts.

We recruited 26 EMS providers from four agencies across two U.S. regions. Ten providers participated in the PD workshops, and sixteen participated in follow-up evaluations. Participants included 16 paramedics and 10 Emergency Medical Technicians (EMTs), with experience ranging from 2 to over 40 years. This variation in agency type, geography, and professional experience helped capture a broad range of EMS workflows and perspectives, supporting the transferability of our findings to similar prehospital contexts. The study was approved by the Institutional Review Board (IRB) of the first author’s university.

B. Data Collection

1). Participatory Design (PD) Workshop:

We conducted four PD workshops (2–3 participants per session), each lasting approximately two hours. Workshops began with reflections on participants’ recent patient care cases to surface decision-making and patient-management challenges. Participants then discussed cognitive aids and decision-support tools currently used in practice, including what works well and where friction occurs in EMS practice.

Next, the research team facilitated an open-ended discussion about how AI might support EMS decision-making. To ground the discussion and support ideation, we introduced examples of AI applications proposed for other clinical domains, such as ambient documentation, patient risk prediction, and AI-assisted medical knowledge search [4], [38]. Participants assessed the perceived usefulness and workflow fit of these ideas and articulated potential risks and limitations. They were also encouraged to think beyond these examples and propose additional AI concepts relevant to EMS. We then conducted a sketching activity in which participants visualized interface ideas or created storyboards to illustrate how they envisioned the AI working and what it should look like. Sessions concluded with a group discussion of higher-level considerations for adopting AI in EMS, such as how AI should fit into existing workflows and what would be required to support trust, efficiency, and safety.

2). Formative User Evaluations:

Building on workshop findings, we iteratively developed a set of design concepts, progressing from paper sketches to high-fidelity wireframes in Figma. These wireframes depicted alternative ways AI could support EMS decision-making (see an example in Figure 2. We used the wireframes as design probes in follow-up formative user evaluations with individual EMS providers to elicit additional feedback on usefulness, workflow fit, and concerns.

Fig. 2.

Conversational agent (CA) prototype used as a design probe in user evaluations to elicit EMS providers’ perspectives on conversational AI. (a) Landing page enabling typed or voice-based natural-language queries. (b) Example response to “Show me the nearest hospital with a Level I trauma center,” displaying Jacobi Medical Center as the recommended destination with address, trauma level, estimated time of arrival, and a navigation map.

Each evaluation lasted approximately 60–90 minutes. Participants reviewed multiple wireframes and discussed the potential benefits, limitations, and applicability of each concept to their daily work. For select concepts (e.g., AI-driven clinical reasoning), participants also completed a brief questionnaire with Likert-scale items assessing perceived usefulness, trust, and workflow fit (1 = strongly disagree; 5 = strongly agree). Throughout the sessions, participants were encouraged to think aloud and explain their reasoning as to how these AI concepts might support or disrupt their work.

C. Data Analysis

Workshops and user evaluations were audio recorded, transcribed verbatim, and analyzed in NVivo. We conducted thematic analysis using an abductive approach that combines inductive coding with deductive coding [39]. More specifically, two researchers (the second and third co-authors) independently coded an initial subset of transcripts (one to two transcripts each) and met to reconcile codes and develop a preliminary codebook. The team applied this codebook to the remaining transcripts, adding or refining codes as new concepts emerged. We derived final themes by clustering related codes into higher-level categories capturing (1) challenges in EMS decision making, (2) perceived AI opportunity areas and non-applicable use cases, and (3) factors shaping the adoption of AI support in EMS. In this paper, we use participants’ quotes to illustrate key insights. Supporting quotes are included throughout, with identifiers “PD#” referring to participants from the participatory design workshops and “U#” denoting participants from the user evaluations.

For questionnaire responses collected for select AI concepts during user evaluations (completed by 16 participants), we used descriptive statistics (counts and proportions across Likert-scale options) to summarize overall sentiment toward each concept. We then triangulated these results with qualitative data from participants’ think-aloud comments and discussions to interpret the rationale underlying their ratings. For example, participants who reported moderately high trust in AI often emphasized that such trust depended on transparent system behavior (e.g., how the recommendation is generated), particularly given accountability and privacy concerns.

IV. Results

In this section, we synthesize findings from both PD workshops and user evaluations to illustrate which AI concepts EMS providers viewed as promising and which they viewed as misaligned with their workflow. We organize results into two categories: (1) AI use cases participants considered usable and applicable in the field, and (2) use cases they considered risky, disruptive, or unlikely to fit EMS workflows.

A. Promising AI Concepts for EMS Decision Making

1). Leveraging AI and Conversational Agents for Faster Information Seeking and Retrieval:

Timely access to information is vital in EMS care, where even minor delays can affect clinical decision-making and patient outcomes. EMS providers often need to retrieve critical information on the fly, for example, reviewing treatment protocols, calculating weight-based medication doses, or identifying the nearest facility equipped for a specific case (e.g., a Level I trauma center or a stroke center). Currently, these information-seeking tasks can be cognitively demanding because providers must switch between multiple resources, such as reference cards, mobile applications, and printed guidelines, while simultaneously monitoring patients and coordinating with their teams. This multitasking increases the risk of cognitive overload, as one participant noted: “With our current system, everything is separate. We would have to go up to the front of the ambulance to see where the nearest stroke center was located.” [U#5]

Given these challenges, we explored EMS providers’ views on using AI to expedite access to information. As illustrated in Figure 2, we presented a concept of a conversational agent (CA) that can communicate with providers; for example, providers could ask questions using natural language queries (e.g., “What is the pediatric epinephrine dose for an eight-year-old?” or “Show me the chest pain protocol.”) and receive immediate responses with direct links to clinical sources. Overall, providers identified several potential benefits of integrating CAs into EMS workflows, including reducing the need for manual searching or memorization and enabling faster, more intuitive information retrieval without delaying care delivery. As two participants explained: “If I’m busy with an airway, I don’t have time to flip through a binder. If I can just ask, ‘what’s the dose for this med?’ and get it right there, that’s huge.” [U#3] “Like having a partner who knows the protocols by heart. It lets me focus on the patient instead of second-guessing my math.” [U#1]

Regarding the use of CAs in practice, a critical consideration is how quickly providers can obtain the information they need with minimal interaction. To that end, participants suggested design strategies to make the CA extremely efficient to use. One commonly mentioned idea was supporting keyword-based search instead of requiring a long query to explain what the provider needs: “My only thing with the chatbot is, I think it would have to be, as long as it can be integrated in a sense, where I don’t have to, like, type out a full word. So if I’m in a critical instance where I can’t think of what that protocol is, but I know a keyword for that protocol, it would have to be able to pull that up without me having to say a full sentence to get it to pull up.” [PD#7]

2). AI-enabled Speech Recognition Technology for EMS Documentation:

A major barrier to effective clinical decision support in EMS is the lack of timely documentation. EMS providers work in high-pressure, mobile environments where hands-on patient care must take priority over data entry. As a result, key details, such as vital signs, symptoms, treatments administered, and changes in patient status, are often not documented in real time. Instead, providers frequently defer documentation until after patient handoff at the receiving hospital, then spend significant time completing records by relying on short-term memory: “The biggest issue that we always face is that we treat the patient, we get them down to the hospital, and then afterwards, we try and remember what the hell we did.” [PD#8]

To address this gap, providers described AI-enabled speech recognition as a promising approach to streamline documentation. In this concept, providers could dictate key details, which the system would transcribe in real time and parse into clinically relevant elements (e.g., medication names and dosages, vital signs, and medical history) to generate structured encounter summaries. One provider shared his experience and perspective on this technology: “I’ve seen AI technology that listens throughout a call, records events, and then almost writes the patient care report for you based on what it collects. That would be incredible. It would save a lot of time and reduce manual reporting errors.” [PD#6]

Even so, participants emphasized that this approach would only be viable if the AI performs reliably in the dynamic EMS environment. A primary concern was accuracy in noisy conditions (e.g., sirens, radio chatter, and overlapping speech among team members), which could lead to mis-transcriptions of critical details. Such errors could have serious consequences, including billing issues and legal liabilities: “I would say it could pick up something wrong. It could misinterpret something.” [U#5] Several participants suggested that the technology should be designed to produce a draft for quick review and editing—rather than automatically finalizing notes or populating EHR fields—to preserve accountability and ensure providers remain in control of what is documented.

3). Using AI to Capture and Recognize Patients’ Medication Information:

Accurately capturing a patient’s medication history is a routine but often difficult part of EMS care and documentation. Providers may need to identify home medications (the medications that patients have been taking to treat their chronic conditions) to guide treatment, check contraindications, and complete the patient care report. In practice, this task is frequently slow and error-prone: patients may not recall names or doses, medications may be unfamiliar, and labels are often incomplete or worn, especially when patients present with a bag of mixed bottles: “One of the biggest time sinks as a paramedic is when you should be looking at your patient and trying to do stuff for them, while also trying to make sure your documentation is in order. And part of that is writing down all the medications that they take, which can take forever.” [PD#2]

To address these challenges, we explored a concept that leverages AI, in particular computer vision techniques, to help EMS providers identify patients’ home medications in the field. The concept supports identification via barcode scanning of medication bottles and visual recognition of pills when barcodes or labels are unavailable. Once identified, the system surfaces key details (e.g., drug name, indications, contraindications, and safe dosage ranges) and records them directly into EHR.

In both studies, participants consistently viewed AI-enabled medication recognition as beneficial. They emphasized that integrating medication identification into the EMS workflow could reduce the need to look up drug information on a separate device (e.g., a mobile phone) and help them recognize and document medications more quickly: “It saves us pulling up another app on another device. A lot of the common medications we know, but there are literally tens of thousands of others, and being able to look them up would be really handy.” [PD#5] Others highlighted its value in time-sensitive scenarios such as overdoses, where quickly confirming safe dosage ranges could reduce delays: “Almost every overdose call I run, we’re Googling lethal doses. If it could just tell me the safe range and lethal dose based on weight, that would save a lot of time.” [PD#4] Finally, participants noted that this feature could be particularly helpful when labels are missing or unreadable: “Very frequently I’ll ask for people’s medication lists, and they have like, a big plastic bag, and then the labels are worn off.. . So I like the idea of just another way to get that information.” [PD#10]

The only concern raised about this feature was the risk of misidentification, particularly because some medications have similar names and appearances: “Um, so I think my, my only concern with that is that a lot of medications have similar names, and that scanning it might bring up the wrong information entirely.” [U#10]

4). Using AI to Summarize Patient’s Past Medical History for a Quick Snapshot:

Obtaining a patient’s past medical history in the field is often difficult, even though it can be critical for informed decision-making. EMS teams frequently arrive with limited context, and patients may be unable to provide accurate information due to distress, altered mental status, respiratory compromise, intoxication, or unconsciousness. Family members or bystanders may also struggle to recall or know the patient’s past medical history. As one provider explained: “I think the most helpful part of that would be the medical history, because especially when we go on some of our critical patients, and even if there is family or friends around, if they’re in a stressful situation, they don’t tend to be able to provide us much information.” [PD#2] As a result, providers may spend valuable time asking repeated questions or making decisions with incomplete background information, which can delay care and increase the risk of missed contraindications.

To address this challenge, we explored an AI-enabled “patient snapshot” concept. Once a patient identifier is entered into the EHR (e.g., name and date of birth), the system retrieves and summarizes salient information from past encounters, such as recent ED visits or admissions, key diagnoses, relevant medications, allergies, and notable care patterns. Participants generally viewed this snapshot as useful because it could provide immediate context when history is difficult to obtain on scene. One provider emphasized the value of quickly surfacing only the most important information: “I can see it being beneficial with some patients, just because some of the calls recently run, knowing, like, if they have, like, psychiatric things, because they can look completely normal, but then you’ve noticed over a long period of time something’s off, and knowing that beforehand, probably would have changed the way that we entered the call.” [PD#5]

B. Impractical Use Cases of AI in EMS

1). Cautious Use of AI for Clinical Reasoning Support:

We explored the idea of using AI to support EMS clinical reasoning—for example, using a patient’s symptoms, vital signs, and contextual details to suggest possible conditions (e.g., differential considerations), flag red flags or high-risk patterns, and highlight relevant assessment or treatment considerations. Several PD participants noted that such support could be helpful in principle, which was echoed in the questionnaire responses collected during user evaluations (Table I): most participants somewhat or strongly agreed that AI could improve emergency medical care and reduce cognitive burden. Participants also emphasized that this type of reasoning support may be most useful during longer transports, when there is more time to reflect on the patient’s presentation and verify AI suggestions, as one provider explained: “For shorter transport times, such as 10 minutes, it might not be as helpful, but for longer ones, it could be very beneficial.” [PD#8]

Table I.

EMS Providers’ General Perceptions about Using AI for Decision Making

	User Ratings
	Strongly Disagree	Somewhat Disagree	Neither Agree Nor Disagree	Somewhat Agree	Strongly Agree
I would trust AI to assist in time-critical EMS decisions	4	0	5	6	1
I believe using AI can improve the quality of emergency medical care.	2	0	1	7	6
I would feel comfortable relying on AI recommendations during high-stress situations.	3	2	3	5	2
I would want to understand how the AI arrived at its recommendations before using them.	3	0	0	2	11
I worry about over-relying on AI in critical EMS situations.	3	0	1	2	10
I believe AI should be used only as a secondary aid, not as a primary decision-maker.	0	1	3	2	10
I am concerned about liability issues if something goes wrong following an AI recommendation.	0	0	2	3	11
I think AI has the potential to reduce cognitive burden in emergency situations.	4	1	0	7	4
I would recommend using AI to colleagues in EMS.	2	0	6	4	4

However, providers’ enthusiasm for AI-supported clinical reasoning was tempered by several considerations. First, participants emphasized that EMS providers are rarely expected to reach definitive diagnoses in the field, given limited access to diagnostic resources (e.g., labs or imaging). Instead, their primary goal is to rapidly stabilize the patient and transport them safely to the nearest or most appropriate facility (e.g., a stroke or trauma center). As a result, the practical need for diagnosis-oriented AI is often limited, as one participant explained: “Out here, we’re not making the final diagnosis. We’re treating what we see, stabilizing the patient, and getting them to the right place.” [PD#6]

Furthermore, participants expressed lower confidence in relying on AI-generated recommendations for time-critical clinical judgments (Table I): Only one participant strongly agreed that they would trust AI to assist in time-critical EMS decisions, while four strongly disagreed. Similarly, only two participants strongly agreed that they would feel comfortable relying on AI recommendations in high-pressure scenarios, whereas three strongly disagreed. One provider explained that his hesitancy stemmed from both clinical risk and the unpredictability of field constraints, which AI may not fully account for: “I have a hard time believing that AI, unless it’s very smart, would know, for example, if someone has heart failure and therefore shouldn’t be given a certain medication, because giving that medication could cause that person to die. That worries me. And I’ll give you another example. If AI tells me to connect the patient to a certain device, but the patient is entrapped under a building collapse, how am I going to connect them to that device when I can’t even get them out? So there are certain things I can’t do even if AI is recommending them.” [U#4]

Participants linked this trust gap to the practical difficulty of verifying AI outputs during care. Accordingly, thirteen participants somewhat or strongly agreed that they would want to understand how the AI arrived at its recommendations before acting on them (Table I). One provider emphasized the need for human-understandable rationales and traceable sources: “As long as the AI tool is able to give me the thought process and the stem from which its recommendation came, I think that would make it a lot easier for me to trust if that was given.” [U#14]

Liability and professional accountability further shaped participants’ concerns. Many worried about being held responsible if an AI-driven recommendation contributed to adverse patient outcomes: 11 of the 16 user evaluation participants strongly agreed that they were concerned about liability issues, and another three somewhat agreed (Table I). Accordingly, participants emphasized that final clinical judgment must remain the provider’s responsibility: “It [AI] should never be the one making the final call. That’s my license, my responsibility.” [U#10] Consistent with this view, twelve participants strongly or somewhat agreed that AI should be used only as a secondary aid, not as a primary decision-maker (Table I). For example, participants preferred AI that surfaces relevant resources (e.g., protocols, contraindications, or medication dosages) without asserting definitive diagnoses or directing care. Reflecting this boundary, one provider explained: “I strongly agree that AI should be used as a secondary aid in case I really need backup resources. But this is my job, so I would not rely on that [AI] first.” [U#16]

2). Uncomfortable Use of Ambient AI for Detecting Errors and Workflow Deviations in Real-Time:

During discussions of ambient technology in PD workshops, some participants proposed using this technology to detect workflow deviations or potential medical errors in real time: “If you go forward with the technology that listens in the background, it could also be used to track medication errors.” [PD#6] For example, an ambient system could flag a potentially unsafe pediatric dose if it detects that a provider is preparing to administer 0.5 mg of epinephrine (1:1,000) intramuscularly to a 3-year-old child weighing 15 kg, where protocol-based dosing (0.01 mg/kg) would indicate 0.15 mg. In principle, such alerts could prevent harmful errors before medication administration.

Building on workshop insights, we created wireframes illustrating the use of ambient AI technology to listen in the background to trigger decision support or safety alerts. During follow-up user evaluations, participants reviewed these design probes, completed a questionnaire (Table II), and discussed potential benefits and concerns.

Table II.

User Perceptions about Using Ambient AI Technology in EMS Care

	User Ratings
	Strongly Disagree	Somewhat Disagree	Neither Agree Nor Disagree	Somewhat Agree	Strongly Agree
I would be comfortable with an AI system listening in the background during EMS care.	2	5	2	7	0
I would find it helpful to receive real-time alerts or suggestions from an ambient AI system.	2	4	1	9	0
I trust that an ambient AI system can detect potential errors or oversights accurately.	2	3	2	9	0
I am concerned that ambient AI monitoring may lead to distractions during critical moments.	1	1	2	8	4
I worry about privacy or data security with continuous AI monitoring.	0	2	2	3	9
I believe that an ambient AI system could help reduce human error in EMS care.	1	3	1	9	2
I would want to know exactly what the AI is monitoring and how it processes information.	0	0	2	4	10
I would use an ambient AI system if it helped reduce cognitive workload.	3	4	4	4	1

Overall, survey responses suggest mixed comfort with ambient AI for real-time monitoring of errors or workflow deviations (Table II). Notably, no participants strongly agreed that they would be comfortable with an AI system listening in the background during EMS care (0/16). Instead, 7/16 somewhat or strongly disagreed that they would be comfortable (5 somewhat disagree; 2 strongly disagree). Similarly, while many participants somewhat agreed that it would be helpful to receive real-time alerts or suggestions (9/16) and that they trust ambient AI can detect potential errors or oversights accurately (9/16), no one strongly agreed with either statement, and a non-trivial minority disagreed (6/16 for helpfulness; 5/16 for trust). Taken together, these results indicate that even when providers saw potential value, they were reluctant to endorse ambient monitoring as something they would fully rely on.

Participants’ concerns centered on privacy, transparency, and the risk of distraction. Privacy and data security were especially salient: 12/16 participants somewhat or strongly agreed that they worry about privacy or data security with continuous ambient monitoring (9 strongly agree; 3 somewhat agree). Transparency expectations were even more pronounced: 14/16 participants somewhat or strongly agreed that they would want to know exactly what the AI is monitoring and how it processes information (10 strongly agree; 4 somewhat agree). One provider explained: “I would want to know exactly what the AI is listening for and how the information is processed. That would be very helpful and ease any doubts and concerns of mine.” [U#11]. Relatedly, many participants worried that ambient monitoring could become distracting during critical moments (4 strongly agree; 8 somewhat agreed). Consistent with these concerns, willingness to personally adopt the technology was only moderate: only 5/16 somewhat or strongly agreed that they would use ambient AI if it reduced cognitive workload (with 11/16 neutral or disagreeing).

3). Concerns about Having a Conversation with AI in Front of Patients:

Because EMS work is fast-paced and hands-busy, providers may not have the time or free hands to type questions, navigate menus, or interact extensively with a decision support interface while actively treating and monitoring patients. To account for these constraints, we explored alternative interaction modalities that could reduce manual input and enable rapid access to support. In particular, we explored the idea of using voice-based interaction to access AI support during EMS care. In this concept, providers could speak natural language questions (e.g., how to treat a symptom, what protocol to follow, or what medication to give), enabling hands-free information access during hands-busy tasks.

Although participants saw the potential value of voice interaction for on-scene work, many were hesitant to use spoken queries with AI in front of patients or family members. In prehospital care, communication is not only instrumental for coordinating tasks but also central to building trust, calming anxious patients, and demonstrating competence. Providers noted that verbal exchanges with an AI system could be misinterpreted as uncertainty or lack of expertise, particularly when patients are frightened, in pain, or closely watching the provider’s actions. As one participant explained: “My only concern is that if the patient or the patient’s family hears you basically asking what to do, yeah, they may start yelling like, do you even know what you’re doing?” [U#10]

Participants also raised concerns about privacy and confidentiality. Verbal queries can unintentionally disclose sensitive information, especially in public scenes, where bystanders may overhear conditions or medications. Even if the AI response is clinically correct, speaking patient details aloud may feel inappropriate or violate agency expectations for discretion. In addition, providers noted that voice interaction may be impractical in noisy environments (sirens, traffic, radio chatter) or when multiple team members are speaking at once, increasing the risk of recognition errors and creating frustration during time-critical tasks. As one provider shared his concerns: “I mean, you know, voice would be good, but you know, the back of the ambulance, especially if you have the sirens going, isn’t always the quietest place, and so I think it’d be really frustrating to use voice versus typing. The problem that we’ve had with chat boxes and things like that in the past is that if there’s multiple people talking, you know, basically, if you say something to it, hypertension, for instance, and somebody talks at the same time, it may go a different route.” [PD#5]

V. Discussion

In this section, we situate our findings within the existing literature to highlight key design implications for developing AI-CDSS tools that better align with the needs of EMS providers.

A. AI That Helps EMS Providers Most: Faster Information, Less Documentation, Preserving Provider Control

Across both studies, our findings illustrate that AI is most likely to be adopted in EMS when it functions as a time saver. Prehospital care is characterized by extreme time pressure, frequent interruptions, and simultaneous demands for hands-on treatment, team coordination, and documentation [7], [28], [30]. In this context, AI’s value is less about producing sophisticated reasoning outputs and more about enabling rapid action: reducing information-seeking time, generating concise patient-history snapshots for quick situational overview, and drafting documentation to offload clerical tasks that compete with patient care.

Crucially, participants emphasized that these benefits are only acceptable when AI preserves provider control. Documentation was the clearest example. Providers welcomed AI-facilitated documentation not because they wanted automation to take over, but because it could reduce documentation burden while still leaving them accountable for the record. In practice, the preferred model was “drafting, not finalizing”: AI may capture and structure information quickly, but providers must be able to review, correct, and decide what becomes part of the official record. This preference highlights a core principle of effective human–AI collaboration: systems should support appropriate reliance by keeping clinicians in the loop and making it easy to verify, override, and finalize outputs. In contrast, systems that silently lock in decisions can erode trust and ultimately limit adoption [5], [4].

B. Cautious Use of AI in Emergency Care Settings

While EMS providers in our study expressed interest in AI-enabled support, they were equally attentive to the practical implications of deploying such technology in the field. Concerns were especially pronounced for ambient AI used to monitor care in real time to detect potential patient-safety events (e.g., medical errors or workflow deviations). Participants worried that “always-on” monitoring could erode privacy and professional autonomy and shift AI from support to surveillance [40]. These concerns echo broader discussions of ambient AI in healthcare, which emphasize the need for explicit disclosure of data collection practices, clear limits on how data are used, and granular consent mechanisms so that automation does not override professional boundaries or patient rights [41].

Another major concern centered on clinical authority and accountability. Providers drew a firm line against AI outputs that could be interpreted as diagnostic assertions (e.g., inferring a definitive condition from symptoms and vital signs), arguing that such features could blur responsibility and increase liability risk. In healthcare, clinicians are accountable for both clinical judgments and patient outcomes; accordingly, our participants were wary of systems that could be perceived as asserting authority over their decisions. This stance is consistent with prior findings that clinicians often resist decision support that appears to replace or override professional judgment, particularly in high-stakes, time-sensitive care [23], [33]. For participants, AI was acceptable when it strengthened clinical reasoning by surfacing relevant information or resources (e.g., protocols, contraindications, dosing references), but not when it repositioned itself as a diagnostic actor.

Taken together, our findings point to a clear tension: EMS providers may see AI as useful, yet still hesitate to rely on it when the stakes are high. Participants in our study viewed greater transparency in AI-supported decision making as a way to mitigate these tensions. More specifically, providers wanted to quickly understand what an output was based on—whether it reflected evidence-based protocols, recognizable clinical rationale, or uncertain inference—so they could assess appropriateness in the moment. This need aligns with prior work on explainable AI showing that contextualized explanations can support accurate mental models and better-calibrated trust [42]. However, in EMS, transparency must be designed to fit time-critical workflow: explanations should be available on demand without adding steps or interrupting care. One promising direction is progressive disclosure, where brief, actionable outputs are shown first, with links or expandable details available when needed [43]. Future work can probe the level of detail required (e.g., high-level summaries vs. model rationale, confidence cues, and provenance links) and how transparency should be delivered without disrupting time-critical workflow.

C. Designing AI Decision Support for the Realities of EMS Work

Designing AI-enabled decision support for EMS requires accounting for the distinctive physical and social conditions of prehospital care. Unlike many clinical settings where clinicians work in relatively controlled environments, EMS care is delivered in mobile, unpredictable contexts shaped by noise, movement, constrained space, and continuous multitasking [7], [28], [30]. These constraints do not merely affect usability; they fundamentally shape what kinds of AI interaction are feasible, trustworthy, and safe. Our findings suggest that AI-CDSS concepts developed for hospitals or clinic workflows cannot be assumed to transfer to EMS without rethinking interaction design around field realities.

Noise and acoustic variability are particularly salient for AI-assisted documentation and speech-based data capture. Participants viewed such technology as promising, but only if it performs reliably amid sirens, radio chatter, traffic, crowd noise, and overlapping team conversations. Inaccurate transcriptions can create downstream clinical, billing, or legal consequences. This highlights a core EMS design requirement: speech-enabled systems must be robust to noisy environments and support rapid correction [44].

Beyond acoustics, EMS introduces distinct social constraints that shape whether voice interaction is appropriate. Although hands-free interaction seems well matched to hands-busy EMS work [45], participants raised concerns about speaking AI queries aloud in front of patients and family members. Prehospital care is a public-facing encounter where trust, competence, and calm communication are essential. Audible exchanges with an AI system (e.g., “what do I do?”, “tell me the protocol”) may be interpreted as uncertainty or lack of expertise, potentially increasing patient anxiety or undermining confidence in the care team. These concerns echo broader HCI and healthcare findings that adoption depends not only on individual usability but also on how technology reshapes communication, roles, and perceived authority in clinical encounters [23], [8]. In EMS, where providers must manage patient emotions and family dynamics alongside clinical tasks, these social signals become a core design constraint.

D. Study Limitations

Several limitations should be noted. Our study focused on eliciting early-stage needs, perceptions, and design preferences rather than evaluating a deployed AI system. Participants did not directly interact with functional AI tools; instead, they reacted to design probes and scenario-based concepts grounded in their professional experience. As a result, our findings reflect anticipated usefulness and concerns rather than observed user experience or real-world usage patterns. For example, while some providers expected AI to reduce cognitive workload, we did not directly measure workload or stress outcomes in this study. Additionally, some questionnaire items showed responses spanning the full likert-scale range, suggesting substantial heterogeneity in attitudes and introducing uncertainty about the degree of consensus across providers and contexts. Future work should implement functional prototypes and evaluate them in high-fidelity simulations or field pilots to assess actual use under realistic conditions and examine how perceptions—and potential impacts on workload and stress—evolve with hands-on experience over time. Finally, our participant sample included many highly experienced EMS providers, which may shape expectations and acceptance of AI support. Experienced clinicians may have stronger mental models of protocols and field constraints, potentially increasing skepticism toward AI recommendations. Future work should examine how perspectives differ by experience level (e.g., novice vs. experienced providers) and how training background influences willingness to adopt AI. Relatedly, our work centered on providers’ perspectives and did not include other stakeholders who influence AI adoption in EMS, such as EMS leadership, medical directors, dispatch, receiving clinicians, compliance/legal teams, or patients and families. These groups may have distinct priorities (e.g., liability, billing, policy compliance, patient trust) that affect feasibility and acceptance. Future studies should incorporate multi-stakeholder requirements.

VI. Conclusion

This paper uses EMS as a time-critical use case to examine where AI technologies could—and should not—support on-scene decision making. Through a user-centered design approach, we found that EMS providers valued AI most when it functions as an assistive, secondary aid that saves time, reduces cognitive and clerical burden, and preserves provider control. At the same time, providers articulated clear boundaries for responsible use, emphasizing that AI should not assert authority over clinical judgment and must be aligned with the practical realities of EMS work (e.g., hands-busy care, noisy and mobile environments, and high-stakes accountability). Together, these findings underscore that successful AI support in EMS depends on low-friction, workflow-compatible designs that enable appropriate reliance—helping providers act faster while keeping them in control.