A voice-based digital assistant for intelligent prompting of evidence-based practices during ICU rounds

STRUCTURED ABSTRACT

Objective:

To evaluate the technical feasibility and potential value of a digital assistant that prompts intensive care unit (ICU) rounding teams to use evidence-based practices based on analysis of their real-time discussions.

Methods:

We evaluated a novel voice-based digital assistant which audio records and processes the ICU care team’s rounding discussions to determine which evidence-based practices are applicable to the patient but have yet to be addressed by the team. The system would then prompt the team to consider indicated but not yet delivered practices, thereby reducing cognitive burden compared to traditional rigid rounding checklists. In a retrospective analysis, we applied automatic transcription, natural language processing, and a rule-based expert system to generate personalized prompts for each patient in 106 audio-recorded ICU rounding discussions. To assess technical feasibility, we compared the system’s prompts to those created by experienced critical care nurses who directly observed rounds. To assess potential value, we also compared the system’s prompts to a hypothetical paper checklist containing all evidence-based practices.

Results:

The positive predictive value, negative predictive value, true positive rate, and true negative rate of the system’s prompts were 0.45±0.06, 0.83±0.04, 0.68±0.07, and 0.66±0.04, respectively. If implemented in lieu of a paper checklist, the system would generate 56% fewer prompts per patient, with 50%±17% greater precision.

Conclusion:

A voice-based digital assistant can reduce prompts per patient compared to traditional approaches for improving evidence uptake on ICU rounds. Additional work is needed to evaluate field performance and team acceptance.

Graphical Abstract

1. INTRODUCTION

Evidence adoption is a persistent challenge in health care, particularly in the complex intensive care unit (ICU) environment.[1–3] Rounding checklists are a leading strategy for improving adoption of evidence-based practice,[4] but they are frequently ineffective because their generic nature leads to excessive repetition, cognitive load, and time waste.[5] With recent advancements in the technologies that support digital voice assistants,[6,7] it might be possible to automatically adapt an ICU rounding checklist to the unique needs of each patient. Such a system would use a microphone to ‘listen’ to an ongoing rounding discussion, artificial intelligence technologies to determine which of the set of evidence-based practices are pertinent and have yet to be addressed by the care team, and a computer screen or speaker to prompt the team to consider the items on the customized checklist.

1.1. Objective

We sought to evaluate the technical feasibility and potential value of a voice-based digital assistant for intelligent prompting of evidence-based practices during ICU rounds based on the care team’s discussion.

1.2. Evidence Adoption During ICU Rounds

Daily multidisciplinary rounds are a cornerstone of modern ICU care. The rounding team consists of nurses, physicians, family members, and others who meet to discuss each patient’s clinical state and create a consensus care plan.[8] Technology is pervasive on rounds, with many—if not all—team members pushing around workstations-on-wheels to access electronic health records (EHR), write orders and perform documentation during the one to four hours it takes to complete rounds.

A leading strategy for evidence adoption is to have a generic checklist of practices and to review it for every patient during rounds.[4] Although checklists have had a tremendous impact in aviation and other industries due to their effectiveness and ease of use,[9] they often fail to affect change in hospital settings like the ICU.[10–12] The generic nature of rounding checklists is a significant reason for their lackluster efficacy. In short, they are too blunt a tool and are not attuned to whether the patient is eligible for the evidence-based practices, whether they are receiving them, and whether the care team intentionally decided, with justification, not to deliver them.[5] Furthermore, a generic checklist does not provide the contextual information required for making an informed decision. As a result, checklists are frequently either circumvented by the care team or simply shelved, thereby precluding any effect.[13,14]

Improvements to generic checklists have been tested. For example, some recommend keeping checklists as short as possible and tailoring them to the specific needs of an ICU.[5] A shorter checklist may increase the likelihood of completion, but this guidance is a mismatch to the steadily increasing body of evidence clinicians are expected to implement regularly.[15–17] An alternative to shortening an ICU’s generic checklist is implementing dynamic checklists within the EHR that adapt their content based on available patient data.[18–21] With a personalized checklist generated for each patient, more practices can be considered without increasing clinician workload, making them flexible to evolving evidence and much more palatable to clinicians.[4]

Dynamic checklists are an improvement to generic checklists; however, a reliance on EHR data is a limitation because EHR data suffers from delayed, variable, and selective documentation.[22] In other words, therapies are often documented significantly later than the decision to order them, and are not documented at all when the team consciously decides not to order something or if the nurse was following charting by exception.[23,24]

Another strategy to improve checklists is the use of a human checklist prompter. Human checklist prompters are dedicated individuals, often nurses with experience in ICU care, who listen to the care team discuss the patient during rounds, adapt the rounding checklist based on that discussion, and then prompt the team only for applicable practices that have yet to be addressed.[25] A landmark clinical trial demonstrated that human prompting is associated with lower mortality in the ICU.[26] However, a human checklist prompter is extraordinarily effort-intensive, challenging to scale, and affected by staff availability and the limits of human attention, making this strategy impractical for widespread use.

1.3. Ambient Artificial Intelligence in Healthcare

The use and study of ambient technologies in healthcare is of growing interest.[27,28] Devices such as cameras, internet of things (IoT) sensors [29,30], and microphones can capture data that were previously unknown, and often unknowable.[31] With expanded information about a patient, their environment, and their care team, new types of artificial intelligence systems can be developed that lead to better outcomes, lower costs, and an improved experience of both patients and clinicians.[32,33] For example, dictation has long been used to assist with documenting doctor-patient interactions,[34] and now voice input is increasingly being used to support doctor-EHR interactions.[7]

Additional opportunities for ambient technologies include supporting the electronic capture of contextual information about why clinical decisions were made [35] or what are a patient’s treatment priorities [36]; automated measurement of caregiver workload [37]; intelligent presentation of relevant medical knowledge or patient data[38,39]; and video surveillance to prevent adverse events such as patient falls.[40]

With new power to capture data comes new responsibility to do so ethically.[41] For example, what does it mean for employee rights when clinicians, rather than patients, are the subject of research studies?[42]

2. MATERIAL AND METHODS

This section begins with a conceptual overview of the intelligent prompting system. It then briefly described the evidence-based practices that we aimed for the system to support in this developmental version. Next, it describes data collection. Then, it describes each of the system’s intelligent components and their evaluation. Finally, it describes evaluation of the entire system.

2.1. Conceptual Overview

A voice-based digital assistant for intelligent prompting can be described using an input, processing, and output framework. For system input, we use audio recordings of the multidisciplinary team’s rounding discussions. While other input modalities are possible, such as structured EHR data, we focus on audio because the data are immediately relevant to the patient’s care trajectory (being that rounding discussions focus on the patient at that moment) and are relatively untapped as a data source for real-time clinical decision support.

For system processing, we divide the steps into three intelligent components: transcription, fact extraction, and checklist adaption from inference. Transcription converts the audio recording into a text transcript. Fact extraction uses natural language processing (NLP) to extract facts about the patient and the content of the team’s discussion from a transcript. Checklist adaption from inference uses a set of rules and the extracted facts to determine which evidence-based practices are applicable yet not addressed for the current patient.

For system output, the personalized sets of prompts are displayed within the EHR. Like input, other output modalities are possible but are outside of project scope. Figure 1 shows the stages and components of the system.

Figure 1.

Overview of a voice-based digital assistant for intelligent prompting of evidence-based practices. The system is depicted on an input, processing, output diagram. This paper focuses on the processing stages’ three components: transcription, fact extraction, and checklist adaption from inference.

2.2. Evidence-Based Practices of Focus

During this development stage, we opted to focus our voice-based digital assistant on the evidence-based practices included in the ABCDEF bundle,[43] a mnemonic that stands for assess, prevent, and manage pain; both spontaneous awakening trials (SAT) and spontaneous breathing trials (SBT); choice of analgesia and sedation; delirium: assess, prevent, and manage; early mobility and exercise; and family engagement and empowerment. The Society of Critical Care Medicine endorses this bundle in clinical practice guidelines, and its elements are often incorporated in rounding checklists,[44] making them an exemplary set of initial practices to integrate.

2.3. Data Collection

Between February and December 2021, we audio recorded ICU rounds on 90 different calendar days capturing a total of 743 patient discussions. In addition to the audio recordings, we also enlisted expert observers to shadow rounds and record a reference standard of which items of the ABCDEF bundle were addressed during each patient’s discussion on 15 of the 90 data collection days (106 patient discussions were assessed).

2.3.1. Audio recording protocol

Audio recordings were captured by a project team member using an omnidirectional microphone (Jabra Speak 510). The microphone was placed on one of the care team’s workstations on wheels before rounds began. Once placed, the care team was expected to conduct rounds as usual without paying attention to whether they were near the microphone. The project team controlled the microphone (i.e., starting and stopping a separate recording for each patient discussion) via Bluetooth using a tablet device. All recordings were securely uploaded to encrypted cloud storage at the end of rounds.

2.3.2. Reference standard protocol

A reference standard was recorded on a subset of the data collection days by one of two critical care nurses (authors JBS and KMP) who served as expert observers. They assessed each patient discussion by watching and listening to the care team. They were instructed not to look into patient rooms or into the EHR because they were supposed to only have access to the same information as the intelligent prompting system (i.e., the team’s spoken discussion). They recorded their assessments using a previously validated tool that specifies which evidence-based practices in the ABCDEF bundle were applicable for each patient and whether the team discussed providing those practices during the rounding discussion.[45]

2.3.3. Clinical setting

The project included two different ICUs at a large teaching hospital: one unit was a 22-bed surgical trauma ICU and the other was a 12-bed transplant ICU specializing in abdominal transplants. Both units are staffed by multidisciplinary teams, including attending intensivists, fellows, residents, critical care nurses, nutritionists, clinical pharmacists, rehabilitation therapists, and respiratory therapists. Most, and sometimes all, of these team members participate in rounds. Neither ICU uses a strict rounding script to guide the patient discussions, but paper print outs of patient information are frequently referenced during rounds.

Clinician demographics were not recorded but the data set included rounding teams lead by ten different attending physicians. Patient demographics were not recorded for the complete data set; however, a previously described measurement study in these same units reported the follow characteristics [45]: median age was 60 (interquartile range: 45–70); 68% (n=36/53) were male; race distribution between Black, White, and other/unspecified was 11%, 61%, and 28%, respectively; and the median Sequential Organ Failure Assessment score on the day of observation was 6 (interquartile range: 4–10). Additionally, of the patients in the testing set (described in Section 2.4) 21% were receiving continuous intravenous sedation and 45% were receiving invasive mechanical ventilation.

2.3.4. Scheduling and ethical approvals

Data collection days were selected out of scheduling convenience and included a mix of weekdays and weekends (when rounds include fewer team members). Neither the team member recording discussions nor the nurse observing them interact with the care team other than gaining written consent at the start of rounds and when occasionally asked not to observe a discussion, such as an end-of-life conversation with a patient’s family. The microphone was also paused when the care team entered a patient’s room, when the patient or their family spoke, and when an unconsented care team member joined a discussion.

All data collection and analyses were part of a larger project that directly sought to improve the quality of patient care by enhancing evidence adoption at the hospital. As such, this work was approved and monitored by the study hospital’s quality improvement review committee (#1792). As part of the approval, all identifiable data were maintained in a secure storage system owned and operated by the hospital. An essential part of making this project possible was an extensive outreach effort that included conversations with the hospital’s quality improvement review committee, legal counsel, chief healthcare innovation officer, chief medical information officer, cloud specialists, unit directors, medical directors, and frontline nursing staff, as well as the university’s institutional review board, office of innovation, and director of technology.

2.4. System Development

In this section, system development is described through the perspective of the data. First the data are separated into training and testing sets (Figure 2), where the testing set includes all the patient discussions collected on the 15 days where an expert observer was recording a reference standard. The training set is used for evaluating a commercially available automatic transcription model and training and selecting a custom NLP model for fact extraction. The testing set is used to evaluate the NLP model and the output of the entire intelligent prompting system.

Figure 2. Data flow from the training and testing set.

The training set was used to evaluate the word error rate of the unmodified Amazon Transcribe Medical model, to develop a codebook for annotating utterances as relating to the adoption of the evidence-based practices on the ABCDEF bundle, and to train and select the multinomial NLP model for annotating individual utterances as belonging to one of the annotation classes. Audio recordings were treated as “unintelligible” and eliminated if Amazon Transcribe Medical did not return a result. Some transcripts were eliminated (i.e., “156 uncorrected” and “88 held out”) due to resource constraints. The testing set was used to evaluate the NLP model and the automatic checklists generated by the entire voice-based intelligent prompting system.

2.4.1. Automatic transcription

For automatic transcription, we used Amazon Transcribe Medical. Audio files were batch processed using the primary care medical specialty model because it was the only available batch model at the time of the study. Performance of this system is calculated on the training set by comparing automatically generated machine transcripts to the corresponding (manually corrected and de-identified) human transcript version. Machine transcripts without a human transcript counterpart were excluded from the analysis. Performance is reported using word error rate as implemented by the JiWER python package. The equation for word error rate is (S+D+I)/(S+D+C), where S, D, I, and C are the number of substitutions, deletions, insertions, and corrected words, respectively.

2.4.2. Fact extraction

The NLP model for fact extraction was developed on the training set using the human transcripts. The objective of the model is to automatically assign each utterance (i.e., spoken sentence) one of ten possible classes based on its content:

1. the patient is on a continuous sedative

2. the patient is not on a continuous sedative

3. SAT was discussed

4. the patient is receiving invasive mechanical ventilation

5. the patient is not receiving invasive mechanical ventilation

6. SBT was discussed

7. a delirium score was discussed

8. patient mobility was discussed

9. family communication was discussed

10. none

2.4.2.1. Codebook development

Training the model required supervised training data consisting of utterances and their manually annotated class labels. Annotation was performed by trained annotators who were first involved in codebook development (five batches of approximately ten notes). The final codebook defines how the annotators should assign one of thirteen annotation classes to each utterance, where each class represents a fact that can be inferred based on the content of the utterance. For example, the utterance “vent settings are 25% and 5 right now” implies that a patient is receiving invasive mechanical ventilation (with the quoted numbers corresponding to the fraction of inspired oxygen and positive end-expiratory pressure, respectively).

The annotators then annotated the 311 and 113 (corrected and de-identified) human transcripts from the training and testing sets, respectively. The numbers of annotations assigned to each data set are shown in Table 1.

Table 1.

Annotation Counts

Class (discussed or implied)	Training set	Testing set
Delirium score	196	37
Agitation score	0	9
On continuous sedative	508	156
Not on continuous sedative	104	22
SAT	253	55
SAT contraindication	6	1
On invasive mechanical ventilation	518	115
Not on invasive mechanical ventilation	221	62
SBT	381	94
SBT contraindication	19	8
Mobility	439	93
Family on rounds	13	0
Family communication	468	111
Total utterances	46441	13583
Total discussions	311	113

Early versions of the codebook included classes related to ABCDEF bundle element A (assess, prevent, and manage pain); however, these classes were later removed due to poor inter-annotator agreement. It will require additional future work to reintroduce pain-related cases.

2.4.2.2. Model architecture and training

With a supervised training set prepared, we trained an NLP model to perform this classification task. The model includes three steps. First, we tokenized each utterance (i.e., replacing individual words and word segments with a numerical value corresponding to that word segment’s index in a pre-defined vocabulary) using HuggingFace’s BioClinicalBERT tokenizer.[46] The publicly available BioClinicalBERT tokenizer and model were trained on a combination of PubMed Central manuscripts and the Medical Information Mart for Intensive Care version 3 (MIMIC-III) free-text notes.[46] Second, we inputted the tokenized utterances into the pre-trained BioClinicalBERT model. Deep learning models like BioClinicalBERT are artificial neural networks with many layers of interconnected nodes numerically representing relationships between words learned from a large corpus. Third, we replaced the model’s output layer with a multinomial logistic regression model that has a single outcome variable with ten possible values (the annotation classes mentioned earlier sans the four classes with less than 20 examples in the training set).[47–49] Using cross validation on the training set, we selected the final training parameters (specifically, we selected the regression model rather than a support vector machines model) and trained the final version of the model on the entire training set before applying it to the testing set.

2.4.2.3. Application on the testing set

On the testing set, the NLP model is applied in two different ways. First, to evaluate the performance of the model we apply it to the testing set’s annotated (human-corrected) transcripts. Performance is reported as precision, recall, and F₁ score with 95% confidence intervals generated using 1,000 bootstrap iterations.

Second, to use the model as part of the intelligent prompting system we apply it to the testing set’s machine transcripts. The resulting output (a set of annotation classes for each transcript) is passed to the checklist adaption component as a list of facts.

2.4.3. Checklist adaption

Checklist adaption is achieved using an expert system, where a set of rules deterministically defines how the checklist should be modified based on the list of facts provided. For example, the pseudocode for one of the rules is as follows: “IF <patient is receiving invasive mechanical ventilation> AND NOT <SBT was discussed> THEN [personalized set of prompts] ADD ‘SBT’.”

These rules were manually crafted and implemented by the project team a priori. In other words, the rules were developed before looking at the testing set and were only applied once to the testing cases to determine the results. Future work can investigate how to further improve the system by updating the rules or using other inference strategies; however, doing so was outside the scope of the current project.

2.5. Entire System Evaluation

In evaluating the system’s output, we focused on two overarching metrics: technical feasibility (i.e., how accurate is the system?) and potential value (i.e., how useful would the system be for improving the workflow of the ICU rounding team?). To evaluate accuracy, we compared the system’s output to a reference standard of a nurse’s assessment of which prompts are appropriate for each patient. For every patient, both the system and the nurse considered six possible prompts: SAT, SBT, assess agitation, assess delirium, early mobility, and family engagement. Evaluation of system output meant counting prompts as true positive (TP) when the system and the nurse agreed that a prompt is appropriate, true negative (TN) when the system and the nurse agreed that a prompt is not appropriate, false positive (FP) when the system judged a prompt to be appropriate, but the nurse did not, or false negative (FN) when the nurse judged a prompt to be appropriate but the system did not. From these counts, we calculated the positive predictive value: TP / (TP + FP), true positive rate: TP / (TP + FN), negative predictive value: TN / (TN + FN), true negative rate: TN / (TN + FP), and F₁ score (2 * TP / [2 * TP + FP + FN]).

To evaluate potential value, we compared the system’s output to a static paper checklist that is the same for each patient. By definition, a paper checklist generates the same six prompts for all patients, while the system generates only the prompts that it estimates are appropriate. We also compared the positive predictive value and true positive rate of the prompts generated by the system to those generated by the checklist, where the checklist’s prompts could only count as true positive or false positive depending on whether the nurse reference standard considered it appropriate or not.

3. RESULTS

3.1. Entire System Performance

The performance of the voice-based digital assistant compared to an expert nurse when prompting for evidence-based practices is shown in Figure 3. These results indicate that voice-based intelligent prompting is technically feasible and provide a baseline for evaluating future work. In decreasing order, the digital assistant’s most common prompts were “assess delirium” (n=98), “family engagement” (n=53), “assess agitation” (n=42), “early mobility” (n=33), “SAT” (n=31), and “SBT” (n=23). The expert nurse generated fewer prompts overall, but the frequency of each prompt followed a similar order: “assess delirium” (n=56), “family engagement” (n=40), “assess agitation” (n=40), “early mobility” (n=38), “SBT” (n=8), and “SAT” (n=4).

Figure 3.

Performance of the voice-based digital assistant compared to an expert nurse when prompting for evidence-based practices on 106 patient discussions during ICU rounds.

Despite generating more prompts than a nurse, the digital assistant generates 56% fewer prompts than the paper checklist (2.6 versus 6.0 prompts per bed, respectively), demonstrating its potential value. The system also has a positive predictive value that is 50% higher (95% confidence interval: 33%, 67%) and a true positive rate that is 32% lower (95% confidence interval: 26%, 38%) than the paper checklist.

3.2. Performance of Individual Prompts

The digital assistant’s performance on each of the six supported prompts is shown in Figure 4. Based on F₁ scores, the digital assistant only outperforms a paper checklist (first and last rows within each prompt, respectively) for family engagement and SBT; however, the F₁ score of the generic checklist is primarily driven by its perfect true positive rate (i.e., no false negatives). Paper checklist performance is theoretical because it assumes the clinical teams would review all prompts. This number will be lower in real-world situations because clinical teams sometimes skip some or all prompts.[5] The figure also shows performance using different sets of intelligent processing components. The impact of automatic transcription can be seen by comparing each prompt’s first and second rows. Likewise, the impact of NLP-based fact extraction can be seen by comparing each prompt’s second and third rows.

Figure 4.

Performance of each prompt on 106 patient discussions during ICU rounds when adapted by different means. Each prompt is divided into four rows to show the contributions of each intelligent processing component. The first row corresponds to the entire system, including automatic transcription, NLP-based fact extraction, and checklist adaption from inference. The second row replaces automatic transcription with human transcription. The third row replaces automatic transcription and NLP-based fact extraction components with human methods. The fourth row includes none of the intelligent components; it corresponds to a generic paper checklist. The stacked bar charts represent confusion matrix counts for false positive and false negative (left of zero) and true positive and true negative (right of zero).

3.3. Performance of Automatic Transcription and Fact Extraction

The median word error rate on 399 machine transcripts was 0.70 (95% confidence interval: 0.31, 0.96) when evaluated on a human-corrected reference standard. The wide confidence interval is unsurprising given the dynamic nature of the data collection environment. The speakers and their location changes throughout rounds and during individual patient discussions as the care team moves through the hallway, talks with patients and their families, and is interrupted by public address system announcements, the sounds of medical equipment, and the requests of other clinicians. Additionally, automatic transcription performance is further penalized because the human transcriptionists de-identified their version of the transcripts and removed sections of text that were unrelated to patient care.

Performance of the NLP model for fact extraction on the human corrected and annotated transcripts of the testing set is shown in Table 2. Annotation classes that were well defined in the codebook performed better than classes which were more open ended. For example, the delirium score class was applied to utterances that contained a mention of an Intensive Care Delirium Screening Checklist (ICDSC) number or a “delirium score.” In contrast, the SAT discussed class could be applied to utterances with a greater variety of concepts, such as “sedation vacation,” “sedation holiday,” “wake the patient up,” “titrate to pain,” “SAT not applicable,” “see how he tolerates being off propofol,” and “bring the propofol off.” Furthermore, some concepts relating to the discussion of an SAT also apply to the class indicating that a patient is on a continuous sedative, making a classification of a single class more difficult.

Table 2.

Performance of the NLP model for fact extraction. Confidence intervals (CI) are calculated using bootstrap resampling with 1,000 replications.

Class	Precision (95% CI)	Recall (95% CI)	F1 score (95% CI)
On continuous sedative	0.83 (0.69, 0.93)	0.83 (0.70, 0.94)	0.83 (0.72, 0.91)
Not on continuous sedative	0.88 (0.63, 1.00)	0.84 (0.58, 1.00)	0.85 (0.65, 0.97)
SAT discussed	0.59 (0.38, 0.83)	0.56 (0.33, 0.80)	0.57 (0.38, 0.75)
On invasive mechanical ventilation	0.77 (0.65, 0.90)	0.78 (0.63, 0.90)	0.78 (0.67, 0.86)
Not on invasive mechanical ventilation	0.85 (0.68, 1.00)	0.83 (0.61, 0.97)	0.84 (0.69, 0.95)
SBT discussed	0.74 (0.61, 0.87)	0.74 (0.59, 0.87)	0.74 (0.63, 0.84)
Delirium score	0.95 (0.82, 1.00)	0.93 (0.76, 1.00)	0.94 (0.83, 1.00)
Mobility discussed	0.89 (0.79, 0.96)	0.88 (0.75, 0.96)	0.88 (0.79, 0.94)
Family discussed	0.87 (0.76, 0.96)	0.88 (0.75, 0.96)	0.87 (0.78, 0.94)
Random none	0.95 (0.93, 0.96)	0.95 (0.93, 0.97)	0.95 (0.93, 0.96)
Macro average	0.83 (0.79, 0.87)	0.82 (0.77, 0.86)	0.82 (0.78, 0.86)

4. DISCUSSION

A retrospective analysis of a voice-based digital assistant showed that it could reduce the number of evidence-based prompts per patient compared to a generic rounding checklist. Overall performance was impacted by both the automatic transcript quality and the NLP model’s accuracy. Large technology companies invest heavily in improving the performance of automatic speech recognition systems;[6] therefore, it is most prudent for academic researchers to focus their efforts to improve this system on NLP model development. For the current model, the “SAT discussed” class had the poorest performance, which mirrors our findings from a prior study that validated the nurse-based assessment tool we used to collect the reference standard for this study.[45] This finding suggests that a deeper investigation into how sedation and SATs are discussed during interprofessional rounds is warranted.

Importantly, these results represent a “floor” for digital assistant performance because we intentionally restricted system input to audio recordings of rounding discussions. The system architecture we developed is readily compatible with additional data sources like EHR data, manual user input, or information carryover from previous discussions of a patient. Combining multiple data sources to support clinical decision support is an emerging and rich area of research that might make clinical decision support systems more useful.[50] For example, the PhenoPad system combines both speech and handwriting recognition to improve notetaking during patient encounters,[51] the Learning Electronic Medical Record System combines EHR click stream and eye-tracking data to train predictive models that anticipate a clinician’s information-seeking behaviors,[52] and the ICU Cockpit captures and combines multimodal waveform data for computational disease modeling.[53]

Although we demonstrated technological feasibility, more work is needed to understand the human-computer interaction aspects of this system. Specifically, work is needed to determine how the care team should receive prompts and how the microphone would be incorporated into the rounding process. Beyond starting a recording, microphone positioning relative to the care team substantially impacts audio and transcription quality. Consequently, designing the team’s interactions with the system to be seamless and value-adding is essential for the team to want to consider microphone operation and placement throughout rounds.

Additional studies are needed to determine the minimum acceptable performance for a system like this. That said, the minimal acceptable performance level likely varies by application and setting, and is probably best defined by the performance that results in sustained improvements in delivery of evidence-based practice. It is also likely that an effective system would incorporate uncertainty into its interface. For example, the system might estimate its confidence in its output and default to an un-adapted state when certain thresholds are not met. Regardless, dynamic checklists enable the routine monitoring of a greater number of evidence-based practices than could ever be realistically included on a static checklist.

This work has potential beyond increasing the adoption of evidence-based practices. The voice-based digital assistant could also automatically generate order sets containing discussed, implied, or recommended orders.[54] Order sets and other forms of automated documentation could be implemented as open-loop actions, which need to be reviewed and accepted by a clinician before committing to the EHR.[55] Additionally, the audio data could be used to calculate individual and team performance metrics automatically,[56] such as rating the quality of an oral case presentation,[57] detecting alignment between palliative care goals and care received,[58] or evaluating the quality of communication between the care team and family members.[59] Automatic evaluation metrics are needed because manual approaches alone are too resource intensive for routine use.[45] When deciding on use cases to pursue, it is likely most fruitful to focus on tasks that support, rather than replace, clinician decision-making.[60]

One limitation of the present study is that rounding discussions were only audio recorded when they occurred in the hallways outside of a patient’s room. This was not an issue at our local institution, where most rounding discussions occur in the hallway; however, other institutions may have different practices that impact the generalizability of results, such as rounding in a conference room or within every patient’s room. Furthermore, we rejected 6% (7 out of 120) of the audio-recorded patient discussions in the testing set because they were less than thirty seconds long or completely unintelligible to Amazon Transcribe Medical (i.e., no transcript generated). The word error rate in our system was significantly higher than the 12.8% found by others.[6] In addition to the causes mentioned earlier (such as multiple speakers in a dynamic environment and the human-correction and de-identification process) we have yet to attempt to improve transcription model performance by adapting it to ICU terms and jargon. By design, we chose intelligent prompting to be our initial use case for the voice-based digital assistant because we (correctly) hypothesized that checklist adaption could be performed accurately from noisy transcripts. Our choice to “place the microphone once and forget about it” is a strength because it reflects real-world conditions.

5. CONCLUSIONS

Automatic transcription, NLP, and a rule-based system were combined to create a digital assistant that intelligently prompts applicable yet unaddressed evidence-based practices during team rounds in the ICU. This approach to increasing evidence adoption is more scalable than dedicated human checklist prompters and potentially timelier than adaptive checklists that rely only on data available in the EHR. Additional work is needed to evaluate system performance and team acceptance in a live clinical environment. Implementing a voice-based digital assistant in an ICU will result in the routine capture of rounding discussions, enabling the development of new types of conversationally informed clinical decision support systems.

STATEMENT OF SIGNIFICANCE.

Problem or Issue:

Evidence adoption is a persistent challenge in the Intensive Care Unit.

What is Already Known:

Of the existing checklist-based implementation science strategies: generic rounding checklists are too imprecise and repetitive, dynamic checklists that use EHR data are oblivious to the care team’s ongoing rounding discussion, and dedicated human checklist prompters are too resource intensive for routine use.

What this Paper Adds:

A new strategy for checklist prompting utilizing voice-based digital assistant technologies to automate checklist prompting. We estimate the real-world performance of our intelligent prompter system on a set of audio recorded ICU rounding discussions.

ABBREVIATIONS

ABCEDF Bundle

a mnemonic covering several evidence-based practices

CI

Confidence Interval

EHR

Electronic Health Record

FN

False Negative

FP

False Positive

ICU

Intensive Care Unit

NLP

Natural Language Processing

SAT

Spontaneous Awakening Trial

SBT

Spontaneous Breathing Trial

TN

True Negative

TP

True Positive

DATA AVAILABILITY

The data underlying this article cannot be shared publicly due to institutional policies that protect the privacy of individuals whose data were used in the study.