Facilitators and Barriers to Interacting With Clinical Decision Support in the ICU: A Mixed-Methods Approach

Abstract

OBJECTIVES:

Clinical decision support systems (CDSSs) are used in various aspects of healthcare to improve clinical decision-making, including in the ICU. However, there is growing evidence that CDSS are not used to their full potential, often resulting in alert fatigue which has been associated with patient harm. Clinicians in the ICU may be more vulnerable to desensitization of alerts than clinicians in less urgent parts of the hospital. We evaluated facilitators and barriers to appropriate CDSS interaction and provide methods to improve currently available CDSS in the ICU.

DESIGN:

Sequential explanatory mixed-methods study design, using the BEhavior and Acceptance fRamework.

SETTING:

International survey study.

PATIENT/SUBJECTS:

Clinicians (pharmacists, physicians) identified via survey, with recent experience with clinical decision support.

INTERVENTIONS:

An initial survey was developed to evaluate clinician perspectives on their interactions with CDSS. A subsequent in-depth interview was developed to further evaluate clinician (pharmacist, physician) beliefs and behaviors about CDSS. These interviews were then qualitatively analyzed to determine themes of facilitators and barriers with CDSS interactions.

MEASUREMENTS AND MAIN RESULTS:

A total of 48 respondents completed the initial survey (estimated response rate 15.5%). The majority believed that responding to CDSS alerts was part of their job (75%) but felt they experienced alert fatigue (56.5%). In the qualitative analysis, a total of five facilitators (patient safety, ease of response, specificity, prioritization, and feedback) and four barriers (excess quantity, work environment, difficulty in response, and irrelevance) were identified from the in-depth interviews.

CONCLUSIONS:

In this mixed-methods survey, we identified areas that institutions should focus on to improve appropriate clinician interactions with CDSS, specific to the ICU. Tailoring of CDSS to the ICU may lead to improvement in CDSS and subsequent improved patient safety outcomes.

KEY POINTS

Question: The objective of this study was identify clinician (pharmacist, physician) perspectives of clinical decision support systems (CDSSs) in the ICU, specifically facilitators and barriers to interactions with CDSS.

Findings: In this sequential explanatory mixed-methods study, we identified that the majority of respondents to an initial survey experienced alert fatigue (56.5%). In an analysis of in-depth interviews, we identified five facilitators and four barriers for clinician interactions with CDSS.

Meaning: We provide recommendations for institutions to focus on to improve clinician CDSS interactions, including tailoring specifically to the ICU environment and a focus on clinical relevance.

Clinical decision support systems (CDSSs) are meant to improve clinician decision-making for patient care and patient safety. In ICUs, CDSS have wide utility due to high-volume data with continuous patient care monitoring, and high levels of exposure to multiple medications, which differs from the general ward (1). Some examples include early warning tools for patients at risk of developing sepsis and acute kidney injury, use for administrative functions such as compliance with regulations, and warnings for medication safety practices (2–4). However, the liberal use of poorly designed CDSS has resulted in excessive alerts delivered to clinicians (5). Clinically irrelevant alerts result in clinician desensitization, resulting in overlooking useful signals and critical warnings, thus compromising patient safety, known as alert fatigue.

Alert fatigue is a common problem for clinicians with an estimated 7,000 passive alerts delivered daily to an individual critical care practitioner (6). This number of alerts is substantially higher in the ICU than non-ICU areas because of the frequency of data generated for monitoring severely ill patients (7). It is especially concerning since it results in inappropriate overrides of CDSS, increasing the risk of adverse events, which makes alert fatigue both an issue with quantity and quality of the alert (8–10). Definitive evidence-based strategies are lacking for improvement in CDSS in the ICU (5, 11).

The intent of CDSS is to optimize care and prevent patient harm but due to alert fatigue, this has resulted in the opposite effect—bypassing potential safety checks. Limited data exist evaluating facilitators and barriers to interactions with CDSS alerts and are not ICU-specific (12–14). To optimize the potential benefits of alerts, we need to both improve alerts themselves and change clinician behavior when interacting with alerts. The primary objective of this study was to conduct a detailed assessment of perceived barriers and facilitators to CDSS alert response by pharmacists and physicians in the ICU. This study aims to understand how alert systems can be improved so clinician response to alerts can be enhanced.

METHODS

We used a sequential explanatory mixed-methods approach to determine facilitators and barriers to clinician interactions with CDSS. Participation in this study was voluntary and no compensation was provided. A flow diagram of the study is found in Figure 1. This study was approved by the University of Pittsburgh Institutional Review Board and is reported using the Consolidated Criteria for Reporting Qualitative Research 32-item checklist (Supplementary Digital Content Appendix A, http://links.lww.com/CCX/B245) (15).

Figure 1.

Overview of study design. CPP = Clinical Pharmacology and Pharmacy, SCCM = Society of Critical Care Medicine, TDF = Theoretical Domains Framework.

Survey Solicitating Interest in Participation

The survey was designed to obtain a respondent’s general understanding and thoughts on alert fatigue. A call for participation to members of the Society of Critical Care Medicine (SCCM) Clinical Pharmacology and Pharmacy (CPP) section was sent in December 2017. A total of 259 responses were obtained, with 33 respondents expressing interest in participating in a subsequent, more in-depth survey.

Facilitators and Barriers Survey

This survey and in-depth interview questions were developed using the BEhavior and Acceptance fRamework (BEAR) to evaluate the perceived utility implementation of CDSS, uniquely incorporating both the Theoretical Domains Framework and the Technology Acceptance Model (16). In brief, this framework was developed using 10 published articles evaluating determinants of behavioral change and acceptance of CDSS, which were refined into a total of 22 domains. Additional questions were added to gauge the number and types of alerts with which the respondents interact. The survey face validity was evaluated by research team investigators not conducting the interviews and five clinicians not participating in the interviews. The final questionnaire was determined by the study team to be thorough in addressing the purpose of the study and the BEAR framework.

The cross-sectional survey (Supplementary Digital Content Appendix B, http://links.lww.com/CCX/B245) was sent by email to pharmacist members of the SCCM CPP originally identified from the interest survey or from expertise with clinical decision support (n = 103). Pharmacists were specifically selected for participation because they interact with all types of CDSS. These pharmacists were intentionally chosen to represent a nationwide sample. A request was made that requested the pharmacist to send the survey to at least one advanced practice provider (APP) (if available at institution) and one physician, both with experience in interacting with CDSS alerts on a regular basis. This request was made to identify attitudes for stakeholders who enter the majority of laboratory and medication orders and to obtain a different point of view within the same institution. Nurses were not specifically contacted for this request, as they often encounter different types of alerts not associated with initial ordering of patient care and more frequently encounter auditory alarms (e.g., vital signs, infusion pumps). An additional question was included to identify respondents who would be interested in an in-depth interview (described below). The survey was made anonymous, using only a unique identifier to track responses. A survey response rate was unable to be determined since it was unknown how many providers a pharmacist may have sent the survey request to participate.

Post hoc analyses were completed to compare differences in responses by clinician type (pharmacist vs provider, defined as APP or physician). A Wilcoxon rank-sum test was used to compare continuous data, while a Fisher exact test was used to categorical data. Statistical analyses were performed using R (Version 4.2.1; Vienna, Austria), with a two-sided p value of less than 0.05 considered to indicate statistical significance.

In-Depth Interviews

Based on the responses from the facilitators and barriers survey, we had limited responses from APPs, which prevented us from including their perspective. We therefore were limited to pharmacists and physicians (referred to as clinicians throughout rest of article, unless otherwise specified) for the in-depth interview. This interview was designed as a sequential analysis to provide a detailed understanding of our findings from the cross-sectional survey. We were interested in a geographically and institution-diverse interview group. Due to the COVID-19 pandemic, in-depth interview requests were commonly declined, so we chose to recruit additional participants through SCCM member volunteers with a replacement approach to complete the predefined sample size of 10 pharmacists and 10 physicians, maintaining the intent of a diverse sample. This sample size was selected to have a 1:1 pharmacist to physician ratio and achieve 15 participants, which is a typical target for thematic saturation.

The BEAR framework was used to develop a semi-structured interview guide to identify barriers and facilitators for clinical decision support alert response by pharmacists and physicians in the ICU (Supplementary Digital Content Appendix C, http://links.lww.com/CCX/B245). Participating pharmacists and physicians were interviewed by trained study investigators (A.W., S.L.K.-G.), conducted from August 4, 2020, to November 6, 2020, via Zoom (Zoom Video Communication, San Jose, CA) from a private space in the researcher’s site of employment. Only the researchers and participants were present during these interviews and only audio was recorded. Field notes were not taken during each interview. Demographic information for each participant was collected after the interview but responses were de-identified and made anonymous through an independent party. In some cases, the participant was known to the interviewer. All participants were interviewed once. No repeat interviews were conducted. Questions were asked in the same order to all participants.

Each interview was transcribed by a transcription service (GMR Transcriptions, Tustin, CA). Transcriptions were reviewed for fidelity of meaning by two investigators (L.S., Y.H.S.), after being trained by another investigator (L.A.B.), with experience conducting qualitative research. After developing an initial codebook, transcriptions were independently coded by two investigators (L.S., Y.H.S.). Revisions to the initial codebook were made throughout coding as discrepancies were resolved. The remaining interviews were coded using the final codebook. Interviews were coded using NVivo 11 software (QSR International, Melbourne, VIC, Australia) to store codes and organize the transcripts. A thematic analysis was performed to generate a list of informative barriers and facilitators. Participants did not receive a transcription of their interview and were not provided feedback on their findings. Quotes were selected to illustrate each identified theme.

RESULTS

Facilitators and Barriers Survey

The estimated response rate for the survey was 56 of 309 (18.1%), while 48 of 309 completed the survey (15.5%). Data for age and sex of respondents were not available due to the anonymous nature of the survey (Table 1). Within this section of the results, a provider is considered as an APP or physician. For more detailed results, please refer to Supplementary Digital Content Appendix D (http://links.lww.com/CCX/B245).

TABLE 1.

Cross-Sectional Survey Respondent Demographics

n of respondents (%)
Advanced practice provider	2 (4.2)
Pharmacist	23 (47.9)
Physician	23 (47.9)
Type of institution
Academic medical center	35 (73.0)
Community, teaching	11 (22.9)
Community, nonteaching	2 (4.1)
Institution size
< 250 beds	3 (6.2)
250–499 beds	13 (27.1)
500–750 beds	14 (29.2)
> 750 beds	18 (37.5)
Type of ICU
Medical	14 (29.1)
Surgical	8 (16.7)
Mixed	7 (14.6)
Cardiac/cardiac surgery	5 (10.5)
Other	14 (29.1)

CDSS Alert Characteristics

The most common alerts that respondents interacted with were medication (72.9% chose as most common), followed by laboratory monitoring. Pharmacist respondents were more likely to most interact with medication alerts than providers (p = 0.009). Respondents indicated that they pay attention to most alerts that they receive (50%), with no difference by clinician type in those who pay attention to greater than 50% of alerts delivered to them (p = 0.08). Respondents spent the most time assessing and evaluating alerts (Table 2).

TABLE 2.

Respondent’s Estimated Daily Time Investment With Clinical Decision Support System Alerts

Alert Characteristic	Overall	Pharmacist	Provider (Advanced Practice Provider, Physician)	p
Median time to read alerts, min (IQR)	6.3 (2.0–16.3)	3 (1–7.5)	15 (6.5–22.5)	0.002
Median time to assess and evaluate alerts, min (IQR)	9.0 (3.0–20.0)	3 (2–5)	15 (10–37.5)	< 0.001
Median time to respond to alerts, min (IQR)	5.0 (3.0–15.0)	5 (3–6)	10 (5–22.5)	0.008

IQR = interquartile range.

CDSS System Performance

Competing tasks or time constraints influenced responding to CDSS alerts, with “10–49% of the time” being the most common response (35.4%), followed by “51–90% of the time or most of the time” (31.3%). There were no differences in response rate for these results by clinician type (p = 0.131 and p = 0.542, respectively).

Institutional Strategies to Address CDSS Alerts

To the respondents’ knowledge, the majority were not aware of any alert fatigue reduction strategies at their current institution (66.7%), with no difference by clinician type (p = 0.187). Of those that were aware of interventions (25%), the most frequent example focused on removing alerts that were frequently overridden, involving interdisciplinary involvement. None of these interventions was identified as ICU-specific. The metric most identified to measure impact was alert override appropriateness.

Benefits of CDSS Systems

Respondents felt competent in their ability to respond to alerts (93.7%), and believed it is part of their job (75.0%; p = 1). Respondents were overall confident (confident or very confident) in their ability to overcome difficulties with alert response (58.4%; p = 0.394), although they may not necessarily be aware of published data supporting the utility in clinical decision-making (47.9%). Half of respondents did not have concerns (never) with legal issues in interacting with CDSS alerts (50.0%), with pharmacists more often to have some concern with legal issues than providers (69.6 vs 32.0%; p = 0.02).

Improving CDSS Systems

The most reported methods to improve how alerts are presented to the user included better visual design (41.7%), improving clinical relevance (35.4%), and ease of response (12.5%). Alert fatigue mitigation strategies included development of a “vetting” process for alerts or the development of an interdisciplinary committee to evaluate alert response (16.7%), improving clinical relevance of alert (8.3%), and improving design of alerts (e.g., tiering) (4.2%). The most identified facilitators to alert response were clinical relevance (54.2%), decreasing number of alerts (8.3%), and ease of response (6.3%). The most identified barriers to responding to alerts were time (37.5%), followed by clinical relevance (12.5%), and number of alerts (10.4%).

In-Depth Interviews

A total of 20 individuals participated in the in-depth interviews (10 pharmacists, 10 physicians). Participants were predominately female (55%). Interviews lasted 33–69 minutes. Based on the NVivo analysis, a total of five facilitators and four barriers to interacting with CDSS alerts were identified (Supplementary Digital Content Appendix E, http://links.lww.com/CCX/B248). The following sections denote the themes, along with supporting quantitative data from the facilitators and barriers survey.

Facilitator: Patient Safety

The theme of patient safety as the purpose of CDSS was prevalent, with respondents indicating they serve as a second check to prevent mistakes from reaching the patient. Respondents indicated they believe that the purpose of CDSS alert systems is for patient safety (41.7%) despite being unaware of published data.

Facilitator: Prioritization

The third most identified method to mitigate alert fatigue was improving design of alerts, such as tiering. Given the diverse group of clinicians that were interviewed, it was expected that there would be variation in how CDSS alerts are presented to them, regarding aspects such as visual design and how higher “importance” alerts.

Facilitator: Optimization Using Feedback

Incorporation of front-line clinicians for prevalent CDSS at their institutions was valued, which focused on a closed feedback loop. This was suggested as a method to help improve the clinical relevance of alerts. Respondents were often unclear whether their institution measured the impact of these implementation strategies (64.6%), as well as being unclear if their institution is committed to reducing alert fatigue (62.5%). A “vetting” process was identified as the most common method suggested at decreasing the effects of alert fatigue.

Facilitator and Barrier: Easy to Understand/Respond; Difficulty to Respond/Multiple Steps

Ease in response of CDSS was identified as both a facilitator when it was easy, as well as a barrier when it was not as easy to respond to alerts (e.g., multiple clicks). Reducing the number of steps needed to interact with CDSS alerts appeared to be favored by participants. Respondents indicated that they find that responding to CDSS alerts to be moderately easy (39.6%) this was consistent with the estimated time in Table 2 being the least.

Facilitator and Barrier: Specific and Actionable; Nonspecific/Irrelevant

Another theme identified as both a facilitator and barrier were the clinical relevance of CDSS, specifically to the ICU patient. In settings where an alert was not clinically relevant, this was an obstruction to workflow. Regarding how CDSS alerts should perform; the majority believed that they are moderately useful (56.3%), with CDSS alerts performing the way that they expected only 10–49% or some of the time (45.8%). Most respondents believe that at least 60% should predict a potential actionable event (54.2%). However, the majority find that CDSS alerts only help improve their decision-making process 10–49% of the time (52.1%) and do not believe that CDSS alerts help allow them to perform their duties quicker (58.3%). Finally, they believed that CDSS alerts can be made to be more useful for clinical decision-making (87.5%), such as presenting their clinical relevance to a patient.

Barrier: Excess Quantity

The number of alerts clinicians interact with was a barrier to appropriate CDSS interactions, which contributed to certain respondent’s alert fatigue. The median number of alerts that respondents estimated that they interact with daily was 17.5 (interquartile range, 5–35.0). The majority believed that this amount was too much (60.4%), with 56.5% believing that they had alert fatigue, with the most identified cause of alert fatigue being lack of clinical relevance of alerts (31.3%).

Barrier: Work Environment Stressors

A variety of variables associated with the work environment were identified as contributing to interruptions in workflow, which included medical emergencies, trainees, and time. Time was identified as the most common barrier to interacting with CDSS (37.5%).

DISCUSSION

In our mixed-methods survey, we identified five facilitators (patient safety, ease of response, specificity, prioritization, and feedback) and four barriers (excess quantity, work environment, difficulty in response, and irrelevance) to clinician interactions with CDSS in the ICU. We focused specifically on the ICU for multiple reasons, such as its unique physical environment, patients who often have altered responses to medications due to organ failure, and the number of high-risk medications they may be prescribed (17). This qualitative analysis was often supported by the quantitative data provided within the facilitators and barriers survey. Despite efforts that institutions have made at improving CDSS, there appears to be much room for improvement, based on the analyses that we performed in this study.

Strategies that have reduced alert fatigue include tiering of alerts for importance/severity and tailoring of CDSS alerts to specific clinicians (18). Tiering drug-drug interaction alerts is also beneficial at improving clinical response, along with removal of certain types of medication-related CDSS, such as dose frequency alerts as well as alerts that are generated from order sets (19, 20). We found that most clinicians are not aware of published literature, which could providing clinicians a better understanding of this literature may help with CDSS acceptance, while more clinical information within the presented alert, could be used for specific CDSS acceptance. While we have approaches to reduce alert fatigue, this evaluation provides additional methods suitable for study.

The theme of work environment stressors appears to be new compared with published literature, which may be due to our focus on the ICU. Limited data support that CDSS acceptance in the form of best practice advisories are lower with increasing patient complexity, as well as with increasing number of CDSS within the ambulatory care setting (21). The ICU is a stressful environment for clinicians due to the physical workplace itself or the stress that is related to clinical care, which has likely worsened with the COVID-19 pandemic (22–25). This, in combination with burnout, which has been prominent within the critical care literature, likely affects clinician response to CDSS, including the responses that we received in our survey (26–28).

Based on the findings of our study, it is apparent that despite growing evidence that CDSS are beneficial, they are also not currently being used effectively. Alternatively, clinicians in general, may not view CDSS as beneficial due to their experiences with ineffective CDSS. One organization that has proposed solutions for CDSS interactions includes incorporation of front-line clinicians in alert governance, as well as the proposal of seven rules for alerts, known as CREATOR (29). Of these seven rules, five match our identified facilitator and barriers (Patient safety: Consistent with organizational strategy and principles, Referenced; Specific and actionable: Relevant and timely, Transparent; Easy to understand, respond, and Optimization using feedback: Overridable). The most acutely actionable interventions to improve CDSS acceptance are focused on ensuring that alerts are more patient-specific and optimized using front-line clinician feedback. It is unlikely that addressing a singular facilitator or barrier will be effective alone, based on data evaluating acute kidney injury CDSS (30).

Obtaining feedback from clinicians is critical to ensure buy-in with CDSS. One study that evaluated methods on reducing CDSS alert burden developed a dashboard visualization tool to evaluate the impact of their interventions on reducing alert burden and the development of a feedback mechanism to identify unrecognized build issues (31). Evaluating a CDSS alert override with a provided rationale may be another method to evaluate clinician feedback, with 66% of these overrides identified to be appropriate, indicating potentially lack of clinical relevance (32). An interdisciplinary monitoring group that regularly evaluates alerts and provides feedback to front-line clinicians is another solution.

Limitations of our study include the lack of validation of our survey and interview as well as our sample size. The survey and interview instruments that we developed were not formally validated but did use the validated BEAR framework. However, we made attempts to enhance face validity and consistency of how interviews were performed via training. Regarding our sample size, there was a low response rate and we do not know the true response rate. This was likely due to overlap of the survey with the COVID-19 pandemic. We believe we did obtain a diverse range of comments on CDSS and alert fatigue, although our data was largely from clinicians from academic medical center, which may limit generalizability. The participants we reached out to were intentionally chosen due to their interest in this topic, as well as different institutions they represented. This likely improved the generalizability of our findings from our initially intended design but may have also introduced some potential bias, given potential familiarity of the interviewers with the participants. However, since there was a standardized survey that was used, with no prompting, this bias was likely limited. Finally, we were only able to include pharmacist and physician perspectives, which limits generalizability of our findings to other clinicians (e.g., APPs, nurses, physicians in-training), as well as CDS alerts outside of medication orders. The statistical comparison by clinician should be interpreted cautiously given the limited sample size included in the analysis. Also, clinician age and gender may influence response, but these variables were not considered in the analysis given the need for anonymization of the respondents. Referring to the literature, there are limited data evaluating clinician characteristics and CDS response, especially within the ICU. Of the available data in areas outside of the ICU, there are conflicting data on characteristics such as age and gender influencing CDS response (33–36).

In conclusion, there is great room for improvement in the design of and how ICU clinicians interact with CDSS. Using a sequential, explanatory mixed-methods approach, we identified a total of five ICU-specific facilitators and four barriers for medication-related CDS alerts, allowing for a more extensive investigation through the use of front-line clinician insight using both quantitative and qualitative data. We expect this study can be used to help design strategies that will result in more comprehensive improvement for appropriate use of CDSS within the ICU, such as the use of a Delphi method to further refine our findings.