Understanding Clinicians’ Usage Patterns of the CONCERN Early Warning System: Insights from a Multi-Site Pragmatic Cluster Randomized Controlled Trial

Abstract

The CONCERN Early Warning System (EWS) uses artificial intelligence (AI) to analyze nursing documentation patterns, predicting hospitalized patients’ risk of clinical deterioration. It generates real-time risk scores displayed on the electronic health record (EHR) interface for the inpatient care team, enhancing situational awareness and supporting timely interventions. A recent multi-site pragmatic cluster randomized controlled trial demonstrated a 35.6% reduction in inpatient mortality, an 11.2% decrease in length of stay, and improved outcomes for patients requiring unplanned ICU transfers. To gain better insights on the mechanisms driving these positive outcomes, we examined clinicians’ engagement with the CONCERN detailed display by clinical role, patient comorbidity level, site, shift, and unit type. This retrospective analysis of EHR log-file data examined 2,572 instances of CONCERN detailed display launches by 393 unique clinician users. Our findings showed distinct usage patterns influenced by clinician role, site, shift, unit type, and patient characteristics. Notably, registered nurses (59%) and ordering providers (37.7%) demonstrated balanced engagement. CONCERN detailed display launches were predominantly observed during day shifts (89.9%) and in acute care units (83.5%), aligning with workflows that prioritize interprofessional decision-making and timely care escalations. These findings offer valuable insights into how user interactions with this AI-driven clinical decision support tool may vary based on clinician, patient, and care setting characteristics, highlighting opportunities to refine implementation strategies and optimize its impact on clinical outcomes.

1. Introduction

The CONCERN Early Warning System (EWS) leverages machine learning to analyze real time changes in nursing documentation patterns and identify hospitalized patients at risk of clinical deterioration. By integrating into the Electronic Health Records (EHR), CONCERN provides a non-interruptive, real-time risk score to inpatient interprofessional care teams, enhancing situational awareness and facilitating timely interventions. A recent multi-site pragmatic cluster randomized controlled trial demonstrated a 35.6% reduction in inpatient mortality and an 11.2% decrease in length of stay (LOS).¹ Additionally, patients in intervention units experienced higher rates of unplanned intensive care unit (ICU) transfers but had lower mortality and shorter LOS post-transfer compared to patients transferred to ICU from usual care.² These findings underscore CONCERN EWS’s effectiveness on improving outcomes through facilitating optimal care escalations. To further understand these positive outcomes and guide future optimized implementations with sustained clinical impact, we sought to understand clinicians’ engagement with the CONCERN detailed display screen.

The CONCERN prediction model analyzes real-time changes in nursing documentation to detect signals of heightened nursing surveillance. Risk scores are categorized as green (low risk of deterioration), yellow (increased risk of deterioration), and red (showing signs of deterioration) and displayed non-interruptedly on the Epic EHR patient list using the FHIR standards. Clinicians can click on the score icon to open the detailed display screen, which includes a score explanation, 72-hour trend graph, and comparative scale showing the patient’s risk relative to others on the unit (Figure 1).

Figure 1.

CONCERN scores displayed on EHR patient list and CONERN detailed display screen.

The purpose of this study was to examine patterns of clinician engagement with the CONCERN detailed display, focusing on CONCERN detailed display launches by clinician role, care setting, and shift. It is critical to clarify that we focused on the CONCERN detailed display launch as a proxy for clinicians’ active engagement with the CONCERN intervention. CONCERN detailed display launches reflect a clinician’s choice to access the detailed score information but do not capture passive viewing of scores on the EHR patient list.

2. Methods

2.1. Study Design and Sample

This study was a retrospective analysis of EHR log-file data from four hospitals across two study sites (two large healthcare systems) that participated in the CONCERN clinical trial.¹ In this trial, 74 acute care units (ACUs) and ICUs clinical units across the two sites were allocated to receive the intervention or usual care. CONCERN early warning scores were generated only for the patients admitted to the intervention units. We extracted log-file data for all clinical trial encounters and included only intervention-group encounters with at least one CONCERN detailed display launch.

2.2. Definition of CONCERN Detailed display Launch

A CONCERN detailed display launch was defined as a clinician accessing the CONCERN FHIR SmartApp by clicking a patient’s CONCERN score on the EHR patient list screen to view detailed score information.

2.3. Analysis Steps

• Step 1. Descriptive Statistics of Detailed display Launch Counts per User: Unique users and detailed display launch frequency were summarized using descriptive statistics and stratified by site. Outliers with high launch counts were assessed.

• Step 2. Count of Unique Users by Clinical Role and Site: Clinician roles were extracted, and the distribution of users by role and site was analyzed.

• Step 3. Heavy vs. Light Users by Clinical Role Stratified by Site: Users were classified as heavy (top 20% contributing to 80% of launches) or light using the Pareto principle (i.e., 80/20 rule).³ Roles and site differences were compared.

• Step 4. Heavy vs. Light Users by Patients’ Comorbidity Levels: Differences in mean Charlson Comorbidity Index (CCI) scores for patients of heavy and light users were assessed overall and by site using t-tests and standardized mean differences.

• Step 5. Detailed display Launch Counts by Shift and Unit Type: Detailed display launches were analyzed by shift (day vs. night) and unit type (ACU vs. ICU), with results stratified by site to identify variations in usage patterns.

3. Results

Of the 60,893 encounters included in the trial, 33,024 were allocated to the intervention group. Among these, 1,213 encounters (Site A: 903; Site B: 320) had at least one clinician-initiated launch of the CONCERN detailed display screen. In total, 2,572 launches were recorded across 393 unique clinician users (Site A: 248; Site B: 145). Launch patterns across users, sites, roles, and shifts are summarized in Table 1.

Table 1.

Summary of Clinician Engagement Patterns with the CONCERN Detailed Display Screen

Category	Overall	Site A	Site B
Users (n)	393	248	145
Mean launches per user	6.5 ± 39.9	8.7 ± 50.2	2.7 ± 5.8
Median launches per user (IQR)	1 (1–3)	1.5 (1–4)	1 (1–2)
Maximum launches per user	741	741, 223	62
User roles (% of total)	RN: 59%; OP: 37.7%	RN: 46.4%; OP: 48.8%	RN:80.7%; OP: 18.6%
Heavy users (n = 83)	RN: 58.3%; OP: 36.9%	n = 63; RN: 49.2%; OP: 46.0%	n = 20; RN: 90%; OP: 10%
Patient CCI (heavy : light users)	4.3 : 4.2 (SMD = 0.02)	4.2 : 3.4 (SMD = 0.24, p = 0.07)	4.3 : 4.4 (SMD = 0.04)
Launches by shift (%)	Day: 89.9%; Night: 10.1%	Day: 96.9%; Night: 3.1%	Day: 50.3%; Night: 49.2%
Launches by unit type (%)	ACU: 83.5%; ICU: 16.1%	ACU: 83.8%; ICU: 16.2%	ACU: 81.9%; ICU: 15.6%

Note. RN: Registered Nurse; OP: Ordering Provider; CCI: Charlson Comorbidity Index; IQR: Interquartile Range; SMD: Standardized Mean Difference.

Heavy users defined as top 20% of users by total launches.

Launches refer to clinician-initiated views of the CONCERN detailed display screen accessed via the EHR patient list.

3.1. Descriptive Statistics of Detailed display Launch Counts per User

Overall, user engagement was highly variable, with a mean of 6.5 launches per user and a right-skewed distribution driven by outliers at Site A (maximums of 741 and 223). Excluding these reduced Site A’s mean to 4.9 (SD=12.1, range=1–90).

3.2. Count of Unique Users by Clinical Role and by Site

Among the 393 unique users, the majority were RNs, accounting for 59% of all users. Site B had a higher proportion of RNs (80.7%) compared to Site A (46.4%).

3.3. Heavy vs. Light Users by Clinical Role Stratified by Site

Overall, RNs comprised 58.3% of heavy users and OPs made up 36.9% of heavy users with notable differences by site.

3.4. Heavy vs. Light Users by Patients’ Comorbidity Levels

Overall, mean CCI scores were similar. At Site A, heavy users launched the detailed display patients with slightly higher CCIs than light users (4.2 vs. 3.4; t=1.80, p=0.07, SMD=0.24).

3.5. Detailed display Launch Counts by Shift and Unit type

Most launches occurred during the day shift (89.9%), with Site A exhibiting higher day usage (96.9%) than Site B (50.3%). ACUs accounted for 83.5% of launches.

4. Discussion

This study provides valuable insights into clinician engagement with the CONCERN detailed display screen during a multi-site pragmatic cluster randomized clinical trial. The variability in detailed display launch counts between sites may reflect differences in workflows and organizational practices. Site A’s exceptionally high launch counts by a few users suggest that certain clinicians take on a more active role in utilizing the detailed display. The distribution of heavy and light users across clinical roles demonstrates balanced interdisciplinary engagement. This lack of a heavy skew toward any single clinical role is particularly encouraging, as it suggests proportional engagement by both RNs and OPs. Such balanced usage aligns with the primary goal of the CONCERN EWS, which is to enhance shared situational awareness and foster collaboration among interdisciplinary care teams. At Site A, patients of heavy users had marginally higher CCIs compared to patients of light users. This finding raises a hypothesis that detailed display usage may increase when managing higher-risk patients, potentially to better monitor and manage these cases.

Detailed display usage patterns by shift and unit type align with clinical workflows, as most detailed display launches (89.9%) occurred during the day shift when interprofessional decision-making activities are more prevalent. Interestingly, site B had proportional distribution of detailed display launches between day (50.3%) and night shift (49.2%) whereas 96.9% of launches at site A were during day shift. This difference may suggest potential differences in workflows and team dynamics between the sites. The higher frequency of detailed display launches in ACUs (83.5%) compared to ICUs at both sites aligns with clinical trial findings of increased unplanned ICU transfers from intervention units. This pattern suggests that clinicians may use the detailed display more actively in ACUs to facilitate appropriate care escalation decisions.

In a recent systematic review, Kouri et al. reported a mean provider uptake rate of 34.2% across clinical decision support (CDS) tools.⁴ In comparison, CONCERN’s active engagement rate—3% of eligible encounters with a detailed display launch—was notably lower. It is important to note that detailed display launches only reflect instances where clinicians actively accessed the detailed information about the CONCERN score. The broader “usage” of the CONCERN EWS also includes passive score observation, which, though not actively measured, plays a significant role in informing clinical decisions and improving patient outcomes, as observed by the significant reduction in mortality and LOS in our clinical trial results. This limitation is inherent to log-based analyses, which can only capture active interactions and not passive engagement. To further clarify these dynamics, future research should differentiate between passive score viewing and detailed display engagement to better understand the mechanisms driving CONCERN’s impact. Building on this, following studies will explore patterns of observable clinician actions in response to risk score changes, assessing whether such actions affect patient care trajectories to improve clinical outcomes.

5. Conclusion

This study offers valuable insights into clinician engagement with the CONCERN EWS, highlighting how clinician and patient characteristics and care setting contexts influence usage patterns. Understanding these factors is crucial for tailoring implementation strategies and optimizing the system’s impact on patient care and outcomes.