Abstract
Background:
Automated machine learning (ML)-based large vessel occlusion (LVO) detection algorithms have been shown to improve in-hospital workflow metrics including door-to-groin time (DTG). The degree to which care team engagement and interaction are required for these benefits remains incompletely characterized.
Methods:
This analysis was conducted as a pre-planned post-hoc analysis of a randomized prospective clinical trial (NCT05838456). ML-based LVO detection software was implemented at 4 comprehensive stroke centers between 1/1/2021–2/272022. Patients were included if they underwent EVT for LVO AIS. ML-software utilization was quantified as the total number of active users and the ratio of the number of comments to the number of patients analyzed by the software by site per week. Primary outcome was the reduction in DTG relative to pre-ML implementation by hospital utilization level. Data are expressed as median[IQR].
Results:
Among 101 patients who met the inclusion criteria, median age was 71 years (IQR 59–79), with 48.5% being female. CSC 4 had the greatest number of total active users per week (32.5[27.5–34.5]), and comment-to-patient ratio per week (5.8[4.6–6.9]). Increased ML software utilization was associated with improvements in DTG reduction. For every one-unit increase in the comment-to-patient ratio, DTG time decreased by 2.6(95% CI[−5.09, −0.13]) minutes, while accounting for site-level random effects. Number of users-to-patient was not associated with a reduction in DTG time (β = −0.22, 95% CI[−1.78, 1.33]).
Conclusions:
In this post-hoc analysis, user engagement with software, rather than total number of users, was associated with site-specific improvements in DTG time.
Introduction
Automated detection of large vessel occlusion (LVO) through machine learning (ML)-based algorithms have demonstrated remarkable precision in identifying LVO from CT angiogram acquisitions.[1–3] This advancement significantly streamlines patient care and has been shown to reduce treatment times and possibly improve patient outcomes.[4,5] As a result, multiple algorithms have achieved FDA clearance and are in increasingly wide usage.[5,6] However, the dependence of the software on user engagement, and the extent to which engagement is required to see the benefits of the algorithms, remains incompletely characterized.
We recently demonstrated in a prospective randomized stepped-wedge clinical trial that the implementation of automated LVO detection software for acute ischemic strokes (AIS) software improved in-hospital endovascular thrombectomy (EVT) workflows.[7] Here, we assess the degree to which care team adoption and engagement influenced the effect of the software on clinical outcomes.
Methods
We conducted a pre-planned post hoc analysis from a recently published cluster randomized prospective stepped-wedge clinical trial.[7] This study was performed across four CSCs in the Greater Houston area from January 1, 2021, to February 27, 2022. Full details on the study design, including the CONSORT diagram detailing trial enrollment, allocation, and follow-up, can be found in the previous publication.[7] Briefly, all patients treated with EVT during the trial period were included unless excluded for the reasons listed below in the Participants section. There was no blinding to allocation, as clinical care teams began to receive AI-enabled alerts once each cluster was activated and were aware of the activation time frames. Four steps were used, corresponding to the activation of one CSC/cluster at a time to the AI software. The duration of time between each step was determined a priori based on the volume of EVT procedures, as described in the Sample size calculation section below.
Participants
Our trial population included patients with LVO AIS who presented to the Emergency Rooms of the four CSCs included in the trial and underwent imaging with CTA. In all four CSCs, imaging with non-contrast head CT followed immediately by CTA was performed in all patients presenting as “code stroke” evaluations throughout the study period as standard of care. Patients were included in the trial if they underwent emergent EVT for treatment of LVO AIS with occlusions of the internal carotid, middle cerebral, anterior cerebral, basilar, intracranial vertebral or posterior cerebral arteries. Patients were excluded if they presented as in-hospital “code stroke” alerts, or if they were transferred from non-EVT performing centers to a CSC for EVT evaluation. These patients were excluded as the workflow for these patients is very different from the Emergency Room workflow, which was the primary focus of the intervention. In the case of patients transferred from other hospitals, the decision for EVT is usually made prior to transfer, and the patients are brought directly to the angiography suite without repeating imaging. In addition, during the trial period, two of the CSCs participated in EVT trials of large core patients. Patients treated with EVT through randomization in one of these trials were also excluded, as these workflows also differed, due to the additional time required in consenting, enrolling and randomizing prior to EVT.
Interventions
At the start of the trial period, non-contrast CT and CTA acquisition protocols were modified such that all imaging performed in the workup of patients presenting to the Emergency Rooms for possible AIS were automatically transmitted to a cloud-based AI algorithm trained to detect LVO AIS from CTA (Viz.AI). This software package analyzes CTA images and arrives at a decision on the presence or absence of LVO within several minutes of receiving images. Total processing time after image acquisition including transmission and software analysis was less than 5 minutes on average. The results of the algorithm are transmitted to a mobile phone application, which the clinical care team was required to download onto their phones and arrived in the form of a pushed alert notification. Within the application, a mobile picture archiving and communication system (PACS) allowed users to verify imaging findings, and a secure messaging platform allowed for communication by the entire care team. While images were sent from each CSC to the software at the start of the study period, the alerts were not sent out until the pre-planned activation date for each site. The corresponding patient images on the mobile PACS and the individual patient-level secure messaging platforms were also not available until activation. This design allowed for optimization of the imaging transmission protocols from the CT scanner to the cloud-based AI servers prior to each site’s activation but prevented exposure to the intervention prior to site activation. Following each cluster’s activation, clinical care teams began receiving alerts and could view images and discuss patient care within the mobile applications.
Measurements
User engagement with the ML software was tracked through two metrics. The first metric consisted of the total number of actively engaged users (defined as the number of distinct users who accessed the mobile application and viewed patient data) each week. We also calculated the ratio of the number of comments to the number of patients analyzed by the software by site per week. This second metric was the one used in the primary outcome analysis, as it best accounted for software engagement and also better controlled for variability in patient volume across sites.
Outcomes
The primary outcome for the study was the effect of software utilization on door-to-groin time reduction. The analysis was performed using a mixed-effects linear model.
Statistical Analysis
Univariable comparisons between categorical variables were performed using chi-square and between continuous variables with the Wilcoxon rank-sum test. We also analyzed the differences in user engagement between the four CSCs using the Kruskal-Wallis test followed by Dunn’s post-hoc analysis. To explore the impact of center utilization following AI implementation on the time from patient hospital arrival to the initiation of endovascular treatment (measured by door-to-groin time), a mixed-effects linear model was used. The model examined weekly average door-to-groin time in relation to utilization measures, such as the ratio of comments to patients or the total number of weekly users. To account for the variability among the four CSCs, site was included as a random effect in the model. To evaluate the robustness of our primary and secondary outcomes, a sensitivity analysis was performed in which we excluded the CSC with the highest volume, which was also the only academic center included in the analysis. Because workflows at academic centers rely on trainees and an overall increased number of providers, we performed this analysis excluding this center to assess the robustness of our findings. The same modeling approach was applied, and results were compared to the primary and secondary analysis to ensure consistency. Analyses were performed using STATA v.17 (StataCorp LLC, College Station, TX) and Prism v.10.2.3 (GraphPad Software, San Diego, CA, USA).
Results
Among the 101 patients meeting inclusion criteria, median age was 71 years (IQR 59–79), with 48.5% being female. As shown in Table 1, median NIHSS was 16 (IQR 11–21), median ASPECTS was 10 (IQR 8–10), and the rate of IV tPA was 47.5% (Table 1).
Table 1.
Pre-/Post-AI Demographic
| Total (N=241) | Pre-AI (N=140) | Post-AI (N=101) | p-value | ||
|---|---|---|---|---|---|
| Age | 70 (59–79) | 70 (58–78) | 71 (59–79) | 0.67 | |
| Female | 122 (50.6%) | 73 (52.1%) | 49 (48.5%) | 0.58 | |
| Ethnicity/Race | Non-Hispanic White | 107 (44.4%) | 58 (41.4%) | 49 (48.5%) | 0.75 |
| Non-Hispanic Black | 68 (28.2%) | 42 (30.0%) | 26 (25.7%) | ||
| Hispanic | 41 (17.0%) | 25 (17.9%) | 16 (15.8%) | ||
| Non-Hispanic Other | 25 (10.4%) | 15 (10.7%) | 10 (9.9%) | ||
| CTA Occlusion Location | ICA | 41 (17%) | 29 (21%) | 12 (12%) | 0.098 |
| M1 | 110 (46%) | 56 (40%) | 54 (53%) | ||
| M2 | 51 (21%) | 29 (21%) | 22 (22%) | ||
| Basilar | 11 (5%) | 9 (6%) | 2 (2%) | ||
| Cervical and Intracranial Tandem | 9 (4%) | 4 (3%) | 5 (5%) | ||
| PCA | 5 (2%) | 2 (1%) | 3 (3%) | ||
| Vertebral | 3 (1%) | 3 (2%) | 0 (0%) | ||
| Other | 11 (5%) | 8 (6%) | 3 (3%) | ||
| Last Known Well to CSC Arrival (minutes) | 132 (61–505) | 131.5 (61–472) | 147 (72–569) | 0.53 | |
| NIHSS | 17 (11–22) | 17 (11–23) | 16 (11–21) | 0.14 | |
| Pre-mRS | 0 | 163 (67.6%) | 97 (69.3%) | 66 (65.3%) | 0.19 |
| 1 | 26 (10.8%) | 16 (11.4%) | 10 (9.9%) | ||
| 2 | 18 (7.5%) | 13 (9.3%) | 5 (5.0%) | ||
| 3 | 22 (9.1%) | 8 (5.7%) | 14 (13.9%) | ||
| 4 | 1 (0.4%) | 0 (0.0%) | 1 (1.0%) | ||
| ASPECTS | 9 (7–10) | 9 (7–10) | 10 (8–10) | 0.040 | |
| IV tPA | 111 (46.1%) | 63 (45.0%) | 48 (47.5%) | 0.70 | |
| Health History | |||||
| Stroke | 42 (17.4%) | 24 (17.1%) | 18 (17.8%) | 0.89 | |
| Transient Ischemic Attack | 16 (6.6%) | 11 (7.9%) | 5 (5.0%) | 0.37 | |
| Hypertension | 180 (74.7%) | 107 (76.4%) | 73 (72.3%) | 0.46 | |
| Hyperlipidemia | 87 (36.1%) | 55 (39.3%) | 32 (31.7%) | 0.23 | |
| Atrial Fibrillation | 70 (29.0%) | 41 (29.3%) | 29 (28.7%) | 0.92 | |
| Diabetes | 69 (28.6%) | 46 (32.9%) | 23 (22.8%) | 0.087 | |
| Smoking | 51 (21.2%) | 28 (20.0%) | 23 (22.8%) | 0.60 | |
| Congestive Heart Failure | 26 (10.8%) | 15 (10.7%) | 11 (10.9%) | 0.97 | |
| Door to Groin (minutes) | 97 (76–113) | 100 (81.5–115.5) | 89 (66–110) | 0.005 |
Data are presented as median (IQR interquartile range) for continuous measures, and n (%) for categorical measures. Pre-AI patients were not included in the analysis.
During the study period, a total of 832 CTA scans were analyzed in CSC 1, with 142 scans flagged as possible LVOs. In CSC 2, 274 CTA scans were analyzed, with 31 flagged as possible LVOs. CSC 3 had 288 CTA scans analyzed, with 47 flagged as possible LVOs. Lastly, CSC 4 analyzed 998 CTA scans, with 106 scans flagged as possible LVO. The median door to groin time across all four CSCs post-AI implementation was 89 minutes (IQR 66–110). The median number of distinct users logging into the software per week varied across the comprehensive stroke centers (CSCs). CSC 1 had a median of 18.5 users per week (IQR 16–22, max 29), CSC 2 had 13 users per week (IQR 11–14.5, max 23), CSC 3 had 15.5 users per week (IQR 11.75–19.5, max 26), and CSC 4 had 32 users per week (IQR 26.75–34, max 37). Median utilization (number of comments per week divided by number of patients per week) across the four centers were 3.31 [IQR 2.4–6.7] (CSC1), 2.23 [IQR 1.7–3.8] (CSC2), 2.5 [IQR 1–3.7] (CSC3) and 5.78 [IQR 4.6–6.9] (CSC4) (Figure 1a). The total number of active users per week also varied. CSC 4 had the highest median of 32.5 [IQR 27.5–34.5], whereas CSC 2 had the lowest median 13[IQR 11–15] as shown in Figure 1b.
Figure 1a and 1b.
(a) Median with interquartile range (IQR) bar plot of ratio of comments to patients per weeks by comprehensive stroke center (CSC). (b) Median with IQR bar plot of number of active users to patients per weeks by CSC. Significance levels were represented as follows: *P<0.05, **P<0.01, ***P<0.001, ****P<0.0001. Statistical analysis was performed using Kruskal-Wallis tests followed by Dunn’s test for post-hoc analysis.
In the mixed-effects linear model, a negative trend was observed between the ratio of comments to patients and door-to-groin (DTG) time. Specifically, for every unit increase in the ratio of comments to patients, DTG time decreased by approximately 2.6 (95% CI[−5.09, −0.13]; R2 = 0.0697) minutes, while accounting for campus-level random effects (Figure 2). However no significant association was observed between ratio of user to patients and DTG time in the mixed-effect linear model (β = −0.22, 95% CI[−1.78, 1.33]; R2 = 0.0014) (Figure 3).
Figure 2. Door to groin (DTG) time vs. ratio of comments to patients per week.
The scatter plot shows the association between the ratio of comments to patients per week (x-axis) and DTG time in minutes (y-axis) using a mixed-effect linear model. Each point represents the observation from each comprehensive stroke center (CSC), color-coded as follows: CSC 1 (red), CSC 2 (green), CSC 3 (blue), and CSC 4 (orange). The solid black line represents the overall trend, and the dashed lines indicate the 95% confidence interval of the mixed-effect linear model.
Figure 3. Door to groin (DTG) time vs. ratio of users to patients per week.
The scatter plot shows the relationship between the ratio of users to patients per week (x-axis) and DTG time in minutes (y-axis) using a mixed-effects linear model. Each point represents the observation from each comprehensive stroke center (CSC), color-coded as follows: CSC 1 (red), CSC 2 (green), CSC 3 (blue), and CSC 4 (orange). The solid black line represents the overall trend, and the dashed lines indicate the 95% confidence interval of the mixed-effects linear model.
In sensitivity analysis, in which we excluded patients from CSC 4, our results were largely unchanged. We observed that for every additional unit increase in the ratio of comments to patients, there was a 2.8 (95% CI [−5.35, −0.16]) minute decrease in DTG time. We similarly found no significant association between the ratio of users to patients and DTG time (β = −0.08, 95% CI [−1.58, 1.43]).
Discussion
In this study of four comprehensive stroke centers (CSC) in a diverse metropolitan area, we observed variability in software utilization and adoption. While ML software activation was associated with decreased door-to-groin times across the centers, its impact varied by user engagement. Increased team-level communicationg was associated with improved workflows, and we quantified a 2.6 minute reduction in DTG for each additional comment per patient. The total number of users logging into the software alone was not associated with improvements.
Previous studies have evaluated the effect of automated LVO detection on in-hospital quality metrics and clinical outcomes.[8–10] Our recent study showed that activation of AI-enabled LVO alerts resulted in an 11-minute reduction in door-to-groin (DTG) time and a reduction in mortality.[7] That reduction was measured on aggregate across the four CSCs studied in this analysis. Here, we find reductions in DTG across the sites individually as well. Other studies have also demonstrated that AI implementation leads to significant improvements in AIS in-hospital workflow.[11–13] Prior studies have shown a 39% reduction in DTG time for direct arriving patients and 32.5% for telemedicine transfer patients, and a notable reduction in hospital length of stay.[11] However, the extent to which user engagement modifies the effect of AI software efficacy has remained unclear.
With AI software providing benefits such as immediate/mobile access to advanced imaging, CT perfusion post-processing, and a HIPAA-compliant coordinated communications platform, these advantages may not be fully realized without active user engagement.[9] There are several additional reasons why user engagement may have this effect. The alerts generated by the software within mobile applications require timely action from clinicians; if clinicians are not actively engaged or do not have the alerts activated, adherence to these alerts diminishes. Prior studies have shown that the mechanism by which the software most effectively reduces DTG time is by converting serial processes to parallel ones.[14–16] With the AI software, you have simultaneous alerting of all members of the care team, and a single communication platform, making parallel processing much easier.[12] This workflow is more efficient than pre-software models, in which individuals would be alerted usually one at a time. However, to realize the benefit of this process improvement, most if not all members of the stroke care team must be engaged and active on the software. As we observed in our study, team-level communication was associated with improved DTG time outcomes, whereas the total number of users was not. Additionally, the continuous activation of software alerts can serve as a persistent reminder for providers about their stroke patients, ensuring sustained focus amidst other responsibilities. However, this continuous focus and improved patient outcomes are achievable only through consistent user engagement and adoption of the AI software.
Our study has several limitations. The analysis and data were based solely on a single healthcare system, which might affect the generalizability of our findings. Additionally, the variability in utilization that we observe and perform our analysis on may not be representative of the full distribution of possible degrees of engagement. Extrapolating these findings to centers with substantially greater or lesser degrees of engagement may not be valid. Finally, it is also possible that engagement may change over time. In our previous publication, however, we observed a consistent degree of utilization over the study period.[7]
Our study finds that door-to-groin times improved with AI implementation, but its effectiveness was dependent on the degree to which the users interacted with the software. These findings highlight the key role that user engagement plays in AI-enhanced process improvement.
Supplementary Material
What is already known on this topic:
Machine learning software for detecting LVO strokes improves workflows, such as door-to-groin time (DTG), but the role of user engagement in achieving these improvements is unclear.
What this study adds:
This study shows that user engagement, measured by comments per patient, significantly reduces DTG time, while the total number of users does not independently improve outcomes.
How this study might affect research, practice or policy:
These findings suggest that fostering active engagement with AI tools is essential for optimizing clinical workflows and outcomes, guiding future research and practice.
References
- 1.Czap AL, Bahr-Hosseini M, Singh N, et al. Machine Learning Automated Detection of Large Vessel Occlusion From Mobile Stroke Unit Computed Tomography Angiography. Stroke. 2022;53:1651–6. doi: 10.1161/STROKEAHA.121.036091 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Matsoukas S, Morey J, Lock G, et al. AI software detection of large vessel occlusion stroke on CT angiography: a real-world prospective diagnostic test accuracy study. J Neurointerventional Surg. 2023;15:52–6. doi: 10.1136/neurintsurg-2021-018391 [DOI] [PubMed] [Google Scholar]
- 3.Yahav-Dovrat A, Saban M, Merhav G, et al. Evaluation of Artificial Intelligence-Powered Identification of Large-Vessel Occlusions in a Comprehensive Stroke Center. AJNR Am J Neuroradiol. 2021;42:247–54. doi: 10.3174/ajnr.A6923 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Sheth SA, Lopez-Rivera V, Barman A, et al. Machine Learning-Enabled Automated Determination of Acute Ischemic Core From Computed Tomography Angiography. Stroke. 2019;50:3093–100. doi: 10.1161/STROKEAHA.119.026189 [DOI] [PubMed] [Google Scholar]
- 5.Murray NM, Unberath M, Hager GD, et al. Artificial intelligence to diagnose ischemic stroke and identify large vessel occlusions: a systematic review. J Neurointerventional Surg. 2020;12:156–64. doi: 10.1136/neurintsurg-2019-015135 [DOI] [PubMed] [Google Scholar]
- 6.Chandrabhatla AS, Kuo EA, Sokolowski JD, et al. Artificial Intelligence and Machine Learning in the Diagnosis and Management of Stroke: A Narrative Review of United States Food and Drug Administration-Approved Technologies. J Clin Med. 2023;12:3755. doi: 10.3390/jcm12113755 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Martinez-Gutierrez JC, Kim Y, Salazar-Marioni S, et al. Automated Large Vessel Occlusion Detection Software and Thrombectomy Treatment Times: A Cluster Randomized Clinical Trial. JAMA Neurol. 2023;80:1182–90. doi: 10.1001/jamaneurol.2023.3206 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Kim PE, Yang H, Kim D, et al. Automated Prediction of Proximal Middle Cerebral Artery Occlusions in Noncontrast Brain Computed Tomography. Stroke. 2024;55:1609–18. doi: 10.1161/STROKEAHA.123.045772 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Figurelle ME, Meyer DM, Perrinez ES, et al. Viz.ai Implementation of Stroke Augmented Intelligence and Communications Platform to Improve Indicators and Outcomes for a Comprehensive Stroke Center and Network. AJNR Am J Neuroradiol. 2023;44:47–53. doi: 10.3174/ajnr.A7716 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Matsoukas S, Stein LK, Fifi J. Artificial Intelligence-Assisted Software Significantly Decreases All Workflow Metrics for Large Vessel Occlusion Transfer Patients, within a Large Spoke and Hub System. Cerebrovasc Dis Extra. 2023;13:41–6. doi: 10.1159/000529077 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Olive-Gadea M, Crespo C, Granes C, et al. Deep Learning Based Software to Identify Large Vessel Occlusion on Noncontrast Computed Tomography. Stroke. 2020;51:3133–7. doi: 10.1161/STROKEAHA.120.030326 [DOI] [PubMed] [Google Scholar]
- 12.Hassan AE, Ringheanu VM, Preston L, et al. Artificial Intelligence–Parallel Stroke Workflow Tool Improves Reperfusion Rates and Door-In to Puncture Interval. Stroke Vasc Interv Neurol. 2022;2. doi: 10.1161/svin.121.000224 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Akay EMZ, Hilbert A, Carlisle BG, et al. Artificial Intelligence for Clinical Decision Support in Acute Ischemic Stroke: A Systematic Review. Stroke. 2023;54:1505–16. doi: 10.1161/STROKEAHA.122.041442 [DOI] [PubMed] [Google Scholar]
- 14.Cheung VJ, Wali AR, Santiago-Dieppa DR, et al. Improving Door to Groin Puncture Time for Mechanical Thrombectomy via Iterative Quality Protocol Interventions. Cureus. ;10:e2300. doi: 10.7759/cureus.2300 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Rai AT, Smith MS, Boo S, et al. The “pit-crew” model for improving door-to-needle times in endovascular stroke therapy: a Six-Sigma project. J Neurointerventional Surg. 2016;8:447–52. doi: 10.1136/neurintsurg-2015-012219 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Mehta BP, Leslie-Mazwi TM, Chandra RV, et al. Reducing door-to-puncture times for intra-arterial stroke therapy: a pilot quality improvement project. J Am Heart Assoc. 2014;3:e000963. doi: 10.1161/JAHA.114.000963 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.