Impact of Predictive Text Clinical Decision Support on Imaging Order Entry in the Emergency Department

Abstract

Purpose:

Imaging clinical decision support (CDS) is designed to assist providers in selecting appropriate imaging studies and is now federally required. The aim of this study was to understand the effect of CDS on decisions and workflows in the emergency department (ED).

Methods:

The authors’ institution’s order entry platform serves up structured indications for imaging orders. Imaging orders are scored by CDS on the basis of appropriate use criteria (AUC). CDS triggers alerts for imaging orders with low AUC scores. Because free text alone cannot be scored by CDS, an artificial intelligence predictive text (AIPT) module was implemented to guide the selection of structured indications when free-text indications are entered. A total of 17,355 imaging orders in the ED over 6 months were retrospectively analyzed.

Results:

CDS alerts for low AUC scores were triggered for 3% of all imaging study orders (522 of 17,355). Providers spent an average of 24 seconds interacting with alerts. In 18 of 522 imaging orders with alerts, alternative studies were ordered. After AIPT implementation, the percentage of unscored studies significantly decreased from 81% to 45% (P < .001).

Conclusions:

In a quaternary academic ED, CDS alerts triggered by low AUC scores caused minimal increase in time spent on imaging order entry but had a relatively marginal impact on imaging study selection. AIPT implementation increased the number of scored studies and could potentially enhance CDS effects. CDS implementation enables the collection of novel data regarding which imaging studies receive low AUC scores. Future work could include exploring alternative models of CDS implementation to maximize its impact.

Graphical Abstract

INTRODUCTION

Unnecessary ordering of advanced imaging studies can lead to suboptimal health care delivery, including poor resource utilization, costs, exposure to risks from ionizing radiation and contrast administration, impact of incidental findings, and impact on other patients competing for timely evaluation [1–3]. Imaging clinical decision support (CDS) mechanisms have been built to help risk-stratify patients to avoid unnecessary imaging and to help select the most appropriate advanced imaging studies for given clinical indications [1]. CDS implementation can change providers’ practice patterns [4] and reduce unnecessary advanced imaging orders [5]. Additionally, Chepelev et al [6] reported that increased provider exposure to CDS can lead to more appropriately focused imaging study orders. Moreover, it is suggested that emergency providers are interested in receiving real-time imaging decision support [7]. However, CDS can also create additional burdens in the form of additional clicks, pop-up screens, and data entry requirements that affect workflow and the timeliness of care and may lead to alert fatigue and failure of CDS to achieve the desired outcome [8–10].

The Protecting Access to Medicare Act requires outpatient and emergency department (ED) providers use certain CMS-approved CDS mechanisms when ordering advanced imaging studies [11]. These CDS mechanisms reference appropriate use criteria (AUC) developed by multiple professional societies, including the ACR, to score the appropriateness of advanced imaging studies for certain indications, with the aim of steering providers toward selecting the optimal imaging study for a given indication. Although the goal of the act is to encourage high-value care and improve patient safety, there have been concerns from major medical organizations regarding administrative burden and curbing access to care [12].

Because CDS mechanisms are now required nationwide, it is paramount to deeply understand the benefits and risks associated with these mechanisms. Our institution has implemented a commercially available, CMS-approved CDS mechanism, CareSelect Imaging (Change Healthcare National Decision Support), into our Epic electronic medical record (EMR) workflow. This CDS tool fires best-practice advisory (BPA) alerts as pop-up windows when providers order imaging studies that receive low appropriateness scores [13]. These alerts contain suggestions for more appropriate alternative studies to order.

A challenge to CDS implementation in the past has been that many providers type in free-text indications for examinations rather than using the structured fields provided. This is due, at least in part, to the presentation of enterprise-wide indication lists that are not tailored at the clinic level, which is part of the foundation builds of some EMRs. Free-text indications previously could not be scored by AUC. Fried et al [14] showed that most free-text indications can be mapped to existing structured indications that can be scored. To this end, artificial intelligence predictive text (AIPT) was developed for CDS mechanisms. AIPT allows providers to enter free text for the indication for an examination. AIPT attempts to automatically find the matching structured indication or provide a list of likely structured indications if a suitable match is not identified. AIPT initially received positive feedback from providers [15] and was recently implemented at our institution.

The implementation of CDS allows collection of data to help elucidate why imaging studies are ordered and how providers are interacting with these mechanisms. Although there has been some evidence showing the potential benefits of CDS on decreasing inappropriate imaging utilization at a large scale [4–6], it is unclear how CMS-approved CDS affects provider workflow and imaging order decisions each time an alert is launched.

The aims of our study were to determine how CDS affected imaging order decisions, to determine how CDS with AIPT affected provider workflow, to determine how the AIPT tool affected CDS scoring of imaging studies ordered, and to use CDS-derived data to understand which imaging studies were receiving low AUC scores.

METHODS

This study was approved by our institutional review board. Our institution uses the CareSelect Imaging CDS for advanced imaging study ordering. Providers place orders for advanced imaging studies in the EMR. Within the EMR, providers have the option to select from a common list of structured indications for an advanced imaging study or to input a free-text indication. If a structured indication is selected, the CDS then automatically scores the advanced imaging study and indication pair on the basis of an AUC score ranging from 1 (least appropriate) to 9 (most appropriate). These scores are also categorized into color ranges on the basis of the AUC: red for low scores of 1 to 3, denoting “usually not appropriate”; yellow for scores of 4 to 6, denoting “may be appropriate”; and green for scores of 7 to 9, denoting “usually appropriate.” For any pairs with red scores (1–3), the CDS launches an appropriateness score BPA (AS-BPA) window to provide a list of alternative imaging studies that are potentially more appropriate for the ordering provider to select from for a given indication. The provider could choose to either replace the existing order with one of the suggested studies or keep the existing advanced imaging order. If the provider chooses to keep the existing advanced imaging study with a low appropriateness score, they must select an acknowledgment reason before proceeding with the order. Providers also have the option of selecting an “emergency medical exception” option that allows them to bypass the CDS scoring system altogether.

AIPT was implemented at our institution on July 21, 2021. The AIPT model uses the free-text input, along with additional patient and provider data, to suggest a structured indication. If the correlation between the free-text and structured indication is strong enough, the model automatically converts the free-text indication into the structured indication. If there is not a sufficient correlation, the predictive text BPA (PT-BPA) window is launched. Using an artificial intelligence algorithm, the CDS provides a short list of structured indications in the BPA window that may be better matches to choose from. Once a structured indication is chosen, the CDS will default to the same workflow described earlier for structured indications. If the provider still is not able to find a matching structured indication, they can click on another option that allows them to submit the order with a free-text indication.

The EMR (Epic), radiology information system (Radiant), and CDS (CareSelect) database were queried for data associated with each advanced imaging study order. Data included patient demographic information, original imaging study ordered before BPA launch, final imaging study ordered after interaction with BPAs, imaging study indication, appropriateness score of imaging study and indication pair, length of time spent interacting with CDS, and comments associated with imaging order entry. These data were gathered for advanced imaging studies (CT, MRI, and nuclear medicine studies) ordered in the adult ED at our large urban quaternary academic medical center for a 6-month period from April 21 to October 21, 2021. This 6-month period was chosen to include an equal 3-month interval before (April 21 to July 20, 2021) and after (July 21 to October 21, 2021) the AIPT tool was implemented. This allowed comparison of appropriateness scores of imaging study orders before and after AIPT implementation.

Analysis was performed on the basis of each imaging order. For example, for a CT head and cervical spine without contrast, this counted as a single imaging order, although two separate studies (CT head without contrast and CT cervical spine without contrast) were performed.

Outcomes of Interest

Aim 1: To Determine How CDS Affected Imaging Order Decisions.

Our primary outcome of interest was to determine the percentage of imaging orders for which providers changed the imaging study ordered on the basis of interaction with the AS-BPA. To do so, all orders for which an AS-BPA was shown to the ordering provider were reviewed. These orders were further filtered to include only those for which the imaging study ordered before and after interaction with the AS-BPA were different. Finally, the AS-BPAs that launched for each order in this filtered list were reviewed to determine whether the imaging order change was suggested by the AS-BPA.

Secondary outcomes of interest included change in appropriateness score on the basis of AS-BPA interaction and categorization of how orders changed on the basis of AS-BPA interaction.

Aim 2: To Determine How CDS Affected Ordering Provider Workflow.

We also evaluated several outcomes related to how CDS-launched BPAs were affecting ordering provider workflows: frequency of AS-BPA launch, the amount of time providers spent interacting with AS-BPAs, and categorization of reasons of why providers did not accept AS-BPAs.

Aim 3: To Determine How the AIPT Tool Affected CDS Scoring of Imaging Studies Ordered.

We compared the appropriateness scores of all imaging studies ordered in the 3 months before and after AIPT implementation. We specifically compared the percentage of imaging studies that received no score before and after AIPT implementation using a one-tailed z test.

Aim 4: To Use CDS-Derived Data to Understand Which Imaging Studies Were Receiving Low AUC Scores.

We determined which imaging studies received the lowest appropriateness scores and further categorized the indications for which these studies received low scores.

RESULTS

A total of 17,355 imaging orders for 24,600 advanced imaging studies were included in our analysis. Demographic characteristics of included patients are included in Table 1.

Table 1.

Demographic information of included patients

	Before AIPT		After AIPT		Total
Characteristic	Frequency	%	Frequency	%	Frequency	%
Age (y)
16–39	2,457	28.2	2,423	28.0	4,880	28.1
40–59	2,572	29.6	2,673	30.9	5,245	30.2
60–79	2,805	32.2	2,747	31.7	5,552	32.0
≥80	868	10.0	810	9.4	1,678	9.7
Sex
Female	4,098	47.1	3,994	46.2	8,092	46.6
Male	4,604	52.9	4,658	53.8	9,262	53.4
Unknown	0	0.0	1	0.0	1	0.0
Modality
CT	8,110	93.2	8,104	93.7	16,214	93.4
MR	581	6.7	532	6.1	1,113	6.4
NM	11	0.1	17	0.2	28	0.2
Total	8,702	100.0	8,653	100.0	17,355	100.0

Note: Information for the cohort of imaging orders before AIPT implementation, after AIPT implementation, and for all imaging orders was included. AIPT = artificial intelligence predictive text; NM = nuclear medicine.

Overall, AS-BPAs were triggered for 3% of all imaging study orders (522 of 17,355) to suggest imaging studies with higher appropriateness scores. The ordering provider selected an alternative study on the basis of the AS-BPA suggestion for 3% of all AS-BPAs launched (18 of 522). This represents 0.1% of all imaging orders (18 of 17,355).

Of the 18 changed imaging orders, 15 changes were related to changes in contrast administration (eg, MRI cervical spine with and without contrast changed to MRI spine without contrast), and 3 changes were related to ordering additional imaging studies in trauma patients.

For the 522 AS-BPAs launched, the average additional time that the ordering provider spent interacting with the AS-BPA was 24 ± 36 seconds. The reasons providers chose when acknowledging AS-BPAs to describe why they potentially did not follow the AS-BPA suggestion are summarized in Table 2. The most common reasons were disagreement with appropriateness score (32.4%) and requested by a consultant (24.9%).

Table 2.

Reasons providers chose when acknowledging appropriateness score best practice advisory alerts

Acknowledgement Reason	Frequency	%
Disagree with appropriateness score	169	32.4
Requested by a consultant	130	24.9
Other (see comments)	96	18.4
X-rays performed	41	7.9
Provider did not provide comment	34	6.5
Consulted with radiology	27	5.2
Previous imaging result was equivocal or nondiagnostic	20	3.8
Modality unavailable	4	0.8
Patient does not tolerate modality (eg, claustrophobia, cannot complete prep routine)	1	0.2
Total	522	100

The percentages of imaging study orders that received different appropriateness score ranges are shown in Figure 1. After launch of the AIPT tool, the percentage of imaging study orders that did not receive an appropriateness score was 45% (3,904 of 8,653). This was significantly less than the percentage of imaging study orders that did not receive an appropriateness score before the launch of AIPT (81% [7,057 of 8,702]) (P < .001).

Fig. 1.

Appropriateness scores of imaging study orders before and after implementation of an artificial intelligence predictive text (AIPT) tool. Appropriateness scores are categorized into colors: red for scores of 1 to 3, denoting “usually not appropriate”; yellow for scores of 4 to 6, denoting “may be appropriate”; and green for scores of 7 to 9, denoting “usually appropriate. AI = artificial intelligence.

Of the 592 study orders that received red appropriateness scores of 1 to 3, the three most frequently ordered studies were CT head without contrast (156 red-scored orders), CT cervical spine without contrast (55 red-scored orders), and CT chest for pulmonary embolism (CT angiography) with contrast (39 red-scored orders). The indications for these examinations are listed in Table 3. The most common indications for CT head without contrast to receive a red score were nonspecific dizziness, syncope with normal neurologic findings, and minor head trauma with normal mental status in patients aged 19 to 64 years.

Table 3.

Indications for common red-scored imaging study orders

Imaging Study	Indication	Frequency	%
CT head without contrast	Dizziness, nonspecific	78	50.0
	Syncope, simple, normal neurologic findings	36	23.1
	Head trauma, minor, normal mental status (age 19–64 y)	30	19.2
	Headache, chronic, no new features	6	3.8
	Dizziness, dehydration or hypotension	4	2.6
	Cholesteatoma or acoustic neuroma	1	0.6
	Headache, classic migraine	1	0.6
	Total	156	100.0
CT cervical spine without contrast	Neck pain, initial examination	29	52.7
	Neck pain, recent trauma	16	29.1
	Neck pain, normal neurologic findings	9	16.4
	Neck pain, history of cancer	1	1.8
	Total	55	100.0
CT chest PE (CTA) with contrast	Chest pain or SOB, pleurisy or effusion suspected	22	56.4
	PE suspected, low pretest probability	10	25.6
	Cough	5	12.8
	Lung nodule, <6 mm, low cancer risk, follow-up examination	1	2.6
	Dyspnea on exertion	1	2.6
	Total	39	100.0
CT entire lower extremity with contrast	Lower leg swelling/redness, cellulitis suspected	6	35.3
	Upper leg swelling/redness, cellulitis suspected	4	23.5
	Soft tissue infection suspected, femur, initial examination	3	17.6
	Osteomyelitis suspected, diabetic	2	11.8
	Osteomyelitis, ankle, follow up	1	5.9
	Fracture, femur	1	5.9
	Total	17	100.0

Note: CTA = CT angiography; PE = pulmonary embolism; SOB = shortness of breath.

Of all CT lower extremity with contrast orders in our data set, 38% (17 of 45) received red appropriateness scores. This represented the highest rate of red-scored orders for all studies when excluding studies with a low rate of orders (less than five total orders in our data set). The most common indications for CT lower extremity with contrast to receive a red-scored order were lower leg swelling (cellulitis suspected), upper leg swelling (cellulitis suspected), and soft tissue infection of the femur suspected on initial examination (Table 3).

DISCUSSION

Imaging CDS mechanisms are intended to aid providers in selecting appropriate imaging tests. CDS mechanisms have the potential to reduce unnecessary imaging and support high-value care, but they can also inundate providers with alerts and hinder workflow. With CDS now required by federal mandate, we aimed to understand how CDS affected providers’ ordering decisions and workflow in a busy ED at an academic quaternary medical center.

CDS triggered AS-BPA alerts when providers ordered imaging studies with low appropriateness scores while also suggesting alternative imaging studies with higher appropriateness scores. In the ED at our institution, these AS-BPAs were triggered for only 3% of all imaging study orders, and providers spent an average of 24 seconds interacting with AS-BPAs.

The impact of AS-BPAs’ causing providers to change imaging orders at the time of order entry was limited. This was due largely to the small proportion (3%) of orders for which AS-BPAs launched and because when AS-BPAs were launched, providers selected alternative studies on the basis of the AS-BPA suggestion in only 3% of cases. This represented imaging order changes on the basis of AS-BPA suggestions for only 0.1% (18 of 17,355) of all imaging orders. Most imaging order changes were related to whether a study should be ordered with and/or without contrast.

Although CDS did not have a large impact on convincing providers to change the types of imaging studies originally ordered at our institution, with further optimization it is believed that CDS can be useful to providers in safely deciding whether to image or not image a patient. Importantly, we did not measure how often providers decided to refrain from ordering a study altogether after interacting with an AS-BPA alert, because these data could not be collected from our EMR or CDS vendor. We suggest that future iterations of CDS implementation make these data easily available to help clarify how CDS may reduce imaging utilization.

The implementation of a similar CDS at other academic institutions yielded improved appropriateness of imaging [16]. Several factors may contribute to our study’s showing a low impact of the CDS. Although our CDS did not cause providers to change orders at the time of order entry, it is possible that the BPA feedback altered provider ordering habits for future imaging studies. It is also possible that the impact of CDS alerts varies across providers, institutions, and patient populations.

The CDS triggered PT-BPA alerts when providers input free text as an indication for an examination instead of selecting a structured indication. Free-text input alone could not be scored for appropriateness by the CDS. Before PT-BPA implementation, 81% of imaging orders did not receive appropriateness scores, which severely limits the impact of our CDS. After PT-BPA was implemented, the percentage of imaging orders that did not receive appropriateness scores significantly decreased to 45%. Although PT-BPAs can enhance the effect of CDS by increasing the number of scored imaging orders, they also have potential drawbacks. Many clinical scenarios may not currently have structured indication options that reflect the true indication for the study. Although our PT-BPA still allows providers to choose a free-text option if no structured indication matches, it takes several additional clicks for providers to be able to input free text, which causes additional workflow interruption. In addition, it is possible that providers simply chose an inaccurate structured indication to avoid PT-BPA workflow interruptions, which can confound the accurate interpretation of the imaging study. Future work should focus on expanding the number of structured indication options and improving the accuracy of models to predict accurate indications to minimize workflow interruptions caused by PT-BPAs.

The implementation of CDS enables the collection of novel data sets that elucidate clinical scenarios in which imaging study orders consistently receive low appropriateness scores. In our ED, CT head without contrast, CT cervical spine without contrast, and CT chest pulmonary embolism protocol were the three studies that most frequently received low appropriateness scores. The low appropriateness for these scenarios may be related to the indication for ordering any type of imaging, rather than which type of imaging study should be ordered. In the future, CDS-derived data can be used to target quality improvement initiatives focused on reducing imaging overutilization.

Future studies could examine the effects of further optimized CDS in the ED setting and in different settings, such as community primary care practices and inpatient internal medicine departments. Additionally, providers could be surveyed on their subjective experiences with CDS to augment our objectively gathered data and understand how providers perceive the impact of CDS.

Future artificial intelligence tools could analyze CDS-derived and EMR data on a population level to not only reduce inappropriate imaging utilization but also make suggestions on optimal screening imaging studies for specific patient cohorts [17]. CDS could also be enhanced by additional artificial intelligence tools that analyze data at a holistic patient level to consider additional factors such as other diagnoses and allergies to make better imaging order suggestions [17]. As these CDS versions emerge, it will be paramount to analyze their effect on clinical care.

Limitations

Our study had limitations in addition to those just described. First, our study was performed at a single urban academic institution and within a single ordering department. It is possible that our findings are not easily generalizable to institutions with different patient populations, providers, CDS vendors, electronic health records, and workflows.

Second, in many instances, we noticed consulting services requesting that ED providers order imaging studies with low appropriateness scores. The consulting physician requesting the advanced imaging study never saw the BPA. This is a universal limitation of all EMR-based CDS mechanisms and limits the impact CDS has on altering an imaging order.

Third, it was not possible to collect data regarding how often providers selected the “emergency medical exception” option to bypass the CDS scoring altogether. Additionally, it is possible that our results would differ if we chose to analyze imaging orders over a longer period of time. Furthermore, we analyzed imaging order data from patients who visited the ED during the coronavirus disease 2019 pandemic, which changed ED patient demographics and imaging volumes at other academic health systems [18] and likely affected the included patient population and provider imaging order patterns in this study.

Finally, it was not possible to collect data regarding the amount of time providers spent interacting with PT-BPAs and the frequency of PT-BPA launch. It would be helpful if CDS and EMR vendors made these data readily available to further quantify the impact of PT-BPAs on ordering provider workflows.

TAKE-HOME POINTS.

• In a quaternary academic center ED, we observed low provider interaction times with CDS alerts that were triggered by low imaging appropriateness scores but also a low impact on imaging study selection.

• The implementation of an AI predictive text mechanism to match free text to structured indications increased the number of scored studies and could potentially enhance CDS effects.

• CDS implementation enables the collection of data to understand in which clinical scenarios imaging studies receive low appropriateness scores.