Physician Usage and Acceptance of a Machine Learning Recommender System for Simulated Clinical Order Entry

Abstract

Clinical decision support tools that automatically disseminate patterns of clinical orders have the potential to improve patient care by reducing errors of omission and streamlining physician workflows. However, it is unknown if physicians will accept such tools or how their behavior will be affected. In this randomized controlled study, we exposed 34 licensed physicians to a clinical order entry interface and five simulated emergency cases, with randomized availability of a previously developed clinical order recommender system. With the recommender available, physicians spent similar time per case (6.7 minutes), but placed more total orders (17.1 vs. 15.8). The recommender demonstrated superior recall (59% vs 41%) and precision (25% vs 17%) compared to manual search results, and was positively received by physicians recognizing workflow benefits. Further studies must assess the potential clinical impact towards a future where electronic health records automatically anticipate clinical needs.

Introduction

Healthcare often falls short of recommended, evidence-based care, with overall compliance with guideline recommendations ranging from 20 to 80%. ¹ Variability and uncertainty in medical practice may compromise quality of care and cost efficiency, especially in scenarios where knowledge is inconsistently applied. ² The advent of the meaningful use era of electronic health records (EHRs) ³ creates the opportunity for data-driven clinical decision support (CDS) that utilizes the collective expertise of many practitioners in a learning health system. ^4–8 Tools such as order sets already reinforce consistency and compliance with best practices, ^9,10 but maintainability is limited in scale by a top-down, knowledge-based approach requiring the manual effort of human experts. ¹¹ Moreover, the intended vs. actual usage of EHR order sets may not align with physician workflows. ¹² A key challenge to fulfill a future vision for clinical decision support ^13,14 is the automatic production of content from the bottom-up by data-mining clinical data sources. ¹⁵

Previously, we developed a clinical order recommender system by automatically data-mining hospital EHR data. ¹⁶ The results of this approach align with established standards of care ^15,17,18 and is predictive of real physician behavior and patient outcomes. ¹⁶ Our underlying vision is to seamlessly integrate a system into clinical order entry workflows that automatically infers the relevant clinical context based on data already in the EHR and provides actionable decision support in the form of clinical order suggestions, analogous to Netflix or Amazon.com’s “customers who bought A also bought B” system. ^19,20 As with many machine learning models designed to support clinical decision making, it is unknown if physicians will actually accept such suggestions into their clinical decision workflow. Most prior studies in automated development of clinical decision support content ^{15,16,21–25} have been strictly analytical evaluations, with few studies assessing the response of human clinicians to such recommender tools and their ordering patterns. More broadly, the majority of physicians have significant distrust or negative attitudes toward the EHR, ^26–28 which may affect how well these tools could be adopted.

This study seeks to address these issues by examining physicians’ behaviors while interacting with a clinical provider order entry (CPOE) interface that simulates an electronic health record for hospital clinical scenarios. We specifically examine physician ordering patterns, time spent, and survey responses when a clinical order recommender system is added to standard functionality.

Objective

Determine whether clinicians will use machine learned clinical order recommendations for electronic order entry in simulated inpatient cases. Further assess how such recommender systems will be viewed by physicians and affect their workflow.

Methods

As described previously, ¹⁶ we extracted deidentified structured data for all inpatient hospitalizations from the 2009-2014 STRIDE clinical data warehouse. ²⁹ The data covers >74K patients with >11M instances of >27K items (medication, laboratory, imaging, and nursing orders, lab results and diagnosis codes). We built a clinical collaborative filtering (recommender) system based on this data, modeled on Amazon’s product algorithm ^19,20 using item co-occurrence statistics.

We built a simulated computerized physician order entry (CPOE) interface with open technologies including PostgreSQL, Python, Apache HTTP, and HTML/JavaScript. Our unique addition is an automated recommender (Figure 1), analogous to a “Customers Who Bought This Item Also Bought This...” service that anticipates other clinical orders that are likely to be relevant based on similar prior cases in prior electronic health records.

Figure 1.

– Simulated clinical order entry interface, notes and clinical order recommender. Standard functions include navigation links to review notes and results (top-left). Order entry includes a conventional search box for individual orders and pre-authored order sets (top-right). A recommender algorithm suggests clinical orders (right), in this example triggered by a presenting symptom code (Shortness of Breath, ICD9 786.05). Clinical orders predicted most likely to occur next are highlighted under Common Orders, while those under Related Orders are less likely but disproportionately associated with similar cases and thus may be more specifically relevant.

A panel of board certified internal medicine physicians (AK, JH, LH, and JHC) developed clinical cases for common inpatient medical problems including unstable atrial fibrillation, neutropenic fever, variceal gastrointestinal hemorrhage, bacterial meningitis, and acute pulmonary embolism (Table 2). Each case includes clinical notes to represent the patient’s history and physical exam. Diagnostic test results are only visible and change state if respective orders are entered (e.g., low hemoglobin revealed only if a blood count is ordered and changes if a blood transfusion is ordered). With each order entered, the clinical order recommender list updates based on the accumulating patient information.

Table 2.

- Summary description of simulation cases tested. Last column reflects the most common clinical orders the test participants used in each case, with the total in parentheses counting repeat orders. ICD, International Classification of Diseases; R-CHOP, rituximab, cyclophosphamide, hydroxydaunorubicin, oncovin, prednisone; CBC, complete blood count; CSF, cerebrospinal fluid; INR, international normalized ratio; ECG, electrocardiogram; NSAID, non-steroidal anti-inflammatory drug; DCCV, Direct Current Cardioversion; IV, intravenous.

Presenting Symptom (ICD-10) / Diagnosis	Case Summary	Key findings	Most Common Orders (Total Orders)
Fever (453.3) Chemotherapy Induced Neutropenic Fever	32 year-old patient with diffuse large B-cell lymphoma presenting with fevers and rigors after receiving chemotherapy (R-CHOP) 10 days prior.	Hypotension, lactic acidosis, severe neutropenia	Sodium Chloride IV Bolus (42) Metabolic Panel, Comprehensive (33) Blood Cultures (32) Cefepime, IV (31) CBC with Differential (31)
Headache (R55) Bacterial Meningitis	25 year-old previously healthy patient presenting with fever, headache, neck stiffness, and photophobia	Fever, significant neck stiffness on examination, absence of rashes	Ceftriaxone, IV (33) Sodium Chloride IV Bolus (32) CBC with Differential (32) CSF Culture and Gram Stain (32) Glucose, CSF (30)
Dyspnea (R06.00) Acute Pulmonary Embolism	70 year-old with a past medical history including systolic heart failure and COPD presenting with worsening dyspnea following a vacation to Hawaii	Hypoxia (81% oxygen saturation), tachycardia, absence of jugular venous distension, minimal wheezes	CBC with Differential (31) ECG 12-Lead (31) Metabolic Panel, Comprehensive (27) NT-proBNP (25) Albuterol-Ipratropium, Inhaled (22)
Palpitations (R00.2) Unstable Paroxysmal Atrial Fibrillation with Rapid Ventricular Rate	66 year-old with a history of diastolic heart failure presenting with palpitations	Tachycardia (rate >150 beats/min), hypotension, irregularly irregular pulse	ECG 12 Lead (46) DCCV (29) CBC with Differential (28) Metabolic Panel, Comprehensive (26) Consult to Cardiology (23)
Hematemesis (K92.0)Acute Variceal Bleeding	59 year-old with a history of alcoholism and NSAID use presenting with hematemesis	Tachycardia, spider angiomata, scleral icterus, mid epigastric pain	Consult to Gastroenterology (59) Sodium Chloride IV Bolus (41) Prothrombin Time/INR (40) CBC with Differential (40) Metabolic Panel, Comprehensive (37)

We recruited licensed physician participant users with experience admitting medical inpatients within the past year through local mailing lists. Participants were offered $195 to use the interface to simulate admitting hospital patients and complete a survey over the course of an hour. A researcher guided participants through two demonstration cases (diabetic ketoacidosis and chest pain) to illustrate basic functions (data review, order entry, order sets) as well as the use of clinical order options presented by the recommender system. The subsequent five test cases were presented to participants in a sequence randomly assigned for each user (Table 2). Each case was randomly assigned as either an intervention case with the recommender system available or a control case without the recommender system not available. Conventional clinical order entry options including order set checklists and manual search of individual orders by name were available in all cases, making usage of the recommender system completely optional. Participant activity was recorded through screen capture, audio, and user interface tracking software. Metrics compared between the intervention and control cases include the time to complete the case and the number of clinical orders selected from manual search vs. the automated recommender system, with P-values assessing differences calculated by two-tailed t-test. The study was approved by the Stanford Institutional Review Board.

Results

Participants

A total of 34 physicians participated in this phase of the study, with a median of 3 and interquartile range of [3, 5.25] years since obtaining their medical degree, 22 (64%) identified their primary specialty as Internal Medicine, 9 (26%) identified Emergency Medicine, 18 (53%) were resident trainees, and 14 (42%) were board certified in their respective specialty.

Case-Based Scenarios

Table 2 summarizes key elements of the five case scenarios that participants were tested with.

Physician Experience

Overall, physicians spent an average time of 6.7 minutes per clinical module, with a mean 54.4 navigation clicks between sections (e.g., notes vs. results review) and 16.5 clinical orders per case. In subgroup analysis of resident physicians in training vs. non-residents (Table 3b), there appears to be varying effects with residents spending less case time and ordering more with the recommender available, while non-residents spent more case time. Across different simulated case types (Table 3c), there was not a consistent trend in physicians taking more or less time to complete cases with the recommender system available. Physicians ordered a greater amount of orders when the recommender system was available (mean 17.1 vs. 15.8 orders per case, p<0.001). The recall of the recommender options was consistently greater than manual search options (59% vs 41%), indicating users were more likely to find the clinical orders they wanted from the automated recommender lists than from options returned by manual search. The precision of the recommender options was similarly greater than manual search options (25% vs 17%), indicating users had to sift through fewer irrelevant options to find the clinical orders they wanted than the number of irrelevant options produced by manual searches. ¹²

Table 3b.

- Usage metrics when clinical order recommender system was available vs. not, separated by Resident physicians (trainees) vs. non-Residents. Reported as totals, proportions, or means +/- standard error.

Table 3c.

- Usage metrics stratified per simulated case scenario for when the clinical order recommender system was available vs. not.

Survey Responses

Overall, the clinical decision tool was positively received by the study participants, where 94% agreed or strongly agreed that the tool would be useful for their position. Moreover, 89% agreed or strongly agreed that the system would make their job easier and 85% felt that it would increase their productivity. Participant comments suggest that physicians believe the system would be more useful for cases that involved common diagnoses or standardized treatment algorithms. Others mentioned the tool’s utility for diseases that may require several simultaneous orders (for example, diabetic ketoacidosis). A small subset of responses noted that the tool could be used for patients presenting to the emergency department without a clear diagnosis, as this would facilitate expedient ordering of several screening tests to help differentiate the patient. Additional comments indicate physicians felt that the tool would be less useful for sub-specialized care or for patients that require few simultaneous orders.

Discussion

We found that the use of a clinical order recommender system for common clinical scenarios seen in inpatient emergency medicine did not clearly affect the amount of time physicians spent on an EHR interface. Physicians did place slightly more total orders per case, although this varied by scenario and user training level. Importantly, physicians placed less orders from manual searches as a result of the tool. The recommender demonstrated superior recall of orders, suggesting that users were more likely to find the orders they wanted from the recommender rather than from manual searches. Similarly, the tool’s greater precision for suggested orders indicated that users were not exposed to as many irrelevant order options when compared to manual searching. The tool was positively received by the study participants, who identified clear benefits toward their workflow and productivity. This represents a key study to examine the use of clinical recommender decision support tools on physician ordering habits as applied toward inpatient emergency clinical scenarios.

Implications of this study include that such decision support tools based on clinical ordering patterns can be integrated and readily accepted for usage into physician workflows. While many systems attempt to improve the EHR experience by providing standardized order sets to streamline and improve care, ^30–33 static order sets often do not align with individual cases with many extraneous or irrelevant order options. ¹² Our recommender tool essentially functions as a dynamical clinical order set that continuously updates in response to new patient information, demonstrating increased accuracy and reduced need for conventional manual searches. This may account for the largely positive views of the participants, suggesting that physicians will accept machine generated clinical order tools if they are embedded into clinical workflows.

Time motion studies indicate that clinicians spend most of their time in the EHR, ^34,35 with many spending significantly more time searching for and entering orders. ³⁶ While this study showed a reduction on reliance of manual searches, interestingly, it did not show a reduction in the amount of time that physicians spent per simulated case. The simulated test setting may have lead participants to artificially fill the time within cases, or perhaps the reduction in manual search efforts freed their cognitive attention to attend more to the medical decision making tasks of each case. Notably, most of our study participants were experienced physicians who have likely developed diagnostic/treatment algorithms based on their previous training. Table 3b suggests that the tool could provide time savings to less experienced physicians (in residency training) who may not be familiar with the EHR interface or have less refined diagnostic/treatment schemata.

The total number of orders tended slightly to increase as a result of the recommender system (mean 1.3 additional orders per case), though this varied by case. The quantity of orders placed with a recommender system may be dependent on the clinical context, where some scenarios (i.e. meningitis) may have more standardized approaches to treatment³⁷ compared to others (i.e. dyspnea), which may affect how recommended a la carte orders are viewed. Indeed, the purpose of commercial product recommender algorithm is to increase the amount of products a customer will buy. ²⁰ While order sets have been shown to promote cost-effectiveness, ^38,39 further evaluation is needed to determine how much clinical recommender systems are promoting improved care with more useful orders vs. reducing cost-effectiveness with more unnecessary orders.

Limitations to this study approach include the nature of collaborative filtering algorithms tending to recommend patterns of historical behavior, but risk a “cold start” ⁴⁰ problem when they are unable to recommend newly available treatments while still recommending older, possibly obsolete, practices. This is not unique to the recommender systems however, as existing standards of information dissemination such as clinical practice guidelines warrant manual updating and dissemination every few years. ⁴¹ For example, the manually-authored static order sets available in our own institution for deep venous thrombosis treatment still recommend warfarin therapy, despite direct oral anticoagulants largely becoming the current standard of practice. ⁴² Guidance for up-to-date medical care clearly must come from multiple sources, though this points to one potential advantage of the collaborative filtering approach in that it can rapidly and automatically adapt to newly emerging practices.

Technical limitation considerations include that the tool was based on a clinical data warehouse of electronic health records data that may not be available at all institutions. Similarly, the lack of a broadly accepted open architecture that allows for custom workflow integrations into common commercial EHRs, limits the ease of implementation of the system components studied. Our users were given an orientation of the recommender system and its purpose before engaging with the practice scenarios, which likely contributes a Hawthorne effect on how users interacted and viewed the system.

At a time when the EHR is met with distrust and negativity by clinicians from the burdens of documentation and data entry, clinical recommender systems represent a key opportunity to improve the quality, consistency, and experience of healthcare. We hope this will be an important step towards a future where EHRs anticipate clinical needs without even having to ask, so that clinicians can start to feel like the computers are working for us, instead of the other way around.

Conclusions

Clinical order suggestions from a data-driven recommender system were readily used and accepted by physicians across a variety of simulated inpatient clinical scenarios. These systems did not clearly affect time spent in the EHR, but physicians were more likely to find the clinical orders they wanted using such tools as compared to manual search methods (i.e. superior recall). Usage may vary by clinical scenario, with further evaluation needed on the clinical value of how such tools affect ordering habits. Nonetheless, clinicians overall view such clinical recommender systems positively, perceiving a clear potential benefit toward their workflow.

Figures & Table

Figure 2.

– Simulated clinical order entry interface, results review and clinical order manual search. Users can order diagnostics such as cerebrospinal fluid (CSF) studies to review results (left). This may require simulated passage of time for results to be ready (e.g., CT Head requiring another 44 minutes for results to be ready). Conventional manual search for clinical orders via a text search box (top-right) yields clinical order options (right) identified by prefix. In this example, identifying all clinical orders with a word starting with “cef.”

Table 3a.

- Usage metrics when clinical order recommender system was available vs. not. Reported as totals, proportions, or means +/- standard error. Options reflects clinical order options that were presented to the user for consideration via either manual search results or automated recommender. Recommender precision (positive predictive value) reflects the proportion of clinical order options from the recommender that were actually used. Recommender recall (sensitivity) reflects the proportion of all clinical orders used that arose from the recommender options.

Table 4.

- Physician Survey Responses. Responses were assessed based on a 5-point Likert scale (1 = Strongly Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly Agree).

Survey Question	1	2	3	4	5
I would find the system useful in my job	0%	0%	7%	46%	48%
Using the system would make it easier to do my job	0%	4%	7%	44%	46%
This system would increase my productivity	0%	9%	7%	41%	44%
This system would let me complete tasks more quickly	0%	4%	9%	37%	50%
This system would increase my job performance	0%	9%	7%	48%	30%