Development and Feasibility of a Real-Time Clinical Decision Support System for Traumatic Brain Injury Anesthesia Care

Summary

Background

Real-time clinical decision support (CDS) integrated with anesthesia information management systems (AIMS) can generate point of care reminders to improve quality of care.

Objective

To develop, implement and evaluate a real-time clinical decision support system for anesthetic management of pediatric traumatic brain injury (TBI) patients undergoing urgent neurosurgery.

Methods

We iteratively developed a CDS system for pediatric TBI patients undergoing urgent neurosurgery. The system automatically detects eligible cases and evidence-based key performance indicators (KPIs). Unwanted clinical events trigger and display real-time messages on the AIMS computer screen. Main outcomes were feasibility of detecting eligible cases and KPIs, and user acceptance.

Results

The CDS system was triggered in 22 out of 28 (79%) patients. The sensitivity of detecting continuously sampled KPIs reached 93.8%. For intermittently sampled KPIs, sensitivity and specificity reached 90.9% and 100%, respectively. 88% of providers reported that CDS helped with TBI anesthesia care.

Conclusions

CDS implementation is feasible and acceptable with a high rate of case capture and appropriate generation of alert and guidance messages for TBI anesthesia care.

1. Introduction

During surgery, anesthesiologists perform the important tasks of concurrent real-time interpretation and execution of several steps, including adequately maintaining physiological parameters, providing mechanical ventilation, administering anesthetic drugs and maintaining fluid balance – all in response to anesthesia exposure (e.g., physiological changes), ongoing surgery (e.g., blood loss) and patient responses (e.g., heart function). To execute these simultaneous steps, anesthesiologists must efficiently and effectively capture and disseminate information from multiple devices and systems, and make quick decisions and interventions that are evidence based. However, due to the complexity of anesthesia care, dynamic nature of caring for anesthetized patients, and lack of familiarity of best practice in all case types, adherence to best practice is not always achieved [1], resulting in suboptimal patient care.

Electronic medical record systems for anesthesia called, anesthesia information management systems (AIMS), have reduced the documentation burden of anesthesia providers by automatically collecting data from patient monitors and anesthesia machines. Recently, real-time clinical decision support (CDS) systems for anesthesia have been developed to augment documentation systems and provide decision aids to manage care and to improve quality and safety [2–13]. These CDS systems are built to acquire AIMS data in real-time, find ongoing clinical issues and notify anesthesiologists so that preventive and corrective actions can be taken immediately.

In 2012, the Brain Trauma Foundation published evidence based guidelines for the care of pediatric (age < 18 years) traumatic brain injury (TBI) patients [14]. Key performance indicators (KPIS) in the guidelines address management of a variety of clinically important parameters such as blood pressure, CO₂ levels, temperature, and blood glucose. Studies reveal that adherence to guidelines broadly improve care and patient outcomes and specific to TBI, Vavilala et al reported that every 1% increase in TBI guideline adherence across 5 leading pediatric trauma centers was associated with discharge survival by 6% in a cohort of 236 children¹. However, there are currently no anesthesiology specific tools that facilitate guideline adoption during neurosurgery for TBI patients. This is important because neurosurgery for TBI is typically urgent or emergent in nature, often presenting dynamic changes in patient, anesthetic and surgical conditions.

2. Objective

Our objective was to develop, implement and conduct feasibility evaluation of a CDS system to provide real-time decision support and guidance to anesthesiologists managing neurosurgery for pediatric TBI patients. In an actual clinical setting, the goals of this study were to:

1. determine the feasibility that emergent, pediatric, neurosurgeries can be automatically detected for clinical decision support,

2. determine if unwanted clinical events related to key performance indicators (KPIs) of pediatric TBI anesthesia care can be identified by a CDS system, and

3. examine anesthesiology provider acceptability of a real time CDS system for TBI anesthesia care.

3. Methods

This study was conducted from November 2015 till March 2016 [17 months] at a 400-bed level 1 trauma center that performs more than 15,000 surgical procedures per year in 26 operating rooms. Approximately 50 anesthesiologists and 25 nurse anesthetists provide anesthesia care on a permanent basis, while 100 anesthesiology residents rotate through the operating rooms every year. The study team had clinical and research expertise in anesthesiology, TBI and bioengineering. This study was determined to be a quality improvement initiative per institutional policies and hence was exempt from review by our institutional review board.

The development and refinement of the CDS system followed a rapid incremental development cycle (▶ Figure 1) borrowing on agile methodology principles in which requirements and solutions evolve through collaboration between multi-disciplinary teams [15, 16]. This approach promoted rapid, flexible and evolutionary development, while achieving continuous improvement. Embedded in the rapid development cycle were steps to test the system in different environments to ensure that desired functionality and adequate user acceptance are achieved. Software testing process were derived from published recommendations [17, 18] and adapted for the intraoperative TBI CDS system.

Fig. 1.

CDS development cycle

In the design phase of ▶ Figure 1, the knowledgebase of the evidence based TBI guidelines were translated into decision rule definitions. Additionally, the software design specifications for the CDS were outlined. In the development phase, the decision rules were built as conditional logic statements and the CDS software was programmed for TBI decision support. The CDS was first tested with computer simulated data using predefined test scripts. Test results were analyzed and if CDS functionality is different from expected, corrections to the decision rules and CDS software were performed. After successful completion of simulation tests, the system was piloted in an actual clinical setting for feasibility evaluation. The system was closely monitored for stability. Performance metrics including accuracy of detection of target cases and KPI events were reported as sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). Data related to user feedback were also analyzed. Corrections to software design, decision rules and software program were performed to overcome shortcomings in the CDS system. This iterative development cycle was repeated to optimize the CDS system.

3.1 Design phase

3.1.1 Knowledgebase: TBI guidelines and KPIs for intraoperative decision support

We extracted and adapted

1. 2012 Brain Trauma Foundation [14, 19],

2. acute care KPIs associated with discharge outcomes in children with severe TBI [1], and

3. institutional practice guideline for KPIs during intraoperative TBI anesthesia care. The evidence based TBI KPIs for intraoperative CDS are outlined in ▶ Table 1.

Table 1.

Key Performance Indicators (KPIs) included in the real time CDS

KPIs for intraoperative CDS
Avoid hypoxia Avoid hypercarbia and hypocarbia Avoid systemic hypotension Muscle relaxants monitored Avoid hyperthermia and hypothermia Monitor arterial blood gas and glucose hourly Avoid hyperglycemia and hypoglycemia Correct anemia and coagulopathy Avoid high concentration of volatile anesthetic agent

3.2.2 Software design: Real-time Anesthesia CDS

Software design for TBI CDS was based on an existing CDS tool for anesthesia care. The anesthesia CDS, called Smart Anesthesia Manager (SAM) is a real-time CDS system developed by our institution. It works with an Anesthesia Information Management System (AIMS) to detect and alert ongoing clinical issues in real-time (▶ Figure 2). A data acquisition module extracts anesthesia specific data from AIMS including hemodynamic and ventilation parameters, medications, fluids, laboratory results, and anesthesia and surgical events/interventions. Data are normalized to a standard data structure and a set of predefined decision rules are applied by a rules engine to detect on-going, undesirable, clinical events. A messaging module composes and delivers a message concerning the detected event. A client module running on an AIMS workstation receives the message and presents it as a “pop-up” message overlaid on the AIMS screen. Systems such as this Anesthesia CDS have been shown improve to quality of care, billing, compliance and waste reduction [2, 5, 7, 12, 20, 21]. Towards, TBI CDS, the software design considerations focused on developing software routines in the existing Anesthesia CDS to detect TBI KPIs and to identify target cases for CDS based on surgical procedure codes.

Fig. 2.

SAM CDS configuration showing the interfaces to AIMS and SAM along with the primary modules of SAM. Also, presented is an example CDS alert message displayed on an AIMS workstation.

3.2 Development phase

3.2.1 Data interface, processing and computation of derived parameters

Data from the AIMS database were extracted in real-time, every 6 minutes during the initial stages of development, but later improved to every 1 minute. Database queries were optimized such that there was no negative impact on AIMS operation. Real-time data for TBI decision support included arterial line and non-invasive blood pressures, oxygen saturation from pulse oximetry, end-tidal CO2, expired concentration of anesthetic agents, temperature, laboratory results, medications, fluids, surgical and anesthesia events and interventions. Additionally, the anesthesia CDS system was programmed to compute parameters that are derived from the AIMS data. This included age adjusted blood pressure targets and minimum alveolar concentration (MAC) of anesthetic agents.

3.2.2 Decision rules for TBI anesthesia care

Two main sets of decision rules were developed, first to automatically identify the target TBI neuro surgeries and second to detect unwanted KPIs or clinical events. The target population was pediatric patients (age <18 years) undergoing urgent/emergent neurosurgery for TBI. At our institution (level 1 Trauma Center) pediatric TBI patients present for TBI neurosurgery either from the emergency department or the intensive care unit. Identifying such cases in the CDS system in the operating room is critical to automatically triggering the TBI decision support. Typically, all surgical patients, including emergency surgeries, get registered and scheduled in a surgical information system. The surgery schedule data are interfaced into AIMS and CDS in real-time. For this reason, we chose to use the surgical procedure information (select procedure codes and key words in procedure description [e.g., subdural hematoma evacuation]) and patient age to detect pediatric neurosurgery. For case recognition, new logic functions were built to search procedure codes and procedure description key words in the surgery schedule data. The rules builder module was enhanced to include these new functions. A set of procedure codes and key words specific to TBI neurosurgery procedures was identified with help from the surgery scheduling personnel. These procedure codes and key words were input into the new functions to recognize TBI neurosurgeries. While automatic detection of TBI neurosurgery was the preferred method and goal to trigger CDS, we also built a feature in our AIMS for anesthesiologists to manually enable or disable CDS for a TBI anesthesia case.

Unwanted clinical events during TBI anesthesia care can be based on either continuous patient monitor or episodic laboratory parameters. Continuous patient monitor parameters include oxygen saturation (SpO₂), end-tidal CO₂ (EtCO₂), blood pressure (BP), body temperature and expired anesthetic agent concentration. Episodic laboratory parameters include arterial blood O₂ (PaO₂) and CO₂ (PaCO₂) tension, blood glucose level, hematocrit (Hct) and coagulation results – platelet counts and international normalized ratio (INR). The unwanted events based on patient monitor parameters include hypoxia (SpO₂ < 90%), hypocarbia (EtCO₂ <30 mmHg), hypercarbia (EtCO₂ >40 mmHg), age adjusted hypotension (systolic BP < 70 + 2×Age for age <10 years; systolic BP<90 mmHg for age 10 years are older) and high concentration of inhalation agent concentration measured in terms of minimum alveolar concentration (MAC) (MAC >1). The unwanted events based on episodic laboratory results include hypoxemia (PaO₂ <60 %), hypocarbia/hypercarbia (PaCO₂ <30 mmHg /PaCO₂ >40 mmHg), hyperglycemia (glucose >180 mg/dL), anemia (Hct <25 %) and abnormal coagulogram (INR >1.3 or Platelet count <100,000 IU).

Decision rules were built using the rules builder module of the anesthesia CDS and included TBI KPIs from the 2012 TBI Guidelines relevant to anesthesia care [14, 19] as determined by previous work by the study team [1]. Rules were created to detect deviation of TBI guideline parameters from recommended ranges. Specifically, logic functions that can apply conditional logic were used for TBI decision support. For continuous variable KPIs such those based on arterial blood pressure, end tidal CO2, and oxygen saturation, the logic functions computed median values in the last 5 minutes and compared them against thresholds to determine whether they are out of range. For episodic KPIs such as those based on laboratory values, the logic functions compared the last value against guideline thresholds. Additionally, rules were also built to give guidance reminders to perform arterial blood gas (ABG) and emergency hemorrhage panel (EHP) analysis and well as administer muscle relaxants if none was given. The CDS rules are outlined in ▶ Table 2.

Table 2.

Summary of KPI based CDS rules for pediatric TBI neurosurgery

Treatment guideline	Decision rule	Frequency of notification
Beginning reminder for Arterial line and ABG	No arterial line or arterial blood gas (ABG) at the beginning of case	Once
Arterial blood gas	No ABG at the beginning of case	Repeat 3 times, then Stop
	Last ABG is more than an hour ago	Q18
Emergency hemorrhage panel	No EHP at the beginning of case	Repeat 3 times, then Stop
	Last EHP is more than an hour ago	Q18
Administration of muscle relaxants	If muscle relaxant has not been given by start of procedure	Once
Normocarbia maintenance	PaCO2 out of range (30–40 mmHg)	Once per new Lab result
	EtCO2 (5 min median) out of range (30–40 mmHg)	Q12 min
Avoidance of hypoxemia/hypoxia	PaO2 <60 mmHg	Once per new Lab result
	SpO2 (5 min median) <90%	Q12 min
Avoidance of hypotension	SBP (5 min median) <70+2xAge or 90 maximum	Q6 Min
Normothermia maintenance	Temperature out of range (35–38 deg	Q12
Blood glucose management	Blood glucose >180 mg/dL	Once per new Lab result
Anesthetic dose managementMinimum alveolar concentration (MAC) maintenance	Age adjusted MAC (5 min median) >1.0	Q12 Min
Transfusion trigger	Hematocrit < 25% and no blood transfusion	Once per new Lab result
Coagulopathy treatment	Platelet < 100,000 or INR > 1.3	Once per new Lab result

Note: For continuous variables, the median value within a 5-minute moving window was computed. This median was compared against the threshold to detect an unwanted event. If no unwanted event is detected in the 5-minute window, the window was advanced by 1-minute for the next sample. However, if an unwanted event is detected, the window is moved by 5 minutes. Hence, if there is an extended period of an unwanted event, consecutive 5-minute non-overlapping epochs are used to trigger alerts. On the other hand brief unwanted incidences that are clinically irrelevant are not alerted.

3.2.3 Message notifications

Notification messages were composed using the anesthesia CDS rules builder to accurately describe the unwanted key performance indicators of TBI care and recommend steps for resolution. If a guideline parameter continues to be out of range, the notification message is repeated at a predefined frequency. The frequency of repetition was decided based on how critical the parameter was and an estimate on how long it typically takes anesthesiologists to address/resolve the issue. Additionally, consideration was also given to alert fatigue experiences due to frequent messages. After the first detection of an issue, alerts can be configured to repeat in multiples of 6-minute periods if the issue is persisting. The frequency of alert repetition is also presented in ▶ Table 2.

3.3 Testing phase

An AIMS test environment was used to perform system validation of TBI CDS in a non-clinical setting. This environment allowed us to simulate patient monitor parameters and a TBI case scenario. Test scripts to evaluate the functionality of each decision rule and its different permutations were defined. The TBI CDS system was deployed in the AIMS test environment and the test scripts were manually executed while simulating a TBI case. The test results were analyzed and if CDS functionality was different from expected, corrections to the decision rules and CDS software were performed. The development and test cycle was repeated till the CDS functionality matched design specifications.

While the test environment and case simulation provided a useful platform to perform the initial tests of the TBI CDS, it could not replicate the realistic hemodynamic and ventilation parameters, as well as process flow steps related to patient registration, surgery schedule and notification of laboratory results during an actual TBI case. For this reason evaluation in an actual clinical setting was necessary.

3.4 Clinical feasibility and acceptability evaluation phase

The tested and refined CDS system was deployed in a pilot fashion in an actual clinical setting. Anesthesiology providers were notified and educated on TBI CDS through multiple channels (e-mails, presentations, and one on one discussion).

3.5 Monitoring and evaluating the TBI CDS and its iterative refinement

Once CDS for neurosurgery was introduced in an actual clinical setting, its performance during pediatric TBI cases was closely monitored and analyzed. Process steps were put in place to identify pediatric patients undergoing emergent neurosurgery on a daily basis. If a pediatric TBI neurosurgery occurred, the entire AIMS data for the case, including minute by minute patient monitor and ventilator parameters, laboratory results, medications, fluids and surgical and anesthesia events were extracted. In addition, the alert logs maintained by the CDS system were also extracted. Data were analyzed to determine whether TBI CDS for anesthesia was triggered (automatically or manually) for the case. Additionally, data were also analyzed to determine KPIs event frequencies and whether these events were detected and notified by the CDS system. Specifically, we first extracted from the AIMS database the high fidelity patient monitor and ventilator data for each TBI case in this study. Then we manually reviewed the minute by minute hemodynamic and ventilation parameters as well as the laboratory results to determine the actual KPI events that occurred during a case. This we compared against the logs of detected KPIs produced by the CDS. Additionally, false events detected by the CDS were also identified. Root cause analyses were performed to examine reasons behind undetected events and false events. The CDS was iteratively corrected and adjusted to optimize the system such that TBI decision support has high accuracy and specificity.

3.6 Anesthesia provider feedback

Within 24 hours of a case, the study team sent an e-mail with a website link to all anesthesiology providers (attending anesthesiologists, residents and nurse anesthetists) who cared for eligible patients. Feedback questions were adapted from previous work on clinical decision support [22] and addressed providers’ overall acceptance of CDS, alerts and reminders that either worked or not worked for a particular case using a 5 point Likert scale (1=Strongly agree, 2= agree, 3=neutral, 4=disagree, and 5=Strongly disagree). User feedback was used to refine the system iteratively. The survey questions are presented in ▶ Table 3.

Table 3.

Survey questions to solicit user feedback

I was aware of the clinical decision support before using it The clinical decision support helps me provide better care for children with severe TBI The clinical decision support tool is hard to use The clinical decision support tool improves education of residents (for attending anesthesiologists only) There were too many alerts The alerts were irrelevant I am very satisfied with the CDS

4. Results

Clinical decision support for pediatric patients undergoing TBI neurosurgery was piloted at our institution starting in November 2014. Between November 2014 and March 2016, twenty eight patients underwent urgent/emergent intracranial neurosurgeries for TBI. The CDS system was incrementally refined and optimized during the pilot in response to identified performance deficiencies of the system in an actual production environment. As described in subsequent sections, performance deficiencies were a result of unanticipated deviations from workflow, changes to systems and devices interfacing with anesthesia CDS, failure of decision rules in certain scenarios and defects in software. Although minor corrections were incrementally made to the CDS system throughout the pilot, a major software revision was made after the 1-year time point in December 2015. The major revisions include:

1. increase in CDS sampling of AIMS database from every 6 minutes to every 1 minute,

2. safeguards in AIMS interface in CDS program to prevent processing of over sampled cases,

3. updates in CDS software to accommodate new data format of ventilation parameters due to introduction of new anesthesia machines,

4. corrections to CDS-AIMS interface to accommodate laboratory interface changes, and

5. corrections to decision rules to detect hypothermia and hyperthermia if temperature values are acquired through different channels.

Hence, we present the results of system performance during two phases – one prior to the major software revision (Phase 1) and another after the revision (Phase 2).

4.1 Identification of Target Surgeries

Overall, for the cohort of 28 patients, the TBI CDS was automatically activated in 22 of the 28 patients (Sensitivity = 79%). Automatic detection failed in the other 6 surgeries (21%). In some instances (n=3) cases were not scheduled in the surgery information system using pre-specified TBI neurosurgery terms, but rather as a generic, non-specific, procedure which was later corrected, typically after surgery. In another instance (n=1) the patient’s age was unknown at the time of registration. One surgery was also missed (n=1) at the beginning of the pilot because the CDS system did not include a newly added procedure code in the scheduling system for the neurosurgical procedure. Lastly, in another instance (n=1), the AIMS system was down for maintenance that prevented electronic documentation for the case. The anesthesia provider did not manually initiate CDS for TBI in any of the instance in which automatic detection of neurosurgery failed.

Among the 22 cases in which CDS were automatically triggered, 3 cases experienced activation delay by over 15 minutes from the time when the patient arrived in the operating room. Delayed activation was either because of delayed scheduling of surgery in the surgery system or delayed correction of originally scheduled generic surgery type to intracranial. Issues that were encountered for automatic identification of intracranial surgeries in pediatric TBI patients and steps of resolution are outlined in ▶ Table 4.

Table 4.

Encountered issues and resolution steps related to automatic detection of neurosurgery in pediatric TBI patients

Encountered Issue	Resolution
Emergent scheduling of cases as generic procedures and not as intracranial	Educate surgery front desk to avoid booking cases using generic procedure codes (Phase 1) Educate Anesthesia providers to manually enable TBI decision support if not automatically triggered (Phase 1) Implement a reference TBI case list that CDS program can refer to include specific patients for decision support (Phase 2)
Unknown age at the time of scheduling the case	CDS decision rules refined into include age up to 20 years to accommodate patients with misidentified age when admitting emergently (Phase 2)
Missing procedure code in	Update CDS with latest procedure codes in the surgery system (Phase 1)
AIMS not operational	Implemented monitoring steps to minimize AIMS downtime (Phase 1)

4.2 Alerts for unwanted clinical events for TBI anesthesia care

▶ Table 5 shows the sensitivity of detecting unwanted patient monitoring related events by the CDS system. A total of 119 unwanted events occurred in the 22 cases in which CDS was automatically triggered. Among these, 12 events were excluded from analysis because they occurred before CDS was activated for the case (delayed activation due to delayed scheduling of cases). Among the remaining 107 clinical events, sensitivity of detection by the CDS system increased from 41.3% during phase 1 to 93.8% in phase 2. False alerts due to artifacts and system data latency (delays in the AIMS in capturing patient monitor data) was 1.5 per case during phase 1, while it was 4.0 per case during phase 2.

Table 5.

Sensitivity of CDS in detecting unwanted clinical events based on patient monitor parameters.

Type of event	Phase I Nov 2014 – Dec 2015 (N=17)				Phase II Jan – Mar 2016 (N=5)
	Number of true events	True events detected by CDS	False events detected by CDS	PPV (%)	Number of true events	True events detected by CDS	False events detected by CDS	PPV (%)
Hypoxia (SaO2 <90%)	3	3 (100%)	0	100.0	1	1 (100%)	1	50.0
Hypocarbia (EtCO2 <30 mmHg) & Hypercarbia (EtCO2 >40 mmHg)	31	9 (29%)	1	90.0	15	14 (93.3%)	2	33.3
Hypotenstion (SBP <70 + 2×Age, limited to 90)	29	17 (58.6%)	13	56.7	12	11 (91.7%)	14	44.0
Temperature <35 or >38°C	10	0 (0%)	0	N/A	1	0 (0%)	4	0.0
High inhalation agent concentration (>1 MAC)	2	2 (100%)	12	14.3	3	0 (0%)	0	N/A
Total	75	31 (41.3%)	26 (1.5/case)	54.4	32	30 (93.8%)	21 (4/case)	55.3

▶ Table 6 presents the unwanted events related to laboratory results. There were a total of 233 laboratory results from 22 TBI neurosurgeries. One hundred and thirty one were excluded from analysis because they were either performed prior to CDS activation during the case (n=20) or because of delayed verification of laboratory results (n=111) in the laboratory system resulting in their delayed arrival in the AIMS/CDS systems. The remaining 102 lab results were used in analysis. Instances where the CDS provided a prompt for a laboratory measurement, but no actual measurement was made, were excluded from the calculated sensitivity and specificity. Sensitivity of detection of unwanted events increased from 50.0% in phase 1 to 90.9% in phase 2. Specificity of detection of unwanted events increased from 93.9% (phase 1) to 100% (phase 2). Due to the small sample size included in our study, the sensitivity, specificity, PPV and NPV should be interpreted cautiously.

Table 6.

Sensitivity and specificity of unwanted events related to laboratory results detected by the CDS system

Type of laboratory result(abnormal value)	Phase I Nov 2014-Dec 2015 (N=17)						Phase II Jan-Mar 2016 (N=5)
	Total lab sample	Abnormal lab value N (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Total lab sample	Abnormal lab value N (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
Arterial	9	0 (0%)	N/A*	100.0	N/A	100.0	8	1 (13%)	100.0	100.0	100.0	100.0
oxygen tension (Pa02 <60 mmHg)
Arterial carbon dioxide tension (PaC02 <30 or >40 mmHg)	9	3 (33%)	0.0	100.0	N/A	66.7	8	5 (63%)	100.0	100.0	100.0	100.0
Glucose (>180 mg/dL)	10	4 (40%)	25.0	100.0	100.0	66.7	9	0 (0%)	N/A*	100.0	N/A	100.0
Hematocrit (<25%)	17	1 (6%)	100.0	100.0	100.0	100.0	11	2(18%)	100.0	100.0	100.0	100.0
Coagulogram (platelet <100,000 or INR > 1.3)	16	4(31%)	100.0	75.0	57.1	100.0	5	3 (60%)	66.7	100.0	100.0	66.7
Total	61	12 (19.7%)	50.0	93.9	66.7	88.5	41	11 (26.8%)	90.9	100.0	100.0	96.8

*There were no events in these time periods, so we could not calculate the sensitivity.

The anesthesia CDS system also generated guidance reminders in addition to alerts on unwanted clinical events. Guidance messages are reminders to perform regular (hourly) laboratory results (ABG, EHP and blood glucose) and give muscle relaxants if none has been administered. Among the 22 cases in which CDS was triggered, there were a total of 51 possible instances when guidance messages were warranted. However, due to delayed verification of laboratory results, in 10 of the instances, the system would not have been able to generate guidance reminders. Among the remaining 41 instance the sensitivity of the CDS system in generating guidance reminders increased from 76% (phase 1) to 81% (phase 2). There were also no false guidance reminders generated in phase 2. Results are presented in ▶ Table 7. Main issues that were encountered towards automatic detection of unwanted clinical events and generation of guidance reminders as well as steps of resolution are outlined in ▶ Table 8.

Table 7.

Generation of guidance reminders by the CDS system

Type of alert	Phase I Nov 2014 – Dec 2015 (N=17)			Phase II Jan – Mar 2016 (N=5)
	Events needing reminders	Reminders generated N (%)	False reminders	Events needing reminders	Reminders generated N (%)	False reminders
Hourly ABG§ & glucose	4	2 (50.0%)	3	8	8 (100.0%)	0
Hourly EHP§	13	9 (69.2%)	0	7	4 (57.1%)	0
Administer muscle relaxants	8	8* (100.0%)	0	1	1* (100.0%)	0
Total	25	19 (76.0%)	3	16	13 (81.3%)	0

*Muscle relaxant was given before guidance alert but anesthesia provider didn’t record in real time for 4 patients in phase I and 1 patient in phase II.

^§ABG – arterial blood gas, EHP – emergency hemorrhage panel laboratory results

Table 8.

Encountered issues and resolution steps related to automatic detection of unwanted clinical events and generation of guidance reminders

Encountered Issue	Resolution
Delayed detection of hypotension events.	CDS sampling of AIMS database increased to every 1 minute from every 6 minutes (Phase 1)
AIMS interface to patient monitor updated. Errors in interface configuration resulted in oversampling of data and out of memory CDS software errors	AIMS interface corrected. Safeguards in CDS program to prevent processing of over sampled cases (Phase 1)
New anesthesia machines introduced resulting in unanticipated changes in data format in the AIMS interface to these machines	Updated CDS software to accommodate new data format (Phase 1 & 2)
Changes in laboratory interface resulting in missed identification of ABG results	CDS software corrected to accommodate laboratory interface changes (Phase 1)
Decision logic errors in CDS system in processing multichannel temperature values.	Decision rules corrected in CDS system to detect hypothermia and hyperthermia if temperature values are acquired through any channel on the patient monitor. (Phase 1)
Artifacts in blood pressure and end tidal CO2 measurements resulting in false alerts	Boundary and median filters programmed in CDS to minimize the effect of artifacts. (Phase 2)
Change in patient age unit definition in the registration interface to AIMS resulting in misidentification of age less than 1 year	CDS software corrected to handle new age unit definitions (Phase 2)

4.3 User acceptance of the CDS system

Forty four responses were returned for the 22 cases; one set from either the resident or nurse anesthetist and one response from the attending anesthesiologist. Thirty six percent of respondents were aware of the CDS. Overall, clinicians stated that CDS “helped them provide better care decision support of their patients with severe TBI (2.6 on scale of 1= strongly agree and 5= strongly disagree)” and none found the CDS “hard to use”. Seventy two percent of attending anesthesiologist respondents agreed or strongly agreed that the CDS “improves education of residents.” Attending anesthesiologist uniformly reported that the CDS helped them and all respondents strongly disagreed that the alerts were irrelevant. None of the clinicians complained of having “too many alerts.” User acceptance was high with provider satisfaction scored as a 4 or 5 by 88 % of the respondents. No provider reported alerts to be excessive.

5. Discussion

This study explored the feasibility and acceptability of developing and implementing a real-time CDS system in a high acuity operating room performing urgent/emergent neurosurgery on pediatric patient with TBI. The CDS targeted anesthesiology providers, and generated alerts concerning unwanted clinical events for TBI anesthesia care. We report that automatic detection of emergent TBI intracranial surgeries for CDS can be achieved with moderately high sensitivity (79%), detection of unwanted events can be achieved with high sensitivity (>90% – phase 2) and specificity (100% – phase 2) while simultaneously achieving high provider satisfaction (88%).

The ability to detect pediatric TBI neurosurgery is important to automatically activate CDS. However, this process proved challenging in circumstances where trauma patients are urgently scheduled for surgery soon after their hospital admission. In these instances target CDS triggers such as age may not be accurately known at the start of procedure. Despite incomplete information, operating room staff schedule cases with generic names and procedures, with best estimates for certain inputs such as age. Hence, the CDS system was unable to identify some patients as candidates for decision support. Although this problem is difficult to completely eliminate, we implemented some work around solutions. The first was to build a trigger in AIMS that can be used by anesthesiology providers to manually enable or disable CDS for a particular case. The second step was to relax the inclusion criteria for age to < 20 years so that patients with an estimated age close to pediatric range can still qualify for CDS and for whom TBI Guidelines apply during anesthesia care. Conversely, if providers learn midway through procedures that patients are over age 18 years, they are able to disable the CDS alerts. Lastly, we also implemented a database of TBI patients that could be cross-referenced by the CDS system to trigger intraoperative decision support in patients with a diagnosis of TBI.

Testing of the CDS system was initially performed in an AIMS test environment. This environment allowed decision rules to be tested against simulated data. This step proved inadequate as the actual clinical environment and complexity of TBI anesthesia care often presented unique unanticipated issues. Hence, development and refinement of the system needed to be performed iteratively utilizing a real clinical setting. However, iterative development was slow as TBI neurosurgeries in pediatric TBI pediatric patients are encountered infrequently. For faster development and for more realistic testing of CDS systems it would be useful to have the ability to playback case data in AIMS so that CDS can be applied on realistic data in a retrospective fashion. However, the AIMS system in our institution (Merge AIMS, Merge Inc., Hartland, WI) did not have features to playback case data.

Data latency or delayed availability of data for analysis by the CDS system has negative impact on real-time decision support [23, 24]. Data latency can result in delayed detection of clinical issues. Alternately, it can trigger false, “after the facts” alerts when in fact the provider has already noticed and corrected a clinical issue. In this study, data latency was experienced due to delays in AIMS database refresh, infrequent sampling of AIMS data by the CDS system and delayed verification of laboratory results. Delays in refreshing the AIMS database with the most current data are a function of the AIMS system and could not be overcome. Using optimized database queries that can be executed quickly, we were able to increase the sampling frequency of the CDS system from once every 6-minutes to once every 1-minute, thus reducing data latency. Delays in verifying laboratory results were because of incomplete patient registration in the laboratory information system. Emergent nature of TBI neurosurgery meant that often complete patient registration information was not available in the laboratory information system in a timely fashion. Without complete patient registration information laboratory results could not be verified against the correct patient and sent to AIMS/CDS. In many instances (32%) laboratory results could be verified only after the surgery was complete. Without real-time availability of lab results the decision rules based on abnormal labs could not be executed.

Artifacts in patient monitor data may trigger false alerts. Examples include surgical personnel accidentally leaning on blood pressure cuff affecting non-invasive blood pressure measurements, flushing of arterial line affecting blood pressure measurement, motion artifacts affecting pulse oximetry signal etc. [25] In this study, we instituted several measures to minimize the effect of artifacts. First, boundary filters were used to remove non-physiological measurements in patient monitor data. Second, we implemented validity checks to verify that diastolic, mean and systolic values of a blood pressure measurement are in increasing order. This helped eliminate artifacts due to arterial line flush. As a third step we employed median filters to remove spurious measurements values due to artifacts. In spite of these efforts, we still encountered artifacts such as for example damped arterial line resulting in false alerts of hypotension. Future efforts to improve artifact detection would include higher sampling of patient monitor data so as to apply signal processing and pattern recognition techniques to identify and eliminate artifacts [25].

Changes to data interfaces could potentially compromise CDS operation. In this study, changes to patient monitor, anesthesia machine and laboratory interfaces without concurrent adjustments in the CDS system resulted in some parameters being not acquired and processed by the CDS system. Proper communication of interface changes to downstream systems is important to prevent such avoidable errors.

6. Conclusions

In conclusion, development of a real-time CDS system for a highly complex clinical setting such as urgent neurosurgery for pediatric TBI patients is challenging, but feasible with good user acceptance. Iterative development and refinement is needed for optimizing such a system. Data latencies and artifacts could result in delayed and false alerts by the CDS system. Never the less, real-time decision support holds much promise in influencing provider behavior and standardizing care to evidence based best practice. Future work will aim towards evaluating the efficacy of the system towards improving adherence to TBI care guidelines and patient outcomes.

Question

Data latency in real-time clinical decision support systems pertains to delayed availability of data to make decisions. Data latency could cause:

1. Delayed detection and notification of clinical issues

2. False notification after a clinical issue has been resolved

3. Reduced effectiveness and user acceptance of real-time decision support

4. All of the above

Ans: (d). Data latency could be because of several reasons – delayed refresh of database in certain AIMS configurations, data processing time for AIMS and clinical decision support module, retrospective manual entry of data and inherent delays in data interfaces. Delayed availability of data means delayed detection of issues potentially leading to a prolonged occurrence of an issue. Also false notification that occurs after an issue has been observed and resolved could also occur. If after the fact, false alerts are generated a majority of the time, they will become a source of annoyance and distraction for the providers.

Funding Statement

This work was supported by a grant from National Institute of Health [R01 NS072308–05]