Timely and Efficient AI Insights on EHR: System Design

Abstract

A patient's electronic health record (EHR) contains extensive documentation of the patient's medical history but is difficult for clinicians to review and find what they are looking for under the time constraints of the clinical setting. Although recent advances in artificial intelligence (AI) in healthcare have shown promise in enhancing clinical diagnosis and decision-making in clinicians' day-to-day tasks, the problem of how to implement and scale such computationally expensive analytics remains an open issue. In this work, we present a system architecture that generates AI-based insights from analysis of the entire patient medical record for a multispecialty outpatient facility of over 700,000 patients. Our resulting system is able to generate insights efficiently while handling complexities of scheduling to deliver the results in a timely manner, and handle more than 30,000 updates per day while achieving desirable operating cost-performance goals.

1. Introduction

With the advent of artificial intelligence (AI) in healthcare, many applications based on natural language processing (NLP) and machine learning (ML) have been proposed to improve patient care by assisting clinicians in navigating, organizing, and extracting key information from electronic health records (EHRs). However, such systems often require a complex pipeline with hundreds of analysis engines and are therefore computationally expensive and difficult to efficiently scale with limited hardware. In addition, new information is frequently added to the patient record, presenting challenges for the system to keep up with updates. In this work, we describe the system architecture we developed to efficiently compute NLP-based insights generated from analysis of the entire patient medical record, and deliver them in a timely manner to clinicians. The techniques developed as part of this work are generalizable and widely applicable for systems computing NLP-based insights on EHRs.

The organization of the paper is as follows. In Section 2, we briefly show the landscape of related work. Section 3 gives a high-level overview of the computed insights and a description of the underlying NLP pipeline. Section 4 describes our initial implementation and lessons learned from deployment. Section 5 discusses our system architecture in detail. In Section 6, we summarize our main conclusions from this work and in Section 7 we briefly describe our vision for the future of AI systems for healthcare.

2. Related work

Advances in storage and standardization of patient data have opened up opportunities for processing EHRs at scale. However, implementations of these NLP algorithms for analyzing clinical notes are inherently computationally intensive¹. To enable NLP analyses at scale, a distributed computing framework using Apache Hadoop and a map-reduce² version of the NLP pipeline is used in HARVEST, a point-of-care patient record summarization tool³. Similarly, to support high-throughput NLP, Mayo Clinic implemented a Hadoop based ecosystem⁴ to distribute computational tasks. While several such implementations^5-7 discuss scaling aspects of throughput-oriented bioinformatics pipelines for information extraction, prioritized processing with restricted compute resources remains unaddressed. In this work, we present a system that provides insights in a timely and efficient manner with strict scheduling requirements and a limited amount of hardware.

3. Insights and NLP Pipeline

In this section, we briefly describe the insights generated from EHRs by our system and present an overview of the NLP pipeline used to create the insights.

3.1. Electronic Health Record

The electronic health record (EHR) represents a patient's medical information over time and across healthcare systems to provide a holistic view of a patient's medical history. As shown in Figure 1, an EHR typically consists of several sections of structured data, such as medications ordered, laboratory test results, and procedures conducted, and a large collection of unstructured data, such as progress notes, telephone encounters, radiology reports, and discharge summaries. Structured data is a rich source of a patient's medical history and is the main input for many healthcare informatics systems, including automated methods for summarization⁸. However, structured data alone is insufficient to capture a complete story of the patient's health⁹. For example, a physician may prescribe a medication to manage a patient's condition, but the patient decides to stop taking the drug due to an adverse reaction. While the prescription is recorded in the structured data, the fact and reason for stopping the medication are only found in the unstructured note. Therefore, to effectively summarize a patient's current status, it is often necessary to consider information both in the structured data and unstructured data. In our system, information from structured data is extracted with a timestamp and temporally aligned with unstructured data to form a more complete feature set for downstream analytics.

Figure 1:

NLP pipeline for generating insights from EHRs

3.2. Electronic Health Record-level Insights

A common way to provide a holistic summary of an EHR is in the form of a problem-oriented medical record (POMR)¹⁰. The core of a POMR is a patient-centered problem list, which is frequently used by physicians either by itself or as part of a summary. Previous studies have shown that a manually maintained problem list in the structured section of an EHR is often outdated and inaccurate, and machine learning and NLP based approaches have shown promising results in generating problems lists from EHRs automatically¹¹. Automatic problem list generation requires a system to process the entire EHR to extract and rank documented problems to create an accurate and complete problem list reflecting the patient's current state of health.

3.3. Note-level Insights

After gaining an overview of the patient through the problem list, physicians may need to further investigate specific problems that should be addressed at the current visit. To support this, our system generates disease-specific notelevel insights for hypertension, diabetes mellitus, hyperlipidemia, heart failure, and chronic obstructive pulmonary disease. These disease-specific insights vary between different patients and different diseases, but in general, they include statements indicating the current status of the disease (e.g. "worsened control due to dietary indiscretion"), home monitoring results (e.g. "home bps running 145/85"), outcomes of any previous treatments (e.g. "doing better on losartan"), treatment plan at the visit (e.g. "continue meds for now, but will consider switching to ARB if cough persists"), and any adverse drug reactions (e.g. "reports diarrhea since increasing metformin"). Together, the extracted insights for a given disease from a specific encounter form a disease-specific extractive summary of that encounter. These extractive summaries are then presented in a longitudinal view allowing physicians to track the specified disease and its status, treatments and outcomes over the history of the patient.

A more detailed discussion of these analytics can be found in Devarakonda et al¹¹ and Liang et al¹² and is not the focus of this paper.

3.4. NLP Pipeline

As shown in Figure 2, each clinical note is ingested by a basic NLP processing layer that performs the standard NLP tasks including tokenization, lemmatization, sentence segmentation, part-of-speech (POS) tagging, and parsing, followed by a medical concept recognition and normalization component, where key medical concepts such as labs, procedures, medications, signs and symptoms, and diseases are identified and linked to a taxonomy. The outputs are then fed to the model layer to generate the disease-specific insights. Our system used an english slot grammar (ESG) parser¹³ followed by a proprietary medical concept annotator that maps terms into Unified Medical Language System (UMLS) concepts¹⁴. This entity linking pipeline is optimized to process clinical notes, where sentences are not always well structured and abbreviation expansion and disambiguation play an important role. We separate the components in the next layer into three categories based on how the analytics are developed, namely, heuristics and rules, traditional machine learning, and deep learning. When labeled data is easier to obtain, data-driven approaches, such as deep neural network architectures, often outperform other methods due to their ability to effectively learn representations as well as model parameters. For instance, for adverse drug events (ADEs), we use an existing labeled dataset from the MADE 1.0 NLP challenge¹⁵ to train a BiLSTM-CRF model with attention¹⁶. On the other hand, ground truth for extractive summarization of a specific disease requires human experts to read the entire note and is harder to acquire. With limited ground truth, we have learned that a hybrid system combining heuristics, less expressive models (e.g. linear SVM), and outputs from deep learning-based components as features generates better results than trying to train an end-to-end pure neural network-based system.

Figure 2:

NLP pipeline for generating note-level insights

In summary, the overall analytics pipeline consisting of note processing and summary processing is extremely complex and computationally intensive. In fact, even after considerable code optimizations the average CPU time to compute all required insights for a full EHR is around 867 seconds (≈ 15 minutes) as shown in Table 3.

Table 3:

Processing statistics

Parameter	Value	Parameter	Value
Cumulative		Note Service
Avg. wall-clock time / EHR	915 sec	Avg. CPU time / note	1.1 sec
Min wall-clock time / EHR	143 sec	Avg. CPU time / encounter	1.8 sec
Max wall-clock time / EHR	5100 sec	Avg. CPU / encounter bundle	10.8 sec
Avg. CPU time / EHR	867 sec	Median CPU / encounter bundle	8.8 sec
Encounter bundle aggregation size Overhead	10K chars	Max CPU / encounter bundle Summary Service	119 sec
Avg. storage overhead / EHR	5 MB	Avg. wall-clock time / EHR	6.1 sec
Avg. scheduling overhead / task	0.1 sec	Avg. CPU time / EHR	4.2 sec

4. Baseline System

The solution was developed for an outpatient facility with a total patient population of more than 700,000 individuals. The initial non-functional requirements (NFR) specified that up-to-date insights (a.k.a. EHR summaries) be available at least two hours before scheduled appointments, or within 15 minutes after the registration of unscheduled walk-ins. Walk-ins were expected to comprise 30 percent of all office visits. To cover the patient population for the selected subset of participating clinicians, up to 7,500 EHRs would need to be processed daily. Up-to-date patient data and schedules were made available to the system by 6:00 AM each day.

The system computed insights for every single patient with an impending appointment by processing the complete EHR. As described in the previous section, the system used an NLP pipeline with state-of-the-art NLP methods for generating insights from an EHR both at note-level and at EHR-level. Typically, insights generated through these techniques are provided as services that end-users access through a web interface that has a powerful backend which computes the insights on-demand. Our initial implementation of the system was designed in a similar way but with ahead-of-time computation of insights prior to a patient appointment, considering the computational complexity. Further, the system was designed to be efficient by storing and reusing any analysis done as part of the first two stages of processing (Basic NLP, Medical Concept Linking) in as shown Figure 2 for any unchanged unstructured data i.e. unmodified clinical notes. The system was deployed on a Kubernetes cluster with four worker nodes, each with 32 CPU cores and 256 GB of memory. The system used a Cassandra database (DB) for storing the previously computed insights. The DB had five nodes, each 32 cores, 256 GB RAM, and 5 TB disk array for storing the data.

However, shortly after the initial version of the system went live for testing with a small number of clinicians it was clear these requirements needed to be updated.

1. Most walk-ins were actually same-day scheduled or rescheduled appointments which were spread throughout the day. This suggested a need for the revised flexibility, to distribute the processing evenly throughout the day.

2. Moreover, clinicians often preferred to prepare for patient visits many hours or even days earlier and their expectation was that EHR summaries are always up-to-date.

3. In order to satisfy the second requirement of maintaining an up-to-date posture, insights for a patient need to be recomputed each time there is a change to the EHR. As described in Section 3, an EHR is made up of structured data and a set of unstructured notes, typically in hundreds. However not all EHR updates are relevant for regeneration of insights. Only updated or new notes, lab results and vital measurements are to be considered, and in that order of priority.

4. In addition, any EHR updates for patients having an impending appointment should be be prioritized ahead of EHR updates without an upcoming appointment.

5. Another implied requirement is that the system should prioritize processing for an EHR if it has never been processed before to prepare for an impending appointment.

6. The system should allow fast-lane, high-priority processing of requests, initiated by the production support personnel, to regenerate the insights for a particular patient EHR, to handle any unforeseen issues with EHR data upstream, within the EHR system.

In summary, there are two kinds of triggers for regenerating insights for a patient, 1) appointments 2) EHR updates and a high-level business prioritization that needs to be considered for processing. The EHR for a patient may get updated several times a day. A spot measurement of Health Level Seven (HL7) traffic from the EHR system on a typical weekday found 8,000 changes to clinical notes and 25,000 structured data changes, and 16,000 appointment scheduled for covered patients. Of these, there were up to 30,000 events (EHR updates and appointments) that would trigger regeneration of insights. This is a four-fold increase in processing needs that the system could not handle without significant redesign. Average processing times of 15 CPU minutes per EHR combined with low latency requirements and frequent EHR updates made it essential to develop a much more efficient and flexible computational strategy. The remainder of this paper mainly describes the engineering work done to address this change in requirements which resulted in improved performance by two-orders of magnitude.

5. System Architecture

The overall solution consists of three layers: a) a clinician facing UI embedded within the EHR system running inside the provider infrastructure b) a REST¹⁷ API layer responsible for serving the InsightRequests from the UI and pulling updates to the EHRs from a Fast Healthcare Interoperability Resources (FHIR) server and c) a backend system responsible for computing the insights for each EHR update. The UI and the FHIR are outside the system boundary and in the following sections we describe the architecture of the backend system which is the core of this work.

5.1. Architecture Goals

The two main goals of the system architecture are the the following.

• Enable Collaborative Research: Research and development of new analytics require running many data science experiments, each with hundreds of EHRs. The system has to be extensible to incorporate new analytics from a sizable team of researchers and engineers with very little effort. This requires a modular architecture. Also, the system should be throughput oriented for batch processing to minimize experiment latency and thereby accelerate quick experimentation.

• Scalable Inference Processing: After the development of new analytics, or improvements to the accuracy of existing analytics, they are promoted to the production system. The production system should be able to meet the strict NFR described earlier in Section 4 within the allowed cost-performance constraints.

5.2. Enable Collaborative Research

The analytics pipeline consists of several steps, each containing implementations of clinical NLP algorithms. Typically, each step involves either training and evaluating or applying a ML model. Refer to Section 3 for a summary of methods used in insight generation for more details. Furthermore, the pipeline consists of both serial and parallel steps, with complex software and resource dependencies at each step. The framework for building the pipeline should be able to accommodate such complex steps easily, ideally through configuration and simple conventions. Further, it should be able to parallelize the execution across processes both within a machine and across a distributed cluster and provide real-time visualization of progress on the job status.

To address the above technical requirements, the analytics pipeline was implemented using Unstructured Information Management Architecture (UIMA)¹⁸. UIMA framework enables applications to be decomposed into components and manages the data flow between them and thus helps achieve the goal of modularity. We chose UIMA over other big-data processing frameworks such as Apache Spark¹⁹ for the following reasons:

• Flexibility for building and maintaining complex pipelines with hundreds of steps.

• To be included in the pipeline, each analysis engine only needs to implement a simple, frictionless API. This simple contract relieves the component owner from having to worry about other parts of the system.

• A type-system specifies data structures that may be used for the processing and sharing of information²⁰. UIMA comes with a powerful type-system management that enables interoperability between components. This is especially critical for NLP pipelines where analysis in each step builds on the analyses done in earlier components.

• Easy learning curve for the researchers who have diverse expertise in areas such as NLP parsing, semantic ontologies, search, reasoning and ML but with little experience in distributed processing.

• UIMA decouples processing logic from run-time, scale-out considerations making collaborative research easier.

To simplify the experimental setup, both the input and the output of the pipeline were file based. The input to the pipeline was complete EHRs and the output produced by the pipeline was stored as JavaScript Object Notation (JSON) files in a high performance shared filesystem. The pipeline is run single-threaded per EHR and averages 15 minutes for each EHR. To achieve high processing throughput, Distributed UIMA Cluster Computing (DUCC)²¹ was used to vertically scale multiple pipelines in each process and horizontally scale processes across machines. Thus the combination of UIMA and DUCC helped us achieve the goal of enabling and accelerating collaborative research within our team while decoupling the concerns of performance and scaling, which we describe next.

5.3. Scalable Inference Processing

As detailed in Section 3, re-processing the entire EHR is both time consuming and resource intensive. To achieve scalable inference, we developed a set of strategies based on the nature of the EHR data and information needs of the individual analysis engine.

• Data segmentation.

• Workload segmentation using split-pipelines.

• Event-driven architecture that enables efficient scaled-out processing.

• A custom scheduler tuned for business-provided prioritization of workload.

We detail these key ideas in the following sections.

5.3.1. Data Segmentation Based on Encounters

To efficiently processing an EHR update, it is important to differentiate the components of the EHR that need reprocessing from those that have not changed. We realized that the notion of an encounter is important for this purpose. An encounter is defined as an interaction between a patient and healthcare provider for the purpose of providing healthcare services or assessing the health status of the patient²². All the information generated from a single encounter is linked by an encounter ID within the EHR. Depending on the age of the patient and the frequency and complexity of their care, the patient's EHR can contain information from a few to potentially thousands of encounters. An encounter may include multiple notes. As discussed in Section 3, note analysis must take into account the associated structured data. However, it does not need to reference data from other encounters. When an EHR is modified, only the new or modified encounters need to be processed. Thus, encounters provide a natural way to segment an EHR for data storage and scale-out. By exploiting this fact, we developed an efficient mechanism for creating bundles of pending work that could be efficiently scheduled. As a further processing optimization, encounters were grouped into EncounterBundles.

5.3.2. Workload Segmentation: Split-pipelines

Another key strategy that lowered the latency was to split the two kinds of summarization that shared the same underlying foundational NLP components, namely, note-level extractive summarization and EHR-level abstractive summarization into two different pipelines. Powered by the modularity from UIMA, we split the end-to-end pipeline into two simply through configuration changes.

• A note processing pipeline wrapped inside a NoteService that could be highly scaled-out. The input to NoteService is an EncounterBundle and the output is a collection of note-level insights for one or more encounters.

• A summary processing pipeline wrapped inside a SummaryService which optionally performed EHR-level summarization but mainly assembled all EHR insights (at both note-level and EHR-level) into the standard JSON formats.

Scale-out of note processing through NoteService yielded low latency when processing a full EHR, and it also enabled finer grained processing priority changes (see Section 5.3.4). Thus, this granular segmentation of workload allowed us to efficiently scale-out the work.

5.3.3. Event-driven Processing Architecture

To handle the two different kinds of triggers (appointments and EHR updates) specified in Section 4, the system needs to be event-driven and have an intelligent mechanism for scheduling of EHR processing to handle an ever-changing workload queue. Figure 3 shows the major components of this event-driven architecture that can coalesce and process these Events with supplied business priorities.

Figure 3:

Event-driven Architecture

The Systemlnterface component receives all appointments and EHR updates. Appointments for new patients (previously untracked by the system) would trigger the Systemlnterface to fetch patient EHR data from the FHIR server and submit to the DB, as well as to begin tracking new EHR updates for the patient. New EHR updates for existing patients would cause the Systemlnterface to fetch the update from FHIR and submit to the DB.

For new patients all notes and structured data would be fetched, separated into encounters, each converted to a standard format, and written to the DB. The SystemInterface maintains a record of the last time patient data was fetched in order to request only notes that had been added or changed. Because the FHIR server did not support fetching incremental change for structured data, encounters with modified structured data are identified by fetching the full set of structured data and comparing to the prior copy saved in the DB. Modified encounters are then written to the DB.

The TaskAllocator periodically queries the DB for Events, and EHR processing tasks are scheduled or rescheduled as appropriate. All outstanding tasks are processed in a particular order, with the order potentially changing on every EHR update made by the SystemInterface. Refer to Section 5.3.4 for further details. Every idle NoteService and SummaryService processing thread maintains an outstanding request for work from the TaskAllocator. The TaskAllocator keeps track of which service (host, process id) received each task, along with an estimate for when the task was likely to complete based on the task size. The TaskAllocator monitors the status of all services to which tasks have been distributed to be able to resubmit uncompleted tasks assigned to failed services. It also monitors for tasks that exceeded their expected lifetime (note processing time strongly correlates to note content size) by some threshold in order to mark the task timed-out.

In order to minimize the impact of task management overhead (communication, DB operations, etc.), small encounters are grouped together up to a target size into EncounterBundles (see Table 3). EncounterBundles are then processed from largest content to smallest to minimize latency for each EHR.

The NoteTask sent to the NoteServices specifies an EHR identifier and a list of encounter identifiers (i.e EncounterBundle) to be processed. Each encounter is retrieved from the DB, unpacked, and each of its notes processed in sequence. The insights produced for all notes in the encounter are compressed and written to the DB. The Scheduler initiates a SummaryTask after all of the encounters for an EHR are processed.

The SummaryTask sent to the SummaryServices specifies the EHR identifier and a list of all of its encounters. The insights for each encounter are retrieved, unpacked, and combined into a single set of compressed insights saved in the DB. When the SummaryTask is completed the TaskAllocator notifies the SystemInterface (via a DeliveryTask), making it available for retrieval by the EHR system.

5.3.4. Scheduler

The primary goal of the scheduler is prioritizing the work done by the system based on the supplied business rules. This involves the following tasks.

• Defining the granular system rules for prioritizing all Events.

• Scheduling and rescheduling EHR processing tasks (NoteTask, SummaryTask, and DeliveryTask).

• Maintaining tracking information for every allocated task.

• Rescheduling failed and timed-out tasks.

• Recovering from a system crash with minimal task reprocessing.

The SystemInterface can modify a previously submitted Event as long as the scheduler has not taken ownership of the Event in the DB. This allows merging of multiple updates for an EHR into a single Event. Ownership prevents the SystemInterface from modifying any EHR data after Event processing has started and before final notification that Event processing has completed. Similarly, the scheduler can not take ownership of an Event owned by the SystemInterface. Event ownership metadata also indicates the Process Identifier (PID) of the owning process, so that ownership could be removed if the associated PID unexpectedly terminated.

There are three stages in the processing of each Event, called NoteTask, SummaryTask and DeliveryTask. As mentioned earlier, each Event includes one or more notes which may be processed in parallel (NoteTask). SummaryTask processing cannot commence until all NoteTask processing for the Event has completed. Likewise DeliveryTask processing must await completion of SummaryTask processing.

Events are submitted to the DB by the SystemInterface with an appointment time-stamp field and/or with a priority integer field. These two fields were used to create a complete processing order for all Events as shown in Table 1 where t is current time and t_appt is appointment time. All time-stamps are kept in UTC to support provider facilities that span time zones.

Table 1:

Processing order

Order Name		Rule	Notes
1	High priority	0 ≤ priority ≤ 1	Useful for handling walk-ins, fast-lane requests and periodic processing of a test EHR to monitor the overall system health.
2	Imminent appointment	t — 1 hour < tappt < t+7 hours
3	Just missed appointment	t —7 hours < tappt < t —1 hour
4	Low priority	2 ≤ priority ≤ ∞	For handling EHR updates.
5	Future appointment	tappt >t + 7 hours
6	Past appointment	tappt <t — 7 hours

The scheduler maintains two typed queues: Priority and Appointment as shown in Figure 4. When a new Event arrives, the scheduler places it onto the appropriate queue, along with its current state of processing. When a NoteService or SummaryService thread requests work, it specifies the states of work it can accept. The scheduler searches the Priority and Appointment queues in the global category order until an Event with matching state is located. If the entry is unlocked, the scheduler takes ownership and creates an in-memory object that first fetches necessary EHR details from the DB to manage all pending tasks, and then returns the next task to be done. If the entry is already owned by a TaskAllocator or SystemInterface thread, the in-memory object returns the next task to be done. The scheduler also receives notification of task completions, which are routed to the in-memory object they originated from. If all tasks for the current state are completed, the object updates the DB and advances the Event state.

Figure 4:

Scheduler

6. Results and Discussion

We evaluated the performance of the system with a representative set of 247 EHRs that also included content size outliers from somewhat small to very large. Throughout the development process, we used this same set to continually track the performance of the system and to verify that the insights from the production system matched that of the research system.

Table 2 shows the average size of EHRs in this test set. It is not surprising that for an outpatient facility the average number of notes per encounter is close to one, given that each office visit, telephone or email consult is treated as a separate encounter by the provider EHR system. Table 3 shows details on processing times and results size for the latest production version of the system. We summarize the results below. Baseline system: The average processing time for an entire EHR on a single CPU was about 15 minutes, with very large EHRs taking several times longer. With the baseline system described in Section 4, meeting the updated NFR of over 30,000 updates a day with a reasonable cost-performance ratio was not achievable. Redesigned system: By employing the techniques described in Section 5, we developed a system that could process the EHR updates incrementally.

Table 2:

EHR statistics

Average EHR content

819 notes

483 encounters

819/483 = 1.7 notes / encounter

1. In the new system, the typical processing time for a single update was under 7 seconds. This includes note processing (1.7 * 1.1 ≈ 2 sec.) and summary processing (4.2 sec.), ignoring the negligible scheduling overhead. In terms of storage overhead, the average size of processed results for EHRs was 2.5 MB. The cost of storing previously computed insights doubled the average results size to 5 MB per EHR.

2. Because NoteService threads were easily scaled-up and task priority changed dynamically, processing latency for even the longest, never-previously-processed EHRs could be brought down to just a few minutes.

The improvements helped us meet the NFR for our production system with no increase to the hardware footprint.

7. Conclusion

In this work we presented a system architecture for efficiently generating AI insights from EHRs with complex NLP analytics. The system was deployed in production for an outpatient facility and designed to handle at least 30,000 EHR updates per day while achieving desirable operating cost-performance goals. While there has been tremendous progress in the development of NLP algorithms and methods for healthcare, deployed production AI applications, such as our system, are still in early stages. The key takeaway from this work is that to design efficient solutions it is important to understand the clinical workflow behind the generation of EHRs and the information needs of individual AI analysis engines. We envision that as the AI adoption within EHR systems mature, much of the NLP analysis could be done during the authoring of clinical notes, reducing the overall solution complexity and operational cost.

8. Acknowledgements

The authors would like to thank Jianhe Lou, Jaroslaw Cwiklik, Daniyar Yeralin, Marshall Schor, Ansu Varghese, Kevin Guo and Richard Trotter for their contribution. We are grateful to Shannon Harrer and Bill Murdock for reviewing this work.