Scalable and High-Throughput Execution of Clinical Quality Measures from Electronic Health Records using MapReduce and the JBoss^® Drools Engine

Abstract

Automated execution of electronic Clinical Quality Measures (eCQMs) from electronic health records (EHRs) on large patient populations remains a significant challenge, and the testability, interoperability, and scalability of measure execution are critical. The High Throughput Phenotyping (HTP; http://phenotypeportal.org) project aligns with these goals by using the standards-based HL7 Health Quality Measures Format (HQMF) and Quality Data Model (QDM) for measure specification, as well as Common Terminology Services 2 (CTS2) for semantic interpretation. The HQMF/QDM representation is automatically transformed into a JBoss^® Drools workflow, enabling horizontal scalability via clustering and MapReduce algorithms. Using Project Cypress, automated verification metrics can then be produced. Our results show linear scalability for nine executed 2014 Center for Medicare and Medicaid Services (CMS) eCQMs for eligible professionals and hospitals for >1,000,000 patients, and verified execution correctness of 96.4% based on Project Cypress test data of 58 eCQMs.

1 INTRODUCTION

Secondary use of electronic health record (EHR) data is a broad domain that includes clinical quality measures, observational cohorts, outcomes research and comparative effectiveness research. A common thread across all these use cases is the design, implementation, and execution of “EHR-driven phenotyping algorithms” for identifying patients that meet certain criteria for diseases, conditions and events of interest (e.g., Type 2 Diabetes), and the subsequent analysis of the query results. The core principle for implementing and executing the phenotyping algorithms comprises extracting and evaluating data from EHRs, including but not limited to, diagnosis, procedures, vitals, laboratory values, medication use, and NLP-derived observations. While we have successfully demonstrated the applicability of such algorithms for clinical and translational research within the Electronic Medical Records and Genomics (eMERGE)[1] and Strategic Health IT Advance Research Project (SHARPn)[2, 3], an important aspect that has received limited attention is scalable and high-throughput execution of electronic Clinical Quality Measures (eCQMs) using EHR data. Briefly, eCQMs are Center for Medicare and Medicaid Services (CMS) defined quality measures for eligible professionals and eligible hospitals for use in the EHR Incentive program for electronic reporting[4]. The e-specifications include the data elements, logic and definitions for that measure in an Health Level Seven (HL7) standard – Health Quality Measures Format (HQMF) – which represents a clinical quality measure as an electronic XML document modeled using the Quality Data Model (QDM). While significant advancements have been made in the clarity of measure logic and coded value sets for the 2014 set of eCQMs for Meaningful Use (MU) Stage 2, these measures often required human interpretation and translation into local queries in order to extract necessary data and produce calculations. There have been two challenges in particular: (1) local data elements in an EHR may not be natively represented in a format consistent with HQMF/QDM including the required terminologies and value sets; and (2) an EHR typically does not natively have the capability to automatically consume and execute measure logic. In other words, work has been needed locally to translate the logic (e.g., using an approach such as SQL) and to map local data elements and codes (e.g., mapping an element using a proprietary code to SNOMED). This erodes the advantages of the eMeasure approach, but has been the reality for many vendors and institutions in the first stage of MU[5].

In our prior work[3], we have devised solutions for EHR data normalization and standardization, including generation of structured data templates using natural language processing. In this work, we address the second challenge for scalable and high-throughput execution of eCQMs using MapReduce, the open-source JBoss^® Drools Rules engine, and Common Terminology Services 2 (CTS2)[6, 7]. Specifically, we have developed a set of architectural principles that guide the functional and non-functional requirements of our proposed system that supports interoperability, testability, and scalability for automated execution of eCQMs. To evaluate and test our system for performance, scalability, and correctness of measure result calculations, we leveraged test datasets from Project Cypress which provides a rigorous and repeatable testing tool of EHRs and EHR modules in calculating MU Stage 2 CQMs, and is the official testing tool for the 2014 EHR Certification program supported by the Office of the National Coordinator for Health IT (ONC). Our evaluation results show that the system implemented can not only replicate Cypress benchmarks for calculated measure results (accuracy of 96.4%), but also linearly scales to a large number of patient records (>1,000,000 patients). The source code of our system, released under Apache License Version 2.0, is available via: http://api.phenotypeportal.org.

2 MATERIALS AND METHODS

In order to evaluate a candidate architecture, we first present a series of Architectural Quality Attributes that outline the critical high-level functional and non-functional requirements of the system. These guide design decisions and technology choices, and must hold for the candidate architecture to be viable. Next, we examine the individual components of the system and their interconnections, along with an examination of the general architectural style. To reinforce the system architectural goals, we will investigate one of the eCQMs, CMS MeasureID CMS163v1 (Diabetes: Low Density Lipoprotein (LDL) Management and Control), as a use case at various stages of transformation and ultimately computation. Finally, to test our architecture and implementation viability, we quantitatively measure system performance in terms of measure computation time and execution “correctness,” or verification of the results.

2.1 Tools, Standards, and Models

Project Cypress

Project Cypress1 is a testing and verification framework for eCQMs and is the official testing tool for EHR measure calculation provided by the ONC. It provides a set of 36 de-identified test patients, along with expected execution results for each quality measure calculation. Both the patient dataset and the expected results are distributed in JSON format[8], which is easily parsed and analyzed for testing. Project Cypress also includes several table-based graphical views of expected results, along with in-depth debugging information.

QDM/HQMF

The Quality Data Model (QDM)[9, 10] is a standardized electronic model for representing quality criteria. These models are represented using the Health Quality Measures Format (HQMF)[11] specification, which is an HL7 Version 3 Draft Standard for Trial Use (DSTU). For any given eCQM, the QDM defines, via logical operators and temporal comparisons, criteria used to classify patients into several categories, or Populations. These include the Initial Patient Population (IPP), Numerator (NUMER), Denominator (DENOM), Denominator Exclusion (DENEX), and the Denominator Exception (DENEXCEP)[12].

CTS2

Common Terminology Services 2 (CTS2) is an Object Management Group^® terminology services standard[6], as well as an accepted HL7 Normative Service Functional Model.2 The CTS2 standard defines a common data model and interface for interacting with a terminology service, including read/write access and various maintenance, content loading, and querying functionality.

2.2 Architectural Quality Attributes

In order to better understand the intent and function of the system, we have identified several guiding concepts, or Architectural Quality Attributes[13], that are necessary for a successful system.

Quality Attribute 1: Interoperability

Statement: Communications, interfaces, and data exchanges should be based on industry standards and specifications.

Rational: Standards-based design allows for effective componentization of the system, as component interfaces and data contracts can be linked to published specifications. This promotes low coupling and improved separation of concerns. A standards-based approach also enables implementation and information hiding[14], which allows greater opportunities for substituting component implementations based on use case.

Implications: First, a standardized measure definition format is needed. This will enable interoperability and provide a common, shared grammar for communicating measure intent. Second, as structured vocabularies play a major role in the measure logic, a standard terminology service interface is required. Finally, standard protocols are needed to integrate these components and to interact with the system.

Quality Attribute 2: Testability

Statement: System results should be verifiable, repeatable, and comparable to benchmarked and curated datasets. Rational: The identification of patient cohorts in many instances remains a manual, human-driven process[15]. This presents a system testing problem, as it may be difficult to determine acceptable system outputs without significant effort. It is important, therefore, that a subject matter expert curated test dataset be available, and that the system architecture incorporate such datasets in order to allow for automated and repeatable testing.

Implications: Testing infrastructure will need to accommodate the automation of benchmarked and curated test datasets. This also will require reporting processes to clearly demonstrate test results.

Quality Attribute 3: Scalability

Statement: System architecture will allow for increasing workloads and expanding datasets.

Rational: As shown in the eMERGE[1] project, efficiently processing large datasets is important to research. High scalability is necessary for both present and future viability.

Implications: Processing algorithms and deployment models must enable horizontal scalability and parallel execution.

2.3 Architectural Style

The architectural style of our platform is a combination of pipe and filter transformations and a MapReduce computational algorithm. In the pipe and filter phase, input data is transformed via several discrete components in a streaming fashion, creating an information flow through the system which ultimately leads to the desired output[16]. Output of this phase is a machine-executable representation of the quality measure. One of the main benefits of this style is loose coupling, as components need not know implementation details of each other. This approach is possible via our Architectural Principle 1 – Interoperability and standards-based design – as components will have explicit and standardized contracts. These contracts are implemented as Representational State Transfer (REST) based services wherever possible, giving them a consistent interface[17]. Measure execution logic is then partitioned by the MapReduce computational algorithm, allowing for parallelization and horizontal scalability.

2.4 System Components

The overall system architecture and scope is shown in Figure 1. Functional software modules of the execution system are enumerated below, while other components or infrastructure shown can be assumed to be external dependencies.

Figure 1:

Scalable and High-Throughput Phenotyping Architecture for Clinical Quality Measure Calculation

HQMF/QDM XML to JSON Transformer

The HQMF/QDM XML to JSON transformation component parses the HQMF/QDM XML representation and transforms it into easily parsable JSON. This functionality is exposed via a REST service wrapping the Project Cypress health-data-standards3 project. This component is Ruby4 based, and represents the only non-Java Virtual Machine (JVM) component of the architecture.

JSON to JBoss^® Drools Compiler

The QDM JSON representation is then programmatically transformed into executable JBoss^® Drools Rules via a compilation process. The result of this transformation is a single Drools Rule file per measure that is executable by the Drools Execution Engine or other JBoss^® Drools environment. The compilation component is based on the Groovy5 language, as Groovy provides templating and string interpolation functionality that is difficult to replicate in Java.

Drools Execution Engine

Utilizing the JBoss^® Drools Rule engine, the generated Drools Rules are executed given an input patient set. The result of this execution is the classification of patients into populations (such as NUMER, DENOM, etc). A thin interface façade was created over the “JBoss^® Drools version 5.6.0. Final” library, hiding the implementation details of the engine. There are, in fact, no Drools-related interfaces exposed to the client – meaning the Drools-based implementation is decoupled from the user-facing contracts.

Patient Data Source

Data is provided to the execution engine via an abstraction layer utilizing the Iterator pattern[18]. This approach neither provides nor specifies a specific data source, but facilitates custom data extraction implementations tailored to the specific database and environment.

Value Set Processor

QDM Data Elements semantically define their intended criteria using Value Sets, or enumerated sets of coded concepts from standard vocabularies. Take, for instance, the criteria: “Diagnosis, Active: Acute Pharyngitis” using “Acute Pharyngitis Grouping Value Set (2.16.840.1.113883.3.464.1003.102.12.1011).” In this example, in order for the QDM Data Element to be satisfied, a patient record must contain a Diagnosis entry coded with concepts contained in the Value Set uniquely identified by the given HL7 Object Identifier (OID)[19].

To facilitate this, a mechanism for resolving the concepts of a Value Set is required. This functionality is described in two procedures shown in Algorithm 1, and defines the capability of the Value Set Processor. The procedure EXISTS_IN_VALUESET defers explicit full resolution in favor of an exists-type functionality, which tests membership of a given coded_entry within a Value Set. In this procedure, the set of concepts C denotes each concept c defined by the Value Set identified by the given OID. The boolean existence of coded_entry in C is returned. Furthermore, it often is necessary to find the subset of coded entries that align semantically with a defined Value Set. In the procedure FIND_MATCHES, instead of checking existence in C as above, the intersection of C and the coded_elements set is returned. In both procedures, the function resolve(oid) returns the set of all concepts in a Value Set given an OID.

Algorithm 1.

Value Set Processor Functionality

2.5 Processing Algorithms and System Deployment

MapReduce

MapReduce is a computational abstraction designed to allow for easier parallelization and distribution of large datasets for processing[20]. The computation is a two-step process, consisting of a map function followed by a reduce function. The map function, given an input, divides this input into smaller sub-problems which then are consolidated by the reduce function.

We can apply this concept in the context of patients and populations. In the map function, each patient is classified into the appropriate populations given an eCQM. We then reduce each population (and its given set of patients) into some analytic result – for instance, analyzing demographic characteristics or creating patient lists for the quality measure.

Algorithm 2.

MapReduce for Clinical Quality Measures

Algorithm 2 illustrates the general flow of information through the measure execution logic. The getPopulations function encompasses the actual measure execution logic – in our case, the Drools Execution Engine component. The postprocess function in the reduce portion is left to be implemented by the client, as this will be use case dependent. Note that the current code base includes a basic demographics analytics post processor, but it is expected that clients will extend it to meet their needs, or substitute their own processing entirely.

2.6 Measure Execution using the JBoss^® Drools Rules Engine

The JBoss^® Drools Rule Engine provides the execution framework for the clinical quality measures. Given the input HQMF/QDM XML, the intermediary JSON representation is then compiled into Drools Rules. Figure 2 shows a representation of CMS MeasureID CMS163v1 and the translation of one example criteria to a Drools Rule. In a Drools Rule execution, decisions are made based on data elements, or Facts, that have been placed into working memory either from an external data source or as the result of other rule executions. In this example, a MeasurementPeriod is first checked – this is a parameter that controls the valid start/end date range of the measure. A Patient is then retrieved from the working memory, and the Encounters of that patient are examined for a semantic match (a match in Value Set 2.16.840.1.113883.3.464.1003.101.12.1001), and a temporal match (expressed by the during operator in reference to the MeasurementPeriod). Finally, if this result is satisfied, a PreconditionResult is inserted into working memory. The Drools execution engine, using the Rete[21] algorithm, assures that dependent rules will then fire, as the new PreconditionResult is now available in working memory.

Figure 2:

Drools Rule Representation of Initial Patient Population for CMS MeasureID CMS163v1

2.7 Quantitative Evaluation Methods

Performance (Execution Time)

Scalability and performance are important aspects of the architecture, and the ability to efficiently process large patient sets is critical. Examining the architecture for these aspects requires several components – first, a large de-identified patient set (>1,000,000 patients with EHR data) is needed. Next, a mechanism to verify the execution results is required to ensure a valid test environment. Finally, as shown in our prior work[22], computational complexity is variable across different quality measures, with examples ranging from simple logical comparisons, to large nested decision trees with temporal relationships between events. We must account for this variability by measuring complexity of the measures under study.

To meet these requirements, the previously mentioned Project Cypress test dataset was re-purposed for execution time performance testing. In order to simulate >1,000,000 patients, the Project Cypress’ 36 patients were “cloned” and re-created to form a larger cohort. Execution time was then measured for 12 different patient set sizes, each set size equaling the Project Cypress test patient set size multiplied by an increasing multiple of 2500. Specifically, given the Project Cypress test patient set P, we can define each test patient multiset T_j as T_j = P ⊗ (2500j) where 1 <= j <= 12. For each set of test patient data T_j, the execution time necessary to compute the test set was measured and recorded.

The complexity of eCQMs under study is an important criteria to quantify, especially when comparing execution times. Several efforts provide guidance in this area[12, 23], from which we derive our complexity attributes: Max Depth, Boolean Operators, Negations, and Temporal Operators. Max Depth measures the maximum amount of nesting in the boolean clauses. If thought of as a boolean decision tree, this measurement would equal the longest path length to the root node. Boolean Operators is a count of every AND/OR operator in the measure, while Negations measures how many of these operators include a negation. Finally, Temporal Operators counts the number of temporal logic calculations specified in the measure – either as relative temporal comparisons between events, or comparisons to the start and/or end time of the measurement period.

Due to brevity, a subset of the complexity measurements is shown in Table 1 (See full details at: http://docs.phenotypeportal.org/figures/htp_complexity.png). To test the computation time of the execution engine under a variety of complexity scenarios, 9 measures representing different complexities were executed. The chosen measures represent a broad spectrum of complexity, ranging from the most complex to relatively straightforward. Execution tests were performed on an Amazon 2xlarge EC2 instance with 8 vCPUs and 30 GiB RAM.

Table 1:

CMS Clinical Quality Measure Algorithm Complexity Metrics

CMS ID	Max Depth	Boolean Operators	Negations	Temporal Operators
CMS159v1	8	41	16	56
CMS160v1	6	46	12	63
CMS153v1	4	110	27	112
CMS145v1	4	84	14	114
CMS126v1	3	57	9	64
CMS134v1	3	26	6	29
CMS163v1	2	10	1	12
CMS127v1	2	8	0	10
CMS148v1	2	4	1	8

Performance (Verification)

Automating the verification of the measure is in many ways as important as the automation of the execution itself. Quality Attribute 2 describes the need for a robust testing environment, and the ability to measure verification performance is key to our architectural evaluation. Verification was performed using the Project Cypress verification §170.314© Clinical Quality Measures document[24]. Project Cypress was used as the testing oracle for the verification process, providing a cu-rated test dataset along with expected outcomes. Automation of this verification process also allowed for Test Driven Development[25] practices to be used. To automate this verification process, first the Project Cypress JSON Patient Data was transformed into the target HTP data representation format. Next, the HQMF/QDM XML representation of a given eCQM was selected, and execution of that measure against the test dataset was started. When complete, the results were compared against the expected outcomes, which were loaded from a provided Project Cypress results file.

3 RESULTS

Performance (Execution Time)

Execution times, shown in Figure 3, show the total elapsed execution time over the test patient sets. Execution times tended to scale linearly as the patient set size increased. The execution time of CMS MeasureID CMS145v1 Beta-Blocker TherapyPrior Myocardial Infarction (MI) or Left Ventricular Systolic Dysfunction (LVEF < 40%) was anomalous, as shown in the results. Although similar in measured complexity compared to other executed measures, it exhibited a significantly higher execution time.

Figure 3:

CMS Clinical Quality Measure Algorithm Execution Time

Performance (Verification)

A subset of the Project Cypress validation results are shown in Table 2. For each measure, results are displayed in the form of (actual/expected) patients classified for the given population. Of the 58 total measures analyzed, 56 matched the expected results, while 2 measures failed to match the expected results in one or more of the population criteria. The two anomalous measures, CMS MeasureIDs CMS159v1 and CMS122v1, both failed to correctly identify the Numerator (NUMER) population. Due to brevity, see detailed results here: http://docs.phenotypeportal.org/figures/htp_cypress.png.

Table 2:

Project Cypress Verification Results – Patients Classified per Population (actual/expected)

CMS ID	IPP	NUMER	DENOM	DENEX	DENEXCEP	Result
CMS122v1	2/2	2/1	2/2	0/0		Failure
CMS126v1	3/3	1/1	3/3	1/1		Success
CMS127v1	5/5	1/1	5/5			Success
CMS134v1	2/2	1/1	2/2	0/0		Success
CMS145v1	2/2	1/1	2/2		0/0	Success
CMS148v1	2/2	1/1	2/2	0/0		Success
CMS153v1	4/4	1/1	4/4	1/1		Success
CMS159v1	2/2	0/1	2/2	0/0		Failure
CMS160v1	3/3	2/2	3/3	0/0		Success
CMS163v1	2/2	1/1	2/2	0/0		Success

4 DISCUSSION

Summary

The evaluation of the candidate architecture begins with ensuring that the Architectural Quality Attributes still hold. By using standards such as QDM/HQMF and CTS2, along with the uniform interface of REST, we were able to achieve a degree of Interoperability in the system. Also, as shown by the Project Cypress integration, the architecture was able to achieve a high degree of Testability, while the MapReduce processing algorithm enables a solid platform for high system Scalability. Our quantitative analysis results also verify that the Quality Attributes hold. Achieving 96.4% Cypress verification (56 out of 58 measures) shows high system “correctness,” while the execution time results show predictable and scalable results for the selected measures given datasets of >1,000,000 patients.

Anomalous Results

CMS MeasureIDs CMS159v1 and CMS122v1 both failed to produce the correct results when executed against Project Cypress test data. For CMS159v1, the numerator population is defined by “Risk Category Assessment: PHQ-9 Tool (result < 5)” <= 13 month(s) starts after end of “Occurrence A of Risk Category Assessment: PHQ-9 Tool.” Cypress verification results identify test patient BH_Adult_D for inclusion, and test data for this patient shows two qualifying assessments, the first on Sat, 24 Sep 2011 12:30:00 GMT and the second on Thu, 01 Nov 2012 12:30:00 GMT. Our results exclude this patient via the <= 13 month(s) temporal constraint. It is unclear as to whether this is a misinterpretation of the temporal constraint or an otherwise unknown system defect.

With regard to the verification failure of CMS122v1, the test patient Diabetes_Adult_A was erroneously added to the numerator population due to a miscalculation of the the constraint: “Occurrence A of Laboratory Test, Result: HbA1c Laboratory Test (result > 9 %).” This patient does have an HbA1c Laboratory Test result, but with a value of only 8%. Inclusion of this test patient indicates a system defect in analyzing the value of this laboratory test result.

The anomalous execution time of CMS145v1 is not sufficiently explained by complexity calculations in Table 1, as it does not differ significantly from other observed measures. One possible explanation could be the large number of individual Drools Rules produced by the compilation process of this measure. To explore this further, Figure 4 shows the number of generated Drools Rules as compared to execution time for each analyzed measure. The results indicate a possible correlation between the two, and shows the execution time of CMS145v1 degrades predictably given the large number of discrete Drools Rules needed to express it.

Figure 4:

Measure Execution Time per 1,000,000 Patients vs. Number of Generated Drools Rules

Limitations

Compilation of the QDM XML into JBoss^® Drools Rules has several significant opportunities for improvement. First, Drools Fusion and Complex Event Processing[7] was only partially utilized for the temporal relationships. Utilizing these modules would cause a significant increase in rule readability and conciseness. Next, the generated rules are prone to combinatorial explosions of rule activations – especially in measures that heavily utilize OR clauses. The exists Drools operator would usually alleviate this, but a unique aspect of the QDM measures called Specific Occurrences makes this difficult. A Specific Occurrence is the notion of assigning an identifier to a specific event (such as a Diagnosis or Encounter), and referring to that specific event elsewhere in the measure. To account for this, we follow the Specific Occurrence processing algorithm proposed by the MITRE Corporation[26]. Execution time performance metrics are also limited by the small Project Cypress patient set size, and as such, a larger test patient set would be more appropriate for scalability and performance evaluations.

Future Work

In general, optimizing the Drools Rule compilation has the potential to increase scalability and performance to even higher levels, as well as to provide more readable and concise rules. Also, the relationship between Complexity in Table 1 and Execution Time in Figure 3 is not fully understood. If a correlation exists, it has not been statistically analyzed. Furthermore, asserting relative complexity rankings between measures is error-prone, as algorithms and metrics for capturing true measure “complexity” are not known. More analysis is needed to explore how this complexity can be measured. Along these lines, Figure 4 suggests that the number of generated Drools Rules – not complexity – may be a more accurate model of execution time. If the correlation does indeed hold, future work in Drools Rule compilation optimizations may be approached with that model in mind.