Evaluation of a Commercial Rule Engine as a Basis for a Clinical Decision Support Service

Abstract

Current implementation techniques constrain our ability to rapidly deploy and augment clinical decision support systems at Partners Healthcare System. We report on the use of a commercially-available rule engine system—iLog Rules—as the basis for a series of prototypes typifying Partners decision support applications. For three prototypical systems, we successfully decoupled the decision support component from application and support logic, and reimplemented that component using iLog. We found that the rule engine itself provides high-performance execution for the small rulesets we evaluated, and that overall application performance was found to be generally acceptable. We note that the major bottleneck to application performance is the ability to rapidly deliver patient data to the rule engine for execution. Future investigation will focus on abstracting features from each of the prototypes and incorporating them into a scalable and reusable decision support service architecture.

Partners Healthcare System has long been an enthusiastic proponent and implementer of clinical decision support systems. Decision support modules permeate the many production systems at the Brigham and Women’s Hospital, the Massachusetts General Hospital, and throughout the Partners network.^1–5 Decision support is internally recognized as the facilitating platform for the implementation of Partners ‘Signature Initiatives’—5 high-level goals to uniformly improve quality, safety, and cost throughout its integrated health care delivery system.

However, we have reached a crossroad in our current implementation of decision support. In general, decision support is implemented on a system-by-system basis, with deployment in a native programming language to support performance and rapid access to data. This has engendered several major hurdles for the organization: decision support rules are not easily accessible to knowledge engineers for maintenance; new rules require programming resources to implement rules and custom data fetches; deployments are not reusable across multiple systems.

To overcome these obstacles, we are investigating the development of an enterprise decision support service (DSS) that would provide a common, centralized mechanism to support the disparate decision support needs of numerous Partners’ systems. As an initial step in this work, we are evaluating whether a commercial rule engine system can provide the features and performance necessary to support an enterprise DSS. In the past fifteen years that Partners has deployed decision support modules, vendor offerings have matured and are used on a large-scale in numerous industries.

In this paper, we report on the use of iLog Rules, a commercially-available rule engine system, in the development of a series of prototypes which typify Partners’ decision support applications. For each prototype, we discuss its functionality and use of the iLog Rules product. We provide a series of performance benchmarks from our prototype evaluations in a variety of test configurations. We discuss the viability of using the iLog product for future production systems and identify key areas for further work.

Methods

iLog Rules

During the development of a reference architecture for Partners clinical systems, a preliminary evaluation of eight commercially-available rule engine systems was undertaken.⁶ (Table 1) This initial evaluation considered 1) feature sets of the rule engines; 2) integration capabilities with external systems; 3) performance and scalability; 4) rule authoring environments; 5) vendor profiles. As a result of this initial analysis, iLog Rules was ultimately selected for pilot evaluation.⁷ The iLog product provides 1) a production rule language; 2) an editing environment for developing production rules using a natural language-like syntax; 3) a rule engine implementing the Rete algorithm with an API for integration with native programming languages (Java, C#).⁸ The rule engine integrates with application data via a ‘Business Object Model’ (BOM)—a user-defined set of classes defining application objects. Rules directly reference application objects in the chosen programming language, supporting a very natural integration with applications.

Table 1.

Products Analyzed in Preliminary Evaluation

Product	Vendor
JRules	iLog
Blaze Advisor	Fair Isaac
CIA Server	Haley Enterprises
Logic Engine	RulesPower
irServer	InRules
PegaRULES	Pegasystems
QuickRules	YASU Technologies
Catalyst	Wellogic

Prototypes

Each of the three prototypes was implemented by individual developers. The developers shared an initial set of base classes for their respective BOMs that provide common access to external data repositories. BOM classes were specialized on an as-needed basis by each developer to meet the individual requirements of each prototype. All prototypes were implemented using the C# programming language. Test patient data for each prototype was drawn from test data used in the development of the respective production system. Evaluation benchmarks were run on stock hardware—a typical system is a Pentium 4 processor at 2.4GHz with 1 gigabyte of memory.

The general goals for each prototype are to demonstrate the following:

• Alert and reminder logic can be implemented using the iLog rule language, deployed via the iLog rules engine;

• A Rules Service can be implemented as an independent component and integrated back into its individual application.

• Separation of rules components from other application logic still results in acceptable performance.

• The Rules service can be scaled to meet production load requirements.

Results Manager

Results Manager is an application that surveys all outpatient laboratory results, and provides alerts and reminders to providers. Results Manager is enabled to perform results analysis with respect to clinical context. For example, Results Manager will create alerts when an INR is subtherapeutic for a patient on coumadin, or for an LDL level with respect to multiple cardiac risk factors. In the production system, rules for recommendations and alerts are implemented within program code, accessible to only development staff.

In the Results Manager decision support prototype, the decision support component has been externalized to a stateless, asynchronous Windows service implemented using .NET technology. The core of the service is provided by a single iLog rule engine. The service integrates with the Results Manager by reading data and posting alerts and reminders via ODBC. The prototype polls the Results Manager transaction queue periodically for new laboratory results which are aggregated into standard laboratory panels. The BOM layer provides integration with backend repository services to obtain patient data.

For development purposes, the decision support prototype was integrated with a development version of the Results Manager. Four rulesets comprising 18 rules were created for INR evaluation, cholesterol evaluation, potassium and creatinine notification, and normal lab processing. For each transaction in the test set, the following flow is executed:

1. The prototype reads all available transactions preprocessed by the Results Manager and writes them into working memory.

2. The Rule Service is executed, obtaining patient data from backend repositories as necessary.

3. Alerts and other status information are written back to the calling application.

eMAR

eMAR is an application that supports the process of inpatient medication administration. The eMAR interacts with nurses at the time of medication administration when a bar-coded medication is scanned into the system. In the production system, rules for recommendations and alerts are hard-coded into the system in a programming language. eMAR supports approximately 250 concurrent users and contains 30 rules.

In the eMAR decision support prototype, the decision support component has been externalized to a stateless, synchronous web service implemented using .NET technology. The core of the service is provided by the iLog rule engine which is wrapped to support necessary application integration. The BOM layer provides integration with backend repository services to obtain patient data. This prototype includes a configurable pool of rule engines to facilitate experimentation with the scalability of this implementation.

For evaluation purposes, the prototype was integrated with a development version of eMAR. Ten of thirty rules were recreated using the iLog rules language and deployed to the service as a single rule set. In the evaluation, for every simulated entry of a medication, the following flow is executed:

1. eMAR makes a web service call to the prototype, passing medication data.

2. The service retrieves a rule engine from the pool, preloaded with the eMAR ruleset, and asserts medication data into working memory.

3. The rule engine is executed, retrieving patient data on an as-needed basis.

4. The service returns any alerts and/or recommendations to the application.

SmartForms

SmartForms is a new application that critiques and recommends alterations in care for patients with diabetes mellitus and coronary artery disease. SmartForms is a new metaphor being evaluated for Partners ambulatory medical record system LMR. The system currently exists in prototype form.

In the SmartForms decision support prototype, the BOM is used both as the integration layer for the rules engine, and as the base classes for application objects. Leveraging this dual use of the BOM, the decision support component is integrated in-line with the SmartForms application, running within the same process. The core of the decision support component—the Context Manager—is provided by a single iLog rule engine.

Eleven rulesets comprising 85 rules were developed for the initial SmartForms prototype. When a context change occurs into a new patient, the following flow is executed:

1. SmartForms retrieves a defined set of patient data from external repositories.

2. A disease-specific display is rendered if the patient has either diabetes mellitus or coronary artery disease.

3. The SmartForms critique and recommendation module is executed if the patient has one or both of the target conditions.

Results

Rule Deployment

Over one hundred rules providing alert and reminder logic for each prototype were successfully redeployed in the iLog rule language. No deficit in rule expressivity was identified by any of the developers.

Results Manager

The Results Manager test harness was run against a set of 2742 panels including 30 INR tests (1.2%), 157 cholesterol panels (5.7%), 4 creatinine/potassium tests (0.1%), and 2551 other tests (93%). Allowable configurations for the rulesets included 1) one single ruleset; 2) multiple rulesets loaded and executed sequentially; 3) a ‘Ruleflow with conditional chaining’—a structure which conditionally branches and executes rule sets based on data values.

Results are listed in Table 2.

Table 2.

Result Manager Performance Varying Ruleset Configuration

Configuration	Single RS	Multiple RS	Ruleflow
Transactions/sec	40.32	41.54	41.54

eMAR

The eMAR test harness was run in three configurations: 1) Web service call with null service execution; 2) Web service call with null external data fetch; 3) normal Web service call. These three variants respectively reflect web service invocation time, service execution time, and service execution time with data retrieval. The test harness was varied to simulate between 1 and 50 clients, each making 207 requests to the service. The service itself was configured with 20 preloaded rules engine threads available for computation. Results are as noted in Table 3.

Table 3.

eMAR Performance Varying Workload and External Data Fetch

Client Threads	Null Service Execution (sec/req)	Null Data Fetch (sec/req)	Normal Service Call (sec/req)
1	0.020	0.022	0.305
20	0.220	0.227	3.2
50	0.55	0.567	8.773

SmartForms

The SmartForms test harness runs a series of ruleset protocols sequentially in order to generate a single recommendation set. Each test patient dataset consists of six months of observation data in addition to medication, allergy, problem, and health maintenance lists. However, because the application and rule service share the underlying business object model, all data is essentially cached prior to execution of the service.

Results for loading and executing individual protocols against test patients are noted in Table 4.

Table 4.

SmartForm Protocol Execution Performance

Protocol	No. Rules	Load Plus Execution Time (msec)
		Rule Load Only	Test A	Test B	Test C	Test D
Lipids	16	300	290	330	310	290
Smoking	4	140	130	130	130	130
BMI	6	130	130	120	120	130
Anti-Plt	10	180	170	200	200	180
ACE/ARB	13	230	220	230	230	220
Beta Blockers	9	180	190	180	180	190
Blood Pressure	11	280	290	280	280	290
Glycemia	12	310	300	320	310	300
Eye Exam	1	230	230	240	240	240
Foot Exam	1	240	250	240	260	240
Immunize	2	260	250	260	260	270
Total	85	2480	2450	2530	2520	2480

Discussion

In prototype, we have successfully decoupled the decision support component from several typical Partners decision support applications, and re-implemented that component using a commercial rule engine. The rule language provided all the necessary metaphors to express our evaluation rulesets. We found that the rule engine in isolation provides extremely efficient execution of rule sets. Overall application performance was found to be generally acceptable with some caveats to be discussed below.

Evaluation of the iLog rule language is beyond the scope of this paper. The rule language itself is general purpose, a rich function set is included, and additional metaphors such as decision tables are provided. Our own internal review of our existing rules revealed that all constructs were amenable to representation using a general purpose rule language.

The iLog rule engine provides excellent execution performance. The most expensive operation in rule execution is loading of rulesets—the internal creation of rete trees. From the SmartForms evaluation, we see that ruleset loading takes on average several hundred milliseconds for small rulesets containing fewer than twenty rules. Once rules are ‘loaded’, execution time is negligible. This is seen in the eMAR data in the negligible time difference between the web service call and rule execution excluding any data fetch (rules have been preloaded into engines). This is also seen in the SmartForm data where rule load times are indistinguishable execution of test data (data has been prefetched from external stores). Using techniques such as pools of preinstantiated rule engines, and preloading of highly-used rulesets, engine execution performance can be highly optimized.

The Results Manager prototype provides outstanding performance, processing approximately 40 transactions/second. Given an acceptable rate in the production system of 2 transactions/second, performance is well in excess of acceptable execution. Some of this speed may be attributable to the composition of the test set—only 7% of the panels cause any of the evaluation rule sets to fire. Currently, only a very limited number of rulesets are implemented in the prototype. Hundreds of rulesets are possible in the real-time surveillance of an observation repository. Performance will need to be reevaluated with a larger number of rulesets where one would expect a greater percentage of panels to trigger rule execution.

One must question the performance difference between the Results Manager prototype (approximately 40 transactions/second) and the eMAR prototype under heavy load (approximately 0.1 transactions/second). The eMAR prototype employs a more efficient implementation where multiple rule engines are preloaded with rulesets—why does performance degrade? What is not apparent in the results is the effect of different test patient datasets on performance. Test patient data in the Results Manager evaluation is quite sparse and has limited laboratory evaluations. On the other hand, test patient data in the eMar evaluation is quite rich and contains six or more months of laboratory evaluations. Because the BOM has a default configuration to fetch 6 months of data, the eMAR performance may degrade if data retrieval is costly.

The performance difference between Results Manager and eMAR raises important architectural considerations. It appears that the major bottleneck in rule service performance is delivery of data to the engine. In order to minimize the number of round trips between a rule service and an external repository, the rule service should be primed with a large swath of patient data. It is critical that the data repository be capable of efficiently serving up a large block of patient data. Otherwise, the repository effectively degrades decision support performance. It will be necessary to further investigate methods for bulk data retrieval in order to assure a scalable infrastructure for an enterprise DSS.

Finally, it is interesting to consider the role of the BOM in the multi-institution interoperability of clinical decision support systems. A BOM provides two separable components: a model of a set of application classes and the implementation of that model. Rulesets are dependent on the BOM model—the model in effect specifies the terminology to be used within rules. On the other hand, the BOM implementation encapsulates external dependencies with the clinical infrastructure external to the decision support system. Therefore the concept of the BOM provides an interesting solution to the so-called ‘curly-brackets’ problem.⁹ Multiple institutions may share a common BOM model, but provide alternate implementations suitable to their individual environments, thereby facilitating inter-institutional sharing of rules.

Conclusion

We have successfully developed a series of typical decision support prototypes for our clinical environment where the decision support component is decoupled from application and support logic. We found that a commercially-available rule engine system can provide both the features and performance necessary to support this decoupled architecture as we consider revision of existing, tightly-bound, programmatic-based decision support systems. Key to promoting these findings from prototype to viable systems will be the co-development of an efficient retrieval system for moving large blocks of patient data to the decision support engine on-demand. Our future investigations will focus on abstracting high-performance features from each of our prototypes and incorporating them into a general-purpose, reusable decision support service.