Clinical Decision Support-based Quality Measurement (CDS-QM) Framework: Prototype Implementation, Evaluation, and Future Directions

Abstract

Electronic quality measurement (QM) and clinical decision support (CDS) are closely related but are typically implemented independently, resulting in significant duplication of effort. While it seems intuitive that technical approaches could be re-used across these two related use cases, such reuse is seldom reported in the literature, especially for standards-based approaches. Therefore, we evaluated the feasibility of using a standards-based CDS framework aligned with anticipated EHR certification criteria to implement electronic QM. The CDS-QM framework was used to automate a complex national quality measure (SCIP-VTE-2) at an academic healthcare system which had previously relied on time-consuming manual chart abstractions. Compared with 305 manually-reviewed reference cases, the recall of automated measurement was 100%. The precision was 96.3% (CI:92.6%-98.5%) for ascertaining the denominator and 96.2% (CI:92.3%-98.4%) for the numerator. We therefore validated that a standards-based CDS-QM framework can successfully enable automated QM, and we identified benefits and challenges with this approach.

Introduction

Overview of Clinical Quality Measurement (QM)

Delivering quality healthcare is challenging due to ongoing and ubiquitous variation in health system processes that may lead to errors.¹ Measuring and reducing variation from evidence-based clinical best practices have been shown to improve quality and decrease costs of healthcare.² Despite multiple efforts undertaken to improve healthcare quality since the publication of the Institute of Medicine’s reports entitled ‘To Err Is Human’³ and ‘Crossing the Quality Chasm’⁴, the quality of healthcare in the United States continues to be compromised by unnecessary variation in the implementation of clinical practice guidelines. Having a means to assess healthcare quality is essential for identifying deviations from evidence-based best practices and mitigating preventable errors.³ Clinical quality measures are measures of processes, experience, and/or outcomes of patient care. There are increasing mandates and financial incentives to use electronic health records (EHRs) to measure quality as opposed to employing traditional manual processes for QM.^3,4 For example, the Meaningful Use (MU) recommendations issued by the federal Health Information Technology Policy Committee (HITPC) in November 2012 require the implementation of QM as well as decision support for high-priority conditions, and the use of related standards.^4,5 The practice and value of quality measurement has evolved over time. According to Meyer et al., “in the last half century, the US has gone from defining quality, to measuring quality, to requiring providers to publicly report quality measures, and most recently, beginning to hold providers accountable for those results”.⁶ The National Quality Forum (NQF) was created as a public-private partnership to guide decisions regarding quality measure selection.⁷ Until recently, quality measurement has relied mainly on the use of electronic claims data, manual chart abstraction, and patient surveys.⁸ Currently, QM is required by public and private payers, regulators, accreditors and others that certify performance levels for consumers, patients and payers.⁶ Current quality measurement systems in many hospitals include time-consuming manual paper and electronic record abstraction by a quality improvement specialist.^9,10

At large academic medical centers such as University of Utah Health Care (UUHC), manual data abstraction is often followed by data analysis by an external organization such as the University HealthSystem Consortium.^11,12 University HealthSystem Consortium is an alliance of 120 academic medical centers and 299 of their affiliated hospitals representing academic medical centers with a focus on quality and safety excellence.^13,14 Manual chart abstraction at UUHC is performed by the Quality and Patient Safety Department, which has 28 employees, including 12 Quality Improvement Specialists.¹⁵ There are several limitations with this process. For example, (a) three to six months may elapse between the time of a clinical procedure (e.g., a surgery) and the time when feedback is given to a clinician; (b) human errors may be introduced during manual record review; and (c) only a subset of the clinical events is oftentimes selected for review, leading to gaps in quality assessment coverage.

Previous Work in Automating QM

One of the promises of implementing EHRs is the possibility for automatic generation of QM.¹⁰ A MU-certified EHR must be able to export standardized quality reports, which can then “be fed into a calculation engine to compute various aggregate scores”.¹⁶ Following these recommendations, major EHR vendors such as Epic have started to integrate QM logic into their products.¹⁷ Currently, however, only some EHR vendors offer quality measurements embedded in their system, and the scope of measures supported is not always comprehensive.^10,17 For example, KPHealth Connect has automated six of 13 Joint Commission measurement sets, and Epic has automated 44 NQF quality measures.^10,17 Vendor-based solutions may offer ‘full sample’ analysis, but the logic may be a ‘black box.’ Also, it is not always clear which version of each rule has been implemented or whether the quality measure logic is up-to-date. In addition, users may not have control over the logic to customize quality measurement. Even so, automated QM has the potential to provide quality reports on demand, may avoid human errors in manual abstraction, and can analyze 100% of patient encounters. Most ongoing efforts to produce automated quality measures are tied to a specific EHR system, and the executable logic for the quality measure is not sharable between different EHR systems.¹⁰

The Problem: Duplicative, Divergent Implementation of QM and CDS

It is intuitively obvious that CDS and QM are highly related, as QM focuses on who is eligible for a needed intervention (denominator identification) and who among them has received the needed intervention (numerator identification), whereas CDS focuses on who is eligible for a needed intervention and has not received the needed intervention (equivalent to numerator identification). However, to the best of our knowledge, there have been limited evaluation and validation in the literature of how technical approaches for one problem space can be re-used in the other, especially for standards-based approaches. This is important, because EHR certification criteria will likely drive much work in this field. It has been suggested that the two could be combined.¹⁸ There have been efforts to combine CDS and QM logic, but the efforts were not standards-based and no conceptual framework was developed.^19,20

Potential Solution: Leverage a Standards-based CDS Web Service across a Population for QM

Kawamoto et al. have previously suggested that a standards-based, service-oriented architecture could be used to make CDS logic sharable between different EHRs.²¹ In this study, we hypothesized that this approach could be extended to encompass both CDS and QM given similarities in their functional requirements.

In pursuing this potential approach to CDS-based quality measurement (CDS-QM), a promising resource to leverage is an open-source, standards-based, service-oriented framework for CDS known as OpenCDS.²² As shown in Figure 1, an EHR system can submit patient data to OpenCDS and obtain patient-specific assessments and recommendations that are provided to clinicians via alerts, reminders, or other CDS modalities.²³ OpenCDS is compliant with the HL7 Virtual Medical Record (vMR) and HL7 Decision Support Service (DSS) standards, and it leverages various open-source component resources, including the JBoss Drools knowledge management platform and Apelon Distributed Terminology System. Theoretically, then, OpenCDS could be used to measure quality as well as provide CDS. Moreover, the use of a CDS-based QM approach could potentially provide advantages for quality improvement compared to traditional approaches. Therefore, the objectives of this study were to: a) identify opportunities to enhance quality improvement using CDS-QM, b) design and implement a CDS-QM approach aligned with candidate CDS standards for Meaningful Use,²⁴ and c) evaluate the CDS-QM approach for a representative quality measure.

Figure 1.

OpenCDS architecture: high-level interaction for CDS

Methods

Identification of opportunities to enhance quality improvement using CDS-QM

We engaged specialists from the Quality and Patient Safety Department to identify strengths and limitations of the CDS-QM approach. We documented the current process of quality assessment and reporting, and interviewed two of the 28 quality improvement specialists to identify possible ways in which the CDS-QM approach could enhance the institution’s capabilities related to quality measurement and improvement.

Design and implementation of CDS-QM framework

Functional requirements

For the purposes of this implementation, our requirements were to enable the evaluation of national quality measures across the relevant patient populations in UUHC. The primary requirement was accurate evaluation of quality measure compliance.

Design principles

In designing the CDS-QM framework, a core design principle was standards-based scalability, so that the framework could potentially be leveraged in the context of various institutions and information systems. Related to this principle, a second principle was open availability, with open-source tooling used as to limit barriers to adoption related to licensing and intellectual property restrictions.

Scope and assumptions

Scope was limited to the analysis of structured data, as opposed to free text data requiring natural language processing. It was assumed that relevant patient data are available, such as in a data warehouse.

Tools and resources

In addition to OpenCDS, we leveraged the open-source Mirth Connect integration engine (v3.0.1). We also leveraged the UUHC data warehouse (DW), which contains data from the EHR systems and other ancillary clinical and administrative information systems at the institution.

Evaluation of CDS-QM approach for representative national quality measure

Quality measure

We chose the Joint Commission’s Surgical Care Improvement Project (SCIP) Venous Thromboembolism 2 (SCIP-VTE-2) quality measure to evaluate the CDS-QM approach. This measure was chosen due to its technical complexity and its prioritization by the UUHC Quality and Patient Safety Department. VTE is a major cause of morbidity and mortality in hospitals.²⁵ In spite of evidence of their effectiveness, VTE prophylaxis by anti-coagulation and/or mechanical compression remains underutilized in US academic medical centers, particularly among surgical patients.²⁵ SCIP-VTE-2 is a well-established quality measure and is supported by level 1a evidence.^26,27 This measure is used to assess the percent of surgery patients that receive appropriate VTE prophylaxis within 24 hours of surgery.²⁸ For this evaluation, we implemented the SCIP-VTE-2 quality measure using the logic published for surgeries that occur in 2014 (version v4.2a).²⁸

Data Sources

We used clinical data generated by inpatient surgeries that occurred at UUHC in 2013. A total of 8,924 cases were assessed using the CDS-QM automated method. The data elements required by the quality measure logic were documented in multiple source systems, including the inpatient EHR (Cerner) and two systems used to document anesthesia and nursing activities during the surgical event. The University of Utah Institutional Review Board performed an administrative review of this project and determined that IRB approval was not required because this effort was conducted for the purposes of quality improvement and does not meet the regulatory definition of human subject research.

Evaluation/Validation

As a reference standard for validation, we used quality measurement results produced by the University HealthSystem Consortium through an analysis of data extracted through manual chart abstraction by the Quality and Patient Safety Department. The sample used for validation was the 319 surgery cases randomly chosen for abstraction by the University HealthSystem Consortium for the first two quarters of 2013.

As the first step in our analysis, we evaluated the degree to which the data required for the evaluation was available in a structured format in the UUHC DW. Second, we compared the results from OpenCDS with the results from the reference standard approach (manual chart abstraction followed by University HealthSystem Consortium analysis). Observations were classified as true positive (TP), true negatives (TN), false positives (TP), and false negatives (FN) for denominator and numerator criteria separately. We calculated recall (sensitivity) of the OpenCDS-based process as the proportion of cases classified as positive by OpenCDS among the cases classified as positive by the reference standard (TP/(TP+FN)). We calculated precision (positive predictive value) of the OpenCDS-based process as the proportion of cases identified as positive by OpenCDS which was also classified as positive by the reference standard (TP/(TP+FP)). We assessed recall and precision for the classification of denominators, as well as recall and precision for the classification of numerators among cases that met denominator criteria. We also assessed the proportion of cases that yielded a complete match with the reference standard. Exact (Clopper-Pearson) confidence intervals (CI) were estimated for all binomial proportions. Statistical analysis was conducted using SAS version 9.3 (SAS Institute, Cary, North Carolina).

Results

Opportunities to enhance quality improvement using CDS-QM

As shown in Figure 2, the current quality assessment process at UUHC starts with data from the DW about clinical events (in this case major surgeries) being reported to the external quality benchmarking organization (the University HealthSystem Consortium), followed by this organization choosing a sample of surgery cases for manual chart abstraction. The UUHC quality specialists then perform manual abstraction of data from the EHR for the selected records. Finally, the UUHC specialist gives the information back to the external quality organization for summarization and reporting. The entire process from the time of surgery to quality reporting can take up to 6 months.

Figure 2.

Comparison of traditional and CDS-QM approaches to quality measurement and improvement

Diagramming the traditional process for quality measurement and reporting was useful for identifying several opportunities using the CDS-QM approach to enhance workflow and impact clinical care. The informaticists and specialists with the Quality and Patient Safety Department identified several potential improvements. The automated approach using the CDS-QM strategy could:

• Improve the timeliness and completeness of feedback to the clinical stakeholders. The Quality Department holds monthly meetings with clinical stakeholders. Using the CDS-QM approach, a summary of the quality measure results from the previous month could be made available at these meetings to enable more rapid responses to identified quality deficiencies. The current reporting process allows such quality deficiencies to go unnoticed for potentially many months.

• Improve the completeness of assessment and feedback. While some records may still require manual review due to missing data (see Results), the vast majority of cases can be evaluated in an automated manner, as opposed to the baseline sampling approach. This more complete approach could potentially identify problem areas not yet sampled and therefore not yet identified by the quality team.

• Enable quality and clinical stakeholders to assess the impact of new rules or different versions of rules. This functionality is not available with the current process. At the same time, this functionality is critical for performing longitudinal analysis or for making predictions about future compliance.

• Provide additional useful information. A traditional quality measurement approach only identifies whether patients met denominator and/or numerator criteria. In contrast, the CDS-QM approach allows one to generate intermediate results that may be useful for better understanding the root cause underlying any deficiencies.

Design and implementation of CDS-QM framework

Figure 3 provides an overview of the CDS-QM approach. First, the Mirth Connect interface engine was used to identifying relevant surgery cases for analysis. Second, Mirth Connect was used to sequentially obtain relevant patient data from the DW. Third, the relevant patient data were converted into the HL7 vMR format used by OpenCDS. Then, Mirth Connect transmitted the vMR input data to OpenCDS using the SOAP Web service interface specified in the HL7 Decision Support Service (DSS) standard.

Figure 3.

Major systems and processes involved in the CDS-QM approach

Within OpenCDS, the local and standard codes provided as the input were mapped to the internal concepts used by OpenCDS. The data were then evaluated by executing the relevant OpenCDS knowledge module, and the resulting patient-specific assessments were returned to Mirth Connect as vMR output objects. Finally, DW tables were populated with evaluation results by Mirth Connect.

Human workflows for developing the necessary business logic in Mirth Connect and OpenCDS were developed as well. These workflows involved identifying and characterizing the required input data, creating database queries in Mirth Connect, developing the required terminology mapping files, and developing the data processing algorithms in OpenCDS.

Evaluation of CDS-QM approach for representative national quality measure

The NQF-endorsed SCIP-VTE-2 quality measure was automated. Business logic was presented in the OpenCDS Web-based knowledge engineering platform, which uses the JBoss Guvnor platform. Figure 4 shows a sample screenshot from this knowledge engineering platform.

Figure 4.

SCIP-VTE-2 Business Logic represented in Guvnor

Mirth queries were able to access all the necessary data elements except documentation about participation in clinical trials and the use of preadmission oral anticoagulant therapies. These two data elements were not recorded in the DW and would require manual review of the EHR text-based records for complete assessment. Among the 319 cases in the reference population selected by University HealthSystem Consortium, 14 did not have complete records stored in the DW and could not be used for automated QM. Completeness of the records was estimated as 95.6% (CI: 92.8%-97.6%). The remaining 305 cases were used to compare OpenCDS results with the reference standard.

When classifying denominators (i.e., identifying cases that should be included for quality measurement), the OpenCDS strategy yielded a recall of 100% (CI: 98%-100%) and precision of 96.3% (CI: 92.6%-98.5%) (see Table 1). A total of 183 cases among the 305 cases selected for review were found to meet the inclusion criteria. The seven cases included in the denominator using OpenCDS but excluded by the reference standard were cases with preadmission oral anticoagulant therapy. These cases were identified by the manual review process but not found by Mirth because the data is not currently saved in the DW.

Table 1.

Comparison of the denominator (inclusion/exclusion) classification, using two methods (n=305)

CDS-QM approach (based on automated review using 2014 logic)	Reference Standard (based on manual review using 2013 logic)
	include	exclude
include	TP=183	FP=7*
exclude	FN=0	TN=115

^*had preadmission oral anticoagulants

Similarly, when classifying numerators (i.e., identifying cases that failed the quality measure among the 183 cases that met the denominator criteria), the OpenCDS strategy yielded a recall of 100% (CI: 97.9%-100%) and a precision of 96.2% (CI: 92.3%-98.4%) (see Table 2). The seven cases that passed following evaluation of the 2014 version of the SCIP-VTE-2 logic were cases that, in fact, failed according to the 2013 SCIP-VTE-2 specification version of the logic used for the manual review. These seven cases illustrate VTE-prophylaxis practices that were previously considered insufficient, but will be considered sufficient using the 2014 criteria. Thus, a complete match was found for 291 out of 319 records (91.2%; CI: 87.6%-94.1%).

Table 2.

Comparison of the numerator (passed/failed) criteria, using two methods (n=183)

CDS-QM approach (based on automated review using 2014 logic)	Reference Standard (based on manual review using 2013 logic)
	passed	failed
passed	TP=176	FP=7*
failed	FN=0	TN=0

^*passed using 2014 logic

Discussion

We successfully prototyped an implementation of a CDS-QM-based system and demonstrated its feasibility. This use case demonstrates that, once the quality and CDS standards are fully aligned, the opportunity for meeting both goals using the same approach is quite feasible. We also identified special considerations for QM, such as the need to efficiently obtain and process data for a large cohort of patients, even while evaluation is conducted on a patient-by-patient basis.

As we engaged in the effort with the experts form the Quality and Patient Safety Department, we identified many opportunities to enhance quality improvement using CDS-QM. The currently used and proposed systems could complement one another. A CDS-QM system has the potential to be a cheaper and more efficient alternative to the analysis of quality measures relying on manual chart abstractions. However, it has a high initial cost for translating rules into executable format. Automation of the process of translating the measures into an executable format is a next logical step. Once translated into executable format, the logic can be shared with other institutions, and the logic can be modified to meet clinical needs to assess alternative or future quality measure specifications. The quality experts were particularly interested in the opportunity to apply the new SCIP VTE-2 logic specifications for 2014 to data generated from clinical practice occurring in 2013, which revealed that the new logic would reclassify their previous ‘failures’ as passing the new quality criteria. If the new logic had the opposite effect, it would be extremely helpful for a quality program to be able to anticipate ‘failures’ before they get reported six month later, as would occur using the traditional approach of manual abstraction and review by an external quality organization.

The CDS-QM approach is aligned with candidate CDS standards for Meaningful Use Stage 3. Standards proposed for 2015 voluntary EHR specification criteria were used²⁴. We were able to use OpenCDS to implement quality measure logic published by the National Quality Forum, and we generated results that were either an exact match or could be explained where differences were observed. Many institutions use SQL queries instead of a CDS-QM approach because it requires less initial effort. However, maintaining an external rules repository is easier than when rules include a direct reference to the EHR or DW data, as there are no dependencies on the local database structure (i.e., the ‘curly braces problem’).

To maximize benefits from the use of a CDS-QM approach, high-quality structured data are necessary. Validation of the SCIP VTE-2 quality measure implementation using the CDS-QM approach highlighted gaps in documentation and problems with the transfer of information from source systems and the UUHC DW. These findings have been shared with the DW and EHR teams, and the feedback is being used to improve processes and data being extracted into a surgery data mart. These feedback loops are important for improving data completeness and concordance. For data that was available in the DW, our automated approach shows favorable results compare to other automated approaches. For example, Kern et al. report that recall of electronic reporting ranged from 46% to 98% per measure, and precision from 57% to 97%.²⁹

The CDS-QM approach may have limitations. Quality measures are often dependent on claims data, which are not usually available in real-time. More analysis is required to evaluate the timeliness of the availability of all the data required to implement quality measure logic. In addition, this study was performed in only one setting and focused on only one rule which may limit the generalizability of the results. However, OpenCDS uses a standard HL7 data model (the Virtual Medical Record), which potentially would allow the quality measurement rules to be implemented across systems. Additional research is necessary to demonstrate the CDC-QM approach in other EHRs and for other institutions.

In the future, we will automate more quality measures and provide additional feedback to the Quality and Patient Safety Department at UUHC, and the output metrics will be incorporated into dashboards that assess the cost and quality of care at UUHC. We are also working on modifying the OpenCDS infrastructure to support population-based queries. Instead of building one vMR at time, we are in the process of assembling thousands of vMRs at the same time. This approach to building vMRs should enable our approach to scale to quality measurement involving much larger patient samples, such as health maintenance measures for the general outpatient population.

Conclusion

In this study, we presented a prototype of the CDS-QM approach which can add value to the traditional quality reporting approaches. The CDS-QM approach allows for full case coverage, prospective use, near real-time evaluation, and is based on standards. The benefits of using CDS-QM include sampling a higher proportion of data, avoiding human error, saving abstractor time, providing more control over measurement rules, and independence from the EHR or other data source systems. To the best of our knowledge, this is the first study illustrating a framework and an approach for using an open-source, system-agnostic, standards-based CDS tool for continuous quality measurement.