Extracting Cancer Quality Indicators from Electronic Medical Records: Evaluation of an Ontology-Based Virtual Medical Record Approach

Abstract

Measuring quality in clinical care is a time-consuming manual task. The vast amounts of clinical data collected through electronic medical records (EMRs) create an opportunity to develop tools that automatically assess quality indicators; however, the diversity of EMR implementations limits the ability to implement general, reusable methods. We evaluate an ontology-based virtual medical record (VMR) approach as a standardized, sharable methodology for defining data abstractions needed for quality of care assessment. Using a set of cancer quality indicators, we conducted a requirements analysis for modeling these abstractions with an OWL-based VMR. We found that the VMR approach needs to be extended to support population-based aggregations of clinical events, models of intended versus completed actions, and models of workflow and delivery systems. Incorporating the patient perspective on quality also requires additional extension of the VMR. We are using these results to create a virtual quality record based on EMR data.

Introduction

Despite rapid advances in the therapeutic regimens for cancer, the quality of cancer care continues to vary considerably throughout the United States. Recognizing that variability in care can have a profound effect on the health outcomes associated with cancer, the Institute of Medicine (IOM) put forth a set of recommendations for quality cancer care emphasizing the need for a health care delivery infrastructure that a) is evidence-based, b) has indicators both locally and nationally to assess quality of care, and c) is capable of responding to the quality indicators in a timely manner to improve performance¹.

With the increasing number of Electronic Medical Record (EMR) deployment, the assessment of quality of care information may be greatly improved over the current pen and paper method used at many health care institutions. Despite the many potential benefits of an EMR for quality of care assessment, the challenges of using the EMR for this task are substantial. These include the lack of a standard medical terminology in practical usage, storage of EMR data in a relatively unstructured data format, and differences in database schema across health care institutions².

We are addressing these challenges in the OncoShare project, a joint research collaboration between the Palo Alto Medical Foundation (PAMF) and Stanford University School of Medicine, to share cancer-related data from EMRs and other information systems and to compare effectiveness and quality of cancer care. The informatics research goals of the the project are to create a standard representation of cancer-related data and treatment plans and tools that can automatically generate quality indicator (QI) measures from this information model. The health service research goals of the project are to determine how well physicians at each institution adhere to guidelines of cancer care, what factors are related to adherence, and what patterns of treatment lead to optimal care.

In developing a standard information model for cancer care, we chose the Virtual Medical Record (VMR) approach, a methodology developed successfully in the SAGE project to implement clinical guidelines decision support methods across EMRs ³. We specifically examined whether we could use VMR constructs for assessing quality of cancer care using EMR data. The specific research questions we address in this paper are:

1. To what degree is the VMR, and its class and property constructs, currently able to capture quality of care information from the EMR?

2. What extensions, if any, are required to enable to VMR to abstract quality of care information from the EMR?

Methods

Conversion of VMR from frames to OWL-DL

The VMR was initially built with a frames ontology. The distinct classes of the VMR (Table 1) follow clinical statement formalisms developed for HL7-Reference Information Model(RIM) by the HL7 Clinical Statements Special Interest Group³. It allows information to be shared across health care institutions, particularly when used in conjunction with a standard clinical terminology, as well as limited automated reasoning with its underlying frames logic. To build upon and harmonize with existing ontology development efforts within the biomedical research community, we intend are building ontologies for OncoShare with the OWL-DL formalism. OWL (Web Ontology Language) is built on a description logic foundation that provides formal guarantees when checking the logical consistency of ontologies and when deducing new knowledge. OWL-DL, one of the three sublanguages of OWL, allows for increased expressiveness (such as set-theoretic operations to define membership in complex classes) while maintaining computational completeness and strong decidabililty guarantees. We used Protege-OWL (version 3.4) to convert the frames version of the VMR into an OWL-DL formalism⁴.

Table 1.

Classes in the SAGE VMR

Observation	Procedure
Encounter	Referral
Problem	Agent
Adverse Reaction	Appointment
Substance Administration	Alert
VMR Order	Composite Clinical Model
Goal

One of the authors (WL) then examined each concept in the VMR to convert the slots in the frames ontology to OWL object and data-type properties when necessary and to add in the logical constraints as in the original frames version of the VMR.

We performed a consistency check with the Pellet reasoner which is built in with Protege-OWL.

Selection of Cancer Quality of Care Indicators

The American Society of Clinical Oncology (ASCO) and the National Comprehensive Cancer Network (NCCN) have put forth a set of process of care quality measures that assess the overall quality and effectiveness for breast, colon, and rectal cancer care⁵. These measures (Table 2) were derived by a panel of experts in breast and colorectal cancer care and selected based on feasibility in obtaining the data from hospitals, and the potential for improving population health outcomes. Each measure was then specified in terms of a denominator (the patients who should have had the quality indicator performed), the numerator (the patients who actually had the quality indicator performed), and time intervals in which the quality indicator should be performed.

Table 2.

ASCO/NCCN Cancer Quality Measures

Breast Cancer

Received tamoxifen or AI within 1 year of diagnosis

Started radiation therapy within 1 year of diagnosis

Received adjuvant chemotherapy within 120 days of diagnosis

Colon and Rectal Cancer

Received post-operative chemotherapy within 9 months of diagnosis

Received pelvic radiation therapy for rectal cancer within 6 months

Received adjuvant chemotherapy for stage III colon cancer within 4 months

Had 12 or more lymph nodes removed and examined for stage II or III colon or rectal cancer

AI - Aromatase Inhibitor

Assessing Modeling Requirements

In its definition of quality cancer care, the IOM describes 3 domains for quality measures (1) the structural aspects of the health care delivery system (i.e. case volume); (2) processes of care (i.e. use of adjuvant chemotherapy); and (3) the outcomes of care (i.e., survival). For each cancer quality measure in Table 2, the authors evaluated the VMR in the context of the domains described by the IOM.

Results

The conversion of the VMR frames to VMR OWL-DL did not reveal any logical inconsistencies with the Pellet reasoner. The analysis of the adequacy of the VMR for each domain of quality of care is discussed below.

Structural Aspects of Health Care Delivery

Quality assessment of the health care delivery system requires aggregate clinical data for comparative effectiveness analysis. The VMR is designed to provide a clinical view of data on an individual situation basis, and is not designed for aggregate clinical analysis.

The VMR also does not contain concepts needed to describe the clinical workflow and structural elements of the health care delivery system. Additionally, the VMR is not equipped to model temporal events, thereby limiting the modeling of a temporal sequence of events for quality of care assessment.

Processes of Care

The VMR models events that are documented within the EMR. The events, then, are declarative - an observation is made, or a substance is administered, for example. In evaluating processes of care, however, the intent of the health care agent is as insightful as the actual action that took place. The VMR does not contain concepts to model the intention to provide specific care.

Outcomes of Care

Monitoring health outcomes is an important aspect of response assessment in quality of care. While the VMR contains a concept class for Observation, the instances in that class does not necessarily contain only health outcomes of interest. For example, the observation of a laboratory value for Hemoglobin is not a relevant outcome in hypertension management. The VMR does not contain a specific class for monitoring quality of care related health outcomes.

Discussion

Interpretation of Findings

We found three types of modeling refinements and extensions are needed for the VMR: (1) aggregate clinical events, (2) intent vs. action, and (3) workflow and system issues. In addition, incorporating patients’ perspectives on quality, such as health-related quality of life, requires additional constructs in the VMR.

Modeling Aggregate Clinical Events

Consider the example provision of tamoxifen or a 3rd generation aromatase inhibitor with hormone receptor positive breast cancer. In a typical decision-support scenario, a specific recommendation is made based on the abstracted data elements from the EMR to the VMR. Thus, a physician might receive an alert about a woman with stage I - III breast cancer who has not yet received tamoxifen. From a quality of care perspective, the physician, however, would have no information about whether tamoxifen is provided for all his other patients, or about the performance of other physicians in the practice. The VMR, therefore, must model patterns of care that are discerned from aggregate data.

Our approach to using the VMR to model care patterns from aggregate data is shown in Figure 1. The VMR will abstract data from the EMR for each patient with the specified diagnosis. The concepts from multiple VMRs can then be mapped to concepts in the quality of care information model, which is instantiated by data aggregated from the EMR.

Figure 1.

Information model for quality of care

Each concept within the quality of care information model would essentially represent a table that could be obtained from a structured query of the relational database model that holds the data from the EMR. The VMR currently is able to model demographics, medical problem lists, and medication lists. Table 3 describes the relevant object properties for Clinical Events, an abstraction that would need to be added to the VMR. Clinical events can be random events, such as the onset of breast cancer, in which case hasAgent would not be defined. They can also be initiated by a specific person, as when a medical, surgical, laboratory or radiology procedure is performed. Thus, a physician agent would order tamoxifen for a breast cancer patient. The order is implemented by some other agent (i.e. a nurse), and a desired outcome (i.e. patient took tamoxifen) is measured.

Table 3.

Property extensions for clinical event

Clinical Event (ce)	Restrictions
hasAgent	∃ (Agent)
isImplementedBy	∃ (Agent)
hasVMR_Order	∃ (VMR Order)
hasOutcome	∃ (Outcome)

Although the VMR can capture the agent, order, and outcome concepts, these concepts are spread out in several classes. For simplification, we instead added a single new class (VMR_Event) within our OWL-based VMR to capture clinical events and modeled the agent, order, implementation, and outcome modeled as object properties for this class.

Modeling Intended Versus Completed Actions

The quality indicators listed in Table 1 are process of care measures that describe a procedure, or a set of procedures that should have occurred during the care of the patient. The negation of the quality indicator, however, does not necessarily imply that the process did not occur.

For example, a patient may have a number of justifiable reasons for not receiving radiation therapy within 1 year of diagnosis for breast cancer and breast cancer surgery. These include the death of the patient before radiation therapy could occur, patient refusal to have radiation therapy, non-compliance with a scheduled regimen, or a sudden illness that precludes the use of radiation therapy. The health care provider may actually be compliant with the process of care indicator - but unpredictable events prevented the completion of the process.

It may not be tractable, given the open world assumption in OWL-DL, to model all the possible reasons for a patient not satisfying a process of care quality indicator. Our approach to this problem is to model both the intent of a process as well as the implementation of a process. As shown in Table 2, when an agent makes an order for some procedure (surgical, medical, or diagnostic), an intent for that procedure is explicitly stated. When the order is implemented, and an outcome results, that procedure is completed.

The VMR class VMR_Order is able to capture intent, but does not have a class to assess outcome. From the EMR, an outcome would be either some diagnosis (VMR:Problem) or a procedure (VMR:Procedure). Using the OWL-DL specification, the occurrence of a favorable outcome is necessary and sufficient to state that the intent was present. Using the Semantic Web Rules Language (SWRL) with OWL-DL, this might be reasoned as:

A completed action might be reasoned as:

Modeling Workflow and System Issues

EMRs are limited in the granularity with which intent can be captured. One essential challenge in the performance characteristics of quality indicators is identifying where in the process of care a breakdown occurred. Our solution is to model the workflow for a care process. The contextual situations in the workflow are not necessarily geographic - they can include other contexts, depending on the needs of the user. The OWL-DL formalism also allows context to be described with properties that may suit the needs of the user - for example, a practice setting context for a radiology suite can have the property hasMonthlyVolume to describe the average number of mammograms that are read in that suite per month. Work volume has been described to affect the quality of mammogram interpretations, and is an example of health system descriptors.

Regardless of context, we assume that there is a temporal model that can be attributed to the workflow. By modeling workflow with the process of care, we can localize quality performance deficits to geographic, situational, or temporal contexts, thus simplifying the root-cause analysis of health care researchers, clinicians, or administrators. For example, consider two sample workflow scenarios for the provision of adjuvant chemotherapy within 4 months for Stage III colon cancer (Figure 2). In figure 2A, workflow context is given by physician location. If the patient is not given chemotherapy (as circled in red), then adjuvant chemotherapy cannot be provided. Figure 2B shows workflow context given by situation. In this scenario, if the physician’s office does not remind the patient of an upcoming visit, that patient may miss the appointment during which adjuvant therapy is provided. In each context, workflow specification can be automated to help identify the root-cause for poor performance with the quality measure.

Figure 2.

Workflow scenarios for the provision of adjuvant chemotherapy for colon cancer

Implications of Findings

In our study, we identified class extensions to the VMR that would be necessary to adapt it for use in modeling quality of care from EMRs. We also describe properties that would enable the VMR to use a rules language so that automated reasoning could be done on the EMR. The reasoning would enable health services researchers to classify patients according to care patterns, and to identify those patients who met defined quality of care indicators.

The class and property extensions we describe are important aspects of quality of care assessment. Aggregate patterns of care must be modeled. Intent actions as well as completed actions help define adherence to established protocols. Workflow context is necessary to do root-cause analysis when quality indicators are not met⁶.

Prior studies have assessed the utility of EMRs to assess the quality performance of hospitals, and generally found the EMR to be inadequate in assessing quality of care^7,8. These studies, however, limited themselves to using administrative data from the EMR, without the benefit of a knowledge representation. As such, the authors would not be able to do automated classifications, or define new classes based on reasoning. In short, prior studies found EMRs inadequate because they lacked a descriptive information model for the EMR.

We describe an information model based on the VMR that can be used for clinical decision support at an individual level, and to assess the quality of care at an institutional level. Using the OWL-DL formalism, the VMR extensions we describe can be easily implemented, and modified to meet the needs of the user. For example, patient satisfaction is recognized as an important indicator of quality of care, especially since it is correlated with adherence to clinical regimens and prescribed medications⁹. Although not evaluated as a quality indicator in this study, patient satisfaction can be easily implemented within an EMR, and modeled in the VMR as an added extension.

With the use of rules languages such as SWRL, quality of care can be made specific to the needs of an institution, and the VMR can be used to capture care patterns for internal quality assessments. For example, a hospital that wants to evaluate an outbreak of a nosocomial pathogen such as methicillin resistant Staphylococcus Aureus (MRSA) in the intensive care unit (ICU) might customize the VMR to capture the workflow among the different health care agents in the ICU. With this model, the VMR may detect that the placement of central venous catheters is associated with MRSA infection, thus allowing the hospital epidemiologist to best use resources in identifying the source of the outbreak.

The National Cancer Institute (NCI) along with organizations such as the American Society for Clinical Oncology (ASCO) and the National Comprehensive Cancer Network (NCCN) have expressed great interest in leveraging the capabilities of the EMR for quality of care and health services research¹⁰. With grid computing resources such as caBIG, information in its full richness and context can be captured and analyzed across EMRs. This would effectively create the rapid-learning health system¹¹, in which quality and health services inferences are made automatically and quickly so that individual and policy level decisions can be well informed. We believe that the VMR information model we propose can substantially advance this goal.

Conclusions

Knowledge representation methods can be instrumental towards improving the quality of health care and health care delivery in cancer care. The VMR offers a clinical view of the Electronic Medical Record that can be customized to fit the needs of the health services researcher, or the hospital administrator. The extensions to the VMR that we describe - aggregating clinical events to facilitate data mining, modeling the intent of the health care provider as well as the actions that occurred, and including workflow and structural elements in the VMR model - may improve the capture of rich clinical and operational information.

These extensions have the potential to improve not only care at an individual hospital level, but also health care across institutions, as the use of the Web Ontology Language can facilitate data integration and sharing for multiple health care institutions. Further research needs to be done on the optimal utilization of EMRs for quality of care research and applications.