Demonstrating the Effects of an IAIMS on Health Care Quality and Cost

Abstract

The importance of demonstrating the effect of integrating electronic medical records into clinical practice, and methods for conducting the studies necessary to do so, are presented as a model that may be applicable to other aspects of the Integrated Advanced Information Management System (IAIMS). Integrated electronic medical record (EMR) systems offer the prospect of both improving the quality of health care by reducing variation in processes and outcomes and lowering its costs. Because such systems are expensive and require time-consuming re-engineering of health care delivery, demonstrating effectiveness should be part of system development. The expected benefits should be demonstrated using the most rigorous study design that the local clinical environment can support. Results of useful studies include both processes and outcomes of care, the latter including both objective and subjective measures. Comprehensive testing of EMR innovations requires a multispecialty team of investigators, adequate funding, and a commitment of both informaticists and clinicians. Demonstrating the beneficial effects of integrated EMR systems will facilitate their incorporation into everyday clinical care.

The goal of Integrated Advanced Information Management Systems (IAIMSs) is to improve both health care decision making and health care outcomes by integrating information from a variety of sources into work processes. Although the need for such an infrastructure may seem obvious, it is expensive, and the barriers to redesigning workflow to take advantage of the technology can be daunting. It is imperative that we demonstrate the effects of IAIMS and other information technology innovations on both cost and quality. Only then will we be able to make decisions about when to invest in the technology and when to follow alternative strategies. This paper analyzes the reasons for documenting the effects of integrating electronic medical records (EMRs) into clinical practice and the methods for conducting the required studies as a model that may be applicable to other aspects of IAIMS.

EMRs and the Quality and Costs of Health Care

The rapid inflation in health care costs has stimulated a search for ways to reduce costs while holding the line on, or even improving, the quality of care. Integrated EMR systems have been touted as one way of reaching this Holy Grail of “more for less.” ¹ Among the anticipated benefits of EMRs is the ability to quickly move information within and between institutions so that decisions can be more accurate and timely. For example, urinary tract infections could be treated more accurately if the patient's prior urine cultures and risk factors for having a complicated infection, along with the hospital's recent antibiotics sensitivity patterns for urinary pathogens, were available to the care provider. Decisions for emergency room patients with chest pain could be made more quickly if results of prior electrocardiograms, chest radiographs, and cardiac catheterizations were routinely available.

Other investigators have reported remarkable variability in the provision of health care that cannot be explained by differences in patients. ^2,3,4,5 This has encouraged the wider use of practice guidelines in an attempt to reduce variation, increase health care quality, and lower the costs of care. ^6,7,8 EMRs represent perhaps the only practical way to invoke practice guidelines while clinicians are delivering care to patients, when invoking such guidelines has the greatest effect on provider decision making. ⁹ For such interventions, EMRs can serve as both the platform for intervening and the source of outcome data. ^10,11

The purchasers of care who have interests in quality and costs include patients, employers, insurers, and state and federal government agencies. All of these groups focus on different aspects of health care, and EMRs should meet all of their needs. Patients, for example, seek high-quality care, affordable premiums, choice of personal physicians, convenient services, and cost-effective care. Employers are most interested in low costs, few patient complaints, and predictability of both costs and outcomes. Insurers and government payers want high-quality care along with low costs and happy customers (i.e., employers and patients).

Providers of health care also value high-quality care ; yet current exigencies have caused them to become more cognizant of its costs. They also desire autonomy, however, and the ability to provide high-quality care of their own definition that is individualized to their patients' needs. That is, they want to avoid “cookbook medicine.” Any EMR purporting to serve health care consumers must serve the needs of providers as well and be seen as augmenting care, not creating barriers to its provision.

Demonstrating the Benefits of EMRs

Although it is a formidable task, satisfying all of the parties concerned can be facilitated by integrated information systems that freely exchange patient data and medical knowledge. Such systems, however, are expensive to install, and many of their components have not, to date, been demonstrated to improve the quality of care or lower its costs. Purchasers of such systems should have some notion of the degree of benefit they can expect. Skepticism about EMR benefits is healthy because, despite their promises, not all computer innovations have proven to be effective in either improving quality or lowering costs. ^12,13

Efficacy versus Effectiveness

Like most health care innovations, EMRs have been mostly carefully studied in academic health care systems, where the chances of demonstrating their efficacy is greatest. Although necessary to encourage subsequent testing by providing evidence of effect in these most supportive of environments, efficacy studies should be replicated, if possible, by effectiveness studies performed in more diverse (and usual) health care settings. For example, McDonald and coworkers demonstrated that computer-generated reminders improved compliance with preventive care guidelines by internal medicine residents and their attending faculty internists. ¹⁴ The information system studied was contained in a cluster of minicomputers at a county hospital affiliated with a major university medical center. ¹⁵ Subsequently, McPhee et al. demonstrated that similar reminders generated by microcomputers placed in the offices of community physicians also increased cancer screening. ¹⁶

Study Methods

Comprehensive, integrated EMRs are intended to broadly improve the delivery of health care. Therefore, the “laboratory” for studying them must be the clinical arena itself.

Performing studies in such settings requires the longterm commitment of clinicians, administrators, and, in some cases, educators. At Indiana University, ¹⁷ the directors of a large academic primary care general internal medicine practice have maintained for almost 20 years a “laboratory” for testing outpatient informatics innovations of the Regenstrief Medical Record System (RMRS). ¹⁵ Examples of this support include using the RMRS to schedule all outpatient visits and procedures ; establishing identical, adjacent practices with separate physicians, staff members, and patients ; allowing new physicians to be randomized to open practice sessions ; insisting that innovations be used (e.g., honoring only test orders that were entered on microcomputer workstations when the workstations were first installed) ; expecting innovations to be studied with randomized, controlled trials ; and being willing to subsume effective interventions into the practice. In return, these studies double as required quality-improvement studies, and RMRS information is provided for educational purposes and strategic planning.

Even with such support, there will still be tradeoffs between rigor and practicality when designing trials of informatics interventions. Randomized, controlled trials (RCTs) are the most rigorous method of studying EMRs. ¹⁸ They control for coincidental changes in the health care environment that would otherwise confound study assessments. But performing RCTs requires the sustained commitment of clinical directors, who must be willing to insist that new aspects of EMRs become the default means of providing care while withholding them from control providers. Such studies require the physical separation of practice units, to minimize contamination, and random assignment of patients and/or providers to these units. Maintaining such a practice “laboratory” is difficult, yet once established it can be used in a series of controlled trials where successful interventions become part of the “usual care” background against which subsequent innovations are tested. ^17,19

In many instances, the clinical “laboratory” will not support truly randomized trials. Although less rigorous than RCTs, nonrandomized controlled trials can provide valuable information that can support (or reject) new information technology. For example, if the number of independent practice units is too small, or if the clinical directors simply will not permit the constraints required for RCTs, informatics innovations can be introduced serially in a staged manner into multiple sites. The order of installing the new technology can be randomized while concurrently collecting information from all study sites, thus providing concurrent control data to compare with data from subjects at the intervention sites. The downside of this design is that there is potential confounding between the intervention and sites (i.e., a positive effect at one site may be due to idiosyncrasies of that site rather than a direct effect of the intervention). To some extent, fortunately, such differences in sites (e.g., differences in practitioners, patients, or management) can be controlled for statistically. The key here is the collection of identical information in the same manner simultaneously from both intervention and control sites, preferably by evaluators who are blinded to the intervention or control status of the patients, providers, and/or sites.

Time-series (before-and-after) studies, though less rigorous still, can provide valuable information when the innovation being studied is so dramatically superior to the control condition that further controlled trials would be unwarranted or even unethical. For example, if a time-series study showed that a system that provided results of cardiac enzyme studies and prior electrocardiograms to emergency rooms dramatically reduced the time to delivering thrombolytic drugs to patients with myocardial infarctions, additional studies would not be needed (or justifiable). As part of another study, we found that entering hospital admitting orders in the emergency room on microcomputer workstations reduced the time to the first administration of inpatient drugs from six hours to 30 minutes. ^{20 unpublished data} This effect was so dramatic that performing a separate controlled trial of the effect of such an intervention on clinical outcomes would not be justifiable. The key here is that differences must be dramatic for time-series studies to satisfactorily establish effectiveness. It is important to note that as the rigor of a study's methods declines, its power to discern a clinically significant difference also declines dramatically.

By far the weakest study design is the purely observational study. For such studies, EMR innovations are placed into the clinical environment and selected effects are measured, with little or no prior direct assessment of cogent outcomes. For example, a new system for scheduling patients might be placed in a clinic, with subsequent measurement of patient waiting times and satisfaction. Without comparisons, such data have limited value in helping others decide whether such a system is worth the investment of money, time, and effort.

Outcomes of Interest

To be most useful, studies of new technology should measure all relevant outcomes of interest. ²¹ For EMR innovations, the primary outcomes of interest are costs, quality of care, the ratio of costs to clinical outcomes (cost-effectiveness), and provider satisfaction.

Although information about health care charges is fairly accessible, data on true costs are harder to come by. For an isolated site or aspect of care, it may be possible to perform a formal cost analysis, taking into consideration the costs of personnel, facility maintenance and depreciation, and consumables (e.g., medications, reagents, and electricity). For most studies, however, only charges are available. Fortunately, as a result of the Tax Equity and Fiscal Responsibility Act (TEFRA), the Health Care Financing Administration requires each hospital to report cost-to-charge ratios annually for each hospital cost/revenue center. ²² These ratios can be used to estimate true hospital costs in a manner that, although far from perfect, is comparable between institutions. No such method exists for estimating outpatient costs, however. Physician fees can be estimated by assigning Relative Value Units (RVUs) and their appropriate conversion factors to individual visit and procedure CPT4 codes. ²³

A critical aspect of a cost analysis is its perspective. If the patient's perspective is taken, then the cost studied should be the sum of deductibles, copays, and out-of-pocket expenses. These would be considered direct costs. Indirect costs might also be assessed, such as the cost of missed work or the cost of providing inhome care. If the health care provider's perspective is taken, then the focus is profit : reimbursement minus the costs of doing business. If the analysis takes the insurer's perspective, then the cost is the allowable charges minus deductibles, copayments, and coinsurance. Finally, the analysis can take society's perspective. This is the hardest to define because it must take into account all profits and losses in what is ultimately a zero-sum situation. However, most societal analyses take the perspective of the ultimate payer (the total paid by all consumers : patients and payers). For many strides, it is appropriate (and most helpful) for the analysis to take multiple perspectives by repeating the analyses using varying assumptions. If the results are not affected by the perspective of the analysis, then the results are more robust and more likely to represent reality. Importantly, the cost of the EMR intervention itself should be factored into its overall cost or cost savings.

For the effectiveness side of the cost-effectiveness equation, the relevant outcomes are improvements in clinical care, which can be measured as both health care processes and outcomes. Ultimately, improvements in health care should affect the health and wellbeing of its consumers. For many outcomes with low prevalence rates, however, meeting accepted standards of health care delivery must suffice. For example, it is not reasonable to expect every EMR intervention aimed at increasing mammographic testing to prove it lowers breast cancer mortality. But even assessing compliance with guidelines is not always easy. Calculating compliance rates (i.e., the number of eligible instances where the appropriate care was delivered) may ignore appropriate reasons for noncompliance if such data are not captured. ²⁴

Whenever possible, however, clinical outcomes should be assessed. Such outcomes fall roughly into two categories of clinical measures : objective and subjective. Examples of objective measures include clinical events (e.g., myocardial infarction, the occurrence of breast cancer, and death) and physiologic measures (e.g., blood pressure, weight, and glycohemoglobin). Unfortunately, only rarely have EMR interventions demonstrated clinical effects. Exceptions include the interventions described by Rind et al., who showed that reminders to monitor renal function in hospitalized patients taking potentially nephrotoxic drugs resulted in lower serum creatinine levels, ²⁵ and McDonald et al., who demonstrated less pulmonary morbidity during an outbreak of influenza for patients whose physicians had received flu-shot reminders. ²⁶

Subjective measures, also referred to as patient-centered outcomes, ^27,28 include symptoms (e.g., the New York Heart Association classification of heart failure symptoms ²⁹), functional status, quality of life, and satisfaction with one's health care or providers of care. These measures can be generic or condition-specific, and they can rate an individual against a fixed scale (e.g., the SF-36 ³⁰) or in regard to his or her own baseline functioning. ^31,32,33 Although many of such survey instruments have been shown to be reliable and valid, it is important to understand that they may behave much differently in different populations, especially if they are modified to be specific to local needs. ^34,35

Finally, the provider must not be forgotten when assessing informatics innovations. If such systems are not well received, providers will not use them ¹² or will use them only under duress. EMR systems are more likely to be favorably received if they are easy to use and offer access to both medical knowledge and patient data, ³⁶ but the most critical aspect is time. If it takes a long time to learn to use such a system, or if it takes more time to use the system even after its use is mastered, acceptance will be difficult if not impossible to achieve, and subsequent use will be limited. ¹²

Teamwork Is Key

A research unit capable of performing high-quality research on EMRs requires collaboration among a broad range of researches : medical informaticists, health services researchers, clinicians, administrators, behavioral scientists, psychometricians, statisticians, epidemiologists, and health economists. Although it is rare that a research team contains expertise in all of these areas, broader expertise will allow assessment of more interventions and outcomes.

Assembling and maintaining such multidisciplinary teams is expensive, yet funding to support informatics research is limited. The National Library of Medicine supports both demonstration projects and clinical studies. The Agency for Health Care Policy and Research has funded technology-assessment studies (e.g., controlled trials, time-series studies, and decision analyses), while some foundations (e.g., the Hartford Foundation) have occasionally provided funding for studies of specific aspects of EMRs.

One relatively untapped resource for funding studies of informatics innovations is hospitals and other health care organizations themselves. The Joint Commission for the Accreditation of Health Care Organizations (JCAHO) requires them to perform continuous quality-improvement studies, and information management is a primary focus of the accreditation process. Hospitals and other health care organizations are thus required to support such studies, and often (with persistence) they can be convinced that high-quality studies not only satisfy JCAHO requirements but can guide informatics implementation and (hopefully) result in higher-quality, lower-cost care.

Goals of Clinical Informatics Research

Clinical research involving comprehensive, integrated EMRs has the following goals : 1) understand what works (and what does not) in terms of health care costs, processes, and outcomes ; 2) understand the effect of technology on the user ; 3) understand the relative costs and effects of EMR innovations ; and 4) provide purchasers of EMRs with sufficient information to make the right choices.

However, informatics developers and researchers must understand that EMRs, like all new technology, probably will have effects that are profound, farreaching, and completely unforeseen. The telephone was originally seen as a faster way of sending a letter, but the world-wide telephone network is now used to send all manner of information (text, picture, voice) at rapid rates across the entire planet. Photography was seen as a faster, perhaps more accurate way to paint pictures, yet now video technology from movies to electronic games to virtual reality is reshaping how we work and play. It is likely that a new tool such as integrated EMR systems will have profound effects on the practice of medicine, most likely redefining the roles of both patients and providers of care. Anticipating and rigorously measuring such as-yet-unforeseen benefits is the primary challenge to present and future medical informatics investigators.