Comments on Return on Investment (ROI) As It Applies to Clinical Systems

In the current issue of this journal, Kaushal and colleagues¹ present an exemplary, thorough analysis of the financial impact of installing computerized physician order entry (CPOE) in a major academic hospital. This excellent work quantifies quantifiable parameters clearly, estimates other parameters conservatively, and lists problems associated with measurement of the remaining small set of other factors. This work is arguably the best that can be done with respect to evaluating CPOE return on investment (ROI). Why does it not quite seem to be enough or to be sufficient to persuade?

The answer to the foregoing question lies in the culture of informatics research and in the health care industry's collective culture of decision making. The research community feels compelled to refine estimates of CPOE's financial return to a greater and greater degree along the analytical lines in the work of Kaushal et al. Using adoption estimates and implementation plans, proponents of CPOE installation in many institutions may base their arguments solely on the Kaushal et al. financial framework. Other proponents may take a different approach, emphasizing instead the broader array of workflow and professional productivity factors that lead to greater financial return. Still other CPOE proponents pursue the challenge of extending financial models to estimate the benefits of CPOE in every hospital, in every ambulatory practice setting, and in every community.

A growing body of hard data on the benefits of CPOE presents a compelling case for all who make CPOE decisions to go forward (see references cited in Kaushal et al.¹). They should accept, once and for all, that substantial benefits will accrue to hospitals and patients following the successful implementation of effective CPOE systems. The challenge is not to fine-tune the financial benefit models, but instead to determine how to identify and successfully install an effective CPOE system. Many other factors not included in financial models contribute to the success or failure of CPOE implementations. The challenge to society and to industry is to determine how to extend successful CPOE implementations from the setting of a premier academic medical center to every care delivery setting in the nation. We must understand more fully limitations of ROI analysis in hospitals. We must question and document the extent to which such analyses serve as a valid justification for extending CPOE into diverse, nonhospital-based settings where most medical care is delivered. Such an investigation should start with available findings from the large, sophisticated academic medical centers with adequately abundant technology, talent, and time to develop and install CPOE systems. Even in these settings, we are reminded that “many savings from CPOE are not realized in the operations budget” and hence preclude accurate confirmation of financial benefit.

Examples of difficult-to-quantify benefits mentioned in the Kaushal et al. article include advanced systems for specialized treatment issues, scheduling efficiencies, and documentation capabilities. To advance the argument for successful adoption within hospitals, as well as to promote broader adoption of health care systems in community-based ambulatory settings, the foregoing list must be extended. From a purely financial perspective, hospitals and large clinics can only accrue substantial financial benefits if their clinical systems are linked to administrative and operational systems used to capture costs and charges, to systems that measure efficiency and quality, and to systems that measure clinical and financial performance. Linking CPOE systems to pharmacy and medication administration processes can provide value to patients and fiduciaries. From a financial and operational perspective, however, this benefit is even greater if these same clinical systems are tightly coupled with those systems used to manage the complex distribution of expertise, medications, and supplies within a hospital. Although CPOE is not needed to realize many gains in these areas, the very near-term survival of health delivery organizations may depend on their ability to link, in real time, activities at the point of service with thoughtful management of inventory and other business processes—just as this is critical to retailers and manufacturers.

To a sad and significant degree, however, health care delivery organizations conduct their businesses within an incentive structure that epitomizes the concept “market failure.” Operations within a hospital or clinic can be rationalized. However, such rationalizations do not reflect the totality of care delivery processes that American patients experience as they navigate through the current complex and turbulent U.S. health care delivery system. The necessity of incorporating into ROI calculations the impact of reimbursement and payer mix provides a glimpse of the misaligned and perverse incentive structures at play in health care. There is a difference in calculated CPOE ROI if one receives reimbursement through fee-for-service rather than fixed fees for care delivery. The cynical dependence of ROI on variations in payment mechanisms speaks more about external forces influencing the introduction of clinical systems than do the detailed financial analyses.

Focusing on this confusion distracts one from the central point—that to deliver better care, effective implementation of CPOE becomes a responsibility. Whether the ROI can be accurately measured is immaterial to this goal. Perhaps those health care institutions that have not mounted the organizational will to deploy advanced clinical systems suffer because their best administrative and financial talents focus the majority of their time striving for optimization of reimbursement and regulatory rationality, instead of devoting their time to critical internal strategies for successful health care delivery per se.

Financial ROI analysis for clinical system installations is incomplete in that it does not reflect the values of patients and health care professionals affected by the systems. From the perspective of the patient, the ROI is measured in safer and more effective medical care. Such care leads to better outcomes, better health, and higher levels of personal and professional productivity. Patients who endure complex illnesses or suffer unneeded complications know that effects of health care delivery extend beyond hospitalizations into the activities of the family members, ambulatory care providers, and other caregivers over a lifetime. Similarly, the ROI measured by those who must make a capital investment is not the same as the ROI measured by health professionals who must take the time both to use CPOE systems and to adapt their work patterns to achieve the goal of improved quality of care. To the health care professional, the true ROI may be measured in terms of ease of use, total expended effort, and satisfaction with the results achieved. From the perspective of one who delivers care, the best hallmark of a truly effective computer technology may be the extent to which one no longer is even aware that one is using a computer to accomplish a better outcome.

As in other industries, measurement of the effects of information technology on the clinical setting suffers from a “productivity paradox” described by the economist Robert Solow in 1987.² Solow stated that computer technology can be found everywhere but in the productivity data. Solow pointed out the difficulty of teasing out the impact of a subtle but pervasive technology from broader transformations taking place within an economic system. The Internet, for example, allowed businesses to take new approaches to managing all aspects of supplier-marketer relationships. Solow's point was that one cannot easily measure how much improved economic performance was due to the enabling technologies and how much was due to significant transformations in how businesses managed their internal operations and their relationships with customers, competitors, and suppliers.

In most industries having far less impact on the gross domestic product than health care, the use of advanced technologies is now a matter of necessity rather than debate. One must infer that no matter how laborious and tortuous is the evolution of our health care delivery systems, within the next decade, advanced clinical systems will become deeply embedded in our clinical work whether or not we ever can measure ROI beyond our current capabilities.

Health care leaders are beginning to realize that investing in effective clinical systems is as much a matter of organizational will as a matter of financial rationale. Indeed, one might argue that undertaking ROI analyses deals not with determining a theoretical outcome, as much as it does with addressing the fear that the institution lacks the culture, incentives, and workforce to ensure that successful system implementation, which is admittedly not a trivial task.^3,4 Both the size of the investment and the risk of achieving successful implementation are at play when hospitals are asked to “make large up-front capital investment without clear data on return on investment or confidence in physician use of implemented systems.”

Given the strong arguments in favor of adopting clinical information systems, perhaps it is time to put aside the ROI arguments and focus instead on ensuring that all implementations are successful. Such routine success will require the absolute commitment of every individual within an organization who has any direct role in the transmission and actionable use of health care information in a clinical setting and an equally substantial change on the part of the vendor community. Such commitment only comes when global financial incentives are aligned with the personal motivations of the diversity of individuals who must use health information technology. In major centers, the knowledge engineering and workflow expertise may be hidden costs occurring “piecemeal over many years and probably represented a relatively small part of the entire [system] costs.” For many other institutions, the quandary is not so much one of cost and time of expertise as it is of the very availability of such expertise at critical junctures. In many instances, the cost and availability of intellectual capital may be a far greater constraint to implementation than insufficient financial capital.

The introduction of health information technologies into communities presents even greater challenges. This task demonstrates how much more work must be done to extend cost-benefit analysis into additional settings. To realize successful longitudinal, patient-focused care, health care professionals and patients require an infrastructure that provides timely, secure access to a wide array of health care information. That information is at present distributed across many sites and difficult to integrate. A clinical system capable of delivering optimal care in physician offices, pharmacies, home care, and other community settings will require access to medication histories, personal health information, improved clinical communication, and standardization of many other currently disparate aspects. Building such an infrastructure while mindful of costs and patient preferences requires a daunting effort. Lessons gleaned from hospital-based ROI analysis may provide partial guidance. Yet, in these settings even more than in the hospital, one must place emphasis on the cost of an appropriate infrastructure, the availability of intellectual and financial capital, and addressing the misaligned incentives among patients, employers, government, intermediaries, health plans, practitioners, pharmacists, and other involved parties. One of the greatest obstacles to healthier communities may be the enormous gap between what we know how to do in the large and sophisticated health care systems and what we know must be done in our communities, our pharmacies, our long-term care facilities, and our ambulatory practices.

Professional informaticians must continue to address these broader problems while learning from existing thorough analyses. The profession must extend, not just refine, these analyses to develop a better understanding of the broader financial impact of health information technology and the less tangible but even more powerful benefits accrued to patients and other stakeholders. Society must then address the daunting challenges required to achieve the widespread and effective use of technology in every medical setting, no matter how small in size, no matter how impoverished in capital and technical talent, and no matter how remote. If medical need and health care professionals' desires to improve caregiving become the motivating factors for better use of health information technology, our success in introducing these solutions will produce an ROI that every patient and health care professional will understand.