Toward an Effective Strategy for the Diffusion and Use of Clinical Information Systems

Abstract

The full impact of IT in health care has not been realized because of the failure to recognize that (1) the path from availability of applications to the anticipated benefits passes through a series of steps; and (2) progress can be stopped at any one of those steps. As a result, strategies for diffusion, adoption, and use have been incomplete and have produced disappointing results. In this paper, we present a comprehensive framework for identifying factors that affect the spread, use, and effects of IT in the U.S. health care sector. The framework can be used by researchers to focus their efforts on unanswered questions, by practitioners considering IT adoption, and by policymakers searching for ways to spread IT throughout the system.

Introduction

The conventional wisdom is that information technology (IT) will “transform” the delivery of health care, helping to contain costs and to make medical care significantly safer, of more consistently high quality, and more efficient. ^1,2 Although the expected transformation has not yet occurred—despite “a four-decade history of anticipation and investment” ³ —the potential benefits have been so widely accepted that the federal government, ⁴ private insurers, ⁵ and large employers ⁶ have developed programs to increase IT use.

A recent paper by Sidorov ⁷ provides what may be part of the answer as to why IT has not had the expected systemwide beneficial effects. In it, he presents evidence that one important application, the electronic health record (EHR), contrary to the intent, often leads to higher billings, declines in provider productivity, and inconsistent error reduction, without producing savings. While Sidorov does not claim “the emperor has no clothes,” his paper is a wake-up call for those—and we are among them—who are optimistic about the contributions that IT can make to the health care system. It makes clear the value of stepping back to ask some basic questions: (1) What do we actually know about the reality to date along the path from the availability of potentially useful IT applications to the good things they are expected to produce? (2) Why has IT not transformed the health care system? And, finally (3) if IT can, indeed, do what its proponents say it can, what needs to happen to produce those results? We believe that the full impact of IT has not been realized because of the failure to recognize both that the path from availability of applications to anticipated benefits passes through a series of discrete steps and that progress can be stopped at any one of those steps. In this paper, we present a comprehensive framework that can be used to review existing literature about the following question: What driving and restraining factors affect the spread, use, and effects of information technology in the health care sector in the U.S.? A truly effective strategy to achieve the benefits of IT throughout the health care system can be developed and implemented only after this question is answered.

Conceptualization

Part of the explanation for IT’s failure to transform the health care system is that not enough health care organizations and clinical professionals have adopted IT applications to affect the entire health care system. But even with broad IT implementation, available evidence about effects, which has been mixed, suggests the story is more complicated than the simple failure of more groups to adopt IT. ⁸ So, either the level of enthusiasm is misplaced, or conditions were not always sufficient for the applications to achieve their potential. We have identified the steps that must be taken to get from IT availability to the desired outcomes and will summarize what is already known about them in a way that can maximize their usefulness to policymakers. The underlying premise is that the trajectory from availability to benefit can stop at any step along the way and, therefore, attention must be paid to all of them.

The steps between availability and outcomes are diagrammed in ▶. We should note that the process of going from Step 1 to Step 5 is not strictly linear and that a lot of back-and-forth activity occurs between the steps. Nonetheless, we believe the steps outlined in the figure proceed in a logical order and are useful heuristically in examining progress toward achieving the benefits of IT.

Figure 1.

Preliminary Outline of Steps between the Availability of Health Information Systems Applications and the Production of Benefits.

Step One: The Availability of HIS Applications

To have an impact on the health care system as a whole, IT applications must be available widely. Some, mostly academic medical centers, have developed their own systems with the goal of improving practice internally, but the real potential of IT is in common systems that are familiar to and used by many clinicians and others across many health system organizations. It is useful to think of two major categories of IT applications: 1) Practice Management; and 2) Clinical Management. Much of the commercial development to date has been of practice management applications. These tools help medical practices set appointments, track patients, identify charges, track payments, and manage accounts receivable. They have been relatively popular because they result in more efficient practice operations and potentially higher revenues, they can be purchased and used within a practice without significant coordination across organizational boundaries, and they only minimally affect clinical practice. In contrast, electronic health records (EHR) and other clinical applications have developed more slowly. They are more complex to develop, less likely to produce either greater efficiency or higher revenues for the organization, and often require considerable physician involvement and use. While the value of practice management systems for individual practices is clear, it is the potential of clinical management applications to truly change the delivery of medical care that has stimulated the widespread enthusiasm for IT. A number of private efforts (like Bridges to Excellence—BTE ⁶ ) as well as public sector programs (like Doctors Office Quality-Information Technology—DOQ-IT ⁴ ) are stimulating the demand for such applications.

Among the research questions related to Step One, it would be helpful to know with more precision exactly what applications are in the market, what functions they include, and the extent to which some of the more eagerly awaited features are missing from available products. In addition, it would also be useful to know what motivates entrepreneurs to develop the particular versions they create, how they characterize the market, and what opportunities and obstacles for further development they see. Answers to these questions would be of particular value to policymakers and others who would want to expand the diffusion of HIT in the U.S. health care system.

Step Two: The Adoption of HIT Applications

This is the step that has attracted the most academic attention to date. The sociologist James Coleman’s classic 1966 report, Medical Innovation, ⁹ was one of the first empirical studies of the diffusion of an innovation in health care. It was funded by Pfizer to determine the effectiveness of well-orchestrated, expensive promotional efforts, including advertising, on the adoption of tetracycline by practicing physicians.

From the many studies that followed, sociologists refined “the diffusion model” to describe the pattern by which innovations spread through many fields of endeavor. ¹⁰ They observed a typical S-curve pattern, first described by Tarde in 1903, ¹¹ that begins with the handful of adventurous “innovators” and continues through the slightly larger group of “early adopters.” Then, large strides are made with “the early majority” and “the late majority,” who, being more risk-averse, tend to wait until respected peers try it first, but do not want to be left behind. Finally, “the laggards” bring up the rear. ¹²

One question the diffusion model raises is, “Since this pattern is so common, is it inevitable, and should we just be patient until it runs its course?” The pace of adoption can be deliberately increased through the process of dissemination in which the spread of an innovation is “planned, formal, often centralized, and likely to occur more through vertical hierarchies,” which can be contrasted with diffusion, “unplanned, informal, decentralized, and largely horizontal processes.” ¹³ The IOM studies suggest that the stakes are too high to be content with gradual diffusion and that, therefore, a deliberate strategy should be developed to increase the pace of dissemination. Indeed, the National Health Service (NHS) in England has embarked on an $11 billion program to “connect” physicians, hospitals, and other care providers. Already 98 percent of general practitioners have been connected. ¹⁴ In a market-oriented system, like the United States, the search for clues about the conditions under which such a strategy can succeed needs to focus at the level of medical practices, hospitals, and other health care delivery organizations, each of which is itself a subsystem of the health care system as a whole.

In addition to its sociological insights, Coleman’s 40-year-old study also highlighted non-sociological considerations that are relevant to the diffusion of IT in the 21st century health care system. For example, new technology is not simply added to old technology; it often replaces it. Thus, for innovations that have yet to reach their most stable, mature state, the prospect of incurring adoption costs more than once is a disincentive to adoption. Moreover, new technologies often offer attributes that are not valued immediately by mainstream customers and may not perform as well on some dimensions that are important to those customers. ¹⁵ Where either of these eventualities is the case, the economically rational course for leaders of care organizations and group practices may be to resist investing money, time, training, and good will among clinicians until the technology is fully developed and the probable need to replace it is small.

This conclusion is strengthened when potential benefits are factored into the equation because the health care system does not always reward (and may even penalize) organizations making IT investments. Whatever the benefits for patients, payers, or the system as a whole, clinical applications do not usually result in more patients or higher payments for adopters and, thus, do not reward them with higher revenues. As one cynical author writes about a seriously ill patient taken to a hospital emergency department, “the less the hospital knows about him, the more services it can render, the more it can bill his health insurer, and the more it will collect.” ¹⁶ In many instances, there simply is no “business case” for improving the effectiveness or quality of care with or without IT. ¹⁷ On the other hand, the opposite is the case for practice management applications, which may improve the efficiency of care through better scheduling or billing and may result in greater income by enabling clinicians to see more patients per session or to collect higher fees because of more effective coding. ¹⁸ However, IT adopters may discover that even efficiency improvements are penalized, for example, when third parties pay less to efficient hospitals or medical practices because the services cost those providers less to deliver. ¹⁹ It is simply unreasonable to expect that most hospitals or practices will adopt information systems if they must bear the costs of innovation, but the benefits accrue to payers or patients.

Lists of reasons why health care organizations and physicians have not adopted particular IT applications are not new. ^20,21 What is missing from the literature, however, is a comprehensive understanding that encompasses costs and benefits to each party, purchase and use decisions, and the system conditions that influence decisions and ultimately diffusion. That is especially the case with clinical IT innovations because they are costly to adopters, their benefits often accrue to others, and their maximum value depends on their integration with similar innovations (that is, interoperability) that other health care organizations or providers may be equally reluctant to obtain and use. Since the IOM report identifies the level and pervasiveness of safety and quality as properties of the health care system, the implication is that raising system safety and quality levels can be facilitated by changing certain system factors. One such factor is the multiplicity of rules, forms, and sets of codes that insurers and other third-party payers require. Currently, delivery organizations or physician practices must have multiple software programs (and sometimes hardware, as well) in order to achieve the quality benefits of integration of records across providers and to receive remuneration for care they have provided.

Another reality is that most physicians still work in relatively small, independent practices; in 2001, 63% of non-institutional physicians in patient care practiced alone or in groups of two to four physicians. ²² Not only do they tend to have limited financial resources with which to acquire IT applications, but diffusion must occur on a “retail” basis, one practice at a time. This characteristic increases the difficulty of developing a “dissemination” strategy of the type referred to above. ¹³

At the same time, growing numbers of employers do not offer insurance coverage at all, and partly in response to rising costs, others have either reduced the comprehensiveness of coverage or passed on more of its cost to employees, many of whom decide to “go bare.” These conditions may further inhibit the spread of IT because they increase the risk that practitioners, already discouraged by the cost, ^20,23 will not be compensated for treating patients. Moreover, contrary to the conventional wisdom, competition cannot be the engine to produce system change because until the benefits of IT to the organization are more certain and adoption by other health care organizations is more common, failing to invest in IT or other quality-enhancing innovations does not put the organization or practice at a competitive disadvantage. ²⁴

This is likely to continue to be the case until the acceleration phase in the diffusion S-curve is reached and the early and late majorities adopt IT applications in large numbers, thus changing the competitive landscape. Actually, it is not clear where on that curve the critical “tipping point” that leads to full adoption lies—either by organizations considering IT adoptions or by clinicians in organizations that already have adopted them. But, as Gladwell has suggested, ¹² the challenge is to facilitate adoption up to that position and then expect momentum to draw others the rest of the way toward full diffusion.

Leadership, always an important factor, takes on even greater importance in situations like these in which those who would like changes to be made—either in their own organizations or in the health care system as a whole—have few tangible benefits or incentives to offer the individuals who need to make the changes. ²⁵ In such an environment, leaders are the visionaries who promote change because they see the potential for the future at the same time that they recognize that the investment in dollars, time, and energy is likely to pay off financially only in the long term.

Since the diffusion of many innovations has been widely studied, the key questions related to Step Two concern the relevance of particular factors that have been shown to affect the diffusion of other new services and products to the adoption of HIT by health care organizations. Among other things, it would be useful to know the extent to which doubt about the claims of developers and/or the mixed evidence from studies regarding the capabilities of particular applications play a role in their decision-making. In addition, since the available products are known to vary considerably in their features, it would be useful to know which features are most valued by those making decisions for various types of health care organizations, why they are valued, and why others are less (or not) valued.

Step Three: The Use of IT Applications by Individuals

Some members of organizations that adopt information systems do not use them. A series of articles in Business Week ²⁶ makes clear the potential value of the IT: care processes can be simplified and shortened, rework can be reduced, money can be saved, and quality can be improved. Yet, even with an investment of $72 million over several years, Business Week reported that “only 10% of drugs and orders are entered electronically” at Hackensack University Medical Center. ²⁶

Scholars have long recognized the social dimension of the diffusion of innovations. Coleman’s careful study ⁹ uncovered the importance of social interaction among physicians as a primary driver of the adoption and use of a new antibiotic. Physicians in the four study communities did not want colleagues whose opinions they valued to think of them as stubbornly resistant to an innovation of demonstrated value. They also wanted to “keep up with” professional leaders in their communities. These insights may have great utility regarding efforts to encourage members of a health system to use a newly available clinical IT application.

The importance of these and other non-technical factors has been noted in the implementation and assimilation of information systems, as well, and some of the mechanisms to assist in those processes have been described. “Technology use mediation (TUM)” is an intervention that helps to adapt a new technology to the context in which it is to be used, modifies the context as needed, and “facilitates the ongoing effectiveness of that technology over time.” ²⁷ Davidson and Chiasson use two case studies to show the importance of activities specifically designed to encourage use of an EMR. The two cases differed on several dimensions, including the size of and resources available to the organization, whether or not the people doing the facilitation were part of the organization or consultants to it, and whether the EMR was sold by an outside vendor or created specifically by and for the organization. ²⁷ In one, a 500-bed hospital in the western U.S., “an average rate of 60% direct entry by physicians has persisted for several years.” ²⁷ In the other, outcomes were even more mixed; the clinicians involved all had different practice styles and all wanted the system to support their usual procedures.

The literature contains additional evidence about the extent of use of IT applications, reasons for use or non-use, and a variety of strategies that facilitate actual use of an IT application. To move ahead, policy makers and planners must know: (a) Among the medical practices that adopt IT applications, what proportion of health care professionals actually uses them at all and to what extent do they use them? And (b) What factors differentiate professionals who use the applications from those who do not? Among others, these may include demographics of the user/non-user, profession, primary care or specialty site, and the nature and size of the organization (for example, general hospital or independent medical practice, ownership, the number of professionals, the degree of their identification with the organization, and characteristics of its leadership). Answers to these questions would be of particular value to organization leaders who have purchased or are considering the purchase of HIT applications and who will need to develop strategies to reduce the amount of nonuse.

Step Four: Changes in Work Processes as a Result of IT Applications

Even using an electronic medical record does not guarantee that the quality of care will be high consistently or that the costs of care will be contained. Simply entering data about patients into a computer is not enough. The practice will not become more efficient or the quality more secure without changing elements of the way physicians practice. For example, where it is available in a practice, the decision support function must be used to influence diagnosis and treatment decisions in order to improve quality and efficiency.

Since there is considerable variety among applications, not all versions of, say EHRs, will be equally capable of improving quality and efficiency. As one recent review reported, in 2005 “approximately 24 percent of physicians used an EHR, although only 9 percent used EHR systems that have functionalities such as electronic prescribing.” ²⁸ In fact, some firms selling EHR applications deliberately avoid requiring physicians to change the way they work. Instead, to reduce or eliminate a barrier to adoption, they accept at a remote location electronic faxes of prescription orders, lab orders and results, and other information that doctors or others in their practice produce by hand. ^† Then data-entry employees input the physicians’ faxed data into the EHR. Once entered and available electronically, the doctors may use data in the records differently or more efficiently than when they needed to find, retrieve, and use paper records, although some may prefer to use printouts of the information much like they used paper medical records in the past.

Key questions include: To what extent and in what ways do the work processes (that is, elements of the delivery of services that could be affected by the HIS application) change for those who use the applications? What factors differentiate those organizations and professionals whose work processes change from those that do not? More specifically:

• a) Is there evidence that physicians used the computer to search for prescription drugs appropriate to treat patients’ conditions, to identify appropriate doses or routes of administration, and/or to identify potential drug interactions?

• b) Are prescription orders entered by computer, sent electronically to a pharmacy, and entered electronically into patients’ medical records?

• c) Do medical record entries show that prescriptions have been filled?

• d) Is there evidence that physicians, nurses, or others accessed the record electronically to determine the status of patients’ prescriptions?

• e) Do more patients comply with physician recommendations?

Similar questions should be asked regarding tests and referrals.

Step Five: Effects on the Quality, Efficiency, and Cost of Care

Many studies—although far from all—do show benefits of various types from the introduction of IT applications. Often, however, the magnitude of the gains is modest and a substantial part of the practice is unaffected by the IT. For example, in one study, computerized systems that reminded physicians to provide specific services appropriate to patients’ conditions showed “statistically significantly higher rates of compliance than the control group for all standards combined,” but the increase was only from 53.5% to 58.8%. ²⁹ In a different type of study about a computerized physician order entry program (CPOE) with a computerized decision support system (CDSS) designed to reduce preventable adverse drug events, researchers found a 17% decrease (not statistically significant) and a second study “found a decrease in the rate of adverse drug effects per 1000 patient days from 14.7 to 9.6 during the study.” ³⁰

Studies that demonstrate relatively small benefits are not surprising given research that revealed the “productivity paradox,” the absence of productivity gains in spite of major investments in information technology. ³¹ A number of hypotheses have been advanced to explain this paradox, and it is important to understand why the benefits are not more robust. Is it that too few people use the system? That too few patients are affected by it? That too few physicians are convinced of the value of decision support systems to use them? That not enough time has elapsed to produce more substantial benefits? Or is the cause a flaw in the underlying logic such that the systems are unlikely ever to produce the desired level of gain? Until these questions are answered, the benefits may not be substantial enough to affect the entire health care system. Yet, answering them is methodologically difficult since the practices in which the studies can be conducted are self selected, in both their use of technology and their willingness to participate in research.

Key questions to be addressed in relation to Step Five are, in those organizations and for those physicians whose work processes do change, to what extent are measures of quality improved? To what extent are measures of safety increased? To what extent is care delivered more efficiently? And to what extent are the costs of care lowered? Since many efforts to improve quality and safety, to improve efficiency, and to lower costs are unrelated to information technology, the bottom-line question is the extent to which any changes that are observed can be attributed to the presence and use of HIT.

Discussion

We believe the primary value of the five-step conceptualization that we have outlined is heuristic: it provides a comprehensive picture of the entire space between the availability of an application and the benefits it produces after it is adopted and used. It is certainly true that a major reason that information technology has not transformed medical practice after 40 years of trying is that not enough organizations have adopted IT applications. On the other hand, we suspect that the story would not end with more widespread adoption. (The British NHS, which recently achieved 98 percent adoption by general practices, could be a site for empirical tests of that proposition.) Individuals in the adopting organizations would still need to use the applications, they would still need to change the processes of delivering medical services to patients, and, under those circumstances, the applications would still need to work as advertised.

Although the framework may come across as obvious, the picture it represents emerged only incrementally as we became immersed in the literature. We began by thinking that promising IT applications have not diffused throughout the system largely because health care organizations bear the substantial costs, but others reap the benefits. Instead, we came to recognize that the reality is much more complicated and created the Five-Step Framework to reflect that complexity. We believe that, as they move forward, researchers, policymakers, and practice managers, among others, will find the framework to be useful.

We believe the conceptualization can be useful to researchers by helping them to recognize the importance of the processes that occur in each step and to attempt to understand the forces at work in each, one at a time. Thus, knowing the factors that affect adoption of IT applications by organizations is critical. Understanding the influences on organization members in their individual decisions to use or not to use the available applications is also critical. The conceptualization can help create a research agenda by enabling us to recognize, for example, that some steps are less well understood than others and therefore a more inviting focus for research. Thus, for instance, it could lead some researchers to concentrate on organizations in which applications have not only already been adopted but also are being used by substantial proportions of the members and, thus, to focus their inquiry on the impact of those applications on work processes. Similarly, it could lead others to concentrate on factors affecting personal decisions to use or to not use applications that are available within an organization or practice. We believe researchers should focus on Steps 2, 3, and 4. Work on Step 5, in contrast, is premature and until more is known about the earlier stages in the process, results that are produced about the effects of IT applications will give a distorted picture, especially, of the cost and benefit implications of widespread diffusion. Researchers planning studies that do include outcomes need to design their projects to allow for the possibility that the results show little effect of IT on whatever the outcome is (cancer screening rates, adverse drug events, or lower costs, among others). In a well-designed study, higher rates of appropriate utilization in the group with enhanced IT may be attributed to the IT, assuming the two practice groups are similar otherwise. But if, on the other hand, the results are inconclusive, certainly, it would be inaccurate to say that IT investments are worthless. Therefore, the question shifts to “Why were these results achieved?” The answer could be that, even though the application is available in the practice, too few physicians use it; and one reason for that (among others) could be that the decision to purchase was made without involving them and that practice managers gave no attention to the problem of trying to facilitate use. Alternatively, it could be that physicians do use it, but in ways that do not affect the desired outcome. These examples illustrate the importance of asking about the processes related to the adoption decision, use of the IT by individuals, and the work of delivering care to patients.

The framework presented in this paper can also alert public policymakers who want to encourage the adoption and use of IT applications that they will be disappointed if they fail to pay attention to all five steps along the path. Further, it should lead them to spend much less effort at this juncture trying to persuade health care organizations to buy EMRs and other applications. Instead, since not all applications on the market can do what the IOM reports want them to do, government should contribute to, if not lead, efforts to establish a complete set of standards based on an application’s capacity to achieve the desired goals. One such effort, the Certification Commission for Healthcare Information Technology (CCHIT), was created by the American Health Information Management Association (AHIMA), the Healthcare Information and Management Systems Society (HIMSS) and The National Alliance for Health Information Technology as a voluntary, private-sector organization to certify HIT products. Launched in July 2004, it has recently been designated by the Department of Health and Human Services as its first recognized certification body. ³² Insurers and employers that now offer incentives through programs like BTE and DOQ-IT to practices that adopt certain applications, like EMRs and EHRs, would benefit from this effort, too, because they could be more selective in what they support. Finally, the federal government should invest much more heavily in research—particularly on Steps 2, 3, and 4 in the framework—so that better decisions can be made in the future. One effort in this direction is represented by The Health Information Technology Adoption Initiative created by the Office of the National Coordinator for Health Information Technology (ONCHIT). That initiative is “aimed at better characterizing and measuring the state of EHR adoption and determining the effectiveness of policies aimed at accelerating adoption of EHRs and interoperability.” ³³

Finally, the conceptualization can help medical practice managers and other practitioners, as well, to understand the full range of IT implementation challenges and to prepare to meet each one as it arises. It can be a powerful decision-making aid for them by helping them to clarify the goals they want to accomplish with particular IT applications and to select products that can really do those things. It can also help them to understand fully the steps needed to achieve their objectives and to be sure that they can actually take those steps. The framework can help them develop strategies to ensure that their physicians will use the applications and that they can change work processes in the ways needed to improve quality and efficiency. It will also help them to estimate the probability that the benefits from their investments will, in fact, be achieved.

We wrote this paper because we are worried. While many of the points we make may seem “intuitive” and, therefore, widely known, we do not see evidence that many people act as if that is the case. Instead of recognizing the challenges summarized in the framework, many tend to promote HIT applications with more enthusiasm than is warranted by the available evidence. A disinterested observer, looking at the literature, would conclude that the potential of IT is tantalizing, but would not be surprised if a cash-strapped medical practice decides the case is not yet strong enough to justify the large investment required. This point is reinforced by the JAMA editorial referred to earlier, which contains a statement to the effect that the probability of finding benefit drops by half when the author of the study is not also the developer of the application—and even then, as noted above, the magnitude of the benefit is often small. ⁸

Further evidence that policymakers are ignoring these facts comes from a recent article on the front page of the Business section of the Sunday New York Times. ³⁴ Secretary of Health and Human Services Michael Leavitt is quoted as saying “the rollout of electronic health records was ‘the most important thing happening in health care’” even though the reporter wrote that others not only doubt that the large savings predicted by “various studies” will materialize, but also believe that expenditures may even rise. And, while the article acknowledges that EMRs are now used by only 20% of doctors, the assumption appears to be that since a “push” to increase that number is “accelerating,” dramatic growth is imminent.

To the contrary, the very slow pace of adoption by medical practices is an “unobtrusive measure” ³⁵ that most practice managers are not persuaded that they should spend their money on many of these applications. Why do policymakers appear to ignore that simple truth? The most benign explanation may be that most people read one article at a time and extrapolate from it without putting together their own conceptualization of the space. This is especially likely since most articles emphasize statistically significant benefits of whichever IT application is being studied and skip lightly over the often small magnitude of those benefits. Another explanation may be that policymakers are under pressure to “do something,” and IT looks like a good bet because although the payoff is in the future, they can get credit for their activity now.

Conclusion

None of what we have written should be taken to mean that our national interest in information technology for health care is misplaced, but it does mean that much more needs to be done to maximize the value since the literature often reports only a modest degree of benefit for some applications in some settings and little or no benefit in others. While part of the story may be a natural resistance to change on the part of people who are content with things as they are, undoubtedly, some of the failure to diffuse IT is a rational response to incompletely developed applications that need additional refinements and the inconsistent record of achievement reported in the literature. ³⁶ We ignore the evidence, unobtrusive or otherwise, at our peril. If placed in the appropriate context using the framework presented above and if not intent simply on trying to demonstrate that they produce some level—any level—of statistically significant improvement in an appropriate outcome (especially cost reductions), then, for those applications with promise, we can identify real obstacles to achieving their potential and find rational ways to overcome them.