Evidence-based Practice for Mere Mortals

Abstract

The poor translation of evidence into practice is a well-known problem. Hopes are high that information technology can help make evidence-based practice feasible for mere mortal physicians. In this paper, we draw upon the methods and perspectives of clinical practice, medical informatics, and health services research to analyze the gap between evidence and action, and to argue that computing systems for bridging this gap should incorporate both informatics and health services research expertise. We discuss 2 illustrative systems—trial banks and a web-based system to develop and disseminate evidence-based guidelines (alchemist)— and conclude with a research and training agenda.

Despite the broad acceptance of evidence-based medicine as the gold standard approach to clinical practice, patients often fail to receive the care that the scientific evidence dictates. For example, randomized clinical trials (RCTs) and other studies have consistently shown since the 1980s that β-blockers given at hospital discharge from an acute myocardial infarction (MI) reduce subsequent mortality,^1–3 yet in 1998, only 50% of eligible patients nationwide were prescribed β-blockers at hospital discharge.² What is required to ensure that patients receive evidence-based care during their encounters with the health care system?

In recent years, computer-based approaches to managing the clinical literature have become widespread. With over 10,000 RCTs indexed in medline in 1999 alone, computers are no doubt a necessary, but not sufficient, technology for evidence-based practice. Broadly speaking, information technologies will be needed to 1) help the research community manage what is and is not known, 2) assist clinicians with applying the totality of the research evidence to individual patients, and 3) encourage, facilitate, and monitor practice change. Informatics systems that meet these objectives will have to be not only technically sophisticated but also grounded in an understanding of real-world clinical reasoning and practice. Furthermore, to affect point-of-care decision making, these systems must work within the complex organizational and policy environment of today's fractured health care system. Therefore, we believe that efforts to help physicians achieve evidence-based practice through technology need to draw from both medical informatics and health services research.

In this paper, we first draw upon the methods and perspectives of clinical practice, medical informatics, and health services research to analyze the critical gaps between evidence and action. We then provide 2 illustrative examples of informatics solutions being developed for bridging this gap that are grounded in health services research: trial banks to capture the design and results of RCTs for systematic reviewing and computer-based reasoning,⁴ and the alchemist system for creating and customizing decision-model–based guidelines.⁵ These examples will emphasize the joint role of health services research and medical informatics as foundational disciplines for informatics systems that will help methodologists and practitioners translate evidence into practice more effectively and efficiently. We then suggest an agenda for future methodological work in informatics and health services research.

FROM EVIDENCE TO ACTION

Four major difficulties span the gap between evidence and action, the difficulties in 1) obtaining evidence that is relevant to the clinical action, 2) systematically reviewing this evidence, 3) applying this summary evidence within a specific decision context at the point of care, and 4) supporting and measuring the practice change that should ensue from applying this evidence. While informatics systems can ameliorate each of these difficulties, their success at doing so has been limited to date. We discuss how strategies that combine informatics, health services research, and clinical-practice expertise promise greater success than current strategies at resolving each of these 4 problem areas.

Obtaining Relevant Evidence

Estimates are that only about 20% of clinical care is addressed by research evidence, and controversy exists as to the merits of efficacy versus effectiveness trials for rectifying this problem.⁶ Notwithstanding this debate, an even more fundamental challenge faces the clinical research community: we do not know what we know,⁷ what has been or is currently being investigated,⁸ or which important research questions are not being investigated at all.⁹ If the clinical research community as a whole is without such knowledge, how can individual physicians be expected to know the status of relevant clinical research?

The policy response to these shortcomings of clinical research is beyond the scope of this paper, but a first step toward a better managed clinical research portfolio would be to construct comprehensive databases of past and ongoing research projects that facilitate systematic review and interpretation. The www.clinicaltrials.gov trial registry and others like it^10–12 are a start, but because study registries are often geared only toward study enrollment, they do not typically contain sufficient details of study design, execution, and results for analysis. Furthermore, existing study registries do not use standardized medical vocabularies and are thus difficult to cross-search comprehensively. A high priority should therefore be placed on constructing databases of clinical research that meet the performance standards of health services research (sufficient detail for systematic reviews) and medical informatics (use of standardized vocabularies).

Systematic Review of Evidence

Over a decade of work has emphasized the importance of systematically reviewing clinical research results. The methodological and logistical challenges to performing systematic reviews have been discussed elsewhere.¹³ From the informatics perspective, systematic reviewing consists of 4 main tasks: 1) identifying and retrieving research studies, 2) judging the internal and external validity of the studies, 3) performing a qualitative and/or quantitative synthesis as appropriate based on the heterogeneity of the studies, and 4) interpreting the totality of the evidence in its proper socioeconomic and ethical context. Most computer programs for systematic reviewing support only the quantitative synthesis task, leaving users to struggle with retrieving poorly indexed study reports^14,15 that are too often incomplete^16–18 and nonstandardized.¹⁹

Information technologies can better support these 4 main tasks of systematic reviewing if clinical studies are available in computer-understandable databases as well as in prose articles. Thus, high-performance clinical research databases that are built for managing the clinical research portfolio, as discussed above, could double as completely reported, standardized databases for systematic reviewing. To build such multipurpose databases, careful attention must be paid to which study attributes are captured, so as to ensure that the necessary and sufficient data are available for all intended uses. In addition, these databases should have compatible data designs so that cross-searches among them would yield valid and reliable results. For example, the RCT Bank database²⁰ stores study inclusion and exclusion criteria separately, whereas the ClinicalTrials.gov database²¹ stores all criteria together as generic entrance criteria. With such incompatibility, a search for inclusion criteria across these 2 databases may yield incorrect results. Informatics techniques of knowledge modeling and interoperation are thus critical for computer-based approaches to support systematic reviewing.

Application of Evidence in the Decision-making Context

The point of care is where the logistical barriers of practicing evidence-based medicine becoming overwhelming. These difficulties are likely to increase as Internet-based consumer health programs shift the point of care beyond the clinic and the hospital.²² Computing systems for supporting evidence-based decision making must mesh with complex workflow patterns while encouraging and supporting methodologically sound applications of research evidence.

At present, the majority of computer-based approaches to facilitating evidence-based practice involve the electronic dissemination of vast quantities of static and generic information to practitioners. For example, the National Guideline Clearinghouse²³ includes over 1,000 guidelines, but while the website provides some support for choosing among potentially relevant guidelines, it does not help users to adapt these guidelines to particular patients or populations. Furthermore, many computer-based approaches disseminate information that is not necessarily based on rigorous evidence appraisal, synthesis, and application methodologies. For example, Kuntz and colleagues compared expert panel and decision analytic recommendations for coronary angiography after myocardial infarction, and found that some interventions that the experts judged as inappropriate had cost-effectiveness ratios comparable with many generally recommended medical interventions.²⁴ Decision models provide a formal analytic framework for representing the evidence, outcomes, and preferences involved in a clinical decision. Basing guideline creation on such a decision analytic framework allows the performance of sensitivity analyses to identify critical variables and thus to focus refinement of the evidence. This analytic process can enrich the resulting guideline.^5,25–28 In the National Guideline Clearinghouse however, only a minority of the guidelines reference decision models for their development.

Three contingencies are crucial for enabling a new breed of decision support systems that are more workflow-sensitive and methodologically rigorous. These contingencies have profound informatics implications. First, the best current evidence must be available to decision support systems in computer-understandable form. Second, to increase physician acceptance of decision support systems, their recommendations should be patient specific. Third, decision support systems must be designed to complement the workflow and preferences of their intended users, be they physicians, nurses, or patients.

To meet these contingencies, the following technologies are needed:

• computer-understandable clinical research databases as discussed above;

• electronic medical records (EMRs) and other clinical systems that use a rich standardized clinical vocabulary, e.g., SNOMED,²⁹ to ensure that different systems share a common understanding of the clinical situation;

• standardized interfaces among clinical and practice management systems to facilitate communication among multiple systems;

• new and higher-performance technologies (e.g., speech recognition and wireless computers) to make it easier for physicians to enter data and to enable better workflow compatibility.

With this suite of technologies, frontline decision support systems would be better able to dynamically generate patient-specific assessments and recommendations by matching studies, systematic reviews, and guidelines with individual patient characteristics as stored in the EMR. The recommended action could then be delivered to handheld computers, which can send wireless instructions to practice management and other systems to execute the recommended action.

Unfortunately, the needed technologies are not yet widely available. At present, commercial EMR systems tend to use only ICD-9 and CPT-4 coding³⁰—rather than SNOMED, the most expressive and robust clinical vocabulary available^31,32—in part because the benefits of SNOMED over ICD-9 are not widely appreciated. More standard interfaces such as HL-7 are still needed to integrate the information systems of our highly fragmented health care industry.³³ In summary, the development of practical decision support systems for routine clinical use still faces major informatics and health services research barriers.³⁴ It remains to be determined how such more-rigorous and workflow-sensitive decision support systems will affect physician practice behavior.

Effect and Measure Practice Change

In addition to their role as a methodological tool, computers can be used to implement and measure practice change. Although evidence suggests that computerized reminder systems are effective at changing practice behavior, other types of computerized decision support systems have been less promising.^35–37 Indeed, physicians are notorious for being resistant to computer-based initiatives,^38,39 in part because physicians are reluctant to type, but also because system designs often fail to accommodate the time and resource constraints of daily practice. Information systems can thus both help and hinder attempts to overcome the multitude of organizational and cognitive psychological barriers that impede practice change.⁴⁰ Findings from human factors^41,42 and organizational change⁴³ research are thus applicable to designing user-friendly and error-proof decision support systems.

Computers have been far more successful at measuring than at effecting practice change, especially in situations in which data warehouses and other large healthcare databases are available for outcomes analysis. In addition to the health services research methods required for these analyses, informatics expertise in database engineering, networking, and electronic security are also relevant for constructing and analyzing these databases. Furthermore, both health policy and informatics are relevant to the important task of protecting health information privacy in the electronic age.⁴⁴

In all 4 problem areas of evidence-based practice—obtaining, reviewing, and applying evidence, as well as effecting practice change—the disciplines of both informatics and health services research are vital; both are often underused.

CLOSING THE GAP: TWO EXAMPLES

To be of greatest impact, computer-based approaches to these 4 problem areas should together provide integrated end-to-end support for evidence-based practice—from evidence generation to synthesis to application (Fig. 1). We now describe 2 informatics systems that provide complementary solutions to the incorporation of up-to-date randomized trial evidence into patient-specific guidelines.

FIGURE 1.

Life stages of evidence in evidence-based practice. To help translate evidence to practice, information systems are needed for supporting all the life stages of evidence, from evidence generation to synthesis to application. Solid lines denote connections that should, in the future, be implemented as open applications programming interfaces that all use the same standardized clinical vocabulary. Example systems as discussed in the text (boxed) are shown to illustrate their relationships to each other. For example, the ClinicalTrials.gov electronic trial registry expedites evidence generation by supporting patient recruitment. The study results are stored in trial banks and synthesized into systematic reviews. The systematic reviews are themselves stored electronically, in repositories such as the Cochrane Library. Systematic review findings can then be further synthesized into decision models, which alchemist can transform into guidelines. Finally, guidelines in electronic repositories such as the National Guideline Clearinghouse can be matched to electronic medical records to facilitate evidence application.

Trial Bank Project

Millions of dollars are spent every year on RCTs. Regrettably, information about these studies are available to the scientific community only as nonstandardized, often incomplete text articles of several thousand words in length. Because computers cannot yet read text well, computers can only keyword-search these articles and cannot perform any advanced computation on these articles' informational content.

To address these shortcomings, the trial-bank publishing project⁴ asks authors of RCT reports that have been accepted to either the Annals of Internal Medicine or JAMA to describe their trials directly into a structured and standardized electronic knowledge base called RCT Bank. A web site called Bank-a-Trial will guide authors to report their trials using controlled clinical terms from the Unified Medical Language System⁴⁵ and in accordance with an extended version of the CONSORT RCT reporting statement.⁴⁶ By capturing RCTs as both prose and electronic data, trial-bank publishing provides both humans and computers with access to details of the design, execution, and results of RCTs. A wide variety of computer systems, including the alchemist system described below, can then have access to up-to-date RCT information without having to build and maintain their own RCT knowledge bases. Therefore, widespread use of trial-bank publishing would facilitate the development of clinical decision support systems that incorporate current RCT evidence into their deliberations. A similar approach could also be used for publishing clinical research of other study designs into computer-understandable knowledge bases, to assist in those situations in which RCT evidence is not available or feasible.

RCT Bank, our initial trial bank for trial-bank publishing, promotes the rigorous use of RCT evidence by being able to store all the information that a systematic reviewer would need about a trial. The types of required trial information were derived from a hierarchical task analysis of systematic reviewing.^47,48 Trial-bank publishing authors can thus be assured that they can fully describe their trials for subsequent detailed review.

If trial-bank publishing does become widespread, it is likely that there will be many trial banks, some affiliated with journals and others with governmental programs (e.g., PubMed Central) or for-profit companies. To facilitate the comprehensive identification of relevant RCTs for systematic reviews, all trial banks should be accessible through uniform data access protocols, so that, for example, a meta-analysis support system would need to issue only 1 command to search across multiple trial banks. Such uniform protocols—also called open applications programming interfaces (APIs)—will be critical for integrating all the systems that will together support evidence-based practice (Fig. 1).

The trial-bank project exemplifies the importance and relevance of both informatics and health services research methods to the development of systems to support evidence-based practice. Trial banks, however, are designed primarily to meet the needs of researchers and policy makers and only indirectly to meet the needs of clinicians and patients. To bring RCT evidence to the point of care, other systems that process and present trial-bank data are required.

alchemist Project

Clinical-practice guidelines are one avenue by which RCT and other research evidence can be brought to practitioners in a synthesized, actionable format. In 1999 alone, 691 clinical-practice guidelines were indexed in medline, a remarkable increase from 65 guidelines in 1990, and reflecting the current view that guidelines may be able to reduce unjustified practice variation, improve health outcomes, and contain costs.^49–54

Optimally, guideline development involves the input of clinical experts in the target condition as well as experts in meta-analysis and decision and cost-effectiveness analysis.^55,56 National organizations often convene varied numbers of such experts to develop “global” guidelines for patients with the “average” clinical characteristics of the target condition. Such traditional guidelines have several problems, however, including their static presentation on paper or the web and their propensity to being outdated as new evidence emerges. A more complex problem is that local use of global guidelines may require some guideline modification if the characteristics of the local patient population or practice differ from those of the “average” case. Because clinical and methodological experts are often locally unavailable or are prohibitively costly, local guideline developers face large barriers to developing, adapting, and maintaining high-quality evidence-based guidelines.

To address these problems, the alchemist web-based system^5,57 creates clinical-practice guidelines automatically from decision models. alchemist contains a conceptual framework that represents the knowledge needed for a guideline, the knowledge inherent in a decision model, and a mapping algorithm between these 2 representations. Using this conceptual framework, alchemist analyzes a decision model and determines what it can extract automatically from the decision model and what information it must obtain from the decision analyst. alchemist then creates a web-based editor that uses free-text queries, radio buttons, and check boxes to solicit this additional knowledge from the decision analyst. Finally, alchemist combines the information extracted from the decision model and the decision analyst to create a web-accessible guideline applicable to the global patient population. This alchemist guideline includes a flowchart showing the optimal recommended strategy as well as supportive information such as the guideline's available options, outcomes, evidence, values, benefits, harms, costs, recommendations, methods of validation, sponsors, and sources. Through an open API to the trial bank system described above, alchemist would also include links to the RCT evidence base, links that can help either humans or semiautomated systems to keep guidelines up to date with the evidence.

Once a global guideline is available on the web, local guideline developers and users can adapt it to the local patient population by altering the inputs to the underlying decision model. If the underlying decision model changes as a result of new research findings, for example, these changes would be propagated rapidly and consistently to local sites. Thus, alchemist provides in these web-based guidelines the requisite tools and knowledge for customizing guidelines to specific patients or populations, and for modifying guidelines as the underlying decision model or evidence evolves over time.

The alchemist system depends on the availability of clinically valid decision models. Although decision models provide a powerful tool for analysis of clinical problems, their development requires substantial expertise, and may be expensive. In addition, some clinical problems, particularly those with many sequential decisions, are difficult to represent with decision models.

DISCUSSION

The trial bank system aims to capture RCTs in a computer-understandable form, to enable a new generation of systems for retrieving, analyzing, and presenting RCT evidence. The alchemist system aims to support a formalized method for creating guidelines, and to facilitate the dissemination and translation of these guidelines into clinical practice. Both informatics and health services research methods were central to the design of these projects.

How might the trial bank and alchemist systems help address the multi-faceted problem of improving post-MI β-blocker prescription rates? First, if the β-blocker trials of the 1970s had been available in trial banks, it might have sped the recognition that β-blockers reduce post-MI mortality, a finding that was recognized only many years after sufficient evidence had accumulated⁷. For the present day, Bradley et al identified several critical factors for improving post-MI β-blocker prescription rates, including the timely provision of benchmarking and performance feedback data and the availability to physicians of “credible empirical evidence to support recommended practice.”⁵⁸ Certainly, the EMR is a critical technology for delivering timely practice and performance data to physicians and managers. The trial bank and alchemist systems can help deliver credible, patient-specific evidence to physicians if¹ the supporting RCTs provide evidence for relevant subgroups of patients (e.g., diabetics),² the RCTs are captured into trial banks,³ and the guidelines are based on decision models parameterized by relevant patient subgroups. If these 3 conditions were true for post-MI β-blocker prescription, then alchemist could interface with the EMR to dynamically present physicians at the point of care with recommendations specific to patient groups such as asthmatics and the elderly for which physicians are reluctant to prescribe β-blockers.⁵⁹ Such guideline customization would begin to counter the common complaint that guideline recommendations are not patient specific.⁴⁰ Physicians who wish to see the “credible empirical evidence” behind a guideline could then access trial banks to dynamically retrieve and view the supporting RCT data by specific patient subgroups. The trial bank and alchemist systems thus offer a general infrastructure for bringing customized evidence to the point of care, but it remains to be seen how these technologies would affect practice behavior. As has been characterized for the β-blocker prescription problem,⁵⁸ information technologies are only part of the solution to achieving evidence-based practice. Organizational and other interventions are also important.³⁴ Even so, technologies that are well crafted on the basis of both informatics and health services research promise to be more effective at promoting evidence-based practice than those that are not.

CONCLUSION

The scientific evidence from medical research is not clinician-friendly. Rather than the evidence being usable “off-the-shelf,” clinicians bear the full onus of identifying, retrieving, comparing, appraising, and synthesizing multiple studies. Even when relevant evidence is available, the microstructure and the macrostructure of health care delivery conspire to make it difficult for mere mortal physicians to change their practice behavior. Evidence-based practice is currently the hardest way to practice. It will not become the norm until it is the easiest way to practice.

What are the required next steps for the medical community? Attention and resources must be paid to train, foster, and develop a community of scholars who are cross-trained in the basic principles and methods of informatics and health services research. These scholars should include both generalist and specialist clinicians, who will play critical roles in helping to design and develop workable clinical information systems. Funders, professional organizations, and individuals from both fields must encourage and facilitate continued interactions among health services research methodologists, medical informatics researchers, and end users. Research into basic infrastructure technologies as well as end-user systems must be accompanied by rigorous evaluations of these projects, using both qualitative and quantitative methods where appropriate.

Now is a critical and exciting time for the health services research and medical informatics communities to jointly build an infrastructure based on sound health services research methods and informatics principles. By working together and involving the general medical community, we can make the most of a major opportunity for developing integrated computing systems that promise a much-needed avenue for promoting efficient and effective translation of evidence into practice. The ultimate goal of such a concerted and coordinated effort would be to make evidence-based medicine the easiest and thus the most common way to practice.