Integrating Economic Evaluation Methods Into Clinical and Translational Science Award Consortium Comparative Effectiveness Educational Goals

Abstract

With the ongoing debate over health care reform in the United States, public health and policy makers have paid growing attention to the need for comparative effectiveness research (CER). Recent allocation of federal funds for CER represents a significant move toward increased evidence-based practice and better informed allocation of constrained health care resources; however, there is also heated debate on how, or whether, CER may contribute to controlling national health care expenditures. Economic evaluation, in the form of cost-effectiveness or cost-benefit analysis, is often an aspect of CER studies, yet there are no recommendations or guidelines for providing clinical investigators with the necessary skills to collect, analyze, and interpret economic data from clinical trials or observational studies. With an emphasis on multidisciplinary research, the Clinical and Translational Science Award (CTSA) consortium and institutional CTSA sites serve as an important resource for training researchers to engage in CER. In this article, the authors discuss the potential role of CTSA sites in integrating economic evaluation methods into their comparative effectiveness education goals, using the Columbia University Medical Center CTSA as an example. By allowing current and future generations of clinical investigators to become fully engaged not only in CER, but in the economic evaluations that result from such analyses, CTSA sites can help develop the necessary foundation for advancing research to guide clinical decision making and efficient use of limited resources.

The growing number of medical and surgical advances in recent years has created challenges for physicians as they attempt to select the best strategy to prevent or treat a patient’s health condition. In many cases, they must make choices between two or more clinical strategies with different profiles of safety, efficacy, and cost, without the benefit of comparative research studies. To address this growing concern, the American Recovery and Reinvestment Act of 2009 (ARRA) allocated $1.1 billion in federal funding for comparative effectiveness research (CER).¹ Simply stated, CER—which is also known as patient-centered outcomes research—aims to compare two or more interventions or strategies used to prevent, diagnose, or treat a medical condition.² Federal funding of CER is undoubtedly a significant step toward more evidence-based practice and better informed allocation of constrained health care resources; however, it has also led to heated debate on how, or whether, such research may contribute to controlling national health care expenditures.

Over the past three decades, according to the Centers for Medicare and Medicaid Services (CMS), health care spending has grown at an average annual rate that is 2.5% faster than the rate of overall U.S. economic growth as measured by the nominal gross domestic product (GDP).³ Annual spending on health care reached $2 trillion in 2005 and CMS estimates it will rise to $4 trillion—20% of the GDP—by 2015. Yet there is significant evidence that opportunities exist in both the public and private sectors to curb rising health care costs without adverse health consequences. For example, Medicare’s cost per beneficiary varies significantly across geographic regions. However, this variation cannot be explained by differences in population demographics, and, in general, higher-cost regions have not demonstrated higher quality or improved access to health care.⁴

Studies have estimated that less than one out of every thousand dollars spent on health care, or less than 1.5% of spending on medical research, is used to determine which treatments provide better outcomes.⁵ A large percentage of current research spending is devoted to enabling new treatments to obtain regulatory approval. These studies often compare a novel medication or device with a placebo rather than with the next-best alternative treatment. Yet to guide clinical decision making, it would be more relevant to compare new treatments with existing clinical strategies that are the current standard of care. For example, atrial fibrillation, the most common sustained arrhythmia encountered in clinical practice, may be treated using surgery, catheter ablation, or pharmacologic therapy, but there is a lack of prospective studies comparing the relative effectiveness of these various treatment strategies.⁶ It is not surprising, therefore, that atrial fibrillation is in the top quartile of the Institute of Medicine’s “Initial National Priorities for Comparative Effectiveness Research.”⁷ CER is needed in other aspects of cardiovascular medicine as well. Among national cardiovascular practice guidelines, for example, research has demonstrated that only 314 (11.6%) of the 2,711 recommendations set forth by the American College of Cardiology and the American Heart Association were classified as backed by level A evidence (i.e., from multiple randomized trials or meta-analyses).⁸ In contrast, 1,246 (46.0%) of the recommendations were based on level C evidence (i.e., expert opinion, case studies, or standards of care).

While it is clear that comparative studies are necessary to improve medical decision-making in all fields, the heightened national awareness of and federal funding for CER may lead one to conclude that the resources for advancing CER goals are already in place. However, a recent study analyzing all randomized trials, observational studies, and meta-analyses involving medications published in the six highest-impact-factor medical journals during June 2008–September 2009 showed that there is significant room for improvement.⁹ Of the 328 studies evaluating medications, only 104 (31.7%) involved CER. Among these CER studies, 45 (43.3%) compared the efficacy between medications, 20 (19.2%) focused on safety, and only 2 (1.9%) included an economic analysis. Moreover, just 14 (13.5%) of the CER studies had exclusively commercial funding, compared with 101 (45.1%) of the non-CER studies (P < .001). These findings suggest not only that there is a need for continued growth of CER, particularly with regard to economic analysis, but also that federally funded investigators at academic health centers, rather than those funded by industry, are likely to be the major force driving this growth.

The Inseparable Disciplines of CER and Economic Evaluation

Economic evaluation, although not required in formal CER, is sometimes viewed as a natural outgrowth of CER studies because it provides the opportunity to compare the relative value of two or more interventions for a given medical condition.¹⁰ Such consideration of costs, especially in the context of health insurance coverage decisions, however, is a subject of heated debate. Some scholars argue that existing economic evaluation methods—notably cost-effectiveness analysis and cost-benefit analysis—are not transparent enough and “not ready for prime time.”¹¹ Some critics also note that the results of cost-effectiveness studies may vary dramatically depending upon the assumptions made or analytic perspective chosen (e.g. patient, government, or society). Their concerns are valid—significant variation has been identified in methodologies used in published cost-effectiveness analyses.¹² On a broader scale, some highlight the public’s concern that economic evaluation will ultimately lead to health care rationing.¹³

Despite these criticisms, the argument for economic evaluation begins with the basic premise that health care spending cannot continue on its current trajectory. More prudent decisions must be made about the provision of medical services that are highly costly but offer little or no additional benefit compared with less costly alternatives. Economic evaluation goes beyond comparing the efficacy of medical interventions to address their relative value by formally weighing the benefits, risks, and costs among different clinical options. These evaluations frequently can be conducted alongside clinical trials by recording resource cost endpoints, in addition to clinical endpoints, or by constructing a decision-analytic model to incorporate immediate resource needs, potential risks, expected downstream savings (“cost offsets”) from avoided medical services, and gains in life expectancy or quality of life.

Cost-effectiveness and cost-benefit analyses have been integral to guiding clinical and public health resource allocation decisions in countries such as the United Kingdom, the Netherlands, and Australia.¹⁴ In the United States, as economic evaluation is not a formal component of CER, there are currently no standardized thresholds from which to translate research findings into public policy.¹⁵ As resources become increasingly limited, however, cost-effectiveness analyses may begin to play a greater role in the evolving U.S. health care system.¹⁶

There has been a significant growth in the number of cost-effectiveness and cost-benefit analyses published in the medical literature in recent years: just 21 journals published cost analyses in 1994 whereas 147 journals did in 2006.¹⁷ We searched MEDLINE in June 2010 using the terms cost-effectiveness analysis or cost-benefit analysis, and found that more than 500 publications containing these terms in the title or abstract were published from January 1, 2009, through June 1, 2010. Yet despite the growing prominence of economic evaluation in medical research, studies have demonstrated that there remains a lack of understanding and formal training among clinical investigators about how to design and apply economic methodology within randomized trials and observational studies, and how best to interpret and analyze economic data.¹⁸ As noted above, academic health centers will play a major role in driving the growth of CER, and these institutions, through initiatives such as the Clinical and Translational Science Award (CTSA) Consortium, are well-positioned to advance the field of economic evaluation through education and training. In this article, we discuss the potential role of the CTSA in integrating economic evaluation methods into their comparative effectiveness education goals. As a case example, we describe the Columbia University CTSA curriculum in economic evaluation methods.

The Role of CTSAs in Advancing CER

In 2006, the National Institutes of Health in collaboration with the National Center for Research Resources worked together to create the CTSA program, with the broad goal of creating an “academic home for clinical and translational research.”¹⁹ As of February 2011, 55 medical research institutes in 28 states across the country are CTSA sites. Together they form the CTSA Consortium, which works to foster collaborative research across disciplines and use innovative approaches to tackle research challenges and train clinical and translational researchers. The five strategic goals of the CTSA consortium are to:

1. Build national clinical and translational research capability

2. Provide training and improve career development of clinical and translational scientists

3. Enhance consortium-wide collaborations

4. Improve the health of our communities and the nation

5. Advance T1 translational research to move basic laboratory discoveries and knowledge into clinical testing¹⁹

These goals will be instrumental in shaping and developing the infrastructure that can support the demands, complexities, and collaborations required to carry out successful CER.

Equipping the next generation of clinical investigators with the skills to develop multidisciplinary and translational research collaborations is not just a core CTSA value—it is also critical for the development of high-quality CER. Comparing the clinical benefits and risks from multiple medical interventions is a complex process that is fraught with study design and methodology issues and requires collaboration across medical disciplines. A significant challenge comes from the fact that the outcome of interest in CER is effectiveness, not merely efficacy. Effectiveness addresses how an intervention works in patients in realistic settings with less than perfect compliances, multiple comorbidities, and even lack of access to care, whereas efficacy is usually derived from a controlled, experimental environment with carefully selected candidates. The CTSA Consortium and individual CTSA sites have the resources to provide the training researchers will need to overcome such challenges.

Economic Evaluation Training and CTSAs

Since 1998, the Association of American Medical Colleges (AAMC) has advocated that “in caring for individual patients, [physicians] must apply the principles of evidence-based medicine and cost-effectiveness in making decisions about the utilization of limited medical resources.”²⁰ However, a recent AAMC survey, as reported in the New York Times, demonstrated that only 60% of U.S. and Canadian medical schools include some content on health care costs in their curricula.²¹ There are a multitude of historical, structural, cultural, and philosophical reasons for this finding, but the setting in which medical education takes place is among the most often cited reasons.²² Medical schools may lack faculty with expertise in economic evaluation methods to teach students during the pre-clinical years, and they face time constraints due to students’ clinical duties in the third and fourth years. As economic evaluation continues to become ever prominent in the CER literature, however, clinical investigators will need to be prepared to interpret and contribute to such research. As noted above, with the core mission of improving multidisciplinary research, institutional CTSAs may be an ideal avenue for introducing clinical investigators to methods of economic evaluation as part of their training in conducting CER.

A curriculum in economic evaluation should be flexible and provide different levels of familiarity with economic methods based on the trainee’s research. Its breadth and depth can be structured to ready investigators for launching a research career in designing and conducting economic evaluation of medical interventions, or to prepare trainees to engage in multidisciplinary collaboration by exposing them to the theoretical foundation and basic elements of these analyses. The former approach requires commitment of substantial time and resources to ensure researchers understand the intricacies of specific economic methods and obtain practical expertise and hands-on skills. The latter, more practical approach may involve didactic lectures coupled with project-based exercises that use clinical case studies and decision analysis software.

Although economic evaluation education at CTSA sites will vary according to faculty, financial resources, and trainees’ needs, there is some common ground. In keeping with CTSAs’ educational goals and focus on multidisciplinary research, the training they provide in economic evaluation should cover core health economic principles and be infused with practical exercises based on actual clinical case studies. To this end, we propose five objectives for economic evaluation training within CTSA educational goals. The training provided should enable researchers to:

1. understand and master a set of basic analytic techniques used in performing cost-effectiveness and cost-benefit analyses;

2. describe and form strategies to quantify the tradeoffs in clinical and public health decisions regarding risks, benefits, and costs;

3. identify and describe the uses and limitations of these techniques;

4. critique and apply these methods in clinical decision making under uncertainty; and

5. identify appropriate data sources and analytic methods to be applied in a given research setting and interpret results.

Upon achieving these learning objectives, trained clinical investigators should be informed users of the cost-effectiveness and cost-benefit analysis literature, be able to establish or participate in a collaborative team environment with PhD-level health economists or decision scientists to answer a specific clinical question, be confident in incorporating economic evaluation as part of their research grant proposals if they so choose, and be able to provide constructive feedback as reviewers of research manuscripts or research grant proposals involving costs.

The Columbia CTSA Curriculum in Economic Evaluation

Each CTSA site addresses the consortium’s strategic goals in slightly different ways. Here we describe how the Irving Institute for Clinical and Translational Research, the CTSA at Columbia University Medical Center, addresses these goals and integrates an economic evaluation curriculum into CER training.

First, the KL2 Mentored Career Development program for junior faculty consists of a mentored research project, a research master’s degree, research rotations, and participation in multidisciplinary seminars. Second, the TL1 Educational Certificate Program allows predoctoral students in the population and public health sciences to take courses outside their field (such as courses in the basic sciences) so they can learn to apply a multidisciplinary approach to their research. Third, the program for a master’s degree in patient-oriented research includes comprehensive training in the skills required to conduct high-quality clinical and translational research, including quantitative research, critical thinking, and practical strategies to address the challenges of translational research. Fourth, the intensive Columbia Summer Research Institute (CSRI) for junior investigators offers further training in clinical research; topics include research design, statistical analysis, epidemiology and biostatistics, health disparities research, medical decision making, and cost-effectiveness analysis. In addition, our CTSA supports numerous institutional core research facilities, clinical trial and design consulting services, pilot collaborative awards, and medical facilities for conducting clinical research.²³

The curriculum in economic evaluation methods provided through these education programs typically includes a 10- to 16-session course, “Decision Analysis for Clinical and Public Health Practices” (spanning a semester or, in the summer institute, two weeks), as well as elective courses focused on outcome measures of quality of life and health economics. The core decision analysis course exposes trainees to basic techniques such as decision trees and entry-level Markov models, principles in diagnostic and screening tests (e.g., predictive values depending on both test and patient characteristics, how best to choose cutoff values), measurements of quality of life and quality-adjusted life years, and elements of cost-benefit analysis and cost-effectiveness analysis. Cost analysis training covers direct and indirect costs, discounting, and calculation of cost-benefit ratios, net health benefit, and incremental cost-effectiveness ratios. The trainees receive hands-on training in these skills using decision-analytic and spreadsheet software during computer lab sessions. Guest lectures and seminars focus on practical medical topics such as cancer screening, economic evaluation alongside clinical trials, ethical and political considerations, and disease-specific applications in clinical guidelines and regulations. Furthermore, trainees must select and conduct an in-depth critique of a published economic evaluation study in their own clinical area and present a potential research project design.

To date, clinical investigators who have participated in these programs have collaborated on research manuscripts and grant proposals that have shown initial success in integrating economic evaluation methods into patient-oriented research projects.^24,25 In the future, our CTSA site may offer an economic evaluation consulting service to assist clinical investigators with trial design, grant development, and clinical resident and fellow education. There are also plans to organize campus-wide seminars and workshops that draw together experts in decision analytic modeling, administrative databases, and population health research. With the successful implementation of the original strategic CTSA goals across U.S. academic health centers,²⁶ the future appears promising for enriching CTSA-based educational programs with training in economic evaluation and decision-analytic methods.

Conclusions and Future Directions

CER’s role in U.S. health care and guiding evidence-based medicine is evolving. Comparative evaluation of incremental effectiveness against incremental costs is inevitable in resource allocation decisions going forward. Therefore, there is a pressing need for both current and future clinical researchers to acquire economic evaluation skills, including cost-effectiveness and cost-benefit analysis, to meet the challenges of translating research from the bench to the bedside, and from the bedside to the real world of clinical practices and policy. With an emphasis on translational research and collaboration across disciplines, CTSA sites are well positioned to provide clinical investigators with the skills necessary to play an active role in leading, participating in, and collaborating on CER studies, as well as in shaping the future direction of CER. It is important to note that advancing CER is a national priority that demands the participation of all medical institutions. While CTSA sites have the infrastructure to serve as the starting ground for training future investigators in CER, true advancement will require curriculum development and training at all institutions that provide clinical research education. Moreover, CER should not be limited to post-graduate education but rather introduced and taught in undergraduate nursing, allied health, and medical education curriculums. Just as CER’s role in evidence-based medical care is developing, so too must education and training in CER be continuously advanced and refined within CTSA sites, and more broadly within medical institutions throughout the United States.