What Can Comparative Effectiveness Research Contribute to Integrative Health in International Perspective?

Abstract

The interest in Comparative Effectiveness Research (CER) in the international community is growing. A panel titled “What Can Comparative Effectiveness Research Contribute to Integrative Health in International Perspective?” took place at the 3rd International Research Congress on Integrative Medicine and Health in Portland, Oregon, in 2012. The presentations at this panel highlighted different perspectives on CER, including the funders' and the stakeholders' perspectives from the United States, as well as experiences with economic evaluations from Australia and pragmatic trials in Europe. The funders' perspective emphasized the need for innovation and controlling costs in large-scale studies. The stakeholder's perspective stressed the need to gather the input of stakeholders in shaping the framework for more informative, more decision-maker-driven research. Several examples of cost-effectiveness analyses were offered from Australia. The importance of balancing rigor and pragmatism was also discussed in a presentation of the efficacy–effectiveness continuum. A wide-ranging discussion explored additional questions concerning the translation of evidence into practice; the effect of pragmatic trials on funding or policy; evidentiary distinctions between and among pragmatic trials and traditional randomized clinical trials; and the multiple roles of stakeholders, particularly in generating new information and knowledge. The presentations and discussions showed that more development of methods is needed. This includes developments on study design and statistical approaches, as well as methods for stakeholder involvement and mechanisms to bring these results into practice.

Background

Despite the decades-long effort and the enormous economic and intellectual investments of clinical health care researchers worldwide to provide solid evidence of the efficacy and effectiveness of medications, therapies, interventions, and procedures, the randomized clinical trial (RCT) has yielded relatively little “actionable intelligence” for decision-makers. For patients, payers, caregivers, and policymakers, the evidence that has thus far been generated often provides a strong endorsement of an intervention's efficacy (thus answering the question “Does it [the intervention] work as we expected it to?”), but fails to provide clarity for decision-makers confronted by choices between and among a variety of options. Moreover, when an intervention has been proven to “work,” it often does so in comparison to a placebo, only for an exclusive population of patients, and under ideal, tightly controlled laboratory conditions. This rigor has, unfortunately, largely provided results that are poorly generalizable and that require inordinate amounts of time and financial resources to generate.¹

In the early and mid-1990s, primarily in the United States, the United Kingdom, and Germany, the methodologic constraints associated with the RCT became the focus of study and experimentation. The concurrent emergence of interest in examining the efficacy and effectiveness of complementary and alternative medicine interventions (driven by the increasing use of complementary medicine medications, approaches, and therapies), such as acupuncture, coincided neatly with this methodologic curiosity. Some early studies,^2–4 which broadened inclusion criteria and examined a variety of outcomes, pointed to a new way of understanding the RCT, now known as pragmatic trials and seen as an element of comparative effectiveness research (CER).⁵

CER evolved in the United States and pertains to research that aims to generate evidence of critical importance to decision-makers and succeeds by suggesting that flexibility in study design offers researchers opportunities to sacrifice some degree of internal validity in exchange for generating results that are timely, and more generalizable, feasible, and inclusionary than in traditional studies. Further, in the conceptualization of research questions and study designs, stakeholders—such as patients, patient caregivers, payers, and policymakers—are invited into the process. Stakeholder input emphasizes the sorts of questions and outcomes about which decision makers require guidance.⁶

Integrative health care is a field particularly well suited for CER investigations. Medicine is only one aspect of health, and for future health care it is important to explore health care with a wider view that also considers other aspects, such as social and physical environment. The term “integrative medicine” is widely used and has been defined by the Consortium of Academic Health Centers for Integrative Medicine as “the practice of medicine that reaffirms the importance of the relationship between practitioner and patient, focuses on the whole person, is informed by evidence, and makes use of all appropriate therapeutic approaches, healthcare professionals and disciplines to achieve optimal health and healing.”⁷ The term “integrative health” is also widely used but still needs a comprehensive definition.

Because it often concerns use of interventions from both fields, conventional and complementary medicine in a holistic effort to meet many patients' needs (complicated by comorbidities and comedications, particularly in treating chronic disease), integrative health care points to patient populations that are often excluded from traditional RCTs. By fostering study designs that emphasize broader inclusion criteria, more wide-ranging outcomes, subgroup analyses, stakeholder input, usual care settings, and flexible treatment protocols, CER has the potential to yield important evidence for patients receiving integrative health care.⁸

The interest in CER is growing in the international community, and this article summarizes the presentations at a panel titled “What Can Comparative Effectiveness Research Contribute to Integrative Health in International Perspective?” The panel took place at the 3rd International Research Congress on Integrative Medicine & Health in Portland, Oregon in 2012. It was jointly organized by the International Society for Complementary Medicine Research and The Institute for Integrative Health; contributors came from different continents and offered a variety of perspectives on CER, including the funders' and the stakeholders' perspective from the United States, as well as experiences with economic evaluations from Australia and pragmatic trials from Europe (Fig. 1).

FIG. 1.

The symposium presenters. CER, Comparative Effectiveness Research.

Presentations and subsequent discussion were recorded and analyzed to provide a comprehensive paper that summarizes the key points and provides some synthesized responses to additional issues with regard to CER.

Michael S. Lauer: The Funder's Perspective on CER

The National Institutes of Health (NIH) budget has been declining steadily since 2003; the budget continues to shrink. Overall success rates for grant applicants are running at approximately 15%. Such financial conditions pose existential threats for laboratories and require a considered re-evaluation of priorities. Expensive clinical and epidemiologic studies are not sustainable in the current funding climate; despite large, supportive constituencies that recognize the essentiality of the clinical research enterprise, fresh scrutiny of research impacts is required.

Perspectives on this crisis vary, and it has even been suggested that “ the NIH should not fund large clinical studies that divert hundreds of millions of dollars away from hypothesis-driven scientific research.”⁹ Nonetheless, the budgetary crisis presents a tremendous opportunity for challenging communities that do epidemiologic and clinical studies to change their business practices and explore new models of clinical research design that drive generalizable, meaningful results into the hands of decision-makers in a timely and affordable fashion.

It is possible to do large-scale, high-impact trials at very low cost. New technologies and tools, such as information technology, mobile phone technologies, and electronic medical records make it feasible to totally reinvent the clinical trial enterprise. It's not just aspirational—it absolutely must be done.

Sean Tunis: The Involvement of Stakeholders as an Important Aspect of CER

Dr. Sean Tunis discussed the development of effectiveness guidance documents (EGDs) in an international context. Much of the work has not taken place in the context of integrative medicine, with the exception of the development of an EGD for acupuncture research. The general model and approach are very adaptable, particularly for integrative medicine; generalizable; and important.

The whole methodologic framework of CER operates on the premise that the engagement of the end users of research results is stakeholder informed. The challenge is how to get stakeholders and major decision-makers to align around evidentiary expectations.

The American College of Physicians guideline has identified—in the current and dominant framework of conducting randomized trials—some methodologic deficiencies (lack of comparative studies, outcomes that are not clinically relevant, follow-up periods that are too short). Unfortunately, neither that group nor any other has taken the next step to turn these observations into good recommendations. In the case of studies on Alzheimer disease, some areas that remain unresolved (1) the identification of appropriate comparators for new drugs, (2) the identification of more appropriate and meaningful outcomes, and (3) definitions of appropriate follow-up.

It is essential to explore the information needs of patient advocacy groups, patients, and patient caregivers, but this is clearly not enough. The stakeholders, including the payers, can help shape the framework for more informative, more decision-maker–driven research. In addition, however, regulators, the professionals with extensive experience in trial design and methods, are required to translate these information needs into guidelines that shape viable, methodologically sufficient studies.

Design should be informed by stakeholder (patients and clinicians) input but left up to the experts. From an international perspective, the Green Park Collaborative provides a good example. The group was formed in London in 2011 to recommend study designs for researchers working on Alzheimer disease. The group includes Francoise Meyer (France), Sue Hill (Australia), Ryan O'Rourke (Canada), Carole Longson (United Kingdom [National Institute for Health and Care Excellence]), Finn Christenson (Denmark), Paolo Siviero (Italy), and representatives from Singapore and South Korea. The Green Park Collaborative also includes a patient-citizen representative and representatives from the pharmaceutical industry.

The Green Park Collaborative is considering a recommendation to include caregiver quality of life as a measure of the effectiveness of Alzheimer drugs. Of course, regulators would never consider incorporating these sorts of measures. From a patient and family perspective, it is a hugely important issue, and there are some validated ways of looking at that.

This example provides one example of how methodologic recommendations can be deployed to focus on high-priority domains within integrative medicine. This group may well take on additional topics in integrative medicine.

Suzanne Grant: The Options and Relevance of CER Economic Evaluations, Including Cost Evaluations: An Australian Perspective

In Australia, consumers directly pay for complementary medicines, or, if the consumer is among the 51% who have private health insurance, they may receive a partial rebate for the product or practitioner service. Some limited government funding exists for therapies such as acupuncture, osteopathy, and chiropractic. Since 1993, the Australian government has required that cost-effectiveness must be considered for any medicine to be publicly subsidized through the Pharmaceutical Benefit Scheme or for any treatment to be considered under Medicare. In When considering clinical effectiveness, data from head-to-head comparative studies are preferred.

In the Australian context, limited data on the cost-effectiveness and comparative effectiveness of complementary medicine and therapies are barriers to public funding or support of these therapies.

The National Institute of Complementary Medicine has conducted cost-effectiveness analyses of five complementary medicine interventions in the Australian context.

The complementary medicines and health conditions were selected for study by a reference group if they met such criteria as having a high burden (e.g., loss of quality of life, productivity, comorbidity), a clearly defined intervention with adequate comparative effectiveness evidence available, and a well-defined comparator or standard treatment. The interventions selected were acupuncture for chronic low back pain; St. John's wort for mild-to-moderate depression; fish oils for prevention of heart disease among patients who have experienced myocardial infarction; fish oils for rheumatoid arthritis; and a proprietary product, Phytodolor (a tree bark and plant extract from Populus tremula, Fraxinus excelsior, and Solidago virgaurea), for osteoarthritis.

Cost-effectiveness of the interventions was determined through meta-analyses. Costs were calculated on the standard cost for an acupuncture consultation and standard costs for fish oils. The analysis assessed how much the treatment reduced health care system costs through reduced adverse events. Only the direct health system costs were included in each analysis. Indirect costs, such as absenteeism and carer costs, were excluded. Results were expressed as cost per disability-adjusted life-year (DALY). Parameters were defined for mortality, morbidity, and adverse events for each disease. A sensitivity analysis was conducted for each intervention to take into account different parameters.

World Health Organization guidelines were used to define a cost effective intervention. These guidelines articulate cost-effectiveness in terms of the cost to reduce disease burden by 1 DALY. The number of DALYs lost because of a disease is calculated according to mortality and morbidity of the disease. For each intervention, the costs and benefits associated with that strategy were compared with another treatment, such as standard care. A treatment may be considered cost-effective if the cost is one to three times the gross domestic product (GDP) per capita per DALY averted ($52,000AUD to $56,000AUD in 2009), while interventions costing less than 1 GDP per capita ($52,000AUD) per DALY averted are deemed very cost-effective. The GDP per capita in Australia was $52,000AUD in 2008–2009.

This cost-effectiveness analysis method was selected for two reasons. First, cost-effectiveness analyses compare relative costs and outcomes of two or more courses of action. The denominator is nonmonetary (gain in health measured as a DALY avoided) and the numerator is monetary (cost of the health gain). Second, sufficient evidence was available for some complementary medicines to meet the assumptions needed to complete a cost-effectiveness analysis.

For the first intervention, acupuncture for chronic low back pain, three scenarios were explored. For acupuncture and standard care versus standard care and sham, only two studies were suitable for inclusion. A weak positive effect was found for weeks 12–16, but the difference was not significant; thus, a cost-effectiveness analysis was not conducted. For acupuncture and standard care versus standard care alone, acupuncture was very cost-effective, costing less than $52,000AUD. In a sensitivity analysis, when depression was included, further savings were generated, and the intervention costs $19,000AUD per DALY averted. For acupuncture versus standard care, acupuncture was not generally cost-effective unless comorbid depression was relieved; if that occurred, then the cost per DALY averted was $62,946AUD—and therefore acupuncture was considered cost-effective.

In two meta-analyses, St. John's wort (SJW) was as effective as standard antidepressants for mild-to-moderate depression.^10,11 It was just as safe and effective as standard care; it had lower levels of adverse effects and fewer patient withdrawals from treatment. On the basis of these meta-analyses, treatment with St. John's wort rather than standard antidepressants may result in a reduction of $146 per patient per year. The total savings would depend on current utilization data and on whether both mild and moderate depression was treated. The main driver of this result was that the unit cost of St. John's wort was estimated as $0.17/day while the cost of standard antidepressants was estimated as $0.57/day. Standardization of a St. John's wort product may add costs.

Fish oil use among patients who have experienced a recent myocardial infarction is thought to prevent further cardiovascular disease, although recent research has brought this into dispute.¹² This cost-effectiveness analysis, conducted before this recent research, looked at the value of complementing standard care for patients who have had a myocardial infarction with omega-3 fish oil supplements; fish oil supplements were introduced within 3 months of myocardial infarction. Fish oils were highly cost-effective, even with sensitivity analyses on treatment variables (such as myocardial infarction, stroke, revascularization, coronary heart disease mortality, and other mortality). The total cost to administer fish oil to all patients with coronary heart disease is around $40 million per year, with approximately 19,000 DALYs averted.

The fourth cost-effectiveness analysis examined the cost-effectiveness of using fish oil supplements as an adjunctive therapy for rheumatoid arthritis (with lower nonsteroidal anti-inflammatory drug [NSAID] use) rather than standard NSAID therapy alone. The meta-analysis found that fish oil use can lead to long-term reductions in NSAID reliance, but systematic inclusion of fish oils as a main-line intervention for rheumatoid arthritis does not seem economically justified.

The last cost-effectiveness analysis, which dealt with Phytodolor for managing the pain and inflammation of osteoarthritis, lacked adequate data for a meta-analysis. The short timeframes for the studies; differences in study design, comparators, and outcome measures; and differences in dosages made a meta-analysis difficult. Phytodolor has a good safety profile compared with that of cyclo-oxygenase-2 inhibitors and has been found to be as effective as diclofenac.

The five economic evaluations reported contribute to a small but growing body of evidence on the cost-effectiveness of complementary medicine interventions. The methods used in these five cost-effectiveness analyses have some limitations but offer sufficient rigor. The noninclusion of indirect cost is likely to have rendered more conservative conclusions.

Claudia M. Witt: The Efficacy and Effectiveness Continuum in Clinical Trials: Between Rigor and Pragmatism in Clinical Trials—A European Perspective

To improve understanding of the early acupuncture trials in Europe and the efficacy-effectiveness continuum, it is helpful to consider some fundamental definitions. Efficacy is defined by the extent to which a specific intervention, procedure, regimen, or service produces a beneficial result under ideal conditions. Ideally, the determination of efficacy is based on the results of an RCT—a typical drug trial would be the best example—in which the protocols are standardized and the inclusion criteria are very narrow.¹³ By contrast, effectiveness is a measure of the extent to which a specific intervention, procedure, regimen, or service, when deployed in the field in the usual circumstances, does what it is intended to do for a specified population.¹³

In clinical research, effectiveness on a high evidence level is best demonstrated by pragmatic trials.⁵ In a pragmatic trial, patients usually seen in normal practice are represented in the study population and the treatment protocol allows more flexibility. It is crucial to understand that efficacy and effectiveness are at opposite poles of a continuum. Any study can be placed somewhere along this continuum—and only few studies are at one or the other extreme. Where a study is placed depends on the details of the study design. In this context, four dimensions of the study design play an important role: eligibility criteria, treatment protocol, outcomes, and study context.

To assist researchers in working their way through the twists and turns of designing CER studies, EGDs) are being developed. The protracted and robust discussions that have accompanied working groups' efforts to develop EGDs enrich the final products. The discussions are characterized by much negotiation between and among scientists, practitioners, patients, and payers. The EGD on acupuncture that has been developed thus far provides very detailed guidance on methodologic issues in designing CER on acupuncture.¹⁴

Integrative medicine has contributed to CER. Sean Tunis's landmark paper in JAMA in 2003 moved thinking in conventional medicine forward.¹ The early pragmatic trials on acupuncture took place in the United Kingdom and started patient recruitment in 1999,^2,15–18 followed by large acupuncture trials in Germany, in which patient recruitment began in 2001.^4,19,20 Acupuncture was always provided in addition to usual care to a broad patient population, allowing an individualized treatment in a usual care context. Interestingly, all pragmatic acupuncture trials in Germany and the United Kingdom were accompanied by cost-effectiveness evaluations, showing that an additional acupuncture treatment was relatively cost-effective within international thresholds. That acupuncture research is a spearhead for integrative medicine in the field of CER is also supported by a systematic review on acupuncture for low back pain, which concluded that existing acupuncture trials already contribute to CER.²¹ Other fields of integrative medicine can learn from this experience.

Discussion

Discussion with the audience was wide-ranging and focused primarily on the effect of pragmatic trials on funding or policy; evidentiary distinctions between and among pragmatic trials and traditional RCTs; and the multiple roles of stakeholders, particularly in generating new information and knowledge.

The impact on funding policy

The discussion highlighted that important decisions are often made with very little evidence. Further, having evidence does not mean that such evidence is translated into practice. The pragmatic acupuncture trials in Germany had no impact on reimbursement decisions. In Germany, the fact that acupuncture is reimbursed for chronic low back pain and osteoarthritis pain was based solely on trials that also included sham acupuncture arms. By contrast, in the United Kingdom, the pragmatic trial on low back pain had a positive effect on National Health Service coverage.

Examples from conventional medicine in which results from CER did not change practice were also offered. The most prominent example was the Antihypertensive and Lipid Lowering Treatment to Prevent Heart Attack Trial (ALLHAT).²² ALLHAT, the largest and most important clinical trial in hypertension ever done in the United States, demonstrated that a diuretic was as effective as a calcium antagonist, an angiotensin-converting enzyme inhibitor, and an α-adrenergic blocker; there was no change in clinical practice.

Type of evidence

The hierarchy of evidence and what constitutes sufficient evidence to change a national coverage policy—in the United States or Europe—has often been tied to a high level of certainty as to whether a particular treatment was effective for a specific patient population. However, as payment reform takes hold, and the risk is delegated from private payers alone to a risk shared by providers and patients, it is likely that a wider range of evidence will be considered sufficient.

While some discussants were concerned that CER studies with multiple comorbidities and comedications might weaken measures of effectiveness, it was suggested that subgroup analyses in pragmatic trials will require a larger sample size because there will be more variance in the data. Consequently, a larger sample size is needed to obtain a meaningful statistical difference. Exploratory analysis also offers an opportunity to generate hypotheses.

Participants in the discussion acknowledged that the presenters had not addressed observational studies. The participants noted that with randomized clinical trials, we regard the evidence as more or less on the “safe side”; we have confidence in the method. Yet in observational studies, more methodologic development is necessary to ensure the reliability of the evidence such studies generate. Some development is underway, but more work is needed in this area.

Stakeholder involvement

In cases where it's essentially impossible to give definitive, reliable, accurate, and precise information to patients about risks and benefits for alternatives, and for decisions in which uncertainty exists, the collaboration attempts to identify and explore the decision points associated with the highest levels of uncertainty. A knowledge-generating enterprise is needed to address the questions that stakeholders confront in the face of rising health care insurance and other health care costs. Relevant, useful, accurate information is needed—the sort of information that the evidence-generating, consumer-driven CER agenda will produce.

The role of stakeholders in this process was seen as critical. It is essential to leverage patient insights in the decision-making process to inform the design and implementation of better information. Shared decision-making processes can act as an input into evidence generation. A remaining challenge in this domain is in communicating across a variety of stakeholder communities who don't speak the same language. In addition to developing a common language and understanding of each community's point of view, these conversations must take place because they highlight each community's vested interests, access (or lack thereof), and priorities. The conversations also build trust and provide the mechanism in which negotiations and information-sharing can take place. As an example, the Center for Medical Technology Policy is collaborating with the Foundation for Informed Medical Decision-Making (which develops shared decision-making programs based on multiple focus groups with patients) to deeply understand what the alternatives are and to help patients make informed decisions. Another organization that supports patient-centeredness and stakeholder engagement is the Patient-Centered Outcomes Research Institute, which “helps people and their caregivers communicate and make informed health care decisions, allowing their voices to be heard in assessing the value of health care options”.²³

Methods development

From the methodologic perspective, two points were highlighted: Some value was seen in study designs that consider patient preferences (e.g., two-stage randomized trial designs) and the need to develop these designs further. In addition, a strong need for innovation among statisticians to take up the challenge to solve the “right problem”—instead of the easy solution to a pristine problem—was considered essential. Stronger and more robust interactions with statisticians and nonstatisticians are also encouraged.

Conclusion

From the presentations and discussions, it became clear that even as more pragmatic trials and studies are published, more methods development is needed. This includes developments on study design and statistical approaches, as well as methods for stakeholder involvement and mechanisms to bring these results into practice. In the United States, the Patient-Centered Outcomes Research Institute is devoting a substantial percentage of its budget to the development of methods and tools for CER. Development of these tools is critical to advancing the entire CER agenda. However, other countries, such as Australia and Germany, have not yet developed these mechanisms.

Acknowledgment

The symposium was organized by the International Society for Complementary Medicine Research in cooperation with The Institute for Integrative Health.

Author Disclosure Statement

No competing financial interests exist.