Comparative Effectiveness Research: An Approach to Avoiding “Overgeneralized Medicine”

For many diseases, the only evidence we have about different drugs—particularly new ones—comes from a few small, short (6 weeks is typical), placebo‐controlled trials conducted by a manufacturer to obtain approval from the FDA to market the drug. It is now universally agreed that such studies are an insufficient guide to practice. ¹ Usually, they recruit a highly unrepresentative sample of subjects, especially those who are free of other diseases, are particularly likely to take all of their pills, and, too often, are middle‐aged and white. They often use blood tests or other easily obtained measures instead of outcomes patients care about. These trials provide very little information to judge how the drug will work—and how safe it is—in the long run, especially in people who, like many of us, would not have qualified for these trials.

In March 2008, the American College of Physicians and American Academy of Family Practice (ACP‐AAFP) comparative effectiveness report on drugs for Alzheimer's disease illustrated and pointed to the cause of the problem. The report covered five drugs tested in 59 trials, nearly all of them conducted by pharmaceutical companies to obtain FDA approval. ² The authors found that instead of focusing on outcomes that are most important to patients and physicians, the studies used short‐term, convenient measures that are “not used in routine clinical practice.” They found that “improvements (in these measures), although statistically significant, were not clinically important or their relative importance cannot be determined at this time.” For example, nearly all of the trials measured scores on memory tests instead of, say, whether the patient could find his or her way home from a bus stop, or whether use of the drug delayed the need for full‐time care in a nursing home. For the test scores, the great majority of the trials presented averaged results that made it impossible to determine an individual's chance of responding to the drug. For example, on average, compared with placebo, patients taking one of the drugs, galantamine, improved three points on a memory test. However, anything less than a four‐point improvement is considered by experts as not clinically important, meaning that you would not be able to notice a difference in the patient's function. So, the reviewers asked, “What percentage of patients had an improvement of four or more points?”—a more meaningful question if you are a patient and want to know what chance of improvement the drug offers. But only 5 of the 10 studies of galantamine reported this information. Appropriately, the American College of Physicians concluded that the data were not sufficient to determine an individual's chance of responding to the drug, in part because sparse and selective reporting of patient‐specific outcomes raises suspicion that the other five studies may have had disappointing results. Finally, only 3 of the 39 studies compared a drug for Alzheimer's disease to another active drug; the vast majority of the trials merely showed that, mathematically, the drug was superior to placebo.

When we prescribe these treatments widely even though we know little about them, we practice “overgeneralized medicine”; for most patients, we do not even know the likelihood of benefit in the short run, or anything about the benefits and risks in the long term. “Overgeneralized medicine” persists because physicians are usually willing to prescribe widely even when little is known about the actual long‐term benefits and harms. In many cases, it takes years for it to become evident that the supposed benefits were less impressive than we hoped, and the harms worse than expected.

Fortunately, there is a way forward. Comparative effectiveness research—research that directly compares alternative treatments and which includes a broad range of patients encountered in everyday, community practice—helps focus attention and research on patient‐important outcomes and delineates how an individual's characteristics influence the balance of benefits and harms. ¹ , ³ Only comparative effectiveness research that focuses on these populations can correct the information gaps left from premarketing trials. The ACP‐AAFP report brought attention to the fact that, despite an impressive‐sounding number of research studies, we know almost nothing about whether many drugs meaningfully improve life for patients and their families, and how they compare to other drugs.

What is less widely known is that comparative effectiveness research has an important role to play in addressing disparities in health care. This is because the risks of “overgeneralized medicine” are particularly severe for those who are most often underrepresented in premarketing trials—members of racial and ethnic minorities, the elderly, children, and individuals with multiple medical conditions. Both for the general population, and especially for these more vulnerable populations, funding must be provided for comparative effectiveness research that directly addresses the comparative benefits and harms of alternative treatments in these groups.

For several years the Federal government has supported the development of an infrastructure to conduct comparative effectiveness research in populations that, historically, have been underrepresented in research. The federal Agency for Healthcare Research and Quality (AHRQ) supports practice‐based research networks that focus on underrepresented rural and urban populations. These networks are voluntary collaborations among physicians who are motivated to identify the treatments that most improve the quality and length of their patients’ lives.

Although most federally‐sponsored comparative effectiveness research has been done by AHRQ, recently, the National Institutes of Health (NIH) Clinical and Translational Science Awards (CTSAs), given to 46 leading universities in this country (ultimately to be 60), have brought together researchers from across the biomedical and clinical research spectrum, including those who do comparative effectiveness research and those particularly focused on communities and the disparities in health care among minorities and special populations. Thus it seems wise that, in allocating $1.1 billion to comparative effectiveness research, the Department of Health and Human Services place a high priority on answering questions about effectiveness and safety in communities and populations neglected in other types of research.

Yet, in a Wall Street Journal article this January, ⁴ a former FDA official, Scott Gottlieb, argues against federal funding of this comparative effectiveness research. He recommends that private companies take the lead in conducting these studies. At the same time, he argues that comparative effectiveness studies are expensive and that the amount of funds for this research in the legislation—$400 million for NIH—will not go far. It is certainly hoped that private companies, including pharmaceutical companies, will support comparative effectiveness research that provides the evidence we need to make better‐informed decisions, but it is essential that there be industry‐independent transparent federal government sponsorship as well. Without this, if patients seek, and physicians are willing to prescribe, treatments broadly on the basis of the very limited data from premarketing trials— overgeneralizing—the manufacturer has no motivation to get more specific, patient‐important data that would reduce the size of the market for these drugs. The PROVE‐IT trial ⁵ and many other influential, important comparative effectiveness studies were conducted by private companies, but the industry showed no interest in others that were at least as important to patients, particularly those that directly compare generics and branded drugs. Examples are the federally funded ALLHAT ⁶ trial of blood pressure medications and the CATIE ⁷ trial of antipsychotic medications.

Comparative effectiveness research is a public good. It provides better information for better decisions, and is a wise investment. It is clear that some would like to keep such research from informing decisions and influencing the market for drugs and other interventions. However, the public deserves better, especially the particularly vulnerable populations at higher risks from “overgeneralized medicine.”