Skeptical of medical science reports?

“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as editor of The New England Journal of Medicine” (1).

More recently, Richard Horton, editor of The Lancet, wrote that “The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness” (2).

The first of these two commentaries on clinical research publications appeared in 2009, the second in April of this year. These statements are being taken seriously, coming as they do from the experiences of editors of two of the world’s most prestigious medical journals. The first article showed how the relationships between pharmaceutical companies and academic physicians at prestigious universities impacted certain drug-related publications and the marketing of prescription drugs. Potential conflicts of interest seemed to abound: millions of dollars in consulting and speaking fees to physicians who promoted specific drugs, public research dollars being used by a researcher to test a drug owned by a company in which the researcher held millions of dollars in shares, failure of university researchers to disclose income from drug companies, company subsidies to physician continuing education, publishing practice guidelines involving drugs in which the authors have a financial interest, using influential physicians to promote drugs for unapproved uses, bias in favor of a product coming from failure to publish negative results and repeated publication of positive results in different forms. The author, Marcia Angell, cited the case of a drug giant that had to agree to settle charges that it deliberately withheld evidence that its top-selling anti-depressant was ineffective and could be harmful to certain age groups (1).

Marcia Angell’s comments (1) were directed largely against conflicts of interest and the biases introduced by the influence of drug companies on researchers and universities. Richard Horton’s statement (2) was part of his comments on a recent symposium on reliability and reproducibility of research in the biomedical sciences and addresses a broader area of concern. Some of the problems he identified are seen in the veterinary literature. They include inadequate number of subjects in the study, poor study design, and potential conflicts of interest. He notes that the quest for journal impact factor is fuelling competition for publication in a few high reputation journals. He warns that “our love of ‘significance’ pollutes the literature with many a statistical fairy-tale” and he remarks that journal editors, reviewers, and granting bodies all stress original studies to the extent that “we reject important confirmations” (2).

Individuals and organizations considered responsible for the present state of published medical science include researchers, journal editors, reviewers, granting agencies, governments. Horton goes on to reflect on whether the bad practices can be fixed (2). He concludes that scientists have incentives to be productive and innovative but no incentives to be right. He muses on removal of incentives, emphasizing collaboration rather than competition, improving research training and mentorship, funding studies that attempt to replicate published data. Horton ended by noting that it is a good first step to recognize the problems but no one seems ready to begin the task of reversing the trends.

Clinical journals such as The Canadian Veterinary Journal are less affected by the fight for the impact factor because the primary impact that we seek to make is on the clinical practice community, rather than the research community (the journal impact factor is based on the impact on the researcher community). Nonetheless, we share some of the problems discussed above. Perhaps the most serious weakness is inadequate sample number in some studies. Such studies are sometimes accepted because they may have some value if care is taken to acknowledge the limitations associated with inadequate power. The take home message is that readers must exercise caution in interpreting the published literature, regardless of the reputation of the journal in which an article is found.