Trust, Verify and Replicate

A number of studies have reported an association between obstructive sleep apnea (OSA) and metabolic syndrome.^1–4 However, until 2 years ago, it was unclear whether metabolic syndrome could be reversed by treatment of OSA with continuous positive airway pressure (CPAP). Studies were uncontrolled, small in size and conflicting in their results.^5,6 Thus, the purported well designed and conducted report in a high profile medical journal in 2011 that metabolic syndrome could be reversed with 12 weeks of CPAP treatment was welcomed by some in the medical community as another piece of evidence supporting the practice to aggressively treat moderate to severe OSA.⁷ Nevertheless, there were some doubts expressed about the validity of the results.^8,9 Several weeks ago, the other shoe finally dropped. The authors of the 2011 study retracted their paper writing that they were unable to locate and verify some of their primary data.¹⁰ Although they assert that their conclusions remain valid, this claim is difficult to believe given the reasons for the retraction.

What have we learned about this retraction of a high profile paper? One lesson should be that “replication is a necessity of the scientific process.” One analysis of highly cited clinical research papers found that results were not confirmed in 16% of cases.¹¹ Were all of these authors guilty of scientific conduct or sloppy research? Most likely not. In many cases they were likely complicit to our worship of the “p value.” In today's science, results are considered significant and therefore “true” if the p value of a statistical test is less than or equal to 0.05, or less than 1 in 20. Conversely, there is a 1 in 20 chance or less that results are “not true.” This is the most compelling reason for replication. The more times that identical or similar studies find the same results, the greater likelihood that the findings are indeed correct. Other explanations for the failure to replicate include prevailing bias and study design issues.¹² Nevertheless, scientific misconduct and fraud do exist and potentially may adversely influence public opinion concerning scientific research.¹³

The second lesson that should be taken to heart by all of us who participate in research is that we need to take some personal responsibility for papers we co-author. Frequently, co-authors or senior authors are not the primary persons who acquire or analyze the data. As a group, we need to take greater responsibility in verifying and questioning results that do not seem “right,” i.e., difficult to believe, implausible or perhaps too perfect. We also need to take collective responsibility for conducting high quality research including accurate data and analysis.

Finally, for the practitioner, you need to interpret and utilize the results reported in journal articles in the context of your practice and the potential impact on patients. Caution and awaiting replication is reasonable for most patients. For others with serious or soon to be life-threatening conditions, embracing a new technology or treatment approach may be the only option.

Obviously, the process by which scientific and medical advances are made is imperfect. When asked about nuclear disarmament with the former Soviet Union, President Reagan quoted an old Russian proverb “Trust, but verify.”¹⁴ In the context of medical sciences, we should trust, verify and replicate.

DISCLOSURE STATEMENT

Dr. Quan is the Editor-in-Chief of the Journal of Clinical Sleep Medicine.

CITATION

Quan SF. Trust, verify and replicate. J Clin Sleep Med 2013;9(12):1233.