Reporting findings in sleep medicine: is it time for some spin control?

Clinical trials are predicated on the concept of equipoise: we know the existing situation, and have a proposed new treatment, intervention, or diagnostic method that may result in improvement in some meaningful outcome, whether that be longer life, less illness, or cheaper treatment. That improvement is not known as a certainty, and so must be tested in a robust manner. Randomized controlled trials (RCTs) are, in many circumstances, the best mechanism for testing a hypothesis in an unbiased manner, and are generally considered the gold standard for evidence.

Unfortunately, a prior era of quiet non-publication of results of RCTs that did not support the initial hypothesis [1], or post hoc selection of outcomes to report based on whether results supported the hypothesis, limited the impact that adoption of RCTs had on producing greater rigor and reproducibility in science. In response, there has been an aggressive push for preregistration of clinical trials, whether randomized or not, which is now almost universally required by journals [2] and many funders. This means that outcomes are considered at the design stage: the primary outcome(s) must be defined clearly, with sample size justification expected. Other important outcomes may be listed as secondary or exploratory, but registration also requires specification of what these will be and how they will be measured. Such planning is also expected by ethics committees or review boards, as the information to be gained about potential outcomes must justify the risk to participants.

Does this preplanning extend to the final publication of research outcomes? Analysis of published trial outcomes in other fields in recent years suggests not, or at least not universally [3], with selective reporting or emphasis to make outcomes look more positive, termed “spin,” commonly observed [4–6]. Examples of spin include focusing on an alternative or secondary outcome rather than the primary that was defined at registration, highlighting a subgroup comparison rather than results in the overall population, or highlighting a within-group change and ignoring the control arm. The occurrence of spin had not however been specifically assessed in the field of sleep until recently.

In this issue of SLEEP, Guo et al. [7], screened recently published articles in the main “sleep” journals (including but not limited to SLEEP) for the presence of spin among RCTs where there was a neutral finding for the primary outcome. The authors also specifically noted other characteristics including whether trials were sponsored or investigator-initiated, and whether there was the involvement of a statistician, based on affiliations or listed qualifications.

They found that spin was very common, across multiple journals, and study types. Spin was slightly less common if the manuscript included a statistician as coauthor, possibly reflecting preplanned statistical analyses, and strong expectations regarding the benefits of trial interventions are probably less likely to be held by methodologists than their clinical colleagues. It was also slightly less common in sponsored trials, likely reflecting a higher level of preplanning and rigor where the outcomes are intended for submission to regulatory authorities, as well as the involvement of an “agnostic” methods and analysis team.

Although there are certainly occasional cases of research fraud, it seems likely that in most cases of spin, there is no malicious intent to deliberately deceive. Investigators presumably begin each trial optimistic about its potential benefits, and then at its conclusion want to have something to say for all the time, effort, and money that has been invested. Positive outcomes are appealing to readers (and reviewers and editors), more widely reported, and no doubt there is at least subconscious pressure on authors to “put their best foot forward.”

However, inherent in the equipoise concept is the possibility that proposed interventions may not result in a meaningful improvement, or may even be harmful. Trialists must also be prepared to report these outcomes, both from an ethical perspective and to accurately convey the results to casual readers. Given the volume of scientific literature published annually, no single individual can possibly read and remember everything in full detail, and the reality is that most people will skim a table of contents email alert or keyword search results on PubMed. As this is all that many people will ever read, it is critical that the title and abstract accurately reflect the full contents of the manuscript, and should be an accurate representation of the work that has been done. There may of course be legitimate reasons that the final work does not reflect the original plan (other work published in the interim, a change in the recommended “standard care” since the design stage, a global pandemic that derailed the intended work, etc.), in which case, alterations can be clearly stated and justified.

The other important implication from the work presented by Guo et al. [7] is that subject-matter experts would be well served to involve methods experts as genuine collaborators. For example, including trial methodologists at the design stage, applying relevant expertise in recruitment and retention strategies, and engaging with statisticians for expert analysis, will add value to the research enterprise. Just as no individual reader can keep up with the pace of modern scientific publication, so no single researcher can be an expert in all of these areas. Collaboration will improve the quality of the work itself and its reporting.

Most journals today have requirements for preregistration of RCTs as a requisite for publication. However, the impact of this preregistration on improving the rigor and reproducibility of findings in our field will not be fully realized until authors are required to follow through on the preplanned analysis and reporting of results. Preregistered plans are publicly available in repositories such as clinicaltrials.gov. If publications veer from this pre-specified plan to offer spin, one must consider whether this reflects a failure of reviewers and editorial boards along with authors themselves. Journal editors should consider revising their instructions to authors to make clear that not only does preregistration need to have occurred, but the manuscript must report results in light of that registration. Furthermore, editors need to consider making explicit whose responsibility in the review process it is to ensure adherence. The short-term benefits of spin in terms of generating news headlines for authors, journals and the field, are offset by the long-term waste of scientific resources as future studies are designed on faulty or incomplete premises. Furthermore, it risks eroding public trust in our endeavors, and creates needless suffering as patients pursue ineffective therapies.

Reviewers, editors, and indeed readers should expect that not all results will be positive. In fact, if the majority of trials are not negative, we are being overly conservative and arguably not investing sufficient resources to advance the care of patients with sleep disorders. As a scientific discipline, we must remain objective in evaluating data and humble in recognizing that given the complexity of biological systems and human behavior, our hypotheses will often prove wrong. Neutral or negative findings can force us to reassess what we thought we knew. Relinquishing our assumptions, rather than zealously holding them despite new evidence, will advance our field and unlock the potential for sleep and circadian science to grow into a mature biomedical field that can contribute to improving human health and well-being.

Funding

None.

Disclosure Statement

Nonfinancial disclosure: None declared, though drafting this editorial did result in some moments of self-reflection for its authors. Financial disclosure: SRP has received grant funding through his institution from Bayer Pharmaceuticals, Philips Respironics, Respicardia, and Sommetrics and has received personal income from consulting for ApniMed, Bayer Pharmaceuticals, NovaResp Technologies, Philips Respironics, and Powell Mansfield Inc, all unrelated to this work. KAL reports research funding to her institution from Vanda Pharmaceuticals and Takeda Pharmaceuticals, unrelated to this work.