Current Resources for Evidence-Based Practice, September 2020

Abstract

An extensive review of new resources to support the provision of evidence-based care for women and infants. The current column includes a discussion of “spin” in scientific reporting and its effect on summaries and syntheses of the literature and commentaries on reviews about early versus late amniotomy as part of labor induction protocols and the economic burden associated with maternal morbidity.

A review of new resources to support the provision of evidence-based care for women and infants.

Marit L. Bovbjerg, PhD, MS

On the Perils of Skimming Articles

This column’s literature searches yielded an interesting systematic review by Ghannad et al. (2019). Rather than reporting results of a clinical question per se, the authors analyzed how those results were reported. Specifically, they looked at the literature on biomarkers as screening, diagnostic, and prognostic tests for ovarian cancer. Drawing on a robust existing literature on “spin” in presentation of research findings for their analysis, Ghannad and colleagues (2019) evaluated the most recent 200 studies on biomarkers for ovarian cancer and looked specifically at titles, abstracts, and conclusions compared to the main findings as reported in the results sections.

Spin of research findings comes in two forms: misrepresentation, defined as “misreporting and/or distorted presentation of the study results in the title, abstract, or the main text, in a way that could mislead the reader,” and misinterpretation, defined as “an interpretation of the study results in the abstract or main-text conclusion that is not consistent and/or is an extrapolation of the actual study results” (Ghannad et al., 2019, p. 11). In other words, authors conducted their study and found X, but in the title, abstract, and/or article conclusion they reported X+Y.

I assume readers of a column entitled “Current Resources for Evidence-based Practice” are familiar with the concept of publication bias, wherein studies without exciting results (often defined, unfortunately, as results that are not statistically significant) are rejected for publication. This leads to errors when one is attempting to conduct systematic reviews or meta-analyses because the search of published literature only produces papers that were indeed published. Thus, the review can only include some of all studies actually conducted because the rest never made it into print. The resulting review is therefore biased because by definition it does not truly encompass what is known about a topic. Publication bias as described here applies to the entire literature on a particular topic, not to any one study specifically.

Spin is likely a direct consequence of the process that leads to publication bias. When faced with research findings not deemed interesting enough to be worthy of publication, an author has a few options. One is to give up completely on publishing the work, which leads, of course, directly to publication bias. This is absolutely not ideal; publication bias aside, who wants to see months, sometimes years of work collecting, analyzing, and reporting data lead to nothing? Unfortunately, this is the simplest option and chosen not infrequently (hence, publication bias).

A second possibility is to reformat the work in some way to try and make it worthy of publication. You could combine results from two studies, for instance, or rework the paper so it makes a methodologic rather than substantive point. This latter approach is one I used for a paper stemming from my dissertation work several years ago. With help from my advising committee, I found maternal physical activity during pregnancy had little bearing on risk of cesarean. The results were decidedly not statistically significant. Despite this being a worthwhile thing to know—as evidence-based practitioners we want to know what does not work as well as what does—seven journals rejected this so-called negative results paper. Eventually tiring of beating my head against that wall, I rewrote it to focus on our analytic choices instead of our findings, after which it was published immediately. Importantly, I was still able to present the findings that physical activity did not correlate with cesarean birth in our sample. These findings were just now couched within a larger discussion of how to analyze data on physical activity during pregnancy instead of within a discussion on whether physical activity could prevent cesareans (Bovbjerg et al., 2015). Lest I portray myself as some paragon of publishing virtue, I’ll admit the other paper that stemmed from my dissertation also did not contain exciting results, and when faced with the huge amount of work to redo it, I availed myself of Option 1.

The final option for a troublesome paper is spin. Here, the authors rework their paper slightly so that the results seem more exciting than they really were, hoping these slights-of-hand are not called out by peer reviewers (assuming they were done consciously, which might not always be the case). This almost certainly happens behind the scenes as well, as authors choose perhaps only to present the more interesting results instead of all results. In the Ghannad et al. (2019) article on the biomarkers for ovarian cancer literature, 70% of studies included one or more forms of spin. Paper or abstract conclusions claimed a biomarker worked when the results were not statistically significant or when p-values and confidence intervals were not presented in the results section (and one must assume they were not favorable), numbers in the abstract did not match numbers in the text or tables, the abstract or title mischaracterized the extent of biomarker testing (e.g., claiming it predicts ovarian cancer deaths when the study endpoint was 6-month recurrence), and so on. The end result is that the literature is saturated with claims of biomarkers that can assist with diagnosis or prognosis for ovarian cancer, but few of these have panned out in practice.

It is tempting to place the blame for spin squarely on authors, and indeed they are not blameless. However, much like when we consider how to reduce cesareans, we must look at the system as a whole rather than blaming any one class of person or any one individual person. I think the issue is larger than individual authors. It must be, for 140 out of 200 examined studies to have been written in a misleading way (Ghannad et al., 2019). Ghannad et.al. did not report on funding sources for these studies, but I wonder how commercial interests affect spin.

The system of publish-or-perish in academia has led to an ever-increasing number of global publications, as university administrators and granting agencies demand ever-more research productivity (Kovanis et al., 2016). Not only does this system subtly encourage authors to take shortcuts so another paper can go out the door, but also the burden placed on peer reviewers (who are not paid) has increased to a level that may be unsustainable (Kovanis et al., 2016). Journal editors in turn are evaluated based on the impact factors of their journals—a measure of how many citations, on average, each published paper in that journal receives. The impact factor contributes to publication bias because it incentivizes editors to disproportionately accept only the exciting papers that may be cited. The entire system is flawed.

What does this mean for evidence-based practice if so much of the evidence is potentially questionable? Systematic reviews and meta-analyses help dilute the effect of any one perhaps suboptimal study, whether written with spin or otherwise problematic (bias, uncontrolled confounding, etc.). However, publication bias—being unable to include null results in reviews because null results tend not to get published—might still lead one to the wrong conclusion. In well-done systematic reviews, reviewers formally assess publication bias. While I do not check all reviews included in these bibliographies, I do make sure the authors of the highlighted reviews have assessed publication bias to a satisfactory degree. Clinicians can and should allow more fluidity in their practice; the term “evidence-informed” rather than “evidence-based” implies more room for clinical experience, patient and provider preferences, and unique characteristics (Nevo & Slonim-Nevo, 2011). If they are involved with their universities’ governance structures, researchers can work to align promotion and tenure expectations with the realities of doing good science. More importantly, researchers can collectively admit when a research question does not seem answerable. Treatment of women with ovarian cancer would be infinitely easier if we had reliable diagnostic or prognostic biomarkers; however, an extremely large body of literature suggests such a thing is currently—and perhaps permanently—out of reach.

Acknowledgment

The author acknowledges Sabrina Pillai, MPH, for assistance with the literature searches for this column.

Biography

Marit L. Bovbjerg, PhD, MS, is an assistant professor of epidemiology in the College of Public Health and Human Sciences, Oregon State University, Corvallis, OR.

Table 1.

Summary of Results Reported by Moran et al. (2020)

Outcome	Range	Average Cost, US vs. non-US	Average Cost, Woman Only	Average Cost, Neonate Only	Average Cost Both
Gestational Diabetes	€263 to E13,680	€4,607 vs. €1,444	€2,048	—	€3,388
Obesity	€191 to €16,046	€6867 vs. €768	€1,612	—	€8,964
Mental Health	€452 to €794	—	—	—	—
Hypertension	€2,860 to €8,595	—	—	—	—
Nausea	€191 to €454	—	—	—	—
Epilepsy	—	—	€6,033	€1,694	—

Note. All costs are in 2018 Euros. — = no data.