Evaluation of publication bias in the assessment of probiotic treatment for gastrointestinal disease in dogs and cats

Abstract

Background

Systematic reviews and meta-analyses underpin the evidence-to-decision framework used for guideline development. Publication bias is important to understand when assessing the strength of evidence.

Objective

To evaluate the peer-reviewed-journal publication rate of abstracts from 2 veterinary internal medicine conferences regarding probiotic treatment for gastrointestinal disease in dogs and cats.

Animals and procedure

Abstracts from the American College of Veterinary Internal Medicine Forum (2000 to 2023) and European College of Veterinary Internal Medicine Congress (2002 to 2023) that reported clinical gastrointestinal disease outcomes of probiotic treatment for dogs, cats, or both were included. PubMed and Web of Science databases were searched to identify corresponding peer-reviewed publications.

Results

Twelve abstracts were identified; 6 (50%) were subsequently published as peer-reviewed publications. Five of 6 (83%) that were published reported positive clinical outcomes, whereas 4/6 (67%) that were not published reported no beneficial clinical outcomes. Overall, 5/7 (71%) abstracts that reported a clinical effect were published, compared to 1/5 (20%) that did not.

Conclusion

Publication bias complicates assessment of the literature and guideline development.

Clinical relevance

The potential impact of publication bias should be considered when evaluating the literature and developing guidelines.

Formal evidence-synthesis approaches are increasingly being used to provide clinical guidance, with systematic reviews and meta-analyses underpinning the evidence-to-decision framework. One core area of evidence assessment is evaluation of risk of bias, both in individual studies and the collection of studies available for review. Included in this is publication bias, whereby the results of a study influence whether the study is ultimately published. Of greatest concern is the potential for overestimation of the effect of an intervention if “negative” results are less likely to be published. Whereas assessment of publication bias is a component of standardized guideline development, it can be difficult to assess. Centralized registration of clinical trials is not required for veterinary studies, making it challenging to discern which studies were completed but ultimately not published. Concerns about publication bias are heightened when the available literature consists mainly of a small number of small trials (1), something that is very common in veterinary medicine. Methods such as funnel plots can be used to infer whether publication bias may be present, but these have limitations and it has been stated that funnel plots and statistical tests for asymmetry should require at least 10 (or perhaps at least 5) available studies (1). Therefore, the paucity of controlled trials in veterinary medicine can preclude use of these tools. Accordingly, it can be difficult to determine if publication bias is present, and this can weaken evidence assessment and guideline development.

The objective of this study was to evaluate the peer-reviewed-journal publication rate of conference abstracts regarding probiotic treatment for gastrointestinal disease in dogs and cats.

Abstract proceedings from the American College of Veterinary Internal Medicine (ACVIM) Forum (2000 to 2023) and European College of Veterinary Internal Medicine (ECVIM) Congress (2002 to 2023) were searched for “probiotic” in the title or abstract text using the Veterinary Information Network database (http://www.vin.com), which houses full-text abstracts for those conferences. For each abstract, the full text was then screened to determine if it was a prospective research study that evaluated clinical gastrointestinal disease outcomes of probiotic treatment on dogs, cats, or both. Studies that only evaluated nonclinical outcomes (e.g., microbiota, hematological data) or non-gastrointestinal disease (e.g., renal disease) were excluded.

To determine if studies had been published in the peer-reviewed scientific literature, PubMed and Web of Science databases were searched, using combinations of keywords from each title and each author’s name, as well as a general search of “probiotic” and each author’s name. The search was done on March 28, 2024 and repeated on July 11, 2024, shortly before manuscript submission.

Twelve abstracts that evaluated the effect of probiotics on clinical indicators of gastrointestinal disease (e.g., effects on vomiting, diarrhea, fecal score) were identified — 10 from ACVIM and 2 from ECVIM (Table 1). The abstracts described studies on dogs (n = 6), cats (n = 5) or both dogs and cats (n = 1). Eleven (92%) were reported to be randomized controlled trials, whereas 1 was an uncontrolled prospective study (2).

TABLE 1.

Abstracts included in evaluation of the potential for publication bias in therapeutic trials of probiotics for gastrointestinal disease in dogs and cats.

Conference, year	Title	Published (Yes/No), year	Reference
ACVIM, 2003	The efficacy of a probiotic strain, Lactobacillus acidophilus DSM 13241, in the recovery of cats from clinical Campylobacter infection	No	(10)
ECVIM, 2005	Efficacy of probiotics in acute GI disease in dogs and cats	Yes, 2010	(11)
ACVIM, 2009	Effect of Bifidobacterium animalis AHC7 on resolution of acute diarrhea in the canine	Yes, 2009	(12)
ACVIM, 2010	Effect of Enterococcus faecium SF68 supplementation on diarrhea in cats housed in a northern Colorado animal shelter	Yes, 2011	(13)
ACVIM, 2012	Effects of Enterococcus faecium SF68 on stress diarrhea	No	(14)
ACVIM, 2015	Effect of the probiotic Enterococcus faecium SF68 on the presence of diarrhea in weanling kittens	No	(15)
ACVIM, 2015	The effect of the probiotic Sivoy on clinical and histopathological parameters in cats with chronic idiopathic constipation and megacolon	Yes, 2018	(2)
ACVIM, 2015	Effect of Enterococcus faecium strain SF68 on the gastrointestinal clinical signs of cats administered amoxicillin-clavulanate	Yes, 2017	(16)
ACVIM, 2018	Treatment with Enterococcus faecium NCIMB 10415 does not affect the outcome of chemotherapy-induced diarrhea	No	(17)
ECVIM, 2020	The use of a combined prebiotic and probiotic oral product and its impact on stool consistency in dogs undergoing radiotherapy	No	(18)
ACVIM, 2020	Use of a synbiotic for treating antibiotic-induced diarrhea in cats	No	(19)
ACVIM, 2022	Clinical and gastrointestinal changes in healthy research dogs administered prednisone, prednisone/omeprazole, or prednisone/probiotics	Yes, 2023	(20)

ACVIM — American College of Veterinary Internal Medicine; ECVIM — European College of Veterinary Internal Medicine.

Subsequent peer-reviewed publications were identified for 6 (50%) of the abstracts — 3 involving cats, 2 involving dogs, and 1 involving both dogs and cats. Abstracts for these were presented in 2005, 2009, 2010 (n = 1, each), 2015 (n = 2), and 2022 (n = 1), with a median time from presentation to publication of 1.5 y (range: < 1 to 5 y). Five (83%) of the abstracts that were published as scientific papers reported ≥ 1 positive clinical outcomes, whereas 1 reported no beneficial clinical effects. Five of the 6 (83%) involved a commercial probiotic product or a probiotic strain used in commercial products. The commercial status of the probiotic in the remaining study was unclear from the abstract.

The 6 abstracts not associated with a peer-reviewed publication were presented in 2003, 2012, 2015, 2018 (n = 1 each), and 2020 (n = 2). Four studies involved dogs and 2 involved cats. Four of the 6 (67%) reported no beneficial clinical outcome of probiotic treatment. Of the remaining 2, 1 reported a significant difference in prevalence of a specific fecal score threshold based on a presumably post hoc subset analysis and suggested a positive effect on duration of diarrhea with an unsupportive P-value (P = 0.47). One other abstract reported positive clinical outcomes but provided no corresponding numerators, denominators, or clear analysis. Five of the 6 (83%) abstracts involved a commercial probiotic product or a probiotic strain used in commercial products, whereas the commercial status of the other was unclear.

Overall, 5/7 (71%) abstracts that reported a clinical effect were published, compared to 1/5 (20%) that did not. Statistical comparison of publication rates was not done because of the small sample size.

These data indicated that publication bias may be an important concern when assessing the clinical impact of probiotic therapy in dogs and cats, consistent with concerns about publication bias for probiotics in human medicine (3,4). Here, unpublished abstracts generally described no or limited (and possibly questionable) clinical effects, in contrast to the more positive effects in published studies, even though some of the positive effects reported in published studies would have been of limited clinical importance. This bias toward publishing of “positive” trial results was consistent with studies of human and laboratory animal abstracts (5–7).

A recent systematic review and meta-analysis evaluated the effect of probiotics on acute diarrhea in dogs and identified what was considered a trivial positive effect, based on 4 studies (8). Given the small sample size for that meta-analysis, the unpublished studies identified here could have influenced that result and corresponding clinical guidance, had they been published.

Although there was substantial under-publication of abstract data, it must also be recognized that submission and acceptance of an abstract was required for studies to be scrutinized. It is possible that other studies were completed but results were not presented in abstract form, something that would increase the effect of publication bias, assuming that studies that did not reach the stage of an abstract did not have favorable results.

There can be various reasons why an abstract is not subsequently published as a peer-reviewed paper. These include abandonment of a study because of changes in personnel (e.g., a graduate student graduating before completion), changing interests of the lead author or research team leading to reduced motivation to continue a study, rejection by a peer-reviewed journal, lack of motivation to publish a negative result, and pressure from commercial sponsors. Unpublished studies could also have been abandoned because of poor recruitment or poor initial results, in cases where there was still a desire to release the initial information or have a citation but the authors felt that data were inadequate or underpowered to warrant a full publication. Some of these factors (e.g., company pressures, editorial interest, author motivation, poor initial results leading to lack of funding or motivation for completion) would be more likely with negative studies. Regardless of the reason for lack of publication, failure to publish all studies and a tendency for underreporting negative studies are of concern. Small sample size may have been an important factor in failure to publish, related to either authors’ motivation to publish or success in the peer-review process. Whereas statistical analysis may not be appropriate for small studies and valid conclusions about efficacy cannot be made, the data can be important for subsequent evidence synthesis, including meta-analysis.

Funnel plots and statistical tests for asymmetry can be used to assess publication bias (9). Funnel plots were not used here because of the small number of studies and the variability in how effects were reported, preventing plotting of a standardized response metric. Statistical tests of asymmetry were not done for the same reasons.

This study only evaluated 2 conferences because of limitations in abstract access and searchability, plus an a priori assumption that those 2 conferences, which are the main international veterinary internal medicine conferences, would be leading conferences for presentation of abstracts of this nature. It is possible that probiotic efficacy abstracts were presented at other conferences. A broader standardized and systematic approach evaluating more conferences could have provided more abstracts to evaluate and increased the power for potential statistical comparisons. However, that did not affect the focus of this study and there were no apparent reasons why publication rates would differ for abstracts presented at other conferences.

Evidence-to-decision frameworks for guideline development involve assessment of publication bias, but this can be challenging. For subjects with a limited number of studies and limited sample sizes within those studies, publication bias could have a major effect on assessment of clinical efficacy, resulting in overestimation of the effect of treatment. Lack of a mandatory clinical trial registry and a central, searchable database of conference abstracts complicate objective assessment of publication bias, and the small number of studies that are present for many topics complicate the use of measures such as funnel plots. However, the results of this study supported concerns that publication bias may be important and that additional search strategies may be indicated, particularly for therapeutic trials, when the published studies are small in both number and sample size. CVJ