Background
Hypothesis formation is more than a banal step in the scientific method. Prior beliefs can influence study results by informing how studies are designed and if their findings are interpreted as spurious or true. Outcomes from observational studies are particularly vulnerable to being biased by authors’ prior beliefs1 via mechanisms ranging from subconscious priming2 to overt p-hacking.3 This may explain why researchers invested in certain study outcomes may be more likely to find their sought-after associations.4 Given this, best-practice guidelines for reporting observational research recommend describing authors’ a priori hypotheses.5 Thus, we sought to characterize the frequency of explicitly stated hypotheses in articles across major general medicine journals.
Methods
We conducted a repeated cross-sectional analysis of studies in four general medicine journals (JAMA, Annals of Internal Medicine, The BMJ, The New England Journal of Medicine) published in 1999 and 2019. Observational research published as original articles or brief reports was included. Data extracted from each article included the publication year, author degrees, presence or absence of an explicitly stated hypothesis, and the direction of the primary hypotheses and study findings (association, no association, and other including mixed or unclear directionality of findings). The primary outcome of interest, the presence or absence of a hypothesis, was defined as an explicitly written statement about the authors’ prior belief about the direction of the primary outcome of the study made anywhere in the article. Statements were considered explicit if they conveyed the authors’ prior beliefs regarding the primary outcome in the study (e.g., “we postulated”) but not if they simply described the standard setup of hypothesis tests for statistical analyses.
One of the three authors (AC, AZ, SR) extracted the data from the four journals with a fourth author (JN) reviewing 10% of papers to assess for agreement in the presence/absence of a hypothesis. There was substantial agreement on coding of presence/absence of a hypothesis (kappa 0.88, 95% CI 0.76–0.99).
Descriptive statistics and Pearson’s chi-squared tests were used to characterize and assess differences between categorical variables, respectively. Analyses were performed using R version 3.6.0.
Results
Eight hundred twenty-three articles were reviewed. Of these, 495 (60%) reported associations, 93 (11%) reported no associations, and 235 (29%) reported mixed or unclear directionality to findings in the main analyses. One hundred eleven (13.5%) articles had a clearly stated hypothesis, of which 99 (89.2%) hypothesized finding associations in the main analyses. Articles with a hypothesis were more likely to report associations in the main analyses (76% vs 57%, p<0.01) and less likely to have mixed or unclear outcomes (11% vs 32%, p<0.01).
Articles published in 1999 compared to 2019 did not differ in the prevalence of reported hypotheses (12.8% and 14.6%, p=0.52). Additionally, the presence of a PhD author was also not associated with a difference in hypothesis prevalence (Table 1). While the first and last authors with “other” degrees tended to have the lowest percentage of articles with a hypothesis, these associations did not reach statistical significance (p=0.06 and p=0.46, respectively).
Table 1.
Article Characteristics and Proportion with Hypotheses
| Hypothesis | ||
|---|---|---|
| Yes | No | |
| N (%) | N (%) | |
| Publication | ||
| Annals of Internal Medicine | 21 (17.2) | 101 (82.8) |
| BMJ | 38 (13.3) | 248 (86.7) |
| JAMA | 37 (12.5) | 259 (87.5) |
| New England Journal of Medicine | 15 (12.6) | 104 (87.4) |
| First author degree | ||
| MD or equivalent | 53 (14.4) | 316 (85.6) |
| PhD or equivalent | 42 (16.4) | 214 (83.6) |
| Both MD and PhD | 9 (9.3) | 88 (90.7) |
| Other | 7 (6.9) | 94 (93.1) |
| Last author degree | ||
| MD or equivalent | 52 (13.6) | 330 (83.3) |
| PhD or equivalent | 28 (12.2) | 201 (87.7) |
| Both MD and PhD | 25 (16.7) | 124 (83.2) |
| Other | 6 (9.5) | 57 (90.5) |
| Papers with PhD authorship | ||
| Author with PhD present | 95 (14.2) | 576(85.8) |
| Author with PhD absent | 16 (10.5) | 136 (89.5) |
Discussion
This study found few observational studies contain clear hypotheses. Qualitatively, authors often colloquially described their findings as “surprising” or “expected,” which may suggest authors had a priori hypotheses despite them not being explicitly stated for the reader to consider.
We had hypothesized that (1) research conducted by PhDs (i.e., more research training) and (2) earlier publication date (i.e., more barriers to conducting research) would be associated with more complete reporting. However, our data did not support either hypothesis.
Only a small subset of journals were included in this study, and not every observational study from each journal was included, such as research published as letters. Furthermore, many researchers have extensive research training despite not having a PhD, possibly limiting the validity of our findings regarding our first hypothesis. Despite these limitations, this study highlights how few articles, regardless of when and by whom they were published, adhere to best-practice reporting guidelines or include authors’ explicit a priori beliefs. Because these beliefs are important to designing studies and contextualizing outcomes, more comprehensive reporting in observational research is warranted while other solutions, such as creating registries for observational study protocols and hypotheses or encouraging researchers to consider Bayesian methods where priors are more explicitly stated, may also be considered.
Declarations
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Bruns SB, Ioannidis JPA. P-curve and p-hacking in observational research. PLoS One. 2016;11(2). [DOI] [PMC free article] [PubMed]
- 2.Gilder TSE, Heerey EA. The role of experimenter belief in social priming. Psychol Sci. 2018;29(3):403–417. doi: 10.1177/0956797617737128. [DOI] [PubMed] [Google Scholar]
- 3.Fanelli D. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS One. 2009;4(5):e5738. doi: 10.1371/journal.pone.0005738. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Panagiotou OA, Ioannidis JPA. Primary study authors of significant studies are more likely to believe that a strong association exists in a heterogeneous meta-analysis compared with methodologists. J Clin Epidemiol. 2012;65(7):740–747. doi: 10.1016/j.jclinepi.2012.01.008. [DOI] [PubMed] [Google Scholar]
- 5.von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. Published online 2008. 10.1016/j.jclinepi.2007.11.008 [DOI] [PubMed]