Supplemental Digital Content is available in the text
Keywords: background style, content analysis, meta-epidemiological study, systematic review
Abstract
Objectives:
To clarify the styles used in background sections of systematic reviews (SR) and to identify which styles if any were related to the publication in high-impact-factor (HIF) medical journals.
Method:
This was a cross-sectional study for original SR articles published in top 50 journals in MEDICINE, GENERAL & INTERNAL category in Journal Citation Reports 2018. We randomly included 90 articles from top 10 HIF journals and 90 from others, respectively. We conducted a content analysis to classify the background styles. We assessed the factors associated with the publication in HIF journals.
Results:
We found 6 categories. We defined 6 categories as follows: Update of prior SR, New in scope than prior SR, Higher quality than prior SR, Completely new SR, Limitations of primary studies only, and Not presenting unknown in prior SR or primary studies. All 6 categories were not related to the publication in HIF journals.
Conclusions:
We found 6 categories of styles in background sections of SR, none of which however were related to publication in HIF journals. SR authors may wish to use any of these categories to communicate the importance of their research questions.
1. Introduction
The background section of a medical journal article has the important function to communicate the value of the research question to the readers. There are many textbooks and review articles reflecting expert opinions on how to write them.[1–3] Almost all writers of academic articles find it difficult to write the background section.[4,5]
In addition, there are several analyses that examined its structure in medical research articles. The basic structure of the background section may be characterized as follows: “Presenting Background Information”, “Reviewing Related Research”, and “Presenting New Research”.[6–8]
However, little is known about how systematic review (SR) authors review related researches to emphasize the importance of their research question in their background section. This study aimed to examine the methods the authors use in order to delineate their novelty with regard to the existing literature and compare which methods were related to the publication in top medical journals. We have focused on SR because SR are the most important research design, in terms of practicing evidence-based medicine.[9]
2. Methods
2.1. Study design
This was a cross-sectional study for original SR articles. We published the study protocol before conducting this research.[10] We adhered to the reporting guideline of meta-epidemiological study (Guidelines Checklist)[11] where relevant.
2.2. Types of studies included
We included original SR articles published in top 50 journals in MEDICINE, GENERAL & INTERNAL category in Journal Citation Reports 2018.[12] We included all SR articles irrespective of study designs of the included primary studies. We used the definition of SR by National Library of Medicine, which was “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies”.[13]
We excluded Cochrane Reviews, the U.S. Preventive Services Task Force review, or The Rational Clinical Examination articles because their backgrounds styles are prespecified by the respective organizations and different from usual original articles.[14,15] We also excluded protocols or non-English articles.
2.3. Search methods
We searched PubMed on December 1, 2019. The search formula contained the following blocks: “date” AND “study type filter” AND “the name of journals”. The details of search formula are shown in Supplemental Digital Content (Table S1A, B).
2.4. Study selection
We prespecified in our protocol to randomly select 90 articles from the top 10 high-impact-factor (HIF) journals and 90 articles from the 11th to 50th journal as controls. In accordance with previous studies,[16,17] we chose a cut-off of top 10 in our protocol. In our protocol we considered 9 potential confounding factors. Using the rule of thumb that we need cases that are 10 times the number of confounding factors to examine their influence,[18] we chose 90 articles from the top 10 journals randomly and 90 controls from the top 12 to 50 journals randomly with a 1:1 match. One review author (YK) confirmed whether the articles were SR or not. We attempted to repeat the random sampling of SR to reach the prespecified number.
2.5. Data extraction and assessment
2.5.1. Exposures
Since there were no prior categories to refer to, we conducted a content analysis to classify the background style to show the novelty in SR. Content analysis is a method to analyze texts. This method enables researchers to evaluate texts and make meaningful and replicable interpretations.[19] In generating categories, we used the lens of a position to guide the writing of systematic review articles for novices.
In the protocol we defined 4 categories. Using these categories, 5 authors (YK, ST, SYK, YT, or HY) who have prior experience of conducting systematic reviews independently assessed randomly selected 10 articles, then discussed the disagreements. Finally, we defined 6 categories as follows: Update of prior SR, New in scope than prior SR, Higher quality than prior SR, Completely new SR, Limitations of primary studies only, and not presenting unknown in prior SR or primary studies. The details of definitions are shown in Table 1. We evaluated the categories with an allowance for duplications.
Table 1.
The definitions of background categories.
| Category | Definition | Supporting codes |
| Update of prior SR∗ | Mentioning prior SR of same research question, but the search date was new. | We aimed to update the information provided by our previous report (Siew) |
| New in scope than prior SR∗ | There is a reference to prior SR of a similar research question and explains how this review is different from prior SR in scope (e.g. patient, intervention or outcomes reviewed) If there is no mention of any prior SR, this category is not applicable. |
The most recent metaanalysis suggested (omitted). Previous reviews have also mainly reported results for (omitted) outcomes rather than (omitted) and omitted the growing body of evidence from observational studies on this topic, which often include large study populations in real world settings and longer follow-up. (Mathews) One quantitative review examined the effectiveness of psychological approaches for (omitted). We conducted a comprehensive systematic review and meta-analysis to evaluate the efficacy of CBT-based approaches for (Niknejad) |
| Higher quality than prior SR∗ | There is a reference to prior SR of a similar research question but points out problems with their methods other than being outdated, or stating the need for new research methods, such as NMA. | This method allows comparison of all available (omitted) medications against placebo and between pharmacological agents despite the paucity of head-to-head comparisons of therapies in RCTs. (Gregori) |
| Completely new SR | Mentioning that it is a completely new SR. |
To date, to our knowledge, no studies have comprehensively examined (Fellmeth) No clinical practice guidelines include (omitted). This systematic review focuses on this gap (Williums) |
| Limitations of primary studies only | No mention of prior SR, and only mention of limitations of primary studies. | Previous studies have largely been single centre trials with a small sample size, (omitted) not powered to assess the true effect of (omitted) and whether it interacts with important clinical subgroups. (Nath) |
| Not presenting unknown in prior SR or primary studies | No mention of prior SR, no mention of the limitations of primary studies, no reference to primary studies, or expresses uncertain points without any citations. |
Consequently, there is interest among researchers, clinicians, and health policy leaders in (omitted) (Rosenstein) While most industrialized countries have introduced routine female (omitted) vaccination into their national immunization programs, routine vaccination of boys and men is currently implemented in only a few countries (omitted) (Harder) |
After developing the 6 categories, 2 of the 5 authors (YK, ST, SYK, YT, or HY) independently assessed the background section of each of 170 remaining articles. We resolved the disagreements through discussion.
2.5.2. Other variables
One review author (YK) extracted variables related to the background. The background section included subheadings of background and aim or purpose. Word counts included citation symbols. The number of included studies was counted at the time of their synthesis of each article. One review author (YK) evaluated the study characteristics, and 1 of 4 authors confirmed (ST, SYK, YT, or HY). We resolved the disagreements through discussion.
2.5.3. Outcomes
The primary outcome was whether the article was published in HIF journals or not.
2.5.4. Data analysis
We conducted data analysis following a prespecified protocol.[18] We used descriptive statistics to summarize. We used logistic regression models for univariate and multivariate analysis to explore the associations between study characteristics and the publication in HIF journals. Confounding factors were number of included articles in SR, number of tables and figures, number of authors, and presence of appendix. We used median as cut-off point to dichotomize the confounding variables. Two-tailed P values were considered statistically significant if less than .05. We used Stata ver. 16.1 (StataCorp LLC, College Station, Texas, United States of America).
2.5.5. Ethics
Ethics approval was not applicable because this study only used published articles.
2.6. Difference between the protocol and manuscript
We initially planned to investigate the presence of practice implications. It was difficult to reach agreement through discussion in the pilot review. For that reason, we dropped this feature investigation.
2.7. Patient and public involvement
We did not involve patients or the public in our study.
3. Results
3.1. Selection process
The search result and selection process are shown in Figure 1. We searched a total of 569 articles. We randomly included 112 articles from HIF journals and 106 from top 12 to 50 journals, respectively. After full-text screening, we included 90 articles from the former and 90 articles from the latter, respectively (Table S3).
Figure 1.
Flowchart for the selection of reporting guidelines. ∗Top 11 journal was the Cochrane Database of Systematic Reviews (CDSR), we excluded CDSR because the backgrounds style is different from usual original articles. We randomly selected 112 and 106 potentially eligible articles from the search results. When the article was not eligible, we repeated to resample to reach 90 articles in each category. SR = systematic review, USPSTF = United States Preventive Services Task Force.
3.2. Study characteristics
Table 2 shows the study characteristics of included articles. The median number of paragraphs, the median number of cited articles and the word counts in background were fewer in HIF journals than in others.
Table 2.
Characteristics of included articles.
| Top 10 (n = 90) | Top 12–50 (n = 90) | Total (n = 180) | |
| Number of paragraphs in background | 3 [3–4] | 4 [3–5] | 3 [3–5] |
| Number of cited articles in background | 14 [10–19] | 18 [12–23] | 15 [11–21] |
| Word counts in background | 388 [279–493] | 465 [344–585] | 433 [299–539] |
| Number of included articles in SR | 36 [20–93] | 18 [11–34] | 24 [13–50] |
| Number of authors | 8[6–11] | 5 [4–7] | 7 [5–9.5] |
| Number of tables | 2 [1–3] | 3 [2–4] | 2 [1–3] |
| Number of figures | 3 [2–5] | 2 [1–3] | 3 [1–4] |
| Number of citations | 59 [44–84] | 56 [42–80] | 57 [43–82] |
| Presence of appendix | 87 (97) | 61 (68) | 148 (82) |
| Study type∗ | |||
| NMA (intervention) | 9 (10) | 3 (3) | 12 (7) |
| intervention | 40 (44) | 37 (41) | 77 (43) |
| DTA | 2 (2.2) | 3 (3) | 5 (3) |
| prognosis | 19 (21) | 11 (12) | 30 (17) |
| prevalence/incidence | 11 (12) | 8 (9) | 19 (11) |
| Scoping review, qualitative synthesis | 6 (7) | 26 (29) | 32 (18) |
| meta-epidemiology | 3 (3) | 2 (2) | 5 (3) |
We evaluated the inter-rater reliability by kappa statistics. Table 3 shows the proportion of background categories and inter-rater reliability. We did not add any categories for the 170 articles other than the 10 we initially reviewed.
Table 3.
Categories of background style.
| Top 10 (n = 90) | Top 12–50 (n = 90) | Total (n = 180) | Kappa statistics∗ | |
| Update of prior SR | 20 (22) | 11 (12) | 31 (17) | 0.76 |
| New in scope than prior SR | 33 (37) | 35 (39) | 68 (38) | 0.71 |
| Higher quality than prior SR | 18 (20) | 13 (14) | 31 (17) | 0.68 |
| Completely new SR | 7 (8) | 17 (19) | 24 (13) | 0.71 |
| Limitations of primary studies only | 20 (22) | 15 (17) | 35 (19) | 0.52 |
| Not presenting unknown in prior SR or primary studies | 11 (12) | 8 (9) | 19 (11) | 0.51 |
3.3. Primary outcome
Table 4 shows the association of factors and the publication in HIF journals. Unadjusted analysis showed that completely new SR were associated with the publication in HIF journals, others were not. Adjusted analysis showed that all 6 categories were not associated with the publication in HIF journals.
Table 4.
Associations between study characteristics and the publication in top 10 high-impact-factor journals.
| OR (95%CI) | AOR (95%CI) | |
| Number of included articles in SR (>24 vs ≤24) | 3.45 (1.87 to 6.37) | 2.74 (1.33 to 5.65) |
| Number of tables and figures (>5 vs ≤5) | 1.26 (0.69 to 2.27) | 1.31 (0.63 to 2.7) |
| Number of authors (>7 vs ≤7) | 5.16 (2.7 to 9.86) | 3.93 (1.86 to 8.29) |
| Presence of appendix | 13.79 (4.02 to 47.31) | 8.76 (2.33 to 32.91) |
| Update of prior SR | 2.05 (0.92 to 4.58) | 2.21 (0.5 to 9.69) |
| New in scope than prior SR | 0.91 (0.5 to 1.66) | 1.06 (0.24 to 4.61) |
| Higher quality than prior SR | 1.48 (0.68 to 3.24) | 1.08 (0.37 to 3.17) |
| Completely new SR | 0.36 (0.14 to 0.92) | 0.56 (0.09 to 3.4) |
| Limitations of primary studies only | 1.43 (0.68 to 3.01) | 2.27 (0.41 to 12.41) |
| Not presenting unknown in prior SR or primary studies | 1.43 (0.55 to 3.73) | 1.99 (0.31 to 12.82) |
4. Discussion
4.1. Brief summary of main findings
We have identified 6 categories to review related researches in the background sections of systematic review articles published in general medical journals. None of the 6 categories was associated with publication in HIF journals, while the number of included articles, the number of authors, and presence of appendix were.
4.2. Results in relation to prior studies
This is the first study to reveal the “background styles” in SR. Nwogu clarified that in background section of medical articles authors review related research for 2 purposes, which were “Reference to previous research” and “Reference to limitations of previous research”.[7] Five categories found in our research excluding “not presenting unknown in prior SR or primary studies” could be regarded as “Reference to limitations of previous research”. The existence of “not presenting unknown in prior SR or primary studies” may indicate that the research question that the review focused on was obvious for authors and editors to explain.
The amount of information included in SR would be related to the publication in HIF journals. Manriquez et al reported that number of authors and number of references cited were related to citations of skin disease SR.[20] In our study, number of included articles, number of authors, and presence of appendix were related to the publication in HIF journals. These findings are consistent with a previous SR focused on the relationship between the feature of all biomedical articles and citations.[21] Our results indicate that the “background style” is not an important determinant in submission by authors and/or acceptance by journal editors.
4.3. Clinical relevance and implications
Our findings may be useful for novice SR researchers when write their articles. As structured reporting with reporting guidelines helps to improve the quality of reporting,[22] the effectiveness of teaching academic writing following with known structure in each professional academic community has been suggested.[23,24] Further study to develop and evaluate an education program for writing background section in SR using the styles identified in this study would be warranted.
4.4. Limitations
There are several limitations in our study. First, there was some inconsistency in judgment in the classification of background styles. As far as possible, we classified the background based on the operational definition, but the kappa statistics of “limitations of primary studies only”, and “not presenting unknown in prior SR or primary studies” were only moderate. This may be partly due to the oversight of the researchers, but also due to the existence of some papers which fails to explain novelty for readers. Second, we used journal impact factor (JIF) as a proxy of the importance of individual articles. Usually, the distribution of citations are skewed and we should be careful about using JIF, which is a mean.[25] However, we adopted the lens of a position to guide the writing of systematic review articles for novices. We believe that the JIF, which is determined by each journal, are the reflection of the editors preferences for deciding acceptance. Third, we only focused on general medical journals and a certain period. Our findings would not be applicable for journals that are only read by experts who have enough knowledge of the field. Future research is warranted to verify whether our findings would be applicable in other fields and periods.
5. Conclusions
We found 6 categories in background section of SR. SR authors would be able to use these categories to explain the importance of their research questions, but not necessary to care to choose the submission journal.
Acknowledgments
We would like to thank the members of the Research Group on Meta-epidemiology at The Kyoto University School of Public Health (Tomoko Fujii, Yusuke Tsutsumi, Aran Tajika, Kenji Omae, Edoardo G Ostinelli, and Yan Luo)
Author contributions
YK, ST, SYK, YT, HY and TAF contributed to the conception and design of the research. YK, ST, SYK, YT, HY retrieved the data. YK conducted statistical analyses. YK wrote the draft manuscript. TAF and all the other co-authors revised the manuscript critically for important intellectual content. All authors gave final approval of the manuscript before submission.
Conceptualization: Yuki Kataoka, Shunsuke Taito, Sachiko Yamamoto-Kataoka, Yasushi Tsujimoto, Hajime Yamazaki, Toshi A. Furukawa.
Data curation: Yuki Kataoka, Shunsuke Taito, Sachiko Yamamoto-Kataoka, Yasushi Tsujimoto, Hajime Yamazaki.
Formal analysis: Yuki Kataoka.
Investigation: Yuki Kataoka.
Methodology: Yuki Kataoka, Toshi A. Furukawa.
Project administration: Yuki Kataoka.
Resources: Yuki Kataoka.
Software: Yuki Kataoka.
Supervision: Toshi A. Furukawa.
Validation: Yuki Kataoka.
Visualization: Yuki Kataoka.
Writing – original draft: Yuki Kataoka.
Writing – review & editing: Shunsuke Taito, Sachiko Yamamoto-Kataoka, Yasushi Tsujimoto, Hajime Yamazaki, Toshi A. Furukawa.
Supplementary Material
Footnotes
Abbreviations: HIF = high-impact-factor, SR = systematic reviews.
How to cite this article: Kataoka Y, Taito S, Yamamoto-Kataoka S, Tsujimoto Y, Yamazaki H, Furukawa TA. Background styles in systematic review articles are not related to the publication in high-impact-factor journals: a meta-epidemiological study. Medicine. 2020;99:51(e23801).
Toshi A. Furukawa: personal fees from Meiji, Mitsubishi-Tanabe, MSD and Pfizer and a grant from Mitsubishi-Tanabe, outside the submitted work; TAF has a patent 2018-177688 pending. The other authors declare no conflicts of interest.
TAF reports grants and personal fees from Mitsubishi-Tanabe, personal fees from MSD, personal fees from Shionogi, outside the submitted work; In addition, TAF has a patent 2018-177688 regarding smartphone cognitive-behavioral therapy apps pending, and intellectual properties for Kokoro-app licensed to Tanabe-Mitsubishi.
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.
The authors declare no conflicts of interest.
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.
Prior SR with a similar research question is defined as SR including some overlapping primary studies.
NMA = network meta-analysis, SR = systematic reviews.
Number (%), median [interquartile].
We chose one of the applicable types, in order from the top.
DTA = diagnostic test accuracy, NMA = network meta-analysis, SR = systematic reviews.
Number (%).
kappa statistics of rating agreements for 170 articles excluding pilot reviewed 10 articles in each category.
AOR = adjusted odds ratio, CI = confidence interval, OR = odds ratio.
References
- [1].Albert T. Winning the Publications Game. 4th ed.2017;Abingdon, United Kingdom: CRC Press, Chapter 1. [Google Scholar]
- [2].Silvia PJ. Write It Up. 1st ed.Washington, DC: American Psychological Association; 2019. [Google Scholar]
- [3].Bligh, Glennys Parsell J. AMEE Guide No. 17. Writing for journal publication. Med Teach 1999;21:457–68. [Google Scholar]
- [4].Swales J. Genre analysis: English in academic and research settings. 1990;Cambridge, UK: Cambridge University Press, 1–11. [Google Scholar]
- [5].Cho DW. Science journal paper writing in an EFL context: The case of Korea. English Specif Purp 2009;28:230–9. [Google Scholar]
- [6].Devitt AJ. Genre performances: John Swales’ Genre Analysis and rhetorical-linguistic genre studies. J English Acad Purp 2015;19:44–51. [Google Scholar]
- [7].Nwogu KN. The medical research paper: structure and functions. English Specif Purp 1997;16((2 SPEC. ISS.)):119–38. [Google Scholar]
- [8].Jirapanakorn N, Trakulkasemsuk W, Keyuravong S. Move analysis of English research article introductions in Thai and International Medical Journals. rEFLections 2014;17(June):23–40. [Google Scholar]
- [9].Guyatt G, Rennie D, Meade M, et al. Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. 2nd ed.2015;New York, US: McGraw-Hill Professional, Chapter 22. [Google Scholar]
- [10].Taito Y, Kataoka S, Yamamoto-Kataoka S, et al. Background styles in systematic review articles: a cross-sectional study protocol. medRxiv 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [11].Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med 2017;22:139–42. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [12]. InCites Journal Citation Reports [Internet]. [cited 2019 Oct 3]. Available from: https://jcr.clarivate.com/JCRLandingPageAction.action. [Google Scholar]
- [13]. Institute of Medicine. Clinical practice guidelines we can trust. 2011. [Google Scholar]
- [14]. MECIR Manual | Cochrane Community [Internet]. [cited 2019 Oct 3]. Available from: https://community.cochrane.org/mecir-manual. [Google Scholar]
- [15]. Force the USPST. Methods and processes [Internet]. [cited 2019 Oct 3]. Available from: https://www.uspreventiveservicestaskforce.org/Page/Name/methods-and-processes. [Google Scholar]
- [16].Joksimovic L, Koucheki R, Popovic M, et al. Risk of bias assessment of randomised controlled trials in high-impact ophthalmology journals and general medical journals: a systematic review. Br J Ophthalmol 2017;101:1309–14. [DOI] [PubMed] [Google Scholar]
- [17].Onishi A, Furukawa TA. Publication bias is underreported in systematic reviews published in high-impact-factor journals: metaepidemiologic study. J Clin Epidemiol 2014;67:1320–6. [DOI] [PubMed] [Google Scholar]
- [18].Peduzzi P, Concato J, Kemper E, et al. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996;49:1373–9. [DOI] [PubMed] [Google Scholar]
- [19].Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005;15:1277–88. [DOI] [PubMed] [Google Scholar]
- [20].Manriquez J, Cataldo K, Harz I. Factors influencing citations to systematic reviews in skin diseases: a cross-sectional study through Web of Sciences and Scopus. An Bras Dermatol 2015;90:646–52. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [21].Tahamtan I, Safipour Afshar A, Ahamdzadeh K. Factors affecting number of citations: a comprehensive review of the literature. Scientometrics 2016;107:1195–225. [Google Scholar]
- [22].Falagas ME, Grigori T, Ioannidou E. A systematic review of trends in the methodological quality of randomized controlled trials in various research fields. J Clin Epidemiol 2009;62:227–31e9. [DOI] [PubMed] [Google Scholar]
- [23].Chang CF, Kuo CH. A corpus-based approach to online materials development for writing research articles. English Specif Purp 2011;30:222–34. [Google Scholar]
- [24].Badger R, White G. A process genre approach to teaching writing. ELT J 2000;54:153–60. [Google Scholar]
- [25].Lariviere V, Kiermer V, MacCallum CJ, et al. A simple proposal for the publication of journal citation distributions. BioRxiv 2016. 62109. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.