Frequency of limitations statements in original research articles of United States leading medical journals: A meta-research protocol

Abstract

Background

Limitation declarations are commonly deemed essential to uphold intellectual humility for scientific research, but little has been reported about the limitation statements in published original research articles. This meta-research study aims to investigate the trends of limitation statements among three leading general medical journals in the US.

Methods

This cross-sectional study will compile a data set of full-length original research articles published in the New England Journal of Medicine, Journal of the American Medical Association, and Annals of Internal Medicine between 2002 and 2022. Limitation statement will be recognized by two investigators, and a predefined set of sensitive keywords is used for sensitivity analysis. Frequency of limitation statements within the main text of research articles and trends for different study designs, including their association with the corresponding reporting guidelines, are the main measurements. We employ the Cochran-Armitage test for trend analysis.

Conclusion

The findings of this study will provide an overview of the limitation statements in leading general medical journals in the US. The results may contribute to future research to identify factors that are associated with the presence of limitation statements.

Introduction

The concept of intellectual humility is gaining considerable recognition, celebrated as an integral aspect of wisdom, a facilitator of self-improvement, and a catalyst for more constructive dialogue [1]. Acknowledgement of limitation is an attitude of humility warranted in the scientific research. Reducing potential confounding through meticulous study design is a task assigned to researchers [2, 3]; however, biases remain unavoidable [4, 5]. In accordance with journal author guidelines, providing a statement of limitations is typically obligatory in original research. Nevertheless, the prevalence and pattern of such limitations declarations in published works have not been reported.

The reporting guidelines for pertinent publication types were established approximately 15 years ago [6, 7]. The Consolidated Standards of Reporting Trials (CONSORT) guidelines for randomized control trials (RCT) mandate limitations, generalizability interpretation in the discussion or comment section [6, 8]. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement for observational studies mandate key results, limitations, interpretation, generalizability in the discussion section [9]. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for reporting systematic reviews and meta-analyses of studies, consists of a 27-item checklist. also mandate an item to include discussions of limitations at study, outcome, and review level [10].

Despite the explicit requirement for a limitation statement outlined in the respective checklists of these reporting guidelines, adherence to such guidelines is seldom reported. A review of the first 50 articles from each of the six most-cited research journals and two leading open-access journals published in 2005 found that only 67 articles (17%) of the 400 articles mentioned any limitations in their scientific work [11]. However, most of these reports were completed before the implementation of reporting guidelines. Including journals with different editorial policies may introduce bias. Furthermore, previous studies did not examine any trends in the inclusion of limitation statements in articles.

This study aims to assess the frequency of limitation statements in original articles within three leading general medical journals in the United States. It seeks to explore trends from 2002 to 2022, both before and after the publication of relevant reporting guidelines for the respective publication types. Based on personal observations, we hypothesize that there is an increasing trend in the inclusion of limitation statements, particularly after the publication of reporting guidelines.

Methods and analysis

So far, there is no formal reporting guideline for meta-analysis. We will use the reporting guideline proposed by Murad et al. [12], which provided a framework for reporting meta-epidemiological studies.

Eligibility criteria

The eligible criteria are original articles published in the New England Journal of Medicine (NEJM), Journal of American Medical Association (JAMA) and Annals of Internal Medicine (Ann Intern Med). These three journals are selected because they were rated the top three general medical journals in the US.

Study selection

NEJM, JAMA and Ann Intern Med adapt different kinds of columns, consisting of a variety of article types. Therefore, we only enroll articles which were published in the original article section of NEJM, the original contribution and original research section of JAMA, and original research section of Ann Intern Med. Articles categorized in other sections, such as preliminary communication or special communication, regardless of publication types, are not enrolled. A model of flow diagram that will be used to describe for the study selection process is depicted in S1 Fig in the supplement file.

Data collection process

We systematically acquired, through hand-searching, all the full text of original articles published between the first issue in January 2002 to the last issue in December 2022 from the website of the NEJM, JAMA and Ann Intern Med.

We only exclude original articles, by hand-searching, which have been labelled as retracted, regardless of the reason and availability on the journal website. Examples are Mehra’s and Gander’s articles in NEJM and JAMA, respectively [13, 14].

Data items

The Japan-US-Taiwan meta-research group gathers researchers who have published a few commentaries in both NEJM [15], JAMA [16] and other leading journals [17]. In this study, two reviewers (Hsu NC and Hsu CH) will conduct the investigations independently for each article to determine whether a limitation statement exists. Discrepancies will be resolved by consultation with the corresponding author (Tokuda Y). We identify limitations exclusively within the discussion section, as we observe that articles often discussed methods for minimizing bias or controlling for confounding in the methodology section. Some authors may subjectively characterize these approaches as strengths of the study, while others acknowledge them as limitations. Consequently, our focus is specifically on determining whether these statements are presented in the discussion section.

Variables in this study include the study design and the temporal relationship with the respective reporting guidelines. The study designs are classified into three categories. A randomized controlled trial (RCT) is defined as a comparative study reporting a random assignment of participants anywhere in the report. A clinical trial without random assignment is identified as a non-randomized trial. A meta-analysis study is defined as a quantitative research synthesis method that statistically combines the results of multiple independent studies, typically indicated in the title of a report [18].

The CONSORT statement was first published in 1996 [6], with subsequent revision in 2001 [7], and 2010 [8]. We investigate publication between 2002 and 2022 and chose the year 2010 as a reference point to assess the pre-post changes in limitation statements of randomized trials. The STROBE guideline for observational studies, initially published in 2007 without subsequent revisions [10], serves as the reference point for pre-post changes in limitation statements for non-randomized trials.

Measures

The main measure of our study is the existence of limitation statements in eligible articles. Before investigation, the authors have convened a consensus meeting regarding the definition of the limitation statement. The forms of the limitation statement are categorized and exemplified in S1 Table. After a random review of original articles published in these three journals, there are ten primary forms of limitation statements commonly found in the discussion section of academic papers, which include: (1) a subheading entitled “limitations”; (2) a subheading entitled “strengths and limitations”; (3) a subheading entitled “methodological considerations”; (4) a single paragraph discussing limitations without a subheading; (5) multiple paragraphs discussing limitations without a subheading; (6) statements of limitations or weaknesses alongside strengths within the same paragraph; (7) discussions of “bias” without mentioning limitations; (8) discussions of “weakness” without mentioning limitations; (9) reasons for interpreting the results cautiously, without directly mentioning limitations; and (10) acknowledgments of areas where the study may fall short, without using the terms “limitations,” “weakness,” or “bias.” (S1 Table)

Synthesis of results

The main result of interest is the frequency of limitation statement in the discussion section of eligible original research articles, and the trend of annual publications from 2002 to 2022 counted on a yearly basis.

To avoid missing limitation statements in articles not reporting limitations in any of the ten forms mentioned above, we will conduct a sensitivity analysis. Three reviewers (Hsu NC, Hsu CH, and Tsai HB) have convened a consensus meeting to identify additional words that will be used during the sensitivity analysis: "limited", "confound", and "caution". Combined with “limitation”, “weakness”, and “bias”, which we have already employed in the primary form, the sensitivity analysis will contain six words. We have done a pilot testing of the sensitivity analysis using the original articles published in the first three months of 2012, 2014, 2016, 2018, 2020 and 2022 in NEJM and JAMA. With the gold standard of limitation statements recognized independently by two reviewers (Hsu NC and Hsu CH), the absence of all six limitation-pertinent words predicted no limitation statement with 100% accuracy. The occurrence of at least two of the six words resulted in a 100% accuracy in predicting the existence of limitation statements. Based on this pilot test, articles lacking any of the six limitation-pertinent words will be categorized as having no limitation statements in this study. On the websites of these journals, we will use these limitation-pertinent words to manually screen the full text of each article in electronic form as a sensitivity analysis.

We use the Chi-square test to depict frequency distributions for descriptive categorical variables, such as the frequency of limitation statements within a specific journal or a research design. Trends between journals, research designs, and pre-post change are analyzed by the Cochran Armitage test with a P value. Analyses will be conducted using R (R Project for Statistical Computing version 4.1.2) and SPSS version 24. A P value of 0.05 or less will be used to define the statistical significance of a test result.

We do not seek approval from the research ethics committee or institutional review board because the study does not involve human subjects, and the data are publicly available.

Discussion

The fourth edition of the "Uniform Requirements for Manuscripts Submitted to Biomedical Journals," initially published in NEJM in 1991, served as a coordinating document developed by the International Committee of Medical Journal Editors (ICMJE) in North America [19]. In this document, it suggested the article should “include in the Discussion section the implications of the findings and their limitations, including implications for future research”. Because the limitations of a study can be discussed in various ways, it becomes challenging to audit their presence or absence.

From the introduction of the first CONSORT statement in 1996, statements of specific interpretation of study findings, including sources of bias and imprecision (internal validity) and discussion of external validity, including appropriate quantitative measures are necessary in the comment section among the five structured subheadings of RCT [6]. However, the submission process of an original research for some leading medical journals does not mandate an inclusion of any formal reporting guideline.

Our study aims to analyze the inclusion of limitation statements in the reports of original research published by three leading general medical journals in the United States. We hypothesize that the inclusion of limitation statements has been increasing, particularly following the publication of reporting guidelines. This investigation may contribute to the audit of reporting quality in trials, compatible with the goal and mission of a meta-research study. A previous review in psychology critically examined diverse approaches to defining and measuring intellectual humility and identified the common element: a meta-cognitive ability to recognize the limitations of one’s beliefs and knowledge [1]. Therefore, limitation statements in a research report are crucial and go beyond merely adhering to reporting guidelines.

Limitations

Potential limitations of our study encompass exclusive focus on three medical journals in the US, challenges in recognizing limitation statements, variations in author writing styles, confounding by the peer review process, and absence of a comprehensive analysis of the content and quality of the limitation statements. NEJM doesn’t require the inclusion of a subsection on limitations in the discussion. While JAMA suggested a subsection of limitation since 2017, we emphasize that our study refrains from drawing direct comparisons between the journals in this context.

Conclusion

The findings of this study may provide an overview of the limitation statements in leading general medical journals in the US, including the frequency and trends. As this is an inaugural investigation, the results may provide a foundation for future research aimed at identifying factors such as article types, research topics, and journal author guidelines that are associated with the presence of limitation statements.

Supporting information

S1 Fig. A modified PRISMA flow diagram for meta-research analysis.

(DOCX)

S1 Table. Forms of limitation statement in the discussion section of original research articles.

(DOCX)

Data Availability

Data cannot be shared publicly because of the study protocol.

Funding Statement

National Science and Technology Council (Taiwan) (NSTC 112-2410-H-002-174-MY2 and NSTC 112-2410-H-002-172-MY2) on the article processing charges. The authors declare no conflict of interest. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.