Open Science Policies and Practices Among Medical Journals

Key Points

Question

Do leading medical journals align with open science standards, and can automated tools reliably detect the implementation of practices in published research?

Findings

In this cross-sectional study of open science practices across 15 624 articles published in 10 general medical journals, transparency was greater for randomized clinical trials than for other study designs, although overall adherence to open science practices remained suboptimal. These article-level findings were consistent with the limited implementation of Transparency and Openness Promotion recommendations observed at the journal policy level.

Meaning

In this study, leading medical journals demonstrated only partial alignment with open science standards, suggesting that strengthening journal policies may be necessary to advance transparency across the broader medical research landscape.

This cross-sectional study evaluates which open science practices have been adopted by leading general medical journals and whether automated tools can assess article-level adherence.

Abstract

Importance

Open science practices are essential for improving transparency, reproducibility, and trust in biomedical research. Journals play a critical role in promoting these practices through editorial policies, yet implementation and impact remain unclear.

Objective

To evaluate the open science policies of leading medical journals and assess implementation and detectability of practices using automated tools.

Design, Setting, and Participants

This cross-sectional study of journal policies and open science practices evaluated research articles published in 10 leading general medical journals from January 2020 to December 2023. Additionally, the diagnostic accuracy of automated tools was validated against manual extraction.

Exposures

Journal policies regarding open science practices and article-level implementation of 13 core practices including registration, protocol sharing, and intention to share data.

Main Outcomes and Measures

Journal policies were assessed using the Transparency and Openness Promotion guidelines (TOP2025). At the article level, 13 core open science practices were examined. Additionally, 9 validated automated tools were applied to detect these practices, and their performance was compared with manual extraction of articles.

Results

Overall, 15 624 research articles published in 10 general medical journals were analyzed (validation subset, 312 articles: 103 randomized clinical trials [RCTs], 98 meta-analyses, and 111 with other designs). At the journal level, TOP2025 evaluation identified substantial heterogeneity in policies, primarily applied to clinical trials. At the article level, open science practices were more frequently implemented in RCTs than other designs: registration (RCTs: 99% [95% CI, 97%-100%]; meta-analyses: 69% [95% CI, 56%-79%]; other designs: 16% [95% CI, 9%-26%]), protocol sharing (RCTs: 96% [95% CI, 93%–98%]; meta-analyses: 67% [95% CI, 54%-78%]; other designs: 20% [95% CI, 12%-33%]), and intention to share data (RCTs: 79% [95% CI, 67%-87%]; meta-analyses: 65% [51%-77%]; other designs: 70% [95% CI, 57%-81%]). Automated tools showed variable performance (F1 scores, 0.06-1.00) and generally underestimated practices.

Conclusions and Relevance

In this cross-sectional study of 15 624 articles in 10 leading medical journals, journal policies were only partially aligned with TOP2025, and article-level open science practices were more frequently reported for RCTs than for other designs, supporting the need for stronger journal policies.

Introduction

Medical journals play a central role in disseminating biomedical research, which is essential for advancing health care and informing clinical decision-making. Open science practices, such as study registration, protocol availability, and data sharing, are increasingly recognized as fundamental components of high-quality research.¹

In 2005, the International Committee of Medical Journal Editors (ICMJE) required prospective registration of clinical trials as a condition for publication.² Progress in other areas, including data sharing, has been slower. The Annals of Internal Medicine first encouraged data sharing in 2007.³ In 2018, the ICMJE introduced data sharing statements for clinical trials, requiring authors to declare whether and how they would share data.⁴ Adherence, however, remains limited.^5,6

Some leading journals have adopted stronger measures. BMJ began by encouraging data sharing for clinical trials in 2009,⁷ and now requires data deposition in a publicly accessible repository.⁸ PLOS Medicine adopted a similar requirement in 2014.⁹ These initiatives, however, remain largely confined to clinical trials. Extending open science practices to other study designs, such as observational research,¹⁰ could further strengthen reproducibility. Biomedical journals have also been slow to adopt innovative formats, such as registered reports,¹¹ a publication model in which study methods are peer-reviewed and accepted in principle before data collection begins, thereby reducing publication bias. In contrast, registered reports are more common in psychology.¹² The emphasis on transparency aligns with broader reforms in research assessment. The San Francisco Declaration on Research Assessment (DORA) discourages reliance on journal impact factors,¹³ while the Hong Kong Principles advocate rewarding open and reproducible practices.¹⁴ Although more indirectly related, such assessment reform still supports the broader open-science environment. International bodies, including the Group of 7 (G7)¹⁵ and the United Nations Educational, Scientific and Cultural Organization (UNESCO),¹⁶ have made open science a strategic priority and published Principles of Open Science Monitoring.¹⁷ The updated Transparency and Openness Promotion (TOP) Guidelines (TOP2025)¹⁸ provide frameworks for evaluating policies. At the article level, the ScreenIT Working Group (an international consortium of researchers and developers) has developed automated tools to enable large-scale monitoring of open science practices.¹⁹ In biomedicine, there is also consensus on which practices should be assessed.²⁰

Assessing how journals implement and promote open science is therefore essential to determine how well stated commitments to transparency translate into practice. Systematic assessment can identify gaps, track progress, and strengthen the credibility and societal value of biomedical research. It may also help counter the influence of predatory journals (ie, entities that misrepresent themselves as legitimate scholarly journals for financial gain, soliciting submissions and charging fees while failing to provide genuine peer review and other core editorial and publishing functions)²¹ by highlighting those with suboptimal standards.²² In this study, we aimed to develop a proof-of-concept assessment of open science practices in leading medical journals. Specifically, we sought to evaluate the feasibility of using automated tools to monitor transparency indicators at the article level and to describe current open science policies in those journals using manual extraction.

Methods

Study Design: A 2-Level Cross-Sectional Study

This is a cross-sectional study assessing open science at 2 distinct levels: (1) the policies of the included journals, evaluated with the 2025 TOP Guidelines, and (2) the open science practices reported in their published articles. For the second level, we first assessed the diagnostic accuracy of several automated detection tools, then applied the best-performing tools to measure these practices in recent articles.

This study was preregistered,²³ and postprotocol modifications are detailed in eAppendix 6 in Supplement 1. Data and statistical code are openly available on OSF.²³ The code is provided in R language.²⁵ Per the Common Rule, ethical approval was not requested for this cross-sectional study because it involved data extracted from published research. No human participants were involved. This report follows the Standards for Reporting of Diagnostic Accuracy (STARD)²⁶ and Strengthening the Reporting of Observational Studies in Epidemiology (STROBE)²⁷ reporting guidelines.

Journal Selection

We analyzed 10 leading medical journals, purposefully chosen for their international reputation as leading general medical publications. Specialist journals were not included. Consistent with DORA,¹³ we did not use impact factor for selection. Most included journals were either current or former ICMJE members, such as Annals of Internal Medicine, BMJ, CMAJ, JAMA, Lancet, Nature Medicine, New England Journal of Medicine, and PLOS Medicine. Furthermore, 2 major open access medical journals, JAMA Network Open and BMC Medicine, were also included.

Journal Policy Assessment

Two researchers (C.V. and A.P.M.D.) independently assessed each journal’s open science policies against the TOP2025 criteria¹⁸ based on the editorial policies published on each journal’s website. The practices were divided and split into the following categories: research practices (study registration, study protocol, analysis plan, materials transparency, data transparency, analytic code transparency, and reporting transparency), verification practices (results transparency and computational reproducibility), and verification studies (replication, registered report, multiverse, and many analysts). Full descriptions appear in eAppendix 1 in Supplement 1. Scores were calculated as follows: 0, any policy; 1, disclosed; 2, shared and cited; and 3, certified. Disagreement was resolved by a third researcher (F.N.). In addition, we also checked whether the journal or publisher is a DORA signatory. The extraction was conducted from March 19 to April 4, 2025. All editorial teams were then invited to review our extraction and provide any necessary corrections, within 1 month, with reminders sent in the first and third weeks; their feedback was received between May 5 and June 10, 2025.

Open Science Practices Assessment

Article Selection

We included all research articles published in the 10 journals between January 2020 and December 2023, classified as randomized clinical trials (RCTs), meta-analyses (MAs), or other types of research articles, and excluded congress communications, opinion papers, editorials, and other nonresearch articles. Eligible articles were identified through 3 distinct PubMed queries, reviewed by an information specialist and adhered to the Peer Review of Electronic Literature Search Strategies (PRESS) guidelines²⁸ (eAppendix 2 in Supplement 1). We primarily used PDF files and supplements; for Lancet, where these were unavailable, we used the Elsevier API to retrieve corresponding XML files (eAppendix 3 in Supplement 1).

Practices Monitored

Articles were assessed against a published community consensus on the core open science practices monitor in biomedicine,²⁰ a list established through a 3-round Delphi process involving 80 experts from 20 institutions. Table 1 details the 19 core open science practices. Of these, we retained the 12 than can be ascertained from a published article: registration, intention to share data, open access publishing, code sharing, protocol sharing, statistical analysis plan (SAP) sharing, use of a reporting guideline, presence of a preprint (defined as an explicit preprint DOI or link reported in the published article), presence of author contribution, presence of a conflict of interest (COI) statement, reporting author Open Researcher and Contributor Identifiers (ORCID), and presence of a funding statement. These practices differ in their level of endorsement. Some are required by the ICMJE, research funders, or regulators; others are recommended by the EQUATOR network. The remainder, while not yet formally mandated, are nonetheless endorsed by domain experts as essential for reproducibility and transparency.²⁰ The level of endorsement and the reasons for excluding other items from the consensus list are provided in Table 1.

Table 1. Open Science Practices Considered for Monitoring, With Level of Endorsement and Inclusion Decision.

Open science practices20	Inclusion in the study (label)	Explanation and elaboration	Endorsement
Items included
Reporting whether clinical trials were registered before they started recruitment	Yes (registration RCT)	As automated tools cannot determine from the article alone whether registration was prospective, only whether a registration record existed was verified.	ICMJE requirement
Reporting whether author contributions were described	Included (author contributions )	Whether the article included a contribution statement was assessed.	ICMJE requirement
Reporting whether research articles include funding statements	Included (funding statement)	Whether the article included a funding statement was monitored.	ICMJE requirement
Reporting whether author COI were described	Yes (COI)	Whether the article included a COI disclosure was assessed.	ICMJE requirement
Reporting whether study data were shared openly at the time of publication (with limited exceptions)	Yes (intention to share data)	Given the sensitivity of biomedical research data, intention to share data was defined broadly as any declared intention to share data, whether openly or upon request. These estimates may therefore overstate actual data sharing, as authors may refuse requests.	Encouraged by ICMJE
Reporting what proportion of articles are published open access with a breakdown of time delay	Yes (open access publishing)	For this item, automated extraction tools are unsuitable because the relevant information is usually not contained within the article text. Available metadata were used.	Consensus expert and mandate by some funder (eg, Horizon Europe, NIH Public Access)
Reporting whether ORCID identifiers were used	Yes (ORCID)	Whether the article included at least 1 ORCID identifier was assessed.a	Consensus expert, funder obligation (eg, NIH)
Reporting whether systematic reviews have been registered before data collection began	Yes (MA registration)	As automated tools cannot determine from the article alone whether registration was prospective, only whether a registration record existed was verified.	EQUATOR recommendation
Reporting whether a reporting guideline checklist was used	Yes (reporting guideline)	Whether reporting guidelines were mentioned was assessed, but reporting quality was not evaluated. This may underestimate actual use, as only instances where reporting guidelines were explicitly cited or referenced were detected.	EQUATOR recommendation
Reporting whether study code was shared openly at the time of publication (with limited exceptions)	Yes (code sharing)	As with intention to share data, a broad definition was used, which included both open access and on-request access. These estimates may overestimate code sharing, as authors may deny requests.	Expert consensus
Reporting the number of preprints	Yes (preprint)	References to preprints within the article were tracked.	Expert consensus
Items not included
Reporting whether RCTs results appeared in the registry from 1 y after study completion	No	This indicator was excluded, as reporting on a registry differs from publication of results and cannot be appropriately identified in the published article.	Regulatory obligation
Reporting whether there was a statement about study materials sharing with publications	No	Due to substantial heterogeneity in what constitutes material sharing across biomedical research, it was not feasible to monitor this item consistently across all articles.	Expert consensus
Reporting citations to data	No	This item was considered too similar to “intention to share data” and too difficult to apply consistently to current publications.	Expert consensus
Reporting trial results in a manuscript-style publication (peer reviewed or preprint)	No	All published journal articles inherently meet this criterion; therefore, monitoring it within journal publications was deemed unnecessary.	Expert consensus
Reporting systematic review results in a manuscript-style publication (peer reviewed or preprint)	No	All published journal articles inherently meet this criterion; therefore, monitoring it within journal publications was deemed unnecessary.	Expert consensus
Reporting whether data, code, or materials were shared with a clear license	No	This item was excluded because it is challenging to evaluate it using only the content of current research articles.	Expert consensus
Reporting the use of persistent identifiers when sharing data, code, materials	No	This item was excluded because it is challenging to evaluate it using only the content of current research articles.	Expert consensus
Reporting whether the data, code, or materials license is open or not	No	This item was excluded because it is challenging to evaluate it using only the content of current research articles.	Expert consensus

Abbreviations: COI, conflict of interest; ICMJE, International Committee of Medical Journal Editors; MA, meta-analysis; NIH, National Institutes of Health; ORCID, Open Researcher and Contributor Identifier; RCT, randomized clinical trial.

^a Due to limitations in the automated tools used in the study, it was not possible to ascertain whether ORCID identifiers were reported for all authors or only some.

Automated Tools

Through the ScreenIT Working Group¹⁹ and experts encountered through the Open Science Monitoring Initiative, we identified 9 tools covering these practices. There were 2 proprietary tools: SciScore²⁹ (research resource identifier [RRID], SCR_016251) and DataSeer³⁰ (RRID, SCR_023027), part of which is available on GitHub. There were also 6 open-source tools: rTransparent³¹ (RRID, SCR_019276), ODDPub³² (RRID, SCR_018385), ctregistries³³ (RRID, SCR_024412), ContriBOT³⁴ (RRID, SCR_027089), TNRscreener³⁵ (RRID, SCR_019211), and Unpaywall³⁶ (RRID, SCR_016471) as well as a locally hosted open-weight large language model (LLM; Llama 3.3-70B, run via Ollama). All artificial intelligence analyses were performed locally; no article content was transmitted to any third-party provider, and full texts were obtained from open access sources or institutional substruction. Tool specifications and implementation details are provided in eAppendices 4 and 5 in Supplement 1.

Validation Base

To evaluate and compare the diagnostic accuracy of these tools and select the most suitable one for each open science practice, a random sample of 300 research articles from 2022 to 2023 was used, with equal distribution across journals and article types. If a journal did not publish a specific type of study, additional articles were randomly selected from other journals to ensure that approximately 100 articles per research article type were included. When an incorrect classification occurred due to PubMed extraction, the article was replaced with another randomly sampled one.

Manual extraction served as the reference standard (to determine the true presence or absence of a practice) for assessing the diagnostic accuracy of each tool in detecting open science practices, consistent with most prior metaresearch.³¹ Two examiners (F.J.A. and M.S.) independently extracted data from the PDF and supplementary files using a predeveloped data extraction sheet. Any disagreements were resolved by a third examiner (C.V.). For each practice, manual and automated extraction used the same operational definition. For example, both manual and automated extraction considered data described as available (on request or openly available) as a positive (intention to share), so results between processes are directly comparable.

Statistical Analysis

Practices were summarized as frequencies and percentages. Weighted frequencies and 95% CIs were calculated from the proportions of each journal and article types, to correct sampling imbalances.

For each tool and practice, we estimated sensitivity, specificity, and the F1 score, with 95% CIs. For each practice, we selected the best tool based on its accuracy, ease of use (ie, any qualified researcher can use it independently without the need for external assistance), and whether it was openly available. The selected tools were then used to evaluate all research articles. We also compared tool performance on PDF vs XML inputs using XML files from Europe Pubmed Central.³⁷

Analyses were run using R version 4.2.2 (RRID, SCR_001905; R Project for Statistical Computing), with the libraries tidyverse (RRID, SCR_019186)²⁴ and Patchwork (RRID, SCR_000072) for the data visualization. All analyses were performed on a MacBook Pro with an M3 Pro chip and on the Eskemm Data calculation server.²⁵

Results

This study analyzed 2 distinct units: the policies of 10 leading medical journals, and the open science practices in the 15 624 research articles they published between 2020 and 2023. From these 15 624 articles, a stratified sample of 312 articles (103 RCTs, 98 MAs, and 111 other designs) was manually extracted in duplicate as the validation database for the automated tools.

Journal Policy Assessment

We collected TOP2025 data from 10 journals. After outreach, 5 journals (BMJ, CMAJ, JAMA, Lancet, and PLOS Medicine) responded, verified our data extraction, and 15 suggested amendments, resulting in 10 edits to our ratings (eAppendix 12 in Supplement 2).

As shown in Figure 1, many journal policies (ie, registration, protocol sharing, SAP sharing, data sharing and reporting transparency) applied specifically to clinical trials rather than to all research articles. For study registration, 3 of 10 journals (30%) disclosed registration for all study types, whereas all journals except CMAJ (9 of 10 [90%]) shared and cited it for clinical trials. For protocol sharing, 4 journals (40%) disclosed and 1 (10%) shared and cited it for all research, compared with 1 (10%) that disclosed, 7 (70%) that shared and cited, and 1 (10%) that certified it for clinical trials. For SAP sharing, 2 journals (20%) disclosed it for all research vs 2 (20%) that disclosed and 6 (60%) that shared and cited it for clinical trials. For data sharing, the policies were more uniform across study types: 8 journals (80%) disclosed, 1 (10%) shared and cited, and 1 (10%) certified it for all research, compared with 7 (70%) that disclosed, 2 (20%) that shared and cited, and 1 (10%) that certified it for clinical trials. Finally, for reporting transparency, 6 journals (60%) disclosed and 2 (20%) shared and cited it for all research vs 4 (40%) that disclosed and 4 (40%) that shared and cited it for clinical trials. No journals had clear policies on verification practices, and only few on promoted verification studies (replication study at Nature Medicine and PLOS Medicine; registered report at BMC Medicine). In addition, 2 journals were DORA signatories themselves (BMC Medicine and BMJ), and 5 were covered through their publisher’s endorsement, including the 2 direct signatories (BMC Medicine, BMJ, PLOS, Lancet, Nature Medicine).

Figure 1. Heat Maps and Bar Graphs of Figure Policy Assessment of 10 Biomedical Journals.

A score of 0 indicates any policy; 1, disclosed; 2, shared and cited; and 3, certified.

Open Science Practices Assessment

Validation Database and Tool Accuracy

PubMed searches were performed on April 29, 2024; after exclusion or reclassification of articles misindexed by study type in PubMed, 312 articles were included as 103 RCTs, 98 MAs, and 111 other types of research article (eAppendix 7 in Supplement 1). Not all journals published every study type. For instance, we found no MAs published in the New England Journal of Medicine. In the validation database (Table 2), conflict of interest disclosure was universal (100%), whereas preprints were nearly absent (1 of 312 [<1%]). Registration (101 RCTs [98%; weighted, 99% (95% CI, 97%-100%)]; 68 MAs [69%; weighted, 69% (95% CI, 56%-79%)]; and 20 other designs [18%; weighted, 16% (95% CI, 9%-26%)]), protocol sharing (92 RCTs [89%; weighted, 96% (95% CI, 93%-98%)]; 66 MAs [67%; weighted, 67% (95% CI, 54%-78%)]; and 25 other designs [23%; weighted, 20% (95% CI, 12%-33%)]), SAP sharing (52 RCTs [51%; weighted, 59% (95% CI, 47%-70%)]; 4 MAs [4%; weighted, 3% (95% CI, 1%-8%)]; and 9 other designs [8%; weighted, 6% (95% CI, 2%-15%)]), and intention to share data (86 RCTs [84%; weighted, 79% (95% C, 67%-87%)]; 75 MAs [77%; weighted, 65% (95% CI, 51%-77%)]; 81 other designs [73%; weighted, 70% [95% CI, 57%-81%]) were more common in RCTs than in MAs or other study designs, and data were most often described as "available upon request" regardless of design.

Table 2. Use of Open Science Practices in the Validation Database, Obtained by Manual Extraction and Considered the Gold Standard.

Characteristic	Articles, No. (%)
	Overall (n = 312)	RCT (n = 103)	MA (n = 98)	Other (n = 111)
Journal
Annals of Internal Medicine	30 (10)	11 (11)	10 (10)	9 (8)
BMC Medicine	36 (12)	10 (10)	16 (16)	10 (9)
BMJ	31 (10)	10 (10)	13 (13)	8 (7)
CMAJ	17 (5)	3 (3)	4 (4)	10 (9)
JAMA	32 (10)	11 (11)	11 (11)	10 (9)
JAMA Network Open	43 (14)	13 (13)	13 (13)	17 (15)
Lancet	34 (11)	12 (12)	10 (10)	12 (11)
New England Journal of Medicine	23 (7)	14 (14)	0	9 (8)
Nature Medicine	32 (10)	10 (10)	7 (7)	15 (14)
PLoS Medicine	34 (11)	9 (9)	14 (14)	11 (10)
Registration	189 (61)	101 (98)	68 (69)	20 (18)
Data sharing
No sharing	70 (22)	17 (17)	23 (23)	30 (27)
Open access	56 (18)	6 (6)	35 (36)	15 (14)
Under request	186 (60)	80 (78)	40 (41)	66 (59)
Open access sharing	257 (82)	82 (80)	83 (85)	92 (83)
Code sharing
No sharing	237 (76)	86 (83)	70 (71)	81 (73)
Open access	40 (13)	3 (3)	15 (15)	22 (20)
Under request	35 (11)	14 (14)	13 (13)	8 (7)
Protocol
No sharing	129 (41)	11 (11)	32 (33)	86 (77)
Open access	172 (55)	87 (84)	63 (64)	22 (20)
Under request	11 (4)	5 (5)	3 (3)	3 (3)
Statistical analysis plan
No sharing	247 (79)	51 (50)	94 (96)	102 (92)
Open access	60 (19)	48 (47)	4 (4)	8 (7)
Under request	5 (2)	4 (4)	0	1 (1)
Reporting guideline	148 (47)	41 (40)	68 (69)	39 (35)
Preprint	1 (<1)	1 (1)	0	0
Author contribution	289 (93)	88 (85)	96 (98)	105 (95)
COI	312 (100)	103 (100)	98 (100)	111 (100)
ORCID	137 (44)	46 (45)	44 (45)	47 (42)
Funding statement	306 (98)	103 (100)	95 (97)	108 (97)

Abbreviations: COI, conflict of interest; MA, meta-analysis; ORCID, Open Researcher and Contributor Identifier; RCT, randomized clinical trial.

Automated-tool accuracy varied by practices, with F1 scores ranging from 0.06 for Llama3.3-70B evaluating ORCID to 1.00 for rTransparent evaluating COI (Figure 2). For article-level estimates, we retained rTransparent for study registration (sensitivity, 77% [95% CI, 70%-83%]; specificity, 93% [95% CI, 88%-97%]), COI statement (sensitivity, 100% [95% CI, 99%-100%]; specificity, incalculable, as all articles had a statement), and funding statement (sensitivity, 99% [95% CI, 97%-100%], specificity, 67% [95% CI, 22%-96%]). ContriBOT was retained for author contributions (sensitivity, 87% [95% CI, 82%-90%]; specificity, 78% [95% CI, 56%-93%]) and the detection of ORCID numbers (sensitivity, 86% [95% CI, 79%-91%]; specificity, 99% [95% CI, 96%-100%]). Unpaywall was retained for open access publications (sensitivity, 99% [95% CI, 96%-100%]; specificity, 36% [95% CI, 11%-69%]). For all other practices, we selected Llama 3.3 70B. Performance varied across study type and journals, likely reflecting formatting differences. Full per-tool performance, selection rationale, and the PDF-vs-XML comparison are reported in eAppendices 8 to 11 in Supplement 1.

Figure 2. Heat Map of Automated Tool Performance .

COI indicates conflict of interest; MA, meta-analysis; ORCID, Open Researcher and Contributor Identifier; RCT, randomized clinical trial; and SAP, statistical analysis plan.

Practices Across Research Articles, 2020-2023

Across the 15 624 research articles, implementation of open science practices remained limited (Figure 3). For registration, the overall estimate was 20% (70% for RCTs, 53% for MAs and 8% for other research articles). Compared with the weighted estimates from the validation set, the automated tools tended to underestimate most practices. For example, automated screening yielded lower prevalence estimates for clinical trial registration (70% vs 99% in the validation set), data sharing (48% vs 71%), reporting guideline use (40% vs 58%), author contribution statements (82% vs 95%), funding statements (89% vs 96%), and overall registration (20% vs 32%). In contrast, 4 practices were overestimated by the automated tools: protocol sharing (42% vs 35%), SAP sharing (32% vs 14%), code sharing (25% vs 17%), and preprinting (2% vs <1%). For the remaining practices—such as open access publication (89% vs 90%) and conflict of interest statements (>99% vs 100%)—the automated and validated estimates were closely aligned.

Figure 3. Line Graph of Open Science Practices Assessment.

Percentages (with 95% CIs) of open science practices were adjusted for the number of articles published by study type and journal in the random sample; these weighted frequencies are derived from the validation database and represented in the figure in dark blue and light blue. The orange circle indicates the observed percentage in the global assessment obtained from the selected tools and calculated from all articles. COI indicates conflict of interest; MA, meta-analysis; ORCID, Open Researcher and Contributor Identifier; RCT, randomized clinical trial; and SAP, statistical analysis plan.

Discussion

Among 10 leading journals, most had policies that partially aligned with TOP2025 transparency recommendations,¹⁸ and these policies applied mainly to RCTs while verification practices remained rare. Only BMC Medicine offered registered reports, a format still uncommon in general medical journals, despite its growing recognition in the broader scientific literature and its adoption by prestigious journals such as Nature and PLOS Biology.

At the article level, our results mirrored these policy patterns. Practices such as registration and data sharing were more frequent in RCTs, reflecting both their pivotal role in evidence-based medicine and the fact that data sharing is easier to plan prospectively than in studies using routinely collected data. Interestingly, even MAs that use aggregated and already published data showed low data sharing. Although data sharing poses minimal risk in these designs,³⁸ data sharing remained uncommon. Overall, the practices best implemented were those that are formally required (eg, ICMJE-mandated: COI, funding statements, and author contributions), whereas expert-endorsed but nonmandated practices remained infrequent, underlining that adherence largely tracks the strength of endorsement.

Our results could inform long-term scalability or integration into global monitoring frameworks (eg, UNESCO). However, effective implementation requires standardized article formatting to facilitate automated assessment. Stakeholder collaboration, among journals, publishers, funders, and researchers, is essential to establish consensus on information standardization (structured metadata, main text, or supplementary materials). The ICMJE could play a critical role in facilitating such collaboration. These coordinated efforts would strengthen transparent, trustworthy, and reproducible biomedical research.

Limitations and Strengths

This study has several limitations. First, there were some methodological limitations. Automatic and manual extraction have complementary trade-offs: manual extraction is more reliable but not exhaustive, whereas automated extraction is comprehensive but prone to misclassification; all indicators should be read in this light and given the below-optimal accuracy of some tools. Additionally, we did not formally calculate sample size; the number of manually extracted articles was determined pragmatically. We also assessed the cross-sectional presence of open access practices without accounting for embargos or future status changes. Additionally, registered reports were assessed only descriptively, in line with TOP2025, and not as an expected requirement; that BMC Medicine already offers the format nonetheless shows it is feasible in general medical journals.

Second, there are some tool-specific considerations. For the LLM, we used a short prompt to remain generalizable across journals; however more elaborate prompts, with precise target definitions and a few examples, could likely improve performance without making the approach journal specific. Our analysis captured the presence of transparency indicators but did not evaluate their completeness, accuracy or quality. For data sharing, we measured intention to share rather than verified availability, using LLMs rather than specialized tools (eg, ODDPub, rTransparent, which were developed to detect open data) as these performed less well for this outcome. This mirrors the ICMJE data sharing statement requirement, but intention to share overstates effective access: when data are actually requested, most are not shared.³⁹ Verified availability is a distinct outcome that should be the focus of future studies. For instance, preprint detection relied on the presence of a preprint link in the publication, an approach that likely underestimates the practice, as such links are not commonly reported even when a preprint exists. For MAs and observational studies, the SAP is often embedded within the study protocol rather than provided as a separate document; our tools did not differentiate the 2 formats, which may lead to an underestimation of SAP sharing for these designs, even if protocol sharing remained poor in both types. For reporting checklists, we detected only their mention, not their appropriateness or actual adherence. This illustrates a key limitation: the practices we assessed derive from a structured expert consensus, but the existence of a consensus does not mean it is implemented, or even known, by journals and editors.

Additionally, our reference standard is subject to several limitations. Manual extraction relies on textual mentions, which imperfectly reflect actual practice: for instance, reporting checklists like Consolidated Standards of Reporting Trials (CONSORT)⁴⁰ are rarely cited even when likely followed. Our analysis was also restricted to PDF and PDF-formatted supplements, and commercial tools (SciScore²⁹ and DataSeer³⁰) functioned only on a subset of articles, notably excluding XML-formatted Lancet articles. These technical constraints may have resulted in missed information. Furthermore, the included journals are high-impact biomedical journals and may have more stringent transparency policies and editorial practices than the broader journal landscape. Therefore, our findings may not be generalizable to all biomedical journals.

Despite these challenges, the study has notable strengths: it applied rigorously developed and previously validated tools^{29,31,32,36,41,42} to a large and diverse sample of journal articles, providing an overview of transparency in leading journals. Compared with an earlier rTransparent study of an Open-Access PubMed Central article,³¹ tool accuracy was lower here, likely reflecting access restrictions and the inconsistent placement of transparency information across journal. Accordingly, we used tools descriptively to identify general trends rather than make normative rankings. Our findings demonstrate the value of independent assessment systems for journal transparency practices. The TOP2025 framework¹⁸ and core open science practices²⁰ are the result of structured expert consensus and provide a reliable basis for assessing journal performance. Such monitoring could help authors to identify values-aligned journals and universities (eg, CoARA-endorsed⁴³) and could ultimately rank journals on their open science practices, as the Good Pharma Scorecard does for companies.⁴⁴ Moreover, it may help to distinguish legitimate journals that implement best practices from predatory journals that do not.²² LLMs offer promise as format-agnostic solutions for multitype information extraction. Until then, optimal performances likely requires combining specialized tools with improved LLMs (better prompts, larger or hybrid architectures) for article-level monitoring. Responsible use demands local processing or requires explicit policies against misuse from commercial providers and significant trade-offs: environmental, computational, financial, and maintenance cost. These costs must be weighed against simpler algorithms yielding comparable results.⁴²

Conclusions

In this cross-sectional study of 15 624 articles from 10 leading medical journals, journal policies were only partially aligned with TOP2025 guidelines and concentrated on RCTs. Article-level open science practices were more also frequently reported for RCTs than for other designs. Adherence closely tracked the level of endorsement, being highest for formally required practices and lowest for those endorsed by experts but not yet mandated. These findings support the need for stronger journal policies to advance transparency and reproducibility across all study types.

Supplement 1.

eAppendix 1. List and Description of the Items Evaluated by TOP2025

eAppendix 2. Search Strategies, Reviewed by an Information Specialist Following PRESS Guidelines

eAppendix 3. Table of the Available Document Types for Each Journal

eAppendix 4. Table of the Automatic Tools With the Practices That They Assess

eAppendix 5. Detailed Technical Specification and Implementation Procedures for Automated Tools

eAppendix 6. Protocol Modifications

eAppendix 7. Study Flow Chart, Outlining the Selection Process for the Validation Databases

eAppendix 8. Description of the Accuracy of the Automatic Tools (Sensitivity, Specificity, and F1)

eAppendix 9. Automatic Tool Performance Depending on the Study Type and the Journal

eAppendix 10. Justification for Each Open Science Practice of the Chosen Tool

eAppendix 11. Automated Tool Performance in XML and PDF Files

eReferences.

Supplement 2.

eAppendix 12. Comments from Editors Following the TOP2025 Assessment

Supplement 3.

Data Sharing Statement