Reporting checklists in neuroimaging: promoting transparency, replicability, and reproducibility

Abstract

Neuroimaging plays a crucial role in understanding brain structure and function, but the lack of transparency, reproducibility, and reliability of findings is a significant obstacle for the field. To address these challenges, there are ongoing efforts to develop reporting checklists for neuroimaging studies to improve the reporting of fundamental aspects of study design and execution. In this review, we first define what we mean by a neuroimaging reporting checklist and then discuss how a reporting checklist can be developed and implemented. We consider the core values that should inform checklist design, including transparency, repeatability, data sharing, diversity, and supporting innovations. We then share experiences with currently available neuroimaging checklists. We review the motivation for creating checklists and whether checklists achieve their intended objectives, before proposing a development cycle for neuroimaging reporting checklists and describing each implementation step. We emphasize the importance of reporting checklists in enhancing the quality of data repositories and consortia, how they can support education and best practices, and how emerging computational methods, like artificial intelligence, can help checklist development and adherence. We also highlight the role that funding agencies and global collaborations can play in supporting the adoption of neuroimaging reporting checklists. We hope this review will encourage better adherence to available checklists and promote the development of new ones, and ultimately increase the quality, transparency, and reproducibility of neuroimaging research.

Introduction

The impact of neuroimaging research could be enhanced by robust methods for increasing the reliability and generalizability of findings across numerous studies. A critical issue hindering generalizability is the need for more transparent reporting of crucial methodological details in neuroimaging studies [1, 2]. Incomplete reporting reduces the value of individual studies and prevents replication by other researchers [1, 3]. Furthermore, biased reporting practices, particularly the emphasis on positive results, can mislead researchers [4, 5]. Several factors hinder the generalizability of neuroimaging studies, including low statistical power, flexibility in data analysis, software errors, and a lack of direct replication [3]. In the field of psychiatry/neurology, these issues are compounded by the pitfalls of case-control designs, confounders in associational research, and a lack of individual-level analyses and causal mechanisms [6]. Furthermore, the inconsistency of individual studies due to small and heterogeneous samples, analytical flexibility, and publication bias can also affect the generalizability of neuroimaging findings [7]. To mitigate some of these challenges and limitations to improve rigor and reproducibility and facilitate reliable meta-analyses from available research, there have been growing calls for protocol pre-registration [8], standardized processing guidelines [9], and transparent reporting [10–12].

The growing complexity and diversity of neuroimaging research methods and their incomplete reporting exacerbate these transparency and reproducibility issues. To improve the reporting of critical study design and execution details and promote robust and transparent research practices, we encourage the adoption of reporting checklists. These checklists outline minimum sets of details required to replicate studies, enable cross-study comparisons, and facilitate meta-analyses. Checklists can be broad in scope, outlining general reporting requirements across several neuroimaging modalities [13, 14]; or they may be specific to reporting requirements for one specific neuroimaging modality [15, 16].

Table 1 summarizes the terms used in neuroimaging checklists along with some examples or supporting evidence for each definition.

Table 1.

Key terms in neuroimaging reporting checklists: definitions and examples.

Term	Definition/Description	Example/Evidence
Best Practices	Guidelines promoting transparency, reproducibility, and rigor in research methodology and reporting [13]	Best practices in structural neuroimaging of neurodevelopmental disorders [91] Best practices in data analysis and sharing in neuroimaging using MRI [40]
Biased Reporting	Systematic deviation from accurate and impartial reporting of research findings, methods, or interpretations, undermining the integrity and reliability of research outcomes [92]	Potential reporting bias in neuroimaging studies of sex differences [5]
Checklist Adherence	The extent to which researchers comply with requirements and recommendations in reporting checklists, indicating thoroughness and completeness in study reporting [39]	High adherence to the CONSORT checklist in randomized controlled trials [93]
Checklist Development	Systematic creation of reporting checklists through collaboration/consensus among experts to identify essential elements for transparent and reproducible research reporting [86]	Guidelines for reporting health research [94]
Consensus Statements	Formalized documents representing collective agreement among experts in a field on specific topics or practices [95]	The STARD (Standards for Reporting of Diagnostic Accuracy Studies) statement for diagnostic accuracy studies [14] The International Associaton for the Study of Pain consensus statements on the evaluation of neuroimaging measures of chronic pain [96]
Consensus-Making	Process of reaching agreement among a group of experts or stakeholders through discussion and negotiation [97]	Development of the PRISMA statement through consensus among systematic review experts [98]
Delphi	A process in which experts in the field approach consensus on a matter by participating in a series of commenting and/or item rating rounds with feedback [99]	A checklist for assessing the methodological quality of concurrent tES-fMRI studies (ContES checklist): a consensus study and statement [16]
Data Repositories	Centralized platforms for storing and sharing research data, often with standardized formats and accessibility [100]	OpenNeuro, a repository hosting neuroimaging datasets with standardized formatting [61] The National Institutes of Health (NIH) data repositories for Alzheimer’s Disease Neuroimaging Initiative (ADNI) [101]
Data Sharing	The practice of making research data available to facilitate transparency, reproducibility, and collaboration in scientific research [102]	Sharing of raw neuroimaging data from the Human Connectome Project [103]
Efficiency	Achieving maximum output with minimal resources, time, or effort, often through streamlined processes or workflows [104]	Use of automated data processing pipelines to analyze neuroimaging data more efficiently [105] Optimizing MRI scan protocols to reduce scan time while maintaining image quality improves patient comfort and increases scanner availability [106]
Ethical Principles	Fundamental guidelines governing the moral conduct of research, ensuring respect for participants and integrity [107]	Declaration Of Helsinki: a landmark document in medical research ethics that outlines ethical principles for human experimentation, emphasizing participant welfare, informed consent, and research integrity [108]
Frameworks	Structured outlines or systems providing a foundation for organizing and addressing complex research concepts [109]	The FAIR (Findable, Accessible, Interoperable, Reusable) framework for data management [110]
Generalizability	The extent to which research findings can be applied or generalized to other populations, settings, or conditions beyond the study sample [111]	Evaluating the generalizability of neuroimaging findings across diverse demographic groups and clinical populations [112]
Good Research Practices	Established methods and behaviors recognized as conducive to producing reliable and valid research outcomes [113]	Good practice in food-related neuroimaging [114]
Guideline	Detailed recommendations or instructions outlining best practices or standards for conducting research [115]	The PRISMA 2020 statement: an updated guideline for reporting systematic reviews [116]
Harmonization	Process of aligning or standardizing practices, procedures, or guidelines across different research contexts [117]	Harmonization of neuroimaging data formats and analysis methods within the BIDS framework [2, 118]
Inclusiveness	Ensuring the involvement and representation of diverse individuals or perspectives in research processes [119]	Inclusion of participants from various demographic groups in neuroimaging studies to improve generalizability [120]
Manuals	Comprehensive guides or handbooks providing detailed instructions or procedures for conducting specific tasks [121]	The MEEG manual for standardized procedures in magnetoencephalography and electroencephalography [122] Harmonization efforts to ensure data from different PET scanners to be compared effectively in multi-center studies [123]
Protocols	Detailed plans or procedures outlining the steps to be followed in research studies or experimental investigations [124]	Cerebral small vessel consortium (MarkVCID: II. Neuroimaging protocols [125] An MRI protocol specifying the type of scan sequence, imaging parameters, and participant instructions for an experiment [126]
Questionable Research Practices (QRPs)	Questionable or unethical behaviors in research, such as selective reporting or data manipulation [127]	Prevalence of research misconduct and questionable research practices [128] Excluding data from participants who performed poorly in an fMRI task to achieve statistically significant results would be a QRP [129]
Reliability	Consistency and dependability of research findings or measurements, indicating the extent to which results can be trusted and repeatable [130]	Test-retest reliability of Evoked BOLD signal [131] Test-Retest Reliability of Common Task-fMRI Measures [132]
Repeatability	The ability of a study or experiment to produce consistent results when repeated under similar conditions [133]	Repeatability of neuroimaging findings demonstrated through replication studies using the same methodology [134]
Replicability	The ability of a research study or experiment to produce consistent results when conducted by different researchers, using different methods, materials, or conditions [135]	Open Science Collaboration: a collective effort to promote transparency, reproducibility, and openness in scientific research across various disciplines [136]
Reporting Checklist	Structured list of essential items or criteria designed to guide researchers in accurately reporting research methods, results, and conclusions [137]	A methodological checklist for fMRI drug cue reactivity studies: development and expert consensus [15]
Reporting Formats	Standardized formats or templates for presenting research findings, facilitating clear communication and interpretation of results [138]	Introduction-Methods-Results-Discussion (IMRAD) structure [139]
Reproducibility	The ability of a study or experiment to produce the same outputs using the same methods, by different researchers [140] and possibly on different data	Reproducibility of neuroimaging findings demonstrated by independent research groups using the same dataset [139, 141]
Research Integrity	Adherence to ethical principles and professional standards in the conduct and reporting of research, ensuring honesty, accuracy, and transparency [142]	Considerations for Open Science practices in neuroimaging [143] Ensuring informed consent, data security, and honest reporting of findings as parts of research integrity in neuroimaging studies [144]
Resource Sharing	The practice of making research data, materials, and tools openly accessible to the scientific community for replication, verification, and reuse [145]	TemplateFlow: an open-science platform to facilitate the sharing and dissemination of neuroimaging templates and atlases [146]
Robust Methodologies	Reliable and well-established research methods and techniques that produce valid results across different conditions or contexts [147]	Common pitfalls and limitations associated with low statistical power and strategies for improving power estimation and reporting for conducting network neuroscience research [148] Using validated image analysis pipelines in fMRI studies to ensure robust methodologies [149]
Scientific Rigor	The degree of thoroughness, accuracy, and reliability in the design, conduct, and reporting of scientific research [150]	Adoption of pre-registration practices in psychological science. (i.e., registering study hypotheses, methods, and analyses before data collection), to enhance the credibility of research findings [151]
Standards	Published documents that establish technical specifications and procedures designed to maximize the reliability of the materials, products, methods, and/or services people use every day [152]	Brain Imaging Data Structure (BIDS) for organizing neuroimaging data [2]
Transferability	The extent to which research findings, methods, or interventions developed in one context can be applied or adapted to another context [153]	Neurolaw in the Netherlands: investigating the applicability of neuroscientific findings to legal contexts, particularly in the domain of adolescent criminal law [154]
Transparency	Openness, clarity, and completeness in the reporting of research methods, results, and interpretations, promoting scrutiny and reproducibility [155]	Enhancing transparency in neuroimaging research by providing comprehensive methodological details and sharing raw data [2]
Trial Registries	Databases or repositories where researchers prospectively register clinical trials, providing details about study design, interventions, and outcomes [156]	ClinicalTrials.gov [157] International Clinical Trials Registry Platform (ICTRP) [158]
Validity	The degree to which research findings accurately represent the phenomenon being studied, ensuring the robustness and trustworthiness of conclusions [159]	Importance of validity in study design, data acquisition, and analysis and the necessity of ensuring that neuroimaging findings accurately reflect underlying neural processes and are not influenced by confounding factors or methodological limitations [39]

The following sections will delve into the fundamental values and concepts underpinning the development of reporting checklists in neuroimaging. We will explore the characteristics of a reporting checklist, outline the development milestones, and leverage insights from existing checklists to propose a roadmap for advancing reporting practices in neuroimaging.

Definition and impact of reporting checklists

Scientific research relies on robust methodologies and transparent reporting to ensure the credibility and reproducibility of methodologies, analytics, and findings. Reporting checklists serve as essential tools to achieve these goals. These systematic guides, often developed through consensus between researchers within a specific field, provide a comprehensive list of items to consider, implement, and report throughout the research process. By addressing key elements such as study design, data collection, analysis, and interpretation, reporting checklists promote the quality, integrity, and rigor of research methods. Ultimately, they enhance the reliability and validity of research findings and facilitate reproducibility [17–20].

The importance of reporting checklists is underscored by the emergence of related yet distinct concepts within the scientific community. These include protocols for specific methodologies (e.g., The Comorbidity and Cognition in Multiple Sclerosis (CCOMS) neuroimaging protocols [21]), reporting formats (e.g., APA style [22]), frameworks (e.g., Reviewers’ Competency Framework [23]), methodological manuals (e.g., Cochrane Risk of Bias Tool [24]), data standards (e.g., Brain Imaging Data Structure (BIDS) [25]), trial registries (e.g., https://www.clinicaltrials.gov), ethical codes/principles [26], and consensus statements [27, 28].

Reporting checklists can increase the quality of published studies by increasing the transparency of the research process, methodology, and analytics in peer-reviewed publications. By encouraging the detailed reporting of research methods and findings, regardless of their outcome and its significance, they contribute to the trustworthiness of research. Opaque or incomplete reporting practices can erode trust in studies. While checklists alone cannot prevent data fabrication or misconduct, they can serve to promote transparency in research practices. This could be reached by systematically prompting researchers to report key methodological details and results, thus facilitating greater scrutiny and verification of study findings by peers and the scientific community. This transparency can help detect and deter research misconduct, as well as enhance the credibility and reliability of research outputs.

The benefits of reporting checklists extend to a range of stakeholders, first and foremost the researchers themselves. Checklists are often used at the reporting stage and can facilitate clear and transparent reporting, but would be even more consequential before most study design and analysis decisions have been made: consulting methodological checklists early-on in the research process can help investigators avoid poor design choices and serious errors. For example, checklists across neuroimaging modalities can (and often do) require researchers to justify their sample sizes, ideally with power analyses conducted before recruitment. They can also suggest or explicitly require various quality-checking procedures, depending on the specific modality. Both of these measures would ultimately reduce wasted time and resources and improve research quality, and are critical, especially in light of recent evidence about severe power issues [29] and the distorting impact of inadequate quality control [30] across neuroimaging research. Checklists could also help guide the interpretation of study results (for example by outlining specific criteria for the biological interpretation of neuroimaging derivatives). Reporting checklists enhance the assessment and transparency of research reports for other stakeholders as well: for example, reviewers can more clearly evaluate the quality of grant applications and manuscripts; journals and publishers can use these checklists to promote transparency in publications; editors, who may not specialize in every area, can use checklists to more comprehensively verify basic research rigor; members of industry can push for clearer reporting of sponsored research and place greater trust in academic reports; funding agencies can use checklists to help maximize the impact of their investments; and practitioners can depend on more reliable research findings to guide their practice.

Core values in developing methodological reporting checklists

Methodological reporting checklists reflect a pragmatic and proactive approach to increasing the reproducibility, transparency, and integrity of evidence in the field [31, 32]. To meet such an objective, the development of valid and useful checklists should follow some core values as suggested below.

Transparency

The checklist development process should be transparent, from the initial phase of developing the idea through proposing the sets of checklist items. Many checklist development steps are prone to misreporting, like only inviting familiar fellows to the panel or suggesting a mandatory item that the authors have conflicts of interest. Pre-registering the methodology of the checklist development could be helpful to promote transparency of the checklist development process [33].

Repeatability

A reporting checklist should be developed via a process that can be replicated by other teams; meaning that it should be developed and reported as objectively as possible so that if the checklist development process were to be hypothetically repeated, the process would be doable and nearly similar results would be achieved. The replicability/repeatability issue can be addressed in the process of shaping the steering committee and expert panel, the methodological steps for reaching a consensus, the thresholds of agreement for selecting the mandatory and optional items in the checklist, and the process for continued updating and monitoring [32]. Note that this point does not imply that a checklist creation process should be replicated; simply duplicating an existing checklist would hardly be useful. Rather, the core value is that the checklist development process should be described and followed objectively and precisely enough that the process is Repeatable. Here repeatability is an objective measure of transparent reporting of methodology not a recommendation to replicate the available checklists.

Data sharing

To increase the transparency and reproducibility of the reporting checklist development, all relevant data and data analytics used in the process should be publicly available, including the database of panelists and anonymous comments and ratings of experts. This will allow the scientific community to ascertain the validity of the reporting checklist [34].

Diversity, inclusiveness, and global collaborations

In the development process for reporting checklists in each field, it is important to consider voices from all active researchers in the field around the globe. The recruitment for the steering committee and expert panel should consider diversity and inclusiveness on multiple levels, e.g., gender, race, ethnicity, academic institutions, industries, and geographic location. Lack of diversity may result in missing perspectives and hinder widespread adoption. Efforts to enhance diversity could leverage existing initiatives like the ALBA Network [35] and the Organization for Human Brain Mapping’s Diversity and Inclusivity Committee [36]. While no relevant research exists on the importance of diversity for neuroimaging checklists and guideline development, it has been recognized that lack of diversity among clinical practice guideline authors can lead to inequities [37, 38].

Supporting innovation and new methodologies

While checklists are believed to help the scientific community in several ways, there are concerns that they might limit the diversity and novelty of scientific research endeavors [33]. It is thus important to highlight that a checklist is not usually meant to impose the precise methods employed in the formulation of specific studies. Rather, its purpose is to guide researchers in conscientiously addressing, documenting, reporting, and sharing different aspects of their study design, analysis, and reporting that could significantly influence the outcomes. Although there are cases where checklists can have a more prescriptive form - e.g., in the recent REFORMS checklist (https://reforms.cs.princeton.edu/) a goal is to prevent common pitfalls like leakage of test data into the training ones or as noted below regarding multiple testing in the COBIDAS guidelines. Researchers are encouraged to articulate their methodological choices/variations when planning and presenting the findings of their research outside the framework provided by the available reporting checklists.

Current experiences in developing reporting checklists in neuroimaging

This section explores the current landscape of reporting checklists in brain mapping, focusing on examples like the Organization for Human Brain Mapping (OHBM) Committee on Best Practices in Data Analysis and Sharing (COBIDAS) Checklists, International Society for Magnetic Resonance in Medicine (ISMRM) Magnetic Resonance Spectroscopy (MRS) checklist, Addiction Cue-Reactivity Initiative (ACRI) fMRI Drug Cue-Reactivity (FDCR) checklist, International Network of Neuroimaging and Neuromodulation (INNN) Concurrent tES-fMRI (ConTES) checklist, and Content and Format of PET Brain Data Checklist. It should be noted that this is not an exhaustive list of checklists and several other neuroimaging reporting checklists are available on Enhancing the QUAlity and Transparency Of health Research (EQUATOR) website (https://www.equator-network.org/reporting-guidelines).

OHBM committee on best practices in data analysis and sharing (COBIDAS) checklists

One of the most widely used and comprehensive sets of checklists in neuroimaging is developed by the Organization for Human Brain Mapping (OHBM) (https://www.humanbrainmapping.org/i4a/pages/index.cfm?pageid=1). This checklist grew out of the need for the application of principles for open science to promote reproducibility and transparency in neuroimaging studies, building on earlier reporting guidelines developed informally within the field [39]. Although there are controversies over what reproducibility and transparency really mean, the ultimate aim of reporting was to allow the reader to completely understand the strengths and limitations of the reported study and enable a researcher to exactly conduct or manipulate each aspect of that study. To reach this aim, in 2014, motivated by growing concerns about reproducibility in science, the OHBM formed a committee to establish recommendations for best practices. The Committee on Best Practices in Data Analysis and Sharing (COBIDAS) that was formed first focused on MRI. It considered seven different domains of practice: experimental design, acquisition, preprocessing, statistical modeling & inference, results, data sharing, and reproducibility. It was quickly apparent that prescribing the “right” practice was generally impossible because of the great variation in the objectives and methods of any particular study. Hence, the focus was on transparent reporting of practice, and a set of tabular items that should be reported was created for each domain. One exception was statistical modeling & inference, where it was agreed that some practices are ill-advised (e.g., not correcting for multiple testing), and thus for this section the checklists took on a more prescriptive tone. The report was finished in 2016, and the community was invited to comment. A blog was set up to collect input and address each comment received. The document was updated according to that input, and after approval by the OHBM Council, the “Best Practices in Data Analysis and Sharing in Neuroimaging using MRI” was published on bioRxiv [40], and a short descriptive commentary was published in Nature Neuroscience [13]. Based on the positive response from the community, a COBIDAS Magneto /Electro-Encephalography (MEEG) was established in 2017. It followed the same structure as the MRI report, with an important feature of the use of community feedback and open posting as a blog [9, 41]. Since 2018, the newly established OHBM Best Practices Committee has coordinated both house COBIDAS reports (for instance, for Brain Networks nomenclature [42] or Clinical language fMRI [43]) and endorses other imaging best practices from other groups and societies (see https://www.humanbrainmapping.org/i4a/pages/index.cfm?pageid=4027). The OHBM approach is inclusive, from the diversity of the COBIDAS to the mandatory open reporting for comments by the community. Those reports are also live documents with updates planned every 3 to 6 years. The collective experience is that there are differences between the minimal information needed to understand what was done in a study and what minimal information is needed to reproduce a study. Brain imaging involves an expanding set of complex data acquisitions and processing requiring far too many details. These details can however be listed as appendices or supplementary materials or linked to data and code repositories.

Minimum reporting standards for in vivo magnetic resonance spectroscopy (MRSinMRS)

Over the last 40 years, in vivo magnetic resonance spectroscopy (MRS) evolved from a pure research method to a diagnostic neuroimaging tool [44]. The MRS field, however, continues to receive critiques in terms of diverse and suboptimal methodologies [45], lack of rigor and reproducibility, not reporting the fundamental aspects of studies to be used in meta-analyses [46], and limited guidance on practice for researchers in spite of the rich literature [47]. These limitations have been speculated to hinder MRS from prevailing in routine clinical applications. To tackle these issues, diverse international subgroups were formed in a coordinated community-wide effort that resulted in experts’ consensus recommendations on a total of 13 major technical aspects of MRS published in a ‘Special Issue: Advanced methodology for in vivo magnetic resonance spectroscopy’ in the years 2019–2021 [28]. Among these publications was a set of minimum recommendations and reporting standards for MRS studies comprising a checklist among other recommendations. To this end, an initial set of guidelines established at the 2016 International Society of Magnetic Resonance in Medicine (ISMRM) workshop was reviewed by a diverse group of MRS researchers comprising both established and less experienced researchers including trainees. The reporting guidelines were then evaluated by experts in the field who met predefined criteria to be included in the panel for consensus on the checklist development. This led to a list of minimum standards comprising four sections namely: hardware, acquisition, data analysis methods and outputs, and data quality [48]. The aim of providing this checklist is to play a guidance role for authors, a quality assessment tool for journals, and standardize reporting for subsequent studies and meta-analyses.

Addiction cue-reactivity initiative (ACRI) fMRI drug cue-reactivity (FDCR) checklist

The field of fMRI drug cue-reactivity (FDCR) has seen a dramatic surge in the number of publications in the last 25 years, with 415 published articles by the 2023 [49]. In addition to some inherent issues of conducting FDCR studies, not accurately reporting the different aspects of these studies limits the overall reproducibility and rigor of the reports. After a series of meetings within the Enhanced NeuroImaging Genetics through Meta-Analyses (ENIGMA) Addiction working group (https://www.enigmaaddictionconsortium.com), experts witnessed a high level of heterogeneity in the field of fMRI drug cue-reactivity (FDCR), poor methods reporting, and disagreement over the main methodological parameters. These discussions led to the formation of a steering committee with the aim of preparing a reporting checklist in the field of FDCR [15].

After preparing the first draft of the reporting checklist and reaching an agreement among the steering committee, using a modified Delphi methodology, experts were invited to comment on and modify the checklist using predefined criteria. After implementing the received comments by the steering committee, the checklist was sent to the experts for the rating phase in which experts would rate items from not important [1] to extremely important [5]. It is of note that the thresholds for including/excluding an item in the checklist were pre-registered in the protocol of the checklist. The comments received in the revision phase and the ratings in the rating phase were made anonymous and publicly available. Figure 1 shows a summary of the steps taken to provide the checklist and the quantitative measures of agreement on the essential items to be reported.

Fig. 1. Addiction Cue-Reactivity Initiative (ACRI) fMRI Drug Cue Reactivity (FDCR) checklist development process and outcomes.

a Procedure flowchart The process has been roughly divided into distinct stages: the selection of the steering committee (in black) using the results of an earlier mentioned systematic review to choose the initial checklist items and expert panel candidates (in pink), checklist development phase (in red), expert panel selection (in purple), checklist commenting and revision phase (in green), checklist rating phase (in yellow) and data analysis and Delphi process finalization (in blue). The number of contributors to each section is displayed by ‘n =’. Within the graph, an overview of the structure of the checklist at each stage is presented in terms of number of categories, essential items to be reported and further recommendations and their categories. recom: recommendations. b Checklist Rating by Experts: Each item was rated from 1 to 5 (not important to extremely important). All the items met threshold 1 and were rated as moderately, highly, or extremely important by >70% of the raters. In addition, 24 items reached the more stringent threshold 2 of being rated as either highly or extremely important by 80% of raters (the ones that did not reach this threshold are marked with ‘†’) (the figure is adopted and modified from [15]).

The checklist emphasizes the importance of reporting methodological details that are crucial for conducting an FDCR study that would merit universal inclusion as methodological details. This checklist includes seven main categories: participant characteristics, general fMRI Information, general task information, cue information, craving assessment inside scanner, craving assessment outside scanner, and pre- and post-scanning considerations (Fig. 1b). By adhering to the ACRI-FDCR checklist, researchers can ensure comprehensive and informative reporting of their fMRI studies in the context of addiction research, facilitating the advancement of knowledge in this critical domain. After the finalization of the reporting checklist, researchers checked the reporting checklist against the previous FDCR publications to assess the current reporting status of the checklist items. This step was done to assess retrospective adherence to the proposed reporting checklist items and to motivate future evaluations to assess how introduction of the checklist improved the quality of reporting in the field.

International network of neuroimaging neuromodulation (INNN) concurrent tES-fMRI (ConTES) checklist

Integration of transcranial electrical stimulation (tES) with concurrent fMRI allows for mapping neural activity during neuromodulation, supporting causal studies of both brain function and tES effects. Given the potential for variability in neural responses elicited by tES based on the methodological nuances and the increasing number of concurrent tES-fMRI studies, guidelines on factors that should be reported and/or controlled are essential to ensure accurate interpretation and reproducibility of research findings. Additionally, consistent methodology and reporting practices would facilitate meta-analyses. Raising these issues in the International Network of Neuroimaging and Neuromodulation (INNN), experts concluded that the field of concurrent tES-fMRI (ContES) has an incoherent reporting system. As a result, a steering committee was formed to develop a reporting checklist for the field of ConTES [16]. The overall process of this Delphi study is similar to the ACRI-FDCR study mentioned in the previous section. The checklist emphasizes the importance of reporting crucial methodological details for conducting a ContES study, encompassing essential methodological details for universal application. This checklist is categorized into three main sections: technological factors, safety and noise tests, and methodological factors. By adhering to the ContES checklist, researchers can enhance the methodological reporting quality of future concurrent tES-fMRI studies and increase methodological transparency and reproducibility, ultimately fostering more robust research and facilitating meta-analyses.

Content and format of PET brain data checklist

The need for replicating Positron Emission Tomography (PET) findings, ensuring quality control, multi-central collaborations through data sharing, and the growing interests of funding agencies and journals in transparency and data sharing mandated the field of PET neuroimaging to develop guidelines for data sharing and reporting. Data sharing of PET studies presents additional challenges to those faced by other neuroimaging modalities, including the need for sharing raw data alongside detailed construction methods, sharing blood sampling data and related analyses, and sharing dynamic modeling techniques. Efforts to address these challenges started at the NeuroReceptor Mapping Conference in 2016 (NRM2016), with a panel comprising more than 250 PET specialists discussing the need for standards in PET data sharing. This meeting led to the formation of working groups with the purpose of proposing standards for the content and structure of PET imaging studies to facilitate the sharing of PET data [50]. This checklist includes more than 100 mandatory and recommended items related to Radioligands, Data Acquisition, Data-analysis, and Statistics. This checklist is an important step in improving the interpretation and replicability of published work, while also facilitating the processes of archiving and sharing PET data.

Development Process for Reporting Checklists in Neuroimaging

Generally, the aim of developing reporting checklists is to provide essential elements that should be considered over the whole life cycle of research projects. This may provide transparent and detailed information about study design, methodology, results, interpretation, and practical implications. Therefore, reporting checklists are not merely designed to be used when drafting the paper, but more importantly, they should be used in the initial stages of designing the protocol to ensure that all the necessary factors are taken into account for conducting a rigorous study and necessary details are recorded throughout the study lifecycle. These sets of criteria are often established by observing the deficiencies in the current literature, which may undermine the reliability and reproducibility of the findings.

A typical reporting checklist often results from a consensus among experts in the field on the minimum set of items deemed critical to be reported in a research paper. The checklist often includes items categorized into different aspects of a study, e.g., participants characteristics, processing pipelines, results reporting, accompanied by suggestions on where and how to address the reporting requirement [51].

The development cycle for methodological reporting checklists consists of seven steps each of which consists of different activities while suggesting several tools and techniques (Fig. 2).

Fig. 2. Development cycle of reporting checklists.

The inner circle depicts major steps with an example of relevant activities/tasks shown in the middle circle followed by an example of tools/mechanisms shown in the outer circle.

Genesis: What is the problem and what are its dimensions?

As the first step in developing each checklist, there should be a clear and defined problem statement reflecting what the checklist needs to address. These problems are often explored when experts are conducting research, reading published articles, or communicating with peers in meetings. To elaborate on the problem and define its many dimensions, experts often communicate their concerns with peers and form a committee to address this issue. For example, the MRSinMRS checklist originates from a workshop organized by a society (ISMRM). Then, under the umbrella of a broader community-based consensus effort in the field of MRS, the reporting concerns were discussed and a dedicated subgroup was formed to establish a checklist [48].

Development: What is the current situation in the field?

To make sure that the defined problem is valid and important for the field, it is helpful to map the parameter space of the problem in the early developmental stages. This is preferably done by a thorough systematic literature search to identify the parameters affecting the main problem along with their frequency and their impact on the transparency of procedures and reproducibility of the findings. For instance, a systematic review conducted on the field of FDCR [49] created the foundations of the ACRI-FDCR checklist [15]. This systematic review also further enriched the methodological reporting checklist by finding other parameters that were probably overlooked at the first steps.

Organization: Who are the main contributors?

Engaging the community of active researchers can happen by shaping a steering committee and/or an expert panel. This can be done by establishing a set of systematic criteria for experts to be invited, like the minimum number of publications each expert should have. With the help of the systematic literature search done in the previous step, it is possible to identify experts and organize them in either the steering committee (SC) and/or the expert panel (EP) in a replicable and inclusive way. As an example, the ConTES checklist achieved this aim by predefining its expert panel criteria as holding at least one first, last, or corresponding author position in publications of the field of concurrent tES-fMRI. To make sure that no other key expert is missed from the invitation process, invitees can also be asked to nominate further experts to be reviewed by the SC. In the process of checklist development, the SC is responsible for developing the initial draft of the protocol and checklist and making the initial decisions to move the checklist development forward. The members of EP, on the other hand, will shape the reporting checklist with their revisions. The process should be transparently recorded, documented, and reported.

Consensus-making: Is there a common ground for the checklist items?

Reporting checklists are developed to pave the way to adhere to some reporting standards in a specific research community and this could only be met if a common ground is shaped among active community members. There should be a predefined process of community engagement (as described above) and consensus-making (often through survey mechanisms) to ask for expert opinion. One of the popular ways to reach a consensus is through a Delphi process [52]. Experts are first provided with the initial draft developed by the steering committee listing all items the reporting checklist should include, and are asked to modify the initial checklist by changing the items or suggesting to remove or add new items. The steering committee collects all the inputs and updates the checklist items. At the next round, experts are provided again with the updated checklist and this time, are asked to rate items with a Likert scale from “not important to be reported” to “significantly important to be reported”. Based on the predefined thresholds in the protocol of the checklist development, items will either be removed or will be chosen as items to be included in the final checklist. There could be an iterative process defined to develop agreement in the next rounds of expert elicitation on the items that passed an initial threshold but have not achieved sufficient consensus.

For instance, the ACRI-FDCR checklist used a modified version of Delphi in which two rounds of expert elicitation were conducted. In the first round, the initial draft of the checklist was sent to the expert panel for comment and also suggestions for adding or removing items. After implementing the received comments by the steering committee, the checklist was sent to the experts for the rating phase in which experts would rate items from not important [1] to extremely important [5]. Thresholds for including/excluding an item in the checklist were pre-registered in the protocol of the checklist. The more-stringent threshold was a rating of 4 or 5 by more than 80% of the experts and the less-stringent threshold was a rating of 3, 4, or 5 by more than 70% of the experts. Also for the additional recommendation part which was simply rated as Yes/No, the rating of Yes by more than 50% of the experts would lead to the inclusion of the recommendation (not in the list of essential items).

Validation: Does the checklist reflect current publication norms?

After finalizing the reporting checklist items in the consensus-making stage, the applicability of the checklist could be tested against the currently available literature. This can be achieved by a reasonable predefined number of sample studies to be rated according to the developed checklist as was done in the MRSinMRS checklist. As reported in the checklist’s manuscript, some exemplar articles are rated to evaluate if the checklist can practically be utilized. This step not only validates the checklist’s effectiveness to support further utilization by scientific audiences but also can help improve the checklist items by removing vague or irrelevant items in the next revisions of the checklist.

Adherence: How much the checklist is being used in future publications?

To have an impactful reporting checklist, there should be a follow-up plan to reinforce adherence in the upcoming publications. This plan can include promoting a checklist to be used by other researchers in the field, providing education and support for checklist adopters, checking adherence in the new publications, and finding solutions to make the checklist easier to use. As an example, ACRI-FDCR is actively monitoring new publications in a live annual systematic review and checks their adherence to the checklist (https://med.umn.edu/addiction/network/acrin).

Notably another systematic literature search in this stage is needed to find the level of adherence to the checklist as well as possible pitfalls and defects that should be considered in the updating stage.

Updating; How this can be improved?

Reporting checklists are context-specific and technology-bound [31] and the ever-changing nature of the scientific research necessitates the checklists to be live entities. This requires mechanisms to keep the checklists updated according to the latest findings in the field and according to feedback from stakeholders. Nevertheless, there comes a time when the volume of updates may necessitate a reconsideration of the initial problem and parameter space. Thus, the checklist development model forms a cyclical view (instead of a linear one) reflecting a continuous and ever-increasing effort for further improvements (Fig. 2).

The evaluation of the above checklists based on the development cycle discussed in the previous section is summarized in Fig. 3.

Fig. 3. Checklist development process and research process coverage in sample neuroimaging checklists.

The top panel assesses 6 sample checklists using the developmental cycle model. Dark blue means that the checklist fully reported that item in their initial publication, Light blue means partially reported, and Light yellow means not reported. The bottom panel shows the main steps in a research process and how various checklists covered those steps in their items. Light yellow indicates no coverage, Light blue indicates indirect coverage, and Dark blue indicates full coverage. ACRI-FDCR Addiction Cue-Reactivity Initiative fMRI drug cue-reactivity checklist, COBDIAS Committee on Best Practices in Data Analysis and Sharing, ContES Concurrent tES fMRI Checklist, MRI Magnetic Resonance Imaging, MEEG Magneticencephalography/Eelectroencephalography, MRSinMRS Minimum Reporting Standards for in vivo Magnetic Resonance Spectroscopy.

It is noteworthy to mention that not all of the steps of the proposed seven-step development cycle are adhered to by the checklists addressed above. However, the inclusion of these steps in the proposed development cycle is intended to provide a structured framework and best practices for checklist development, rather than to suggest that all checklists must follow each step rigidly. It also should be noted that the development of a reporting checklist will not necessarily be hampered by skipping certain steps, as the development process can vary depending on factors such as the specific objectives of the checklist, available resources, and the expertise of the developers. However, each step in the development cycle tends to provide criteria to increase the robustness, transparency, inclusiveness, comprehensiveness, and usability of the checklist as noted in the description of each step.

Limitations and concerns regarding methodological checklists

Despite their potential benefits, it is important to note some of the reporting checklists’ limitations and relevant concerns about their proliferation in neuroimaging research. One concern, as previously noted, is that enforcing adherence to checklists might stifle innovation in neuroimaging research [33], particularly if the checklists are out of date. Given the rapid pace of development in the field, editors and reviewers need to exercise nuanced judgment when handling research that does not clearly conform to the scope of extant methodological checklists. A related challenge is that certain checklist recommendations and requirements may be too inflexible: for example, checklists might require sample size justification through power analysis, even though appropriate power can also be achieved through deep phenotyping [53] or the use of multivariate methods [54]. This highlights the necessity of nuance and flexibility when checklists are developed, their updating in response to arising concerns, and judicious enforcement. A more fundamental concern is that checklists are often disseminated before evidence of their usefulness is collected. As noted, such evidence is difficult to gather and requires longitudinal and interventional investigations, as checklist adherence on its own can only indicate reach and usability.

As the list of available methodological checklists grows, there are also concerns about “checklist fatigue”: researchers may be overwhelmed by the number of available checklists, and it may be difficult to select appropriate guidelines in the absence of clear scopes and collaboration between different checklist development and teams and adherence-enforcing entities such as publishers. Enforcement of checklist adherence could also become taxing for publishers and add to reviewing overhead, necessitating creativity in checking adherence using a mix of automated tools and nuanced human intervention (for example by reviewers).

A final point is that checklists are only one tool in improving the state of scientific research, and are indeed likely to have a fairly modest impact on their own. Without regular updating, nuanced enforcement, ongoing revision and impact assessment (among other endeavors), methodological checklists would merely add to the box-ticking exercises researchers have to follow. Even at their best, however, and as noted in the introduction, many factors hamper the generalizability and reliability of neuroimaging research studies. It is important to acknowledge the limitations of checklists in enhancing research integrity and emphasize the importance of complementary measures, such as robust peer review, data sharing, and replication efforts. Checklists can mitigate questionable research practices (QRPs) in part by promoting such measures, for example, preregistration and the availability of raw and processed data and analysis code [55]. But checklists alone cannot bring about substantial shifts in research culture and practices.

The future of reporting checklists in neuroimaging

As discussed above, reporting checklists are increasingly developed and adopted for neuroimaging research. The following sections are dedicated to discussing the potential of these checklists to facilitate neuroimaging research in the future, important challenges, and the promise of new technologies.

Reporting checklists to support neuroimaging data repositories

With the increased availability of neuroimaging technologies, the proliferation of hardware and informatics infrastructure, and years of open science advocacy, unprecedented volumes of neuroimaging data are available to researchers [56, 57]. “Big data” can be used to detect true effects with appropriate statistical power and ensure that scientific findings are generalizable, overcoming challenges inherent in neuroimaging studies with small sample sizes [29, 58]. Some such data comes from large studies overseen by centralized committees which adhere to harmonized imaging protocols, such as the UK Biobank [59] and the Adolescent Brain Cognitive Development (ABCD) Study [60]; but a substantial portion of available neuroimaging data comes from smaller studies shared on repositories such as OpenNeuro [61]. Currently, the OpenNeuro repository alone hosts data from 41,599 participants across 1,025 datasets.

The great strength of aggregating data across studies may, however, also create vulnerabilities to particular types of pitfalls. Aggregating large amounts of data can yield impressive group-level statistics while obscuring possible errors in subsets of the data and important differences between pooled data. Since its founding, OpenNeuro has sought to ensure that data shared on its platform are “FAIR”: findable, accessible, interoperable, and reusable [62]. This has largely been attempted through the development and promotion of the Brain Imaging Data Structure (BIDS), requiring all submitted data to conform to this common BIDS standard [25]. Despite the rapid uptake of BIDS across the neuroimaging community however, there are gaps which could be addressed through the use of checklists to ensure there is sufficient context for data shared on OpenNeuro to be re-used and aggregated, whether in the form of meta-data or additional documentation. Automatic BIDS validators do not currently enforce the reporting of all participant or task characteristics which required to use data from a particular experiment. For example, in clinical research, even “caseness” is defined in various ways across studies, ranging from a formal diagnosis based on a clinical interview to exceeding a criterion score on a substance use questionnaire such as the AUDIT for assessing problematic alcohol use [63]. The full list of important variables and how they should be reported can be determined by checklist developers who are experts in the specific neuroimaging field.

Reporting checklists to support neuroimaging consortia

The most notable example of a neuroimaging consortium for systematic, large-scale aggregation of data from previous research is the ENIGMA consortium, founded in 2009 to provide the increased statistical power required for brain-gene association analyses through data pooling. It has conducted some of the largest neuroimaging-genetic analyses to date by gathering datasets for secondary analyses, including the discovery of the genetic correlates of human cortical structure in a sample of over 50,000 participants [64, 65] This efficient and cost-effective strategy has spawned over 30 disease- and syndrome-specific subgroups which have adapted this approach to study the brain and genetic correlates of various clinical conditions, such as substance use disorders [66]. The large sample size provided by data pooling enables researchers to identify reproducible (e.g., split-half confirmation) brain correlates of substance dependence, some that are common across substances and some that are substance-specific [67]. It can also provide better effect size estimates through meta-analyses [68].

ENIGMA has had to tackle the issue of heterogeneity and errors in subsets of the data. Taking ENIGMA-Addiction as an example, the consortium has pooled data from over a hundred studies contributed by 103 principal investigators at 71 different institutions in 16 countries and 6 continents. Successful integration of these data requires careful attention to the details of each study. The neuroimaging data vary in the specifics of their acquisition which occurs on different scanners from different manufacturers employing different image acquisition parameters. Functional data, both task and resting-state, admit even more variety with inter-study differences in task design including stimulus and response modalities and time-series duration. A newer ENIGMA group, focusing on neuromodulation, is systematically evaluating TMS protocols, examining how the frequency, duration, and placement of TMS coils can affect results. Similarly, ENIGMA’s EEG working group [69] has been evaluating how choices in reference electrode and montage affect individual differences in EEG parameters. Lastly, to better streamline future data collection, ENIGMA’s Brain Injury working group has created a set of guidelines termed ENIGMA’s ‘simple seven’ [70] to improve the reproducibility of resting-state fMRI in traumatic brain injury research. Careful attention to cross-study differences is essential when processing each for inclusion in large, pooled analyses.

However, even more complex than the heterogeneity of the varied neuroimaging data is the variety that exists in the non-imaging phenotyping of participants. Study samples can vary on the depth of assessment and definition of diagnostic groups, psychiatric co-morbidities, family history, sociodemographics, and so on. Assessment of other relevant psychological measures such as impulsivity and risk-taking can be extensive in some studies, meager in others, and even left unreported. Similarly, details on inclusion and exclusion criteria and sample characteristics can make ostensibly similar studies quite different. These and other factors can potentially threaten the validity of pooled analyses. Conversely, adequate recording and annotation can enable in-depth analyses of the importance of these factors across studies, enhancing research with pooled data. The availability of specific reporting checklists alongside databases can serve an important role in facilitating the careful and standardized reporting of important details.

Checklists providing standardized details on published studies would greatly enhance the data-pooling efforts of ENIGMA. Indeed, ongoing initiatives within ENIGMA towards harmonized analyses of structural and functional neuroimaging data make clear that many research details are regularly omitted from publications. Further, the fallibility of memory and the turnover of staff and trainees can impede secondary analyses. A standardized approach to recording and reporting these details (e.g., completion of a methods-related checklist at the time of submission to a journal) is a simple but valuable step to facilitate subsequent harmonized analyses.

It is important to note here that consortia datasets are often datasets of opportunity, where most of the data is retrospectively collated across studies that have already concluded. These studies would not have necessarily adhered to consortium standards when conducted and may have not consulted any neuroimaging checklists during study execution and reporting. In such cases, some key data may invariably have been lost, and the use of checklists should not prevent the inclusion of such studies if researchers can still provide enough information for their data to be of potential use in mega-analyses. Researchers can still consult checklists at this stage when submitting data from completed studies to consortia, and at least transparently report what details they can report and clarify what important information required by the checklist cannot be reported.

Promoting checklist adherence

So far, dozens of checklists have been developed to cover different aspects of neuroimaging studies. To measure the status of the field in terms of reporting the items deemed to be important by experts, some checklists assess the percentages of their items being reported in the literature. ACRI-FDCR checklist has evaluated the field after finalization of the checklist and before publishing it, as depicted in Fig. 4. More than 40% of peer reviewed publications in the field reported less than 70% of the essential items in ACRI-FDCR checklist [15]. This evidence further supports the need to use these checklists to enrich future publications in terms of reporting items that increase transparency and reproducibility.

Fig. 4. Reporting status of the Addiction Cue-Reactivity Initiative (ACRI) fMRI Drug Cue Reactivity (FDCR) checklist.

This figure provides an overview of the reporting status of studies that employed the ACRI-FDCR checklist. The assessments are based on the analysis of 108 sample FDCR articles before the publication of the ACRI-FDCR checklist. a displays the percentage of articles that reported each specific checklist item. Each bar in this panel represents a checklist item, showing how frequently it was included in the published studies. b summarizes the overall reporting status of the articles as a whole. This panel aggregates the data to give a broader view of how well the articles adhere to the checklist criteria.

A study reported that enforcement of adherence to three general reporting guidelines (STROBE for observational studies, CONSORT for randomized clinical trial studies, and PRISMA for systematic reviews) by asking authors to supplement their manuscript with the relevant reporting checklists, increased the adherence to those checklists [71]. This was further emphasized in a systematic review of more than 16000 trials that shows the endorsement of CONSORT checklist by journals increased the adherence to this checklist [72]. The endorsement of using checklists not only increases the number of reported items but also it was shown that reporting more items of the checklist increases the number of citations the subsequent articles receive [73]. Although the previous findings need further supporting evidence, they partially support the checklist endorsement policies in high-impact journals, some of them considering adherence to certain general reporting checklists as mandatory [74]. This growing interest in high-impact journals in endorsing reporting checklists to improve the reproducibility and transparency of their publications is also evident in the field of neuroimaging. For instance, the Molecular Psychiatry journal added a note in its submission process “reporting standards for Magnetic Resonance Spectroscopy” and highly encourages researchers to use MRSinMRS in their submission process (https://www.nature.com/mp/authors-and-referees/preparation-of-articles). In summary, it is evident that journals can play a pivotal role in encouraging or enforcing checklist adherence. Note that we do not differentiate between general-purpose journals and those with a more technical audience: checklists specify the minimum level of detail which should be reported by all studies in a specific field or subfield of neuroimaging, regardless of which journal they are published in; though authors can opt to report details deemed less relevant for a specific readership in supplementary materials. On the other hand, journals should be cautious in overwhelming researchers with checklists that do not necessarily improve the reporting quality of the published papers. Furthermore, while direct empirical evidence linking checklist usage to improved replication rates or increased utilization in meta-analyses is desirable, it may be challenging to obtain comprehensive data on these aspects due to various confounding factors and methodological challenges. In this regard, the indirect indicators of checklist utility, such as the increasing adoption rates and ongoing discussions within the research community regarding checklist implementation and refinement could give a sense of the long-term impact of checklist usage on research quality and reproducibility in the field. However, there is no doubt that checklist adherence is an indirect and imperfect measurement more robust evaluations through longitudinal studies or comparative analyses can give deeper insights into the effectiveness of the checklists and identify areas for further improvement.

Artificial Intelligence (AI) and Large Language Models (LLMs) for checklist development and adherence

LLMs can significantly enhance transparency and reproducibility in the development and application of methodological checklists for neuroimaging publications, especially given recent attempts at training LLMs on scientific literature specifically [75]. These technologies can ensure adherence to reporting guidelines and methodological checklists, both during manuscript preparation and peer review, thereby improving the quality and reliability of scientific findings [76]. AI and LLMs, such as ChatGPT, can also play an important role in the development and enforcement of methodological checklists. These technologies can analyze vast quantities of existing research to identify gaps in reporting, helping to justify the need for specific reporting guidelines. By understanding the complexities of research methodologies, LLMs can aid in creating checklists that are both comprehensive and flexible, ensuring that they capture essential information without being overly restrictive. AI can support the development of these checklists through an evidence-based approach, analyzing research and meta-analyses to identify key items that should be included.

Moreover, AI can facilitate the consensus process and the pre-registration of checklist development protocols, ensuring that these guidelines are developed transparently and rigorously. By leveraging AI and LLMs in this manner, the scientific community can significantly enhance the transparency and reproducibility of research findings. Taken together, AI and LLMs offer powerful tools for advancing the development, dissemination, and adherence to methodological checklists, thereby promoting a more transparent, replicable, and rigorous scientific process. Note that our focus here is on the use of emerging AI and LLM technologies in developing checklists and adherence checking. Much of the discussion about the role of these technologies in science has been focused on their use in facilitating the generation and editing of analysis code and narrative text, which is beyond the scope of the present manuscript. An example of how GPT-4, an advanced LLM, can be utilized to improve scientific manuscript sections is evident in its analysis of a methods section provided for a study on fMRI as an outcome measure using CONSORT 2010 guidelines for reporting parallel group randomized trials (Box 1).

Box 1 Example of AI/LLM Enhancement of Methodological Reporting.

Using the CONSORT_GPT [160], the adherence of a sample report with fMRI as an outcome measure [161] to CONSORT 2010 guidelines for reporting parallel group randomized trials [77] was checked. This example underscores how AI and LLMs can assist researchers in developing more detailed, transparent, and reproducible method sections by ensuring that all essential elements of study design and implementation are thoroughly addressed.

Here's an evaluation of the study methods and reporting based on the CONSORT 2010 guidelines for reporting parallel group randomized trials:

Title and Abstract (Items 1a, 1b): The abstract provides a structured summary including the trial design, methods, results and conclusions. The title indicates it is a randomized trial.

Introduction (Items 2a, 2b): The scientific background and rationale for the trial are provided.

• The specific objectives and hypotheses are stated.

  Methods

  Trial Design (Item 3a, 3b) :- Described as a randomized, triple-blind, sham-controlled parallel group clinical trial.

• Important changes to methods after trial commencement are not mentioned.

  Participants (Items 4a, 4b)-

  Eligibility criteria for participants are clearly defined.

• Settings and locations where data were collected are provided.

  Interventions (Items 5)

• The interventions for each group (active and sham tDCS) are well-described including procedural details.

  Outcomes (Items 6a, 6b)

• The primary and secondary outcome measures are clearly defined.

• Any changes to trial outcomes after commencement are not mentioned.

  Sample Size (Item 7a, 7b)

• The projected sample size of 60 and power calculations are reported.

• No interim analyses and stopping rules are mentioned.

  Randomization

  Sequence Generation (Item 8a, 8b)

• The method of random sequence generation is described.

• The type of randomization with blocked randomization is reported.

  Allocation Concealment (Items 9, 10)

  Allocation concealment mechanism is not described in detail.

  Who generated, enrolled and assigned participants is not clearly reported.

  Blinding (Item 11a, 11b)

• The triple-blinding procedure of participants, investigators and assessors is well-described.

• The method for unblinding if needed is not mentioned.

  Statistical Methods (Item 12a, 12b)

• Statistical methods for analyzing primary and secondary outcomes are outlined.

• Methods for additional analyses such as subgroup analyses are not described.

  Results

  Participant Flow (Item 13a, 13b)

• A participant flow diagram is not provided.

• Reasons for non-participation and exclusions are briefly mentioned.

  Recruitment (Item 14a)

• Dates defining periods of recruitment and follow-up are provided.

  Baseline Data (Item 15)

• Relevant baseline demographic and clinical characteristics are summarized for each group.

  Numbers Analyzed (Item 16)

• Number of participants included in each analysis is reported for primary outcomes.

  Outcomes and Estimation (Items 17a, 17b)

• Results for primary outcome of craving self-reports are provided with effect estimates and confidence intervals.

• Results for secondary neuroimaging outcomes are also comprehensively reported.

  Ancillary Analyses (Item 18)

• Results of additional exploratory electric field modeling analyses are presented.

  Harms (Item 19)

• Reports on adverse events and side effects recorded for each group.

  Discussion

  Limitations (Item 20)

• Trial limitations including potential biases are discussed.

  Generalizability (Item 21)

• Generalizability of the findings to other populations is not explicitly discussed.

  Interpretation (Item 22)

• Interpretation of results in the context of existing evidence is provided.

  Other Information

  Registration (Item 23)

• Trial registration number and name of registry are reported.

  Protocol (Item 24)

• Access to full trial protocol is not mentioned.

  Funding (Item 25)

• Sources of funding and other support are stated.

Overall, the paper follows CONSORT guidelines reasonably well in terms of describing the trial design, methods, analysis and results. Some areas that could be improved include providing a participant flow diagram, clearly reporting allocation concealment and blinding mechanisms, discussing generalizability of findings, and mentioning access to the full trial protocol.

Overlapping scopes and collaboration in reporting checklist development

Despite the vast diversity of neuroimaging modalities and research designs, there is substantial overlap in scope, methodological concerns, and good research practices across sub-disciplines. As such, many existing neuroimaging reporting checklists and guidelines overlap in their scope and items, and most neuroimaging research projects likely use more than a single checklist to ensure methodological rigor and transparency. This is especially the case when one considers that there are also reporting guidelines and checklists with broader scope and use which apply also to neuroimaging research: for example, a clinical trial using simultaneous transcranial electrical stimulation and fMRI with a drug cue reactivity task could use the ACRI FDCR checklist [15], the ContES checklist [16], the COBIDAS MRI checklist [40] which applies to neuroimaging research more broadly, and the still more widely-used CONSORT checklist [77] for clinical trials. To maximize the value of checklists, avoid duplication, and prevent user confusion, checklist development teams can collaborate with developers of other checklists to properly specify the scope of their work. In areas where the scope of several checklists overlap, care should be taken to either harmonize items across checklists or to refer readers to a single set of recommendations when possible. Such collaboration could also facilitate resource sharing between teams, and their supporting the uptake of each other’s checklists where appropriate.

As an example of this, the ACRI-FDCR checklist only points to some general participants’ characteristics and encourages researchers to use PhenX toolkit which has already designated a core assessment for mental health and addiction [78]. ACRI-FDCR checklist alsoincludes only a broad item to suggest reporting fMRI analysis details, and the checklist manuscript explicitly suggests that users consult the COBIDAS checklist for reporting statistical methods and results [79]. In the case of widely used checklists such as CONSORT, it is possible to develop new methodological checklists as “extensions” of the existing checklist. A relevant example is the recent development of SPIRIT-iNeurostim and CONSORT-iNeurostim extensions (to SPIRIT and CONSORT guidelines) for clinical trial protocols and reports of neurostimulation devices [80]. The most ambitious solution would be to develop “a network of consortia”, i.e., infrastructure for the development teams of neuroimaging guidelines and checklists to harmonize. A notable example is the EQUATOR network, an international network which maintains a central library of available reporting checklists, provides support for the development of new checklists, and aims to educate stakeholders about different checklists and guidelines [81].

Checklists and global neuroimaging education

Systematically developed reporting checklists and guidelines are distillations of evidence-based expert guidance on good research and reporting practices in neuroimaging, and as such could be helpful resources in promoting these practices among neuroimaging researchers and students. The EQUATOR network’s experience is again instructive: they have developed workshops to foster good reporting habits among young researchers and students (among others) and increase awareness of the network’s resources [81]. Courses on research integrity and good research practices already often incorporate methodological guidelines and checklists [82], and research institutions could endorse such checklists or use them in assessing trainees to teach adherence to best practice guidelines [83]. Checklists could also be used when developing and evaluating courses on research practices, for example by assessing trainees’ knowledge of checklist items before and after instruction or using the checklist to assess student projects completed during the course. A recent systematic review has noted that such standardized assessments are lacking in much of the pedagogical research literature on reproducible research practices [84].

Neuroimaging checklists are often written in deliberately plain, clear, and concise language, and could serve to promote good, consistent research and reporting practices globally. Increased awareness of these standards, and access to a shared set of instructions and terms, would also serve diversity and inclusivity purposes: It would help develop common frames of reference to facilitate the participation of researchers from under-represented backgrounds and countries in the production of robust research and the development of best practices. Reporting checklists could be easily introduced to a global audience of researchers in online seminars, workshops, and courses developed for this purpose, or they could be introduced in the context of more general neuroimaging and neuroscience courses. The Neuromatch Academy for example runs multi-week neuroscience summer courses for thousands of participants, who develop student research projects using neuroimaging data [85]. These courses already include guidance on best methodological practices, and could naturally incorporate reporting checklists.

Funding, reinforcement, and sustainability

As the repertoire of available reporting checklists in neuroimaging expands, there need to be mechanisms in place to ensure their ongoing development can be sustainably funded and their use can be reinforced. Thus far, the development of every neuroimaging methodological checklist that we are aware of has been funded through institutional support and research grants to individual researchers involved in checklist development, or even sometimes unfunded efforts, but this poses issues for sustainable funding of checklist development and enforcement. For instance, as mentioned in the experience of well-established reporting checklists, a minimum of $CAD120,000 is needed only for the costs of consensus meetings [86]. Although the checklists are more akin to public goods rather than the typical outcomes of research projects and justifying their impact to procure funding from traditional academic sources can be challenging. This aspect even affects the decision of researchers to spend their time on developing checklists for global benefits when there is no obvious funding or spend time on their individual research for their own professional advancements. Further, most methodological checklists are developed by teams representing many, sometimes dozens of different institutions spread across countries and need to be continually updated beyond the life cycle of individual research grants. As checklist development communities widen and involve different stakeholders, it can be difficult to decide how and from which sources such long-term funding can be procured. These challenges are not unique to checklist development, and have for example been extensively in the case of funding for other types of digital [87] and physical [88] research infrastructure, and by the research software engineering community [89]. Though there are no reviews of the landscape and gaps in available funding for reporting checklists and guidelines, sustainable funding will likely not come from a single type of funder in the future and as these checklists disseminate beyond purely academic research and are more widely adopted by publishers. One of the funding agencies aiming to fill this gap is the BRAIN Initiative’s Standards to Define Experiments Related to the BRAIN Initiative grant program from the National Institutes of Health (NIH) [90]. This program supports the development of data standards which have to be adhered to in new experiments used by Brain Initiative such as understanding circuit functions in the nervous system, invasive devices for recording and modulation human nervous system, non-invasive neuromodulation, and next-generation imaging.

Conclusions

The development of neuroimaging reporting checklists is a crucial step towards increasing transparency, reproducibility, and reliability, core elements of scientific rigor. This cannot be achieved without a systematic checklist development process that itself follows core values of transparency, reliability, diversity, and openness to innovations. Available neuroimaging checklists highlight the practical utility and potential impact of neuroimaging checklists in improving reporting practices.

Neuroimaging checklists can set standards for reporting practices and data sharing, considering the growing number of data repositories and consortia. Emerging technologies such as large language models can play a role in the near future in checking the adherence of scientific manuscripts to checklist recommendations and identify reporting aspects that could be improved. This will enhance the practicality and efficiency of implementing neuroimaging reporting checklists. Global collaboration in neuroimaging checklist development and involving a diversity of researchers and stakeholders, harmonizing the aims and scopes of various checklists, promoting the use of neuroimaging checklists by stakeholders, and adopting them in educational settings can all foster a culture of scientific integrity. These steps cannot be taken without funding agencies to ensure the development and continuity of these initiatives.

Taken together, neuroimaging reporting checklists can improve the quality and impact of research if they follow the core values and development process discussed in this article which will in turn pave the way for a more transparent and rigorous scientific future.

Author contributions

All authors contributed to the conception of the manuscript first through two panels held at the American College of Neuropsychopharmacology Annual Meeting (ACNP2023) and then through online rounds of discussions. HE, MZB, AS, AV, DMC, HG, TEN, CRP, PMT, and MPP contributed to drafting the first version of the manuscript. HE supervised the process of receiving and implementing the comments. All authors contributed to the manuscript revision, read, and approved the submitted version.

Funding

HE is supported by funds from Laureate Institute for Brain Research and Medical Discovery Team on Addiction and Brain and Behavior Foundation (NARSAD Young Investigator Award 27305). JK is supported by the Department of Veterans Affairs (National Center for PTSD), NIAAA (Center for the Translational Neuroscience of Alcohol (2P50AA012870-23), and National Center for Translational Science Clinical and Translational Science Award (2UL1TR001863-06). He has stock, options, or has received consultation fees from the following companies Aptinyx, Biogen, Biohaven Pharmaceuticals, Bionomics, Boehringer Ingelheim, Cartego Therapeutics, Damona Pharmaceuticals, Epiodyne, Epivario, Freedom Biosciences, Janssen Research and Development, Jazz Pharmaceuticals, Neumora Therapeutics, Otsuka America, Response Therapeutics, Rest Therapeutics, Spring Care, Sumitomi America, Terran Biosciencews, Tetricus, inc. He is an inventor on patents licensed by Yale University to Biohaven Pharmaceuticals, Freedom Biosciences, Janssen Pharmaceuticals, and Novartis Pharmaceuticals. DO has received R01MH114982 funding and also an honorarium from Boehringer-Ingelheim in the past 12 months. CRP is supported by the Novo Nordisk Fonden NNF20OC0063277. MPP is partly supported by The William K. Warren Foundation, the National Institute of General Medical Sciences Center (Grant 2 P20 GM121312), and the National Institute on Drug Abuse (U01DA050989). He advises Spring Care, Inc., receives royalties from an article on methamphetamine in UpToDate, and has a compensated consulting agreement with Boehringer Ingelheim International GmbH. Other authors declare no conflicts of interest.

Competing interests

TPG is the co-editor of the NeuroPschychoPharmacology. JK also serves as the editor of Biological Psychiatry. Other authors declare no conflicts of interest.