Align with the NMIND consortium for better neuroimaging

Robust and validated tools are essential for reproducible science. Yet, in brain imaging, well-implemented and validated tools are not always available and accessible. Many investigators find it necessary to build their own software, but such software rarely follows best practices of software engineering and concerns about the resources that are required to support it are often a limiting factor in sharing. Here we propose the NMIND consortium as a means to bolster reproducible research through evaluation against open, community-developed, scientific software-development standards. NMIND is a grassroots community that is working towards simplifying standardization and collaboration for scientific software. We invite the field of neuroimaging to unite behind high-quality and transparently benchmarked standards and tools.

We believe that NMIND is necessary and timely for the neuroscience community. The scale and complexity of neuroimaging has increased over the past decade¹, but lagging software infrastructure contributes to a crisis in reproducibility². With the onset of mandates such as the National Institutes of Health (NIH) data sharing and management policy in the USA, there is a clear and present need for well-engineered tools and resources to go alongside well-described data and metadata. This will ensure the FAIR (findability, accessibility, interoperability and reusability)³ dissemination of research.

A critical first step in this effort will be the alignment of the neuroimaging community — through participation in hackathons and regular meetings — behind common standards for software, infrastructure, benchmarking and documentation. With these standards in place, the community will develop resources to facilitate the accessible testing of tools and datasets that will enable community members to obtain tangible feedback on their contributions. Finally, by prioritizing ongoing engagement through the research community, NMIND will promote confidence in tools, minimize redundancy and ensure that the NMIND standards remain in alignment with the evolving needs of the field.

Over the course of a series of public virtual hackathons, a number of NMIND subgroups have emerged to assess the state of the field and work towards collaborative solutions. The initial focus of the NMIND community has been on the development of software infrastructure, documentation and sharing standards that are both language- and scope-agnostic, as well as a series of checklists to aid in their application. In addition to enabling both first- and third-party software evaluation, these efforts have led to an initial prototype of a tool marketplace in which researchers can explore and compare libraries (Fig. 1).

Fig. 1 |. Example portal.

A portal for browsing tools and comparing their performance on the NMIND checklists. Image produced by G.K.; ‘hand’ and ‘shield’ icons ^©Twemoji, via Canva.com.

We have identified four objectives towards an equitable and sustainable future for scientific software and drafted an initial plan of action. We hope that this piece may serve as a call to action for community members who are interested in contributing to these solutions either through participation in our regular hackathons or by directly contributing to our projects on GitHub.

Our first objective is to foster interoperability. We will promote the establishment of shared community values through software guidelines, which will in turn facilitate collaboration on reliable and easy-to-use tools. Through the creation of core standards and tools, other fields (for example, physics⁴) have been able to produce consistent software and reduce costs associated with development, maintenance and sharing. We hope to realize these same benefits by establishing common standards for code quality, testing, documentation and dissemination.

Our second objective is to provide access to resources. We at NMIND will develop transparent and community-driven toolkits for evaluating the quality of neuroimaging software packages. We hope that this will build trust and transparency in commonly used tools. We will share common benchmark-testing datasets openly and we will curate a community-facing tool marketplace. Using this marketplace, researchers who develop or consume software will be able to make informed decisions against a standardized evaluation. Our key focus at NMIND will be the adoption of common terminology standards, to minimize lost-in-translation errors across similar tools. Readers can find examples of this in our checklists for using the current standards

Our third objective is to simplify engagement. Given the spirit of NMIND, it is essential that we maintain a tight loop with the community and accurately capture common values. We will adopt mechanisms for encouraging use by promoting participating scientists to increase the attractiveness of participation. We will foster engagement through events and hackathons, with an emphasis on training junior researchers in the best practices of software development. We will continually evaluate the diversity, equity and inclusivity of the community^5,6, and place a focus on elevating members of underrepresented groups. We will reward participation in the community through objective and merit-based letters of support from senior scientists or sponsorships.

Finally, our fourth objective is to grow sustainably. Following the success of projects such as the Brain Imaging Data Structure⁷ and European Open Science Cloud consortia⁸, we envision a grassroots governance model to achieve long-term sustainability. Ideas and needs for standards or procedural changes will disseminate through the community naturally and subsequently define NMIND’s direction. This allows for flexibility and awareness of changing trends in scientific and technological needs. NMIND will coordinate with large-scale data-generation and informatics initiatives such as the International Neuroinformatics Coordinating Facility (INCF), ReproNim, and the NIH Adolescent Brain Cognitive Development (ABCD) and Healthy Brain and Childhood Development (HBCD) studies, to further expand their relevance.

The NMIND vision is undeniably ambitious, but not unrealistic. We believe that the initial success of NMIND requires community buy-in, a key demonstration of value and the formalization of a governance structure. Achieving the larger NMIND vision will require a major investment, however — beyond what traditional funding agencies and mechanisms have been able to provide to date to support brain imaging data collection, analysis and dissemination. As industry-based stakeholders increasingly turn to multimodal imaging measures for deployment in clinical trials and real-world applications, it is an opportune time to also consider public–private partnerships.

In the past decade of brain imaging, we have witnessed landmark advances in data collection, processing and analysis. The greatest advance has arguably been the emergence of an open science culture, with open data and tools serving as incubators and accelerators for collaboration. Looking forward, we believe that the collaborative model that NMIND offers will be essential for the field to take the next major step towards being a science capable of delivering critically needed theoretical advancement and clinical deliverables. In response to this call, we hope that the community will embark on the path towards community-wide software standardization and evaluation with NMIND. We invite the community to join us in turning the NMIND vision into a reality.