A Guide to Developing Harmonized Research Workflows in a Team Science Context

Abstract

Large, interdisciplinary team science initiatives are increasingly leveraged to uncover novel insights into complex scientific problems. Such projects typically aim to produce large, harmonized datasets that can be analyzed to yield breakthrough discoveries using cutting-edge scientific methods. Successfully harmonizing and integrating datasets generated by different technologies and research groups is a considerable task, which requires an extensive supportive framework that is built by all members involved. Such a data harmonization framework includes a shared language to communicate across teams and disciplines, harmonized methods and protocols, (meta)data standards and common data elements, and the appropriate infrastructure to support the framework’s development and integration. In addition, a supportive data harmonization framework also entails adopting processes to decide on which elements to harmonize and to help individual team members implement agreed-upon data workflows in their own laboratories/centers. Building an effective data harmonization framework requires buy-in, team building, and significant effort from all members involved. While the nature and individual elements of these frameworks are project-specific, some common challenges typically arise that are independent of the research questions, scientific techniques, or model systems involved. In this perspective, we build on our collective experiences as part of the REstoring JOINt health and function to reduce pain (RE-JOIN) Consortium to provide guidance for developing research-centered data collection and analysis pipelines that enable downstream integrated analyses within and across diverse teams.

Introduction

Scientists have made impressive gains in our collective ability to interrogate biological systems and generate data of increasing size and complexity. However, each individual dataset only represents a small fraction of the complexity that is inherent to the biological system from which it was collected. In recognition of the narrow limits of any given study, biomedical funding has increasingly supported large, interdisciplinary team science projects to gain insights into complex biological problems (Hall et al., 2019, 2018; Wuchty et al., 2007). Team science projects, such as the Human Genome Project, the Human Biomolecular Atlas Project (HubMAP), or the Helping to End Addition Long-term (HEAL) Initiative, provide substrates for researchers to come together and generate datasets of a size and complexity beyond the abilities of any individual laboratory (Baker et al., 2021; Elam et al., 2021; Jain et al., 2023). Team science also represents the movement of biomedicine towards open science, as the data and methods produced by such efforts is designed to be shared with the wider scientific community (Baumgartner et al., 2023; Koch and Jones, 2016). The Human Genome Project, an early example of team science and open data sharing, highlights the immense scientific potential of data sharing to accelerate scientific discovery by fostering innovation, collaboration, and knowledge generation (Hood and Rowen, 2013).

Open team science often represents a new way of working for researchers, working in larger teams across scientific disciplines to generate outputs that now also include open access datasets and methods in addition to publications. The Science of Team Science (SciTS) field has developed best practices for establishing the framework, processes, and skills needed to build and support efficient transdisciplinary teams (Baumgartner et al., 2023; Hall et al., 2012; Lotrecchiano et al., 2023; Vogel et al., 2021). Similarly, data and knowledge engineers have developed standards and methodologies for sharing and harmonizing datasets (Diehl et al., 2016; Kush et al., 2020; Mungall et al., 2012; Zeb et al., 2021). However, effective data and method sharing require significant time investments, and remain largely unrecognized and unrewarded by scientific institutions (Bezuidenhout, 2019; Hughes et al., 2023). These challenges highlight the need for comprehensive support and training in data management and sharing practices, which will require a culture shift in current scientific practices (Martone and Nakamura, 2022). For example, individual labs will need to implement data management strategies and data collection standards that are designed with data sharing and integration in mind (Dempsey et al., 2022; Donaldson and Koepke, 2022). Additionally, for academic scientists to pursue and contribute to team science initiatives, the entire academic enterprise must shift so that effective data sharing strategies are valued on par with publications and other standards that are commonly used for tenure and promotion decisions (Devriendt et al., 2020; Puebla et al., 2024). The need for this culture shift towards open data sharing in academic biomedical research is made more urgent by the artificial intelligence (AI) revolution already underway. AI offers unprecedented opportunities for discovery science in both basic and clinical neuroscience, but requires open, high-quality data (e.g., well-annotated and structured) on which to operate. Hence, developing research workflows that will generate harmonized data sets that enable downstream integration and interoperability requires multiple elements to be in place, including agreements on how to collect, structure, and share the data itself, as well as the methods, code, and other information needed to understand precisely how the data was generated. Developing such data harmonization frameworks is a considerable task that requires a substantial amount of effort from all members involved.

In this perspective paper, we present guidance for interdisciplinary research teams who are embarking on their data harmonization journey and provide recommendations for establishing a data harmonization framework that enables interoperable data generation. To do this, we build on the challenges faced and the approaches taken to build such a framework within the REstoring JOINt health and function to reduce pain (RE-JOIN) Consortium, a team science initiative aimed at better understanding the cellular and biological underpinnings of chronic joint pain. The RE-JOIN consortium is supported as part of the National Institutes of Health Help End Addiction Long Term (HEAL) program, an NIH-wide effort to accelerate the development of meaningful scientific solutions to the US opioid crisis (Baker et al., 2021). Some RE-JOIN members have worked across multiple large team science projects and have learned that the problems faced by different consortia, centers, and team science projects are often very similar, even if their scientific goals are vastly different. Here, we share our experiences during the initial phase of the RE-JOIN Consortium in establishing effective communication and harmonization processes to ensure, to the degree possible, that data produced throughout the project’s lifespan are findable, accessible, interoperable, and reusable (FAIR) (Wilkinson et al., 2016) and fulfill our obligations to those impacted by pain and the opioid crisis.

Data Harmonization

Team science and open science projects often share a common goal: to generate and curate data sets that can be integrated to enable the broader scientific community to address complex scientific questions. To do so, team scientists must harmonize as many elements of the data collection and processing pipeline as possible, including the types, levels, and sources of data in formats that are compatible and comparable so that they can be integrated (Suppl File 1)(Zeb et al., 2021). Researchers and data scientists can rely on several techniques to harmonize scientific data sets, which have recently been described in detail by Cheng et al (Cheng et al., 2024). Briefly, a data set can be harmonized across several different dimensions related to syntax, data structure, or semantics (Suppl File 1). Establishing a common framework for collecting the data across sites using these strategies ensures that points of commonality between datasets are clear to both humans and machines so that different types of data can be meaningfully combined.

In 2016, the scientific community developed the FAIR Data Principles as a set of guidelines for improving the integration capability of openly shared datasets by making them FAIR (Wilkinson et al., 2016). As a general concept, a FAIR data set should include all the information necessary to understand and interact with the data. This means that, apart from the experimental data itself, FAIR data sets should be accompanied by a detailed description of how the data was generated (research method, code, and scripts), as well as information about the study design, experimental conditions, and sample processing (metadata). In addition, FAIR also prescribes the use of a set of practices that ensures machine-readability, such as the use of unique identifiers and structured metadata. Hence, achieving FAIR data requires a partnership between experimentalists, data scientists, and data repositories. To promote adherence to FAIR principles, data repositories typically require a data set to comply with specific minimal (meta)data standards prior to publication. For example, the SPARC portal – an open neuroscience and systems physiology platform that contains multi-species experimental, computational modeling, and spatial mapping data sets – requires submitted datasets to adhere to the SPARC dataset structure, which includes a metadata standard (Bandrowski et al., 2021). Because the information that is required to understand, re-use, or integrate data sets depends on the type of data and the technology used to generate it, several metadata standards and data structures now exist to generate distinct types of FAIR biomedical data sets. For example, several team science projects have developed a minimal metadata standard that describes the metadata elements that will be collected across its teams such as the 3D Microscopy Metadata Standards (3D-MMS); a series of 91 fields aimed at standardizing metadata for three-dimensional microscopy datasets acquired as part of the Brain Research through Advancing Innovative Neurotechnologies (BRAIN) Initiative (Ropelewski et al., 2022). A more ambitious attempt to develop a universal MMS based on existing metadata guidelines resulted in the creation of a minimal metadata set (MNMS) for general in vivo animal research that aims to facilitate the repurposing and integration of data obtained from various animal research models to reduce the number of animals used for preclinical research (Moresis et al., 2024).

Additionally, the FAIR principles require that connections between datasets be understandable to humans but also that they are structured in a way that they can be combined computationally with minimum human intervention. This can be achieved in part through the above-mentioned standards but also requires developing and adopting formal semantics and common coordinate frameworks (CCFs) (Suppl File 1). Because the skills for achieving both human and machine readability rarely reside in the same individuals, generating machine-readable data sets typically requires collaboration between researchers and data scientists. Researchers are necessarily focused on accurate metadata capture and annotation of key entities such as anatomical structures, cell types, and molecules. In addition, to enable seeing their data in the bigger picture of data collected by other groups, researchers need to acquire enough supporting data so that high-resolution data are placed in an appropriate context. For example, in the case of RE-JOIN, this realization resulted in the recommendation to obtain low magnification images of the tissue from which a biospecimen was collected, including a label or demarcation to show the precise location from which the sample was collected, which gives data users this context. Those analyzing the data can then utilize both data image assets to identify the same or similar entities in other datasets.

Another practical method for data harmonization across research projects and teams is to develop and implement the collection of common data elements (CDEs). CDEs represent pre-determined data elements (e.g. patient-reported outcomes (PROs), questionnaires, and other measurements) that have a set of controlled values and that are collected across research projects (Kush et al., 2020; Wandner et al., 2022). Because CDEs are collected by all teams and projects of the overarching initiative in a controlled fashion, they enable seamless data integration across the initiative’s projects and teams. For example, the NIH HEAL Initiative uses a set of clinical CDEs that are collected by all HEAL-funded, clinical research projects (Wandner et al., 2022).

In sum, effective data harmonization frameworks contain multiple elements that should ideally be defined during the early establishment phases of a team science project to maximize the (re)usability and integrability of generated data sets (Cheng et al., 2024; Hall et al., 2019; McDavid et al., 2020; Seep et al., 2024). CDEs, minimal data standards, data structures, and CCFs will ensure the generation of FAIR, machine-processable data sets. However, because establishing and adopting such standards and structures is a considerable task, effective data harmonization frameworks can include a data coordination group that consists of data and knowledge engineers, curators, and infrastructure specialists to support researchers deliver FAIR data. In addition, such frameworks can include supporting tools to minimize the administrative burden for individual researchers and to increase compliance, such as (meta)data entry templates that contain the agreed-upon (meta)data elements that should be collected by all teams involved. Finally, the data harmonization framework should include a common vocabulary to develop the formal semantics that will ensure everyone involved uses the same term to describe the same element, procedure, tissue, etc. This common vocabulary should be developed early in the research process, preferably building upon existing community vocabulary standards, to create a cohesive and efficient data- and research-sharing model that supports the generation and maintenance of large datasets that can be shared with and reused by the broader scientific community (Gilligan, 2021).

Harmonizing Heterogeneous Datasets – a Chronic Joint Pain Example

The HEAL Initiative is an NIH-wide effort to accelerate meaningful scientific solutions to the US opioid crisis (Baker et al., 2021). It currently involves over 1800 funded projects, including 40+ consortia and centers. One of HEAL’s goals is to build comprehensive datasets that can be used to investigate factors that predict transition or resilience to chronic pain (“NIH HEAL Initiative Research Plan,” 2024).

The RE-JOIN Consortium, which is part of the NIH HEAL Initiative, aims to better understand the neurobiological underpinnings of chronic joint pain by studying the sensory nerve networks within the temporomandibular (TMJ) and knee joint (Fig 1). RE-JOIN will develop sensory innervation maps of these two joints and will identify disease-, age-, fitness-, or therapy-induced innervation changes that may mediate or protect against joint pain. RE-JOIN’s overarching goal is to combine findings from both human and animal models of joint disease resulting in translational insights for developing improved therapeutic strategies for these types of joint pain (Fig 1). To achieve these goals, the consortium contains multiple teams to combine diverse skill sets and knowledge, including research, data, and team science experts (Cruz-Almeida et al., 2024). In addition, RE-JOIN’s data coordination group (DCG) leverages pre-existing infrastructure and team members from SPARC (sparc.science) to establish the foundation of its data integration framework (Goldblum et al., 2024; Osanlouy et al., 2021).

Fig. 1.

Overview of the RE-JOIN consortium’s sites (A) and the cross-disciplinary approaches (B) it applies to patients and animal models with joint disease to better understand chronic joint pain.

Standardizing data collection and analysis pipelines from both human and animal studies posed an initial integration challenge for RE-JOIN members. In addition, RE-JOIN datasets are generated using a multitude of technologies, including neuronal tracing, tissue clearing and 3D imaging, multi-omics, and others (Fig 2). Finally, each RE-JOIN dataset must adhere to existing data standards and requirements as established by HEAL and SPARC (see Suppl File 2) prior to their publication on SPARC. To further complicate RE-JOIN’s harmonization efforts, individual research teams had already started at least some data collection prior to the consortium’s onset. While it is common for large interdisciplinary collaborations to build upon previously collected data sets, this does increase the urgency for the team to develop and implement its data harmonization framework across sites to reach its full potential (Muenzen et al., 2022).

Fig. 2.

Diagram displaying the RE-JOIN consortium’s data generation and integration pipeline for various tissues and pain behaviors, collected from patients and animal models of joint disease.

The first harmonization element that RE-JOIN investigators defined was a set of CDEs. Beginning with the HEAL required CDEs (https://heal.nih.gov/data/common-data-elements), the RE-JOIN CDEs were developed through a rigorous, multi-phase process involving systematic literature review and expert consensus (Cruz-Almeida et al., 2024). An international, multidisciplinary working group of clinicians, researchers, and methodologists came together to develop RE-JOIN’s CDEs to (1) support consistent chronic joint pain phenotyping across its clinical teams and to (2) enable downstream cross-project analysis of data elements related to pain and substance abuse within the HEAL initiative (Wandner et al., 2022). To achieve this, the working group reviewed existing outcome measures and CDE initiatives to identify potential CDEs relevant for characterizing chronic joint pain. Selection criteria included clinical relevance, established psychometric properties, implementation feasibility, and cross-cultural adaptability, while considering patient burden. The working group employed an iterative process including initial item pool generation, multiple consensus-building rounds, expert review meetings, and feedback incorporation. Elements were evaluated based on their utility across various joint conditions and research settings, ensuring broad applicability while maintaining scientific rigor.

Another essential building block of RE-JOIN’s data harmonization framework is its minimal metadata standards (MMS), an extensive list of all required, recommended, and optional parameters that would be included as part of a RE-JOIN dataset. This MMS helps researchers within and outside of the consortium understand how the data was collected and facilitates dataset integration. To develop RE-JOIN’s MMS, the consortium established working groups centered around each of its technologies (e.g., animal models, imaging and tracing, omics). During the consortium’s development phase, working groups met regularly to first compare individual scientific methods and data analysis parameters and subsequently outline strategies to best integrate and standardize them across projects. Here, RE-JOIN’s DCG helped translate, understand, and comply with HEAL and SPARC’s data requirements, fostering the development of a common language among researchers regarding best data practices. Based on these discussions, each working group defined a set of data and metadata elements deemed essential for enabling downstream integrated analyses. These elements were subsequently combined into the first version of RE-JOIN’s metadata standard, leaving the option open for the standard to evolve along with RE-JOIN’s technologies and research goals (Ziogas and Martone, 2025). Upon dataset submission to SPARC, the DCG will curate it to ensure that all data standards are met (Supp File 2), and that data sharing is seamless and effective. To reduce barriers for individual researchers to record all required metadata elements, the DCG developed a template data entry sheet (Suppl File 3). RE-JOIN’s data entry template, along with the metadata standard, data dictionary, and protocol guidelines are publicly accessible and can easily be adapted for use by groups and communities with distinct metadata standards (Ziogas and Martone, 2025).

In parallel, working groups began defining the semantic and spatial standards required to integrate datasets across labs, techniques and spatial scales (Suppl File 1). To do this, the group decided to develop a spatial common coordinate framework (CCF) that will support machine-readability of generated imaging or genomic datasets and will enable visualization and query efforts of collected data sets with respect to the anatomical location from which they were collected (Suppl File 1). The aspirational CCF will take the form of an MRI-based atlas and a set of registration tools, although creating such a framework for moveable joints will present a challenge. While RE-JOIN’s CCF is under development, it is important that researchers record the spatial location of data collection as accurately as possible so that this data can eventually be aligned to the CCF. Therefore, the consortium agreed to take the following steps: 1) As noted above, researchers acquiring imaging data will acquire a low magnification view of any tissues imaged at high resolution so that at least 2–3 agreed upon landmarks are visible that can be used for subsequent registration of data to the CCF; 2) A labeled anatomical drawing or drawings of the knee or TMJ was requested from each group so that labels and definitions of regional anatomy can be compared; 3) A set of potential landmarks will be agreed upon and defined clearly so they can be applied consistently across studies; 4) A set of 2D spatial templates are being prepared to allow researchers to indicate from where a tissue was obtained, e.g., human surgical tissue. These images will be stored along with the dataset so that they can be used for subsequent alignment.

With these processes, standards, and structures in place, RE-JOIN investigators will be well-equipped to collect and curate datasets to meet the consortium’s goals and support future collaborative efforts.

How to make it work: suggestions for building data harmonization infrastructure based on the RE-JOIN experience

1. Start Early

Most large team science projects share a common goal: to generate large datasets that can be integrated and analyzed to reveal novel insights into an intricate research question (Council, 2015). However, the methods, samples, and experimental workflows needed to create the data subsets that will later be integrated are typically developed in individual laboratories long before any meetings between researchers and a DCG occur. Adjusting established experimental and data-collection workflows to incorporate and/or adopt novel, often unfamiliar data standards is labor- and time-intensive, and may generate push-back from team members. However, this is a fundamental element of the supportive data harmonization framework that is needed to accomplish the team’s goals. Allocating time to focus on data integration pipelines early in the project can help avoid a frustrating and tedious end, where data or metadata needs to be dredged up from lab notebooks to be reformatted multiple times (Fig 3). Importantly, establishing data sharing procedures early in the research process will likely prevent experiments from being excluded from integrated analyses due to lacking critical information.

Fig. 3.

Tips and tricks for building a data integration framework.

To promote harmonization and prevent data loss, some recent research funding announcements (RFAs) for team science initiatives now include a start-up phase to define the processes and harmonization plans prior to the onset of data collection. In RE-JOIN’s case, some data collection had already started at the consortium’ s onset. We therefore established working groups to compare research workflows, develop a shared language, and define elements to harmonize.

To comply with metadata standards, metadata collection must occur throughout the study process and will likely be done by more than one individual involved in the study using a multitude of platforms, including handwritten notes, spreadsheets, lab information management systems (LIMS), and others (McDavid et al., 2020). Given that each person on a multidisciplinary team comes with their own approach to data management, it is important to address early how data will be collected and annotated in a way that promotes interoperability and integration to support FAIR Data Principles. Building upon previously published data sharing guidelines (Champieux et al., 2023; McDavid et al., 2020), we provide the following questions to serve as guide for developing data collection and annotation schemas that promote FAIR data standards:

• How will various types of biological samples and data be collected across research teams? Will they be shared across laboratories? Are data collection protocols standardized or unique to each study team?

• How and where will biological samples and data be catalogued and stored?

• How will the anatomical location and identity of collected biological samples be annotated and stored?

• Which experimental parameters are important to enable downstream integrated analyses?

• If subjects and samples are shared across groups, how will they be tracked (i.e., how will unique identifiers be issued and tracked?)

2. Begin with an easy target

When initiating the data harmonization process, an early achievable milestone can include deciding on which experimental processes to standardize. Here, the team must decide together which technologies or elements benefit from harmonization, and which ones support independent, orthogonal validation of scientific findings. For example, RE-JOIN investigators agreed on harmonizing tissue collection and processing workflows for each joint type (i.e., knee and jaw), but will continue to use distinct types of animal models (e.g., surgical, genetic, and spontaneous) within each joint type to identify common versus specific contributors to chronic joint pain. Building consensus around which experimental processes and methods to harmonize can help pave the way for more challenging conversations later on, such as defining the team’s minimal metadata standard. While it may be challenging to build trust among diverse team of scientists who are used to functioning independently, it is a critical step for the team’s success (Bennett and Gadlin, 2012). Low-risk targets, such as the harmonization of experimental processes and methods, moves teams through a “norming” phase that will enhance future communication surrounding more challenging topics (Bennett and Gadlin, 2012).

3. Share early and often

Once the team has decided which processes (experimental paradigms) to harmonize, it can start to dive into the technical details: how does each of its research groups perform its experiments? Seemingly small differences, such as incubation times or viral titers, may generate vastly different results that create additional integration challenges. In RE-JOIN, we created a private workspace on the online method-sharing platform protocols.io to share detailed protocols and reagents that were being used in experiments (Teytelman et al., 2016). Protocols.io is an open-access virtual community that promotes communication across research groups by providing standardized systems that can be used to share, compare, and edit scientific protocols. Working groups first used protocols.io to compare technical differences and efficiencies across teams, which led to greater communication between scientists. By using protocols.io, RE-JOIN investigators were able to adopt harmonized research methods, where possible, to increase the overall research impact. In other cases, the working groups added additional parameters to RE-JOIN’s metadata standard that would capture experimental differences important for downstream integration.

Another “share early and often” strategy that will help outline possible harmonization amongst the research groups is the use of pilot datasets (Cheng et al., 2024). In many cases, research groups have previously conducted experiments similar to what they proposed as part of the consortium. The ability to see the metadata associated with those pre-existing datasets or of a subset of a newly generated dataset can give the group an idea of what metadata is being captured and which parameters may be missing. One way RE-JOIN did this was to provide the DCG with sample images taken from each of the different microscopes being used across research sites. These sample images did not constitute a complete dataset, but instead consisted of a subset of the raw files generated by the microscope. Analyzing the metadata associated with these images helped the DCG determine if each microscope’s proprietary software was automatically capturing all required parameters, or if the research team could increase automated capture by changing some of its microscope’s settings. Similar early spot checks can help ensure that data is being captured in the best possible way early in the project and can save valuable time and resources at the end of the project.

4. Define minimal metadata standards early

Metadata are one of the most critical elements in determining whether data can be successfully integrated and reused. Minimal metadata standards (MMS) describe the experimental parameters and information that should be collected on individual data entries to provide the necessary context to understand how the data point was collected. For some projects, existing metadata standards can be adopted. However, because team science projects often develop novel technologies and/or experimental frameworks, suitable metadata standards may not exist. In this case, the consortium must develop its own minimal metadata standard listing the necessary parameters that will be collected for each of its data types, technology, and species (Abrams et al., 2022; Hill, 2016).

Establishing a tailored MMS will likely be the most time-consuming and challenging aspect of establishing the consortium’s data harmonization framework (Cheng et al., 2024). At the onset, this task may seem nearly insurmountable, particularly because it may seem impossible to predict what parameters may be (or become) important as integrative data analysis technologies evolve. Through RE-JOIN’s experience, we learned that the best way to initiate MMS development is to acknowledge that:

• A consortium’s MMS can be adjusted (versioned) at later timepoints if needed. It is important to keep in mind that dataset requirements evolve continuously. For example, the HEAL CDE’s were established in 2022, but the individual elements that make up these CDEs, how they are named, and how they are coded continues to evolve (Adams et al., 2023; Cruz-Almeida et al., 2024; Wandner et al., 2022). Similarly, a consortium’s MMS can change when needed, depending on how the group’s technologies, data collection processes, and needs evolve. For RE-JOIN’s MMS, the DCG emphasized the need for and possibility of versioning data standards from the start of the consortium’s data harmonization conversation. This led to the successful development and adoption of RE-JOIN’s minimal metadata standard (v1) about 18 months after the consortium’s establishment (Ziogas and Martone, 2025).

• The first version of the MMS should be established as soon as possible to maximize data (re)usability. Adopting this approach allows for researchers to take a step back from their work at the bench/clinic, early in the consortium’s lifespan, to think through all parameters that must be recorded as part of their research program, and to develop and implement a process for collecting them. More importantly, it allocates time for the team to consider what additional metadata could be recorded that may be needed to align datasets obtained by distinct groups.

• It is unlikely that the consortium’s MMS will be established and adopted before experimental pipelines have been developed. In the first few months after its establishment, RE-JOIN investigators received a lot of pressure from different angles to establish its MMS. This caused frustration and delayed consensus-building at first, because individual research groups were simultaneously developing novel scientific technologies and did not have a clear idea yet of what datasets they would generate. Breaking down the MMS into separate sections that are specific for the technologies that would most likely be used to generate consortium-related datasets helped overcome this challenge. In the end, not all data collected as part of the consortium may completely conform with its adopted data standard. However, it is important to realize that individual research groups will most likely continue the line of research, generating future harmonized datasets that may far exceed the consortium’s lifespan.

RE-JOIN employed a successful MMS-development strategy by dividing the process among each of its working groups, which are centered around the various technologies and models that are used to generate data. Each working group first listed the variables and metadata they considered critical for understanding how the experiment was executed and how data was collected. Once all technology-specific parameters were listed, working group members indicated for each proposed field whether it was 1) Already collected routinely, 2) Important to collect, and 3) Feasible to collect. Feasibility is an important aspect of any data standard, because if it is not within reach of all participating labs, then making such metadata required might impose a significant burden on certain investigators. This way, a consensus document was produced by each working group, which were subsequently reconciled into a single consortium-wide metadata standard by the DCG. This reconciliation process also included mapping to existing SPARC and HEAL standards where relevant. This process established rapport and trust among investigators and the DCG, leading to greater productivity overall.

5. Implementation: be practical and realistic

With an MMS in place, the next step consists of implementing the standards in each individual research setting. This process involves developing workflows and resources that integrate metadata acquisition with data collection as much as possible. Collecting this information throughout the project reduces the need for extensive data wrangling at the time of submission and minimizes the risk of not collecting the required information about how data was acquired. Throughout this entire process, frequent communication between each team’s data liaisons and the DCG will facilitate the proper adoption and implementation of the agreed-upon data standard. Indeed, active collaboration across research teams and the DCG helps guard against potential pitfalls and ensure quality control (McDavid et al., 2020).

RE-JOIN employed three practical strategies for reducing researchers’ barriers to comply with its metadata requirements:

The first strategy was to automate metadata extraction as much as possible. For example, the imaging working group was able to configure their microscopes in a way where 20+ metadata parameters could be automatically extracted from their imaging data with a program used by the SPARC group (Osanlouy et al., 2021). In our experience and that of others, any lowering of the barriers implicit to (meta)data entry facilitates uniform and seamless adoption of any standard (Huang et al., 2023).

The second strategy we employed consisted of limiting the amount of required structured metadata. Structured metadata refers to data that is designed for databases and tables and includes discrete data types such as numbers, terms, or dates. Structured data is typically organized in a way that is easy to query and allows for easy comparison across different data records. However, unless metadata extraction and structuring are automated, requiring too many structured fields can be a burden to researchers that may result in a failure to capture the necessary level of detail. RE-JOIN therefore employed structured metadata only for essential metadata that served as a basis for search or selection of a dataset for download. Metadata that is important but won’t necessarily be used as a search term (e.g., fixation parameters) will be recorded as part of the experimental protocol that will accompany each dataset. The DCG’s curators review protocols against these standards to ensure compliance. Allowing metadata to be recorded in free text allows researchers to work in a more natural medium and also captures nuances than more fully structured metadata. In the near future, we expect AI-based agents to fully exploit free text as a means of search and comparison (ARPA-H, 2024; Doan et al., 2014; Liu et al., 2011).

A final way to ensure consistent metadata collection amongst different research groups is to develop template data entry forms that contain all required (meta)data elements (Suppl File 3). For example, REDCap-or Excel-based databases can be generated without the need for coding skills, providing a simple and secure method for capturing metadata in real-time (Harris et al., 2008). The ubiquitous use of REDCap in clinical settings provides many pre-created template databases that can be easily modified and deployed to fit a group’s specific requirements, even in preclinical settings. These can then be output in a standard format.

6. Validate your harmonization approach

Integration of various data types is another important step in attaining a fully cohesive dataset. To achieve this, it is important to look back at point 1 (start early), and attempt to combine small pilot datasets, generated by different research teams, or using different technologies early. This approach can use very small data sets (e.g. only a single sample) to reveal (1) if any important metadata is missing from the MMS, (2) if the data set is structured in a way that can be understood by a scientist who was not involved in its generation and (3) if researchers are adhering to the agreed upon standards, e.g, using the standard as provided without any “tweaks” and providing required information. Indeed, while the MMS establishes what metadata need to be collected, combining datasets in meaningful ways also requires paying attend to how that information is structured and documented. An example of this can be seen in the use of next-generation sequencing techniques (such as bulkRNAseq or scRNAseq) across multiple organisms. In these cases, something as trivial as ensuring that the output from each analysis is formatted in a way that can easily be directed into an Ortholog Detection or Gene Conversion tools that can account for different GeneIDs from different species can become critical. Attempting to integrate smaller versions of these data sets process can help identify and correct potential issues early in the process.

Considerations across the flow of data

Overall, RE-JOIN taught us that establishing effective communication between a consortium’s data coordinators and its researchers requires considerable effort, as expertise, goals, and skill sets vary significantly across and within teams. In this section, we provide perspectives from both the data collection and data sharing sides when working through the process of establishing all elements necessary to develop meaningful FAIR data collection pipelines.

Researcher’s perspective

Large team science projects begin and end with the acquisition or analysis of different types of data that can lead to new insights into complex biological systems (Baumgartner et al., 2023). Each of the individual research groups involved in a large science team has vast experience in executing the methods they routinely use to achieve their primary research goals and in collecting and sharing the data and metadata required for publication. However, many of the methodologies required to achieve the scientific breakthroughs promised by team science projects have yet to be developed and require novel or emerging scientific approaches. How can an individual researcher or research group foresee how to properly structure data and metadata for experimental assays that may not even exist? Researchers have ample experience in developing and troubleshooting new assays (e.g., protocols, standard operating procedures) so they generate robust and reproducible results. In our own laboratories, the details of these optimization steps are usually forgotten as soon as the assay is working. In a team science environment, it is necessary to document and share all steps performed during assay development with other research groups so they can learn from each other’s successes and failures, and so time is not wasted duplicating efforts. In the case of RE-JOIN, researchers exchanged notes on assay development during working group meetings and shared their detailed methods via protocols.io. Involving the DCG in technology development conversations also helps develop an ever-evolving standard for the acquisition of both data and metadata while ensuring that common vocabularies are used as much as possible (Bennett and Gadlin, 2012).

For most newly developed assays, the data and metadata collection process will change throughout the course of the project. The way to best prepare for these changes is to meticulously document all optimization and experimental parameters as best as possible. This, however, increases the “administrative burden” for scientists, which can result in a decrease in productivity. This can be (partly) mitigated by incorporating protocol templates, electronic lab notebooks (e.g., eLabNext), and or personalized internal databases to minimize the amount of time spent documenting additional parameters. Ultimately, these steps will accelerate downstream data sharing, while they will also make the lab’s methods (and results) more reproducible. It is worth noting here that substantial institutional support is often required for the integration of time-saving support resources given their high cost and funding restrictions (Council, 2015)

As individual assays are solidified and as research teams coalesce, a clearer picture of the required data and metadata will emerge. In parallel, the consortium will have put in place formal policies governing internal data sharing to ensure that researchers are comfortable with sharing non-published materials. When individual research groups document as much of their progress as possible and there are ample lines of communication between research groups and the DCG, the standardization of data and metadata can occur organically, resulting in a much smoother path for data uploading and sharing (Bennett and Gadlin, 2012).

Infrastructure perspective

Data sharing cannot occur efficiently without supporting infrastructure (Abrams et al., 2022). Technologies to support FAIR data sharing have changed significantly since the release of the original Data Sharing Mandate in 2003, and continuing advances in scientific technologies constantly challenge the current capacities of our scientific infrastructure (“Final NIH statement on sharing research data,” 2003; Martinez et al., 2021). Five key aspects should be compared when selecting or developing a data-sharing platform: scalability, sustainability, user expectations, security, and interoperability (Bahmani et al., 2021; Szarfman et al., 2022).

Data sharing infrastructure should be able to handle data at scale (Toga and Dinov, 2015). As the size of an average scientific dataset continues to grow, this increases the burden on data-sharing platforms. These platforms must provide sufficient space to store data and offer advanced mechanisms for users to interact with data in the cloud, which is increasingly important considering current pricing models that tax downloading. The challenge here lies in identifying which form of the data may be the most meaningful for future re-use or analysis. For example: should all raw data be shared in its original format, or is there a way to compress files without losing valuable information? The answer likely depends on numerous factors, as it is difficult to anticipate how researchers will use the data in the future in light of technical and scientific advances.

Like other types of software, data-sharing platforms require continuous maintenance to prevent ‘code rot’. As software is comprised of many dependent libraries and operates within an ever-evolving technological landscape, it needs continuous updates to remain operational. Software developed in academic settings has typically struggled with this issue. In addition, academic software is often subject to highly fluctuating usage demands, which complicates sustainability efforts. Therefore, data-sharing infrastructure should be developed with scalability in mind to enable rapid rescaling of infrastructure.

A meaningful, full-featured data platform that enables both internal and public data-sharing is considerably complex. Therefore, one should reuse existing mature infrastructure where possible rather than creating a new platform for each project. Conversely, from a platform perspective, it is important to implement functionality in a generalized way to enable support for multiple, diverse projects. For example: RE-JOIN shares its data through SPARC to both lower its costs and leverage the platform’s capacity for handling multi-modal data (Goldblum et al., 2024; Osanlouy et al., 2021). A major advantage is that the platform is ready to accept data right away and generally has tools available to assist in data structuring and upload. For example, SPARC provides a user-friendly application (SODA), which guides users through the dataset submission process. Using established platforms enables community-driven improvements and enhancements, which promotes the platform’s sustainability.

Finally, it is important to understand the impact of costs associated with maintaining the developed resources and interacting with data on the platform. Operating costs should be considered from the start, as key decisions can be made to minimize operational costs and optimize the infrastructure for long-term sustainability. For example, RE-JOIN made the decision to make datasets public in the cloud but require users to use their own AWS account to download datasets larger than 15GB. This 1) encourages users to analyze these data in the cloud (which incurs no data transfer costs) and 2) protects the Data Coordination Group from unpredictable operating costs associated with access fees. In addition, SPARC infrastructure increasingly depends on serverless infrastructure that only incurs costs when users are actively interacting with the platform. Strategies like these are important to include in any roadmap for a long-term sustainable academic platform.

In summary, infrastructure development and maintenance are integral parts of the multi-pronged approach to develop scalable data-centered research workflows that meet NIH and consortia requirements and maximize the impact of sharing the resulting data to the scientific community.

Data Curation’s perspective

In between the laboratory where data are generated and the infrastructure where data are stored are all the standards, tools, processes, and personnel to transform datasets into FAIR resources that are useful for the broader scientific community. A consortium’s data integration capacity is greatly amplified when it can achieve the following goals: 1) develop and implement community standards, 2) produce a shared data dictionary that documents the meaning of all the variables and metadata it collects, and 3) link data products to shared semantic and spatial frameworks that reveal relationships between datasets collected with distinct technologies or from different species (Suppl. File 1). While DCG’s may have established processes and standards in place, they will likely need to be adapted for each new project. Just as investigators must learn to work together on developing the scientific protocols, so too does the DCG need to learn to work with both the consortium and HEAL on the data and infrastructure side.

Skilled human curators are essential members of a team science project’s data harmonization framework. For RE-JOIN, SPARC’s curators met with investigators at the start of the project to understand their data, including how it was collected and how investigators intend to share it. They also participated in working group meetings to provide advice on developing metadata standards and controlled vocabularies. Throughout this process, we found that identifying a point person or data liaison for each lab and/or for each technology greatly facilitates the implementation process. Establishing early, solid lines of communication between the DCG and these individuals, who will ultimately prepare data for upload to SPARC, helps establish data collection pipelines that conform with agreed-upon data standards, significantly reducing the amount of data wrangling required prior to upload. Taken together, understanding and appreciating the different roles, responsibilities, and contributions of all who work across the data aisle can help smooth out some of the frustrations involved with establishing new data-centered research workflows.

Conclusion

Large team science requires researchers from various disciplines, employing distinct experimental techniques to come together as one functional unit. To enable the generation of datasets that can be combined across teams, technologies, or tissue types, the group should define the elements that will constitute its data harmonization framework as early as possible. Here, we provided various strategies from our own experiences with establishing RE-JOIN’s data harmonization framework and represented prospective views from all parties involved. Several important foundational steps that support successful data harmonization in large team science include: 1) ‘buy in’ from all team members, 2) open and clear communication among all consortium members, 3) early internal sharing of protocols and methods, 4) clear explanations of data and metadata standards, ideally paired with an action plan for implementation, and 5) a practical and realistic mindset that acknowledges the challenges faced by each group during the entire process. Once these points are achieved, many of the strategies we discussed here can be implemented to achieve a fully integrated team working together to collect diverse but FAIR data and metadata that can be used for unique analyses. This framework helps ensure the data we collect can be reused to answer novel questions in ways we never imagined, laying the groundwork for greater scientific discoveries and high-impact research outcomes that improve public health.