The Federated Tumor Segmentation (FeTS) Tool: An open-source solution to further solid tumor research

Abstract

Objective:

De-centralized data analysis becomes an increasingly preferred option in the healthcare domain, as it alleviates the need for sharing primary patient data across collaborating institutions. This highlights the need for consistent harmonized data curation, pre-processing, and identification of regions of interest based on uniform criteria.

Approach:

Towards this end, this manuscript describes the Federated Tumor Segmentation (FeTS) tool, in terms of software architecture and functionality.

Main Results:

The primary aim of the FeTS tool is to facilitate this harmonized processing and the generation of gold standard reference labels for tumor sub-compartments on brain magnetic resonance imaging, and further enable federated training of a tumor sub-compartment delineation model across numerous sites distributed across the globe, without the need to share patient data.

Significance:

Building upon existing open-source tools such as the Insight Toolkit (ITK) and Qt, the FeTS tool is designed to enable training deep learning models targeting tumor delineation in either centralized or federated settings. The target audience of the FeTS tool is primarily the computational researcher interested in developing federated learning models, and interested in joining a global federation towards this effort. The tool is open sourced at https://github.com/FETS-AI/Front-End.

1. Introduction

Accurate delineation of solid tumors is a critical first step towards furthering downstream analyses towards predicting various clinical outcomes (D. A. Gutman et al. 2013; Mazurowski, Desjardins, and Malof 2013; Jain et al. 2014; Bonekamp et al. 2015; Macyszyn et al. 2015; Bakas, Shukla, et al. 2020), monitoring progression patterns (Akbari, Macyszyn, Da, R. L. Wolf, et al. 2014; Akbari, Macyszyn, Da, Bilello, et al. 2016; Akbari, Rathore, et al. 2020), and associating imaging patterns with cancer molecular characteristics (Bakas, Akbari, J. Pisapia, et al. 2017; Bakas, Rathore, et al. 2018; Akbari, Bakas, et al. 2018; Binder et al. 2018; Fathi Kazerooni et al. 2020). Of special note is the problem of automating delineation of glioma sub-compartments, which is a particularly challenging problem, primarily due to their inherent heterogeneity in shape, extent, location, and appearance (Inda, Bonavia, Seoane, et al. 2014). Performing this process manually is tedious, prone to misinterpretation, human error, and observer bias (Deeley et al. 2011), but importantly leads to non-reproducible annotations with intra- and inter-rater variability up to 20% and 28%, respectively (Mazzara et al. 2004; Pati, Ruchika Verma, et al. 2020). The Brain Tumor Segmentation (BraTS) challenge (Menze et al. 2014; Bakas, Akbari, Sotiras, et al. 2017; Bakas, Reyes, et al. 2018; Baid, Ghodasara, et al. 2021) has been instrumental to advance automated delineation of glioma. However, the usability of the algorithms developed as part of the challenge is an open question. To promote wider application of the methods developed for BraTS and facilitate further research, the public BraTS algorithmic repository was created to house them as containers. However, since there is no graphical interface, their results and various fusion approaches (which has been shown to perform better than expert annotators (Menze et al. 2014; Bakas, Reyes, et al. 2018)) are still out of reach for clinical researchers.

The current environment of open-source software tools targeting medical imaging is highly fragmented, with many users looking at general-purpose tools such as MevisLab (Link et al. 2004), Medical Imaging Interaction Toolkit (MITK) (I. Wolf et al. 2005), MedInria (Toussaint, Souplet, and Fillard 2007), 3D-Slicer (Kikinis, Pieper, and Vosburgh 2014), the Cancer Imaging Phenomics Toolkit (Rathore et al. 2017; Davatzikos, Rathore, et al. 2018; Pati, A. Singh, et al. 2019), and ITK-SNAP (Yushkevich et al. 2006), among many others. This issue is further exacerbated in tools that can perform deep learning (DL), are locally deployable, and provide an end-to-end solution that includes harmonized data pre-processing. Example of these DL tools are: NiftyNet (Gibson et al. 2018), DeepNeuro (Beers et al. 2020), ANTsPyNet (Tustison, Cook, et al. 2020), DLTK (Pawlowski et al. 2017), and the Generally Nuanced Deep Learning Framework (GaNDLF) (Pati, Siddhesh P. Thakur, et al. 2021). Importantly, none of these DL tools have a proper graphical interface to allow for manual quality control of pre-processed data nor allow definition of harmonized pre-processing pipelines for an entire population cohort. The ability to define and manage such pipelines is a critical first step in any cohort-based computational analysis, and particularly in complex federated learning studies (M. J. Sheller, Reina, et al. 2018; M. J. Sheller, Edwards, et al. 2020).

Increased amounts and diversity of datasets available for training machine learning (ML) models are essential to produce appropriately generalizable models (Obermeyer and Emanuel 2016). This has so far been addressed by the paradigm of sharing data to a centralized location, spearheaded by use-inspired consortia (Armato III et al. 2004; Thompson et al. 2014; Consortium 2018; Bakas, Ormond, et al. 2020; Davatzikos, Barnholtz-Sloan, et al. 2020; Habes et al. 2021). However, this approach is not scalable due to various legal and technical concerns (M. J. Sheller, Edwards, et al. 2020; Rieke et al. 2020). Federated Learning (FL) offers an alternative to this paradigm, by facilitating training of ML models by sharing ML model information among participating sites (M. J. Sheller, Reina, et al. 2018; M. J. Sheller, Edwards, et al. 2020; Rieke et al. 2020; Roth et al. 2020).

Building upon our existing efforts, in this manuscript we present the Federated Tumor Segmentation (FeTS) tool, which aims at i) enabling harmonized data curation of population cohorts for computational analyses, ii) bringing various pre-trained challenge-winning algorithms and their fusion closer to clinical experts, iii) provide a graphical interactive interface to allow for refinements of the fused outputs, and iv) empower users to perform DL training in either a localized or federated manner, the latter being done without sharing patient data. In the following sections, we will describe the methods used by the FeTS tool, the use cases it has facilitated so far, and our concluding remarks.

2. Methods

Although the FeTS tool has been designed to be agnostic to the use-case, in the rest of the manuscript we have focused on the design and application of the FeTS tool in brain tumors.

2.1. Data

The starting point for FeTS users would be the identification of local multi-parametric magnetic resonance imaging (mpMRI) scans for each subject included in their study. The specific mpMRI scans considered for the case of brain gliomas, follow the convention of the BraTS challenge, and comprise i) native T1-weighted (T1), ii) Gadolinium-enhanced T1 (T1Gd), iii) native T2-weighted (T2), and iv) T2-Fluid-Attenuated-Inversion-Recovery (T2-FLAIR) sequences, characterizing the apparent anatomical tissue structure (Shukla et al. 2017). The FeTS tool deals specifically with adult-type diffuse glioma (Louis et al. 2021). In terms of radiologic appearance, glioblastomas typically comprise of 3 main sub-compartments, i) the “enhancing tumor” (ET), which represents the vascular blood-brain barrier breakdown within the tumor, ii) the “tumor core” (TC), which includes the ET and the necrotic (NCR) part, and represents the surgically relevant part of the tumor, and iii) the “whole tumor” (WT), which is defined by the union of the TC and the peritumoral edematous/infiltrated tissue (ED), and represents the complete tumor extent relevant to radiotherapy.

2.2. Modes of Operation

The functionality of the FeTS Tool can be summarized by 4 modes of operation, to facilitate an end-to-end ML-driven solid tumor study: i) data pre-processing, ii) automated segmentation using multiple pre-trained models, iii) interactive tools for manual segmentation refinements towards generating gold standard labels, and iv) ML model training.

2.2.1. Data Pre-processing for Downstream Analyses

The considered mpMRI scans are typically stored in the Digital Imaging and Communications in Medicine (DICOM) format (Pianykh 2012; Kahn et al. 2007; Mustra, Delac, and Grgic 2008). Prior to start utilizing these scans, the user needs to curate their data according to a specific folder structure, as described in the FeTS tool’s documentation ^‡. Once the curated data conform to the provided definitions, the user can use this folder structure as input to the FeTS tool. The DICOM scans are then converted to the Neuroimaging Informatics Technology Initiative (NIfTI) file format (Cox et al. 2004). This process ensures that the acquired MRI scans can be easily parsed during each computational process defined in the FeTS Tool. The added benefit to this file conversion step is that all patient-identifiable metadata are removed from the DICOM scans’ header (X. Li et al. 2016; White, Blok, and Calhoun 2020). Each of these converted NIfTI scans are then registered to a common anatomical space, namely the SRI24 atlas (Rohlfing, Zahr, et al. 2010) ^§, to ensure a harmonized data shape ([240, 240, 155]) and voxel resolution (1mm³), thereby enabling in-tandem analysis of the mpMRI scans.

Inhomogeneity of the magnetic field is one of the most common issues observed in MRI scans (Song, Zheng, and Y. He 2017). In (Bakas, Akbari, Sotiras, et al. 2017), it has been shown that the use of non-parametric, non-uniform intensity normalization to correct for these magnetic bias fields (Sled, Zijdenbos, and Evans 1998; Tustison, Avants, et al. 2010) obliterates the MRI signal in T2-FLAIR scans, and particularly in the regions defining the abnormal T2-FLAIR signal. In this tool, we have taken advantage of this adverse effect and used the bias field-corrected scans to generate a more optimal rigid registration solution across the mpMRI scans. Specifically, the bias field-corrected scans are registered to the T1Gd scan, which itself is rigidly registered to the SRI24 atlas, resulting in 2 sets of transformation matrices per MRI scan. These matrices are then aggregated to a single matrix defining the transformation of each MRI scan from their original space to the atlas. We then apply this single aggregated matrix to the original NIfTI scans (i.e., those prior to the application of the bias field correction) to maximize the fidelity of the scans that will be used in further computational analyses. Following the co-registration of the different mpMRI scans to the template atlas, automated brain extraction (i.e., removal of all non-brain tissue from the image, including neck, fat, eyeballs, and skull) is performed. This allows further computational analyses while avoiding any potential face reconstruction/recognition (Schwarz et al. 2019). This brain extraction step is done using the Brain Mask Generator (BrainMaGe) ^∥ tool (Siddhesh P Thakur et al. 2019; S. Thakur et al. 2020), which was specifically developed to address brain extraction in the presence of diffuse glioma by considering the brain shape as a prior, resulting in it being agnostic to the specific sequence/modality as input.

2.2.2. Automated Segmentation

This mode of operation provides users the ability to generate automated delineations of the tumor sub-compartments from pre-trained models. Specifically for the glioma use case, three top-performing BraTS methods are incorporated, trained on the training cohort of the BraTS challenge (Menze et al. 2014; Bakas, Akbari, Sotiras, et al. 2017; Bakas, Reyes, et al. 2018; Baid, Ghodasara, et al. 2021). These methods are i) DeepMedic (Kamnitsas et al. 2017), ii) DeepScan (McKinley, Meier, and Wiest 2018), and iii) nnU-Net (Isensee et al. 2021). Further to the generation of segmentations from these three individual pre-trained models, various label fusion strategies are also provided through the FeTS tool, to generate a single decision segmentation label from the three outputs avoiding the systematic errors of each individual pre-trained model (Pati and Bakas 2021). The exact label fusion approaches considered are: i) standard voting (Rohlfing, Russakoff, and Maurer 2004), ii) Simultaneous Truth And Performance Level Estimation (STAPLE) (Warfield, Zou, and Wells 2004; Rohlfing and Maurer Jr 2005), iii) majority voting (Huo et al. 2015), and iv) Selective and Iterative Method for Performance Level Estimation (SIMPLE) (Langerak et al. 2010). Although the fused output segmentation label is expected to provide a reasonable approximation to the actual reference standard label, manual review by expert neuroradiologists is still recommended, as further refinements might be required prior to final approval.

2.2.3. Manual Refinements Towards Reference Standard Labels

The automatically generated annotations are meant to be used as a good approximated label on which manual refinements are needed by radiology experts following a consistently communicated annotation protocol. As defined in the BraTS challenge (Menze et al. 2014; Bakas, Akbari, Sotiras, et al. 2017; Bakas, Reyes, et al. 2018; Baid, Ghodasara, et al. 2021), the reference annotations (also known as ground truth labels) comprise of the enhancing part of the tumor (ET - label ‘4’), the peritumoral edematous/infiltrated tissue (ED - label ‘2’), and the necrotic tumor core (NCR - label ‘1’). ET is described by areas with both distinctly and faintly avid enhancement on the T1Gd scan, and is considered the active portion of the tumor. NCR appears hypointense on the T1Gd scan and is the necrotic/pre-necrotic part of the tumor. ED is defined by the abnormal hyperintense signal envelope on the T2-FLAIR scans and represents the peritumoral edematous and infiltrated tissue.

2.2.4. ML Model Training

The FeTS tool intends to utilize the data generated by the previous 3 modes of operation in 2 ways: i) to train a local collaborator model, using just the local data, or ii) participate in a federation with multiple sites to contribute the knowledge of the local collaborator data in the global consensus model, enabled by OpenFL (Reina et al. 2021). If a ML model is trained as part of a federation, no information about the actual data ever gets sent, but only each model update parameters from the collaborating sites are used to aggregate into a single global consensus model.

2.2.5. Model Architecture

Considering the impact that the potential clinical translation of the global consensus model could have, we did not follow the paradigm of training an ensemble of models (as is typically reported in related literature Menze et al. 2014; Bakas, Akbari, Sotiras, et al. 2017; Bakas, Reyes, et al. 2018; Baid, Ghodasara, et al. 2021), due to the additional computational burden the execution of multiple models introduce. We instead trained a single model architecture during the federated training, and specifically an off-the-shelf implementation of the popular 3D U-Net with residual connections (3D-ResUNet) (Ronneberger, Fischer, and Brox 2015; Çiçek et al. 2016; K. He et al. 2016; Drozdzal et al. 2016; Bhalerao and S. Thakur 2019). An illustration of the model architecture can be seen in Fig. 1. The network consists of 30 base filters, with a learning rate of lr = 5 × 10⁻⁵, optimized using the Adam optimizer (Kingma and Ba 2014). The generalized DSC score (Eq. 1) (C. H. Sudre et al. 2017; Zijdenbos et al. 1994) was calculated on the absolute complement of each tumor sub-compartment independently, as the loss function to drive the model training.

$$DSC=\frac{2\Vert RL\odot PM{\Vert }_1}{\Vert RL{\Vert }_1+\Vert PM{\Vert }_1}$$ (1)

where RL serves as the reference standard label, PM is the predicted mask, ⊙ is the Hadamard product (Horn 1990) (i.e., component wise multiplication), and ∥x∥₁ is the L1-norm (Barrodale 1968), i.e., sum of the absolute values of all components).

Figure 1.

Illustration of the U-Net architecture with residual connections, plotted using PlotNeuralNet https://github.com/HarisIqbal88/PlotNeuralNet.

Previous work (Isensee et al. 2021) has shown that such mirrored DSC loss is superior in capturing variations in smaller regions. The final layer of the model is a sigmoid layer, providing three channel outputs for each voxel in the input volume, i.e., one output channel per tumor sub-compartment. While we used the floating point DSC (Shamir et al. 2019) during the training process, the generalized DSC score was calculated using a binarized version of the output (against the threshold 0.5) for the final prediction.

2.3. Open-source Software Practices

The entire source code is open-source (detailed software architecture diagram is shown in Fig. 2) and automated periodic security analysis and static code checks of the main code repository is done with public notification of alerts using Dependabot ^¶ (Alfadel et al. 2021). Continuous integration is automatically performed for every pull request, with multiple automated checks and code reviews conducted prior to every code merge. All of these steps contribute in ensuring that information security teams of collaborating sites are comfortable with executing the FeTS tool on-premise.

Figure 2.

Detailed diagram of the FeTS Tool’s software architecture showing the various inter-dependencies between the components and the 4 main modes of operation.

3. Use Cases

3.1. The Largest to-date Real-World Federated Learning Initiative

The FeTS tool has enabled the largest to-date federation, utilizing data across 71 sites (Fig. 3), focusing on training a single consensus model to automatically delineate tumor sub-regions, without sharing patient data, and thereby overcoming legal, privacy, and data-ownership challenges. Specifically, the FeTS tool aimed at i) enabling clinical experts and researchers to work with the state-of-the-art segmentation algorithms (Kamnitsas et al. 2017; McKinley, Meier, and Wiest 2018; Isensee et al. 2021) and label fusion approaches (Pati and Bakas 2021), which directly resulted in easier quantification of tumor sub-compartments and the generation of reference standard labels in unseen radiologic scans.

Figure 3.

Illustration of the sites where the FeTS Tool was deployed.

This FeTS initiative followed a staged approach. The initial stage was conducted over a smaller consortium of 23 sites with data of 2, 200 patients (Baid, Pati, et al. 2021; Pati, Baid, Edwards, et al. 2022). The consensus model developed from this initial federation was evaluated on unseen validation data from each collaborator and showed an average model performance improvement of 11.1%, when compared to a model trained on the publicly available BraTS data (n = 231) (Baid, Pati, et al. 2021). Comparing against the model trained on publicly available data, the final consensus model (developed using 6, 314 patients) produced an improvement of 33% to delineate the surgically targetable tumor, and 23% over the tumor’s entire extent (Pati, Baid, Edwards, et al. 2022).

3.2. The First Challenge on Federated Learning

International computational competition (i.e., challenges) have become the accepted community standard for validating medical image analysis methods. However, the actual performance of submitted algorithms on “real-world” clinical data is often unclear, since the data included in these challenges usually follow a strict acquisition protocol from few sites and they do not represent the extent of patient populations across the globe. The obvious solution of simply collecting more data from more sites does not scale well, due to privacy and ownership hurdles.

Towards addressing these hurdles, the FeTS tool enabled the first challenge ever proposed for federated learning, i.e., the FeTS challenge 2021⁺ (Pati, Baid, Zenk, et al. 2021). Specifically, the FeTS challenge focused on two tasks: 1) federated training, which aimed at effective weight aggregation methods for the creation of a consensus model given a pre-defined segmentation algorithm; and 2) federated evaluation, which aimed to evaluate algorithms that accurately and robustly produce brain tumor segmentations “in-the-wild”, i.e., across different medical sites, demographic distributions, MRI scanners, and image acquisition parameters. The FeTS tool enabled both these tasks. In both the tasks, the FeTS tool enabled the processing of all clinically acquired multi-institutional MRI scans used by the remote sites used to validate the participating algorithms. Specifically for task 1, the federated learning component of the FeTS tool Reina et al. 2021 was used as the backbone to develop the weight aggregation methods.

4. Discussion

In this manuscript, we have described the Federated Tumor Segmentation (FeTS) tool, which has successfully facilitated 1) the first ever challenge on federated learning (Pati, Baid, Zenk, et al. 2021), and 2) the largest to-date real-world federation including data from 71 sites across the globe (Baid, Pati, et al. 2021). The FeTS tool has so far established an end-to-end solution for training ML models for neuroradiological mpMRI scans, while also incorporating a well-accepted robust harmonized pre-processing pipeline.

The FeTS tool was able to successfully process all the collaborators’ routine clinical data during the real-world federation, with sites spanning across 6 continents (Fig 3). This proved its out-of-the-box ability to handle processing of data “in the wild”, i.e., from a wide array of unseen scanner types and acquisition protocols, and importantly without any additional considerations. Furthermore, during its application in the first federated learning challenge (Pati, Baid, Zenk, et al. 2021), the FeTS tool has enabled i) the challenge participants to develop novel FL aggregation strategies, and ii) the “in-the-wild” evaluation of trained tumor segmentation ML models, across international collaborating healthcare sites.

One of its current limitations lie in the fact that even though its software stack has been designed and developed to be agnostic to the use-case, it has currently been tuned specifically for application in neuroradiological mpMRI scans. This is primarily due to the maturity and the widely evaluated specific pipelines and pre-trained segmentation models. Nonetheless, all incorporated algorithms can be trained and applied on data from different anatomies and modalities, notwithstanding their performance evaluation. Moreover, extension of the current pre-processing pipeline to include different imaging modalities would also be useful. Another potentially considered limitation, is the focus of the tool capabilities only defined for delineation tasks, and not offering the basis for classification problems. We are actively working towards exploring data augmentation techniques, as well as application of the tool to other anatomies and pathologies in future research projects.

In conclusion, we have developed the FeTS tool following numerous years of research and development on the BraTS challenge, and evaluated its performance and robustness during the largest to-date real-world federation. The FeTS tool can be currently used for any computational processing that would relate to brain tumor mpMRI scans. Particularly important for addressing the need to perform harmonized processing at remote healthcare sites prior to sharing data for varying consortia, as well as for the generation of centralized datasets hosted by authoritative organizations for de-centralized evaluation of computational algorithms. Towards this end, we expect the FeTS tool to pave the way for the generation of uniform datasets, which could contribute in improving our mechanistic understanding of disease.