Defining the boundaries: challenges and advances in identifying cells in microscopy images

Abstract

Segmentation, or the outlining of objects within images, is a critical step in the measurement and analysis of cells within microscopy images. While improvements continue to be made in tools that rely on classical methods for segmentation, deep learning-based tools increasingly dominate advances in the technology. Specialist models such as Cellpose continue to improve in accuracy and user-friendliness, and segmentation challenges such as the Multi-Modality Cell Segmentation Challenge continue to push innovation in accuracy across widely-varying test data as well as efficiency and usability. Increased attention on documentation, sharing, and evaluation standards are leading to increased user-friendliness and acceleration towards the goal of a truly universal method.

Graphical Abstract

Introduction

Segmentation plays a pivotal role in microscopy analysis and refers to the automatic delineation of individual objects (often cells or cellular components) within complex scientific images. It is an important step prior to measuring properties of those biological entities. Approaches for cell segmentation have benefitted from advancements in more general segmentation problems in traditional Computer Vision (CV), Machine Learning (ML), and in recent years Deep Learning (DL) [1,2]. Accurate segmentation allows the quantification and analysis of cellular features, such as morphology, staining intensity, and spatial relationships, which capture valuable cellular phenotypes. While computational methods now achieve better-than-human accuracy on a number of specific tasks, in general, given the wide range of cell types, imaging modalities, and experimental conditions, the problem remains an ongoing challenge.

As state-of-the-art (SOTA) methodologies for segmentation have progressed, the community has also tried to provide access to these methods to less-computational users in the form of user-friendly software interfaces and intuitive tools that improve reproducibility. Widespread adoption will require methods with few-or-no tunable parameters, models that are efficient in terms of computational runtime and memory requirements, and an ecosystem of tools for their use. The past two years reviewed here have seen a proliferation of new local and cloud-oriented software and workflows, the adoption of several user-oriented models, and the development of next generation model architectures. We herein review progress in approaches utilizing classical computer vision techniques and specialist deep learning networks, as well as progress towards and current needs related to making high quality accessible generalist networks that will reduce the “time to science” for the broader community.

Progress in classical approaches

While advancements in segmentation accuracy are largely driven via deep learning approaches, they are not always a suitable solution, as some require large annotated datasets and interpretability (though often unnecessary for segmentation tasks) can be a challenge [3,4]. We therefore begin with advancements in non-deep machine learning and classic image processing. Kartezio [5] is a recent exemplar of non-deep ML, using Cartesian Genetic Programming to combine classic Computer Vision algorithms into a fully interpretable image pipeline for segmentation. It performs comparably to deep learning approaches such as Cellpose [6], StarDist [7] and Mask R-CNN [8], and importantly requires relatively few images to train a computationally-efficient and explainable workflow.

Existing CV tools such as Fiji/ImageJ [9], CellProfiler [10], and Napari [11] are receiving continuous extensions via the growing ecosystem of plugins and integrations [1,12–15], allowing these tools to adapt to a broader range of tasks. One example, General Image Analysis of Nuclei-based Images (GIANI) [16] is a Fiji plugin for segmentation of cells in 3D microscopy images. Similarly, LABKIT [17] is a Fiji plugin specifically oriented towards efficient segmentation in large, multi-terabyte, images. Other tools, such as Tonga [18] prioritize ease of installation and customization to a specific task to appeal to non-technical and non-expert users.

Lowering time-to-science in deep learning

As new deep learning models rapidly emerge, users who want to employ them must surmount computational hurdles. Firstly, installing the models and their corresponding software dependencies can often be technically complex and time-consuming, often involving usage of the Command Line Interface (CLI), compiling source code, navigating environment related conflicts, and other tasks which may confound non-technical users. Secondly, pre-trained models are most effective on the type of data they’ve been trained on, which may differ from a user’s data in a number of ways such as the phenotype of interest, how the biological samples were prepared, the imaging modality (fluorescence, histological stains, phase contrast, etc), and the experimental conditions. The model’s generality is how well it’s able to perform across these differences, without loss in segmentation accuracy. If the model’s generality is not sufficient to perform well on the user’s data, it must be fine tuned by feeding in new training data (Figure 1). The complexity of this task can vary even more dramatically than installation. Depending on the tools made available by the model’s developers, it may involve writing custom code, which can be a time-intensive task even for expert programmers, or may simply require the use of a purpose-built tool.

Figure 1.

Segmentation and fine tuning process. a) The inference dataset contains a variety of samples; some are similar to the original training data the model was pre-trained on, while others are slightly or very different, leading to low segmentation accuracy. b) New training data matching the characteristics of the full distribution is annotated either manually, through software such as CellProfiler, or a human-in-the-loop model such as Cellpose, and used to fine-tune the model. c) The fine-tuned model produces more accurate segmentations on the inference dataset, which can then be used for downstream tasks.

The less time spent installing, tuning, and configuring models and software, the more bandwidth is available to concentrate on addressing scientific questions. Unfortunately, usability varies wildly across tools and documentation for trained deep learning models, from only the model parameters (weights) without documentation or source code, to models that come with extensive documentation and entire libraries for utilizing the model, including data loading and processing, fine-tuning, and configuration, to models that come with several interfaces including CLI or Graphical User Interfaces (GUI), easy to use installers, and guides or tutorials on how to fine-tune and configure the model. It is no accident that some of the most commonly used networks, discussed below, are those that in addition to high performance have emphasized usability.

Current progress towards useful specialist networks

As shown in Figure 1, models typically underperform on new, unseen data. Before fine tuning, this data is considered out of distribution (OOD). Intuitively, OOD data can be thought of as being drawn from a different distribution than that of the original training set (such as the purple vs green cells in Figure 1). A shift between the in-distribution training set and OOD data can be referred to as a difference in style between the two sets of data; for instance differences in acquisition parameters, staining methods, or imaging modalities [19,20]. A process known as style transfer can be utilized to address changes in the data distribution by training a model that is able to pixel-wise map an image of one style to that of another, ideally with minimal loss in semantic content [21]. For instance, a style transfer model can be trained to transform an image of one modality, such as brightfield, to that of another, such as fluorescence; or a model can be trained to transform an image’s annotation mask into the image itself. The nucleAIzer [22] model utilizes the latter approach as a means of achieving greater generalization capabilities, allowing the model to more easily be adapted to OOD data. Although developed over three years ago, it is still one of the top performing models at cell segmentation [23]. In order to improve usability, a plugin was developed for CellProfiler 3 [24] allowing users of the tool to perform inference through the use of a GUI. While the plugin allows ease of use via the CellProfiler interface, manual installation of nucleAIzer and its dependencies are still necessary, which is a challenge for non-computational users. A simple web interface is also available, but requires upload of the data to a central server, limiting use for large batches of images.

StarDist predates nucleAIzer, however iterations of it are still being introduced [25,26], and general usage remains quite high due to a great deal of time being spent on documenting its usage, as well as making it available across environments and in a variety of graphical tools [12] such as Fiji/ImageJ, Napari, QuPath [27,28], Icy [29], CellProfiler, and KNIME [30]. While a variety of pretrained variants are available, it is still limited to specific modalities (such as fluorescence or histology stains) and even with fine-tuning is oriented to segmenting objects which are star-convex—shapes where line segments can be draw from any point along the border to some single interior point—which makes it a poor choice for very irregular cell shapes or neurons, since shapes with very large bends or curves may not be star-convex.

A primary goal of Cellpose was to develop a generalist model by training on a large dataset of manually segmented images from a variety of modalities. Its preprocessing method, focusing on transforming input data to spatial gradients, allows it to generalize to a larger variety of shapes. While the architecture was developed for generalist purposes, fine-tuning is still often necessary. Cellpose 2.0 [31] was introduced as a package that included several pre-trained models, a human-in-the-loop pipeline for fine-tuning custom models with small datasets, and an improved set of graphical software tools to aid in its usage. Omnipose [32] extends Cellpose to work better on elongated cells common in bacteria by adding distance field prediction similar to StarDist. It is similarly well documented, packaged, and provides a number of interfaces in the form of a library, command line interface, and GUI. While it allows for training, it is not available in Cellpose 2.0, and thus is not focused on human-in-the-loop finetuning. Cellpose and Cellpose 2.0 include a custom GUI, and extensive documentation; broad usage is further supported by their availability via plugins from many of the same tools as StarDist.

Mesmer is a deep learning pipeline trained on the largest public tissue data set of annotated nuclei and whole cells, TissueNet [33]. It provides access to a remotely hosted instance model through several interfaces, including a web portal and plugins for Fiji/ImageJ and QuPath. It also provides a Docker container (described below, in Making Models FAIR) to run the model in a self hosted manner, with access provided through a Jupyter Notebook or CLI.

A Vision for the State-Of-The-Art

Many of those working on deep learning for image segmentation are aiming to create a truly generalist model, often referred to as a foundation model, capable of greater-than-human accuracy, across the wide variety of imaging modalities, in a parameter-free manner [34–36]. For biologists, this would mean a model that has no need for manual annotation or fine tuning, preferably in a form factor that is easily accessible, configurable, and invocable. Although there is still much progress to be made on this front, a number of architectural developments can be highlighted as mile markers along the path.

Dataset Availability

The effectiveness of Deep Learning models relies on sufficient similarity between the training data and the user’s own data. The specific criteria for what constitutes “sufficient similarity” will naturally differ based on the methods and architectural choices employed by the model, but it therefore follows that creating foundation models (capable of segmenting a wide variety of biological images) will require a diverse and comprehensive corpus of training data to ensure that the segmentation model can generalize effectively. The dataset should encompass a broad spectrum of microscope modalities, across a variety of imaging conditions, and include many distinct cell types.

The volume of publicly available datasets is ever increasing, particularly driven from the development of specialist models and challenges wherein teams compete on segmentation oriented tasks. TissueNet is the largest collection of annotated tissue images, while LIVECell [37] is the largest collection of high-quality, manually annotated, and expert-validated phase-contrast images. The Cell Tracking Challenge (CTC) [38] is an ongoing benchmark and reference in cell segmentation and tracking algorithms, which in recent years has extended the available benchmarks with the Cell Segmentation Benchmark (CSB). The Multi-Modality Cell Segmentation Challenge (MMCSC) [36] consolidated a modestly sized labeled dataset with a particular emphasis on diversity in modalities.

Challenges stand as an excellent pointer for future progress in a given area; trends among the top-ranking team’s architectures and techniques often form the basis of future implementations available to the wider field. For instance, data augmentation—where existing training data is perturbed and transformed in various ways, such as rotations, scaling, intensity adjustments, or the infusion of random noise—was highlighted in the MMCSC as a particularly important feature in pre-training top performing models, aiding them in their generalizability. Entirely synthetic datasets are also often useful, as demonstrated by their inclusion in a subset of the CTC datasets, and in the development of frameworks for their generation [39].

Next Generation Models

Many architectures are being researched and explored in the quest for ever more general and robust models, as demonstrated in the MMCSC. The KIT-GE model [40] is among the top-3 performing models of CTC CSB, and was therefore used as one of the baselines in the MMCSC alongside Cellpose, Cellpose 2.0, and OmniPose [6,31,32]. Analysis of the top-ranking solutions in the MMCSC shows that choices in backbone networks are particularly important for next generation models. While U-Net inspired architectures form the basis of the widely used contemporary models such as StarDist, Cellpose, and KIT-GE, the winning solutions in MMCSC employ backbones such as SegFormer [41], ConvNeXt [42], and ResNeXt [43]. CTC reports that segmentation performance increases with techniques such as self-configured neural networks (e.g. nnU-Net) [44], neural architecture search, and multi-branch prediction.

The generalist capabilities provided by Cellpose 2.0 rely on fine-tuning, which may cause the model to suffer from severe loss of performance on tasks outside of the fine-tuning dataset [36]. The top performing model [45] of the MMCSC was able to outperform the pre-trained generalist Cellpose and Omnipose models, as well as a Cellpose 2.0 model fine-tuned on the challenge’s training data. The testing set included images which were distinct from the training data, and sourced from new biological experiments, meaning successful models needed to show a strong ability to generalize across data without additional fine-tuning.

While the winning solution of MMCSC and its associated code is available on GitHub, it remains to be seen whether it or any of the top-ranking models will be available with documentation and interface tools in a way comparable to StarDist, Cellpose 2.0, or Omnipose. If so, we may be one step closer to a truly general, easy to set up, easy to use, one-click segmentation model, with no additional tuning. Short of that, alternative interfaces for model configuration may come to prominence in the form of dialog-driven LLMs [46,47]. It also as yet unclear whether future enhancements will be driven primarily by transformer architectures [34,48], whether advancements in convolutional networks [42] will keep pace, as demonstrated by the second and third place solutions in the MMCSC, or whether hybrid approaches will dominate [49,50].

It is also of great research and commercial interest to develop foundation models. Meta AI Research recently released a family of foundation models for segmentation, referred to as the Segment Anything Model [51] (SAM), the largest of which was trained on 1.1 billion high-quality segmentation labels, across 11 million high-resolution images. While the images included in the dataset were mostly photographs of natural scenes, it did include a small number of microscopy images taken from the 2018 Data Science Bowl [52]. In short order, Segment Anything for Microscopy [53] was developed, in which SAM was extended to generalize across many imaging modalities by fine-tuning the original model using a variety of datasets. Important to SAM’s architecture is its interactive segmentation capabilities, where a subset of the user’s data is first annotated with a small amount of either point annotations or rectangular bounding boxes. Annotations of this type are significantly less time-consuming than pixel-level mask annotations, and provide SAM with enough guidance to output full segmentation masks on the user’s dataset. Segment Anything for Microscopy therefore adopts this capability and includes a Napari plugin for interactive and automatic segmentation. There is a mechanism for automatic segmentation, however in order to get generalist accuracy above that of Cellpose, some manual annotations must be made.

Making models FAIR

Alongside progress in model development, there has been a greater push towards the dissemination of models such that they are Findable, Accessible, Interoperable, and Reusable (FAIR) [54–56]. Not only should models be available on publicly accessible platforms, but the associated code for asset loading, data pre-processing, data post-processing, model training and model inference, should also be made available in a well packaged and documented form. Container platforms such as Docker [57] can alleviate many installation and setup complexities, providing an isolated and controlled environment in which software is installed, and a pre-configured installation process for the target software. This “containerization” of software and its dependencies dramatically decreases the barriers for reuse. In addition, making models readily accessible, configurable, and tunable in a low or no code manner via interactive code notebooks or graphical user interfaces encourages broader adoption of SOTA models.

The CTC proposed guidelines for algorithm developers to make their workflows both available and reproducible; while currently optional, they will be mandatory in the future. At minimum the source code should be available on a public repository, and contain clear instructions for installing dependencies, initializing the model, loading weights, and training with new data. They also pushed for source code to be available via notebooks such as Jupyter and Google Colab. In the same vein, the MMCSC required all participants to place their solutions in Docker containers; the winning teams have made these available on public image registries and also made their algorithms publicly available on GitHub alongside processing source code. In addition the top three solutions were encouraged to develop Napari [11] plugins.

A missing component in full adherence to FAIR principles is the interoperability of models. While research and development in model architectures is healthy and vital, there is no agreed upon specification on the inputs and outputs of models. The difference in the model outputs between e.g. StarDist and Cellpose is stark, and the post-processing that is needed is correspondingly distinct. While the outputs are necessary byproducts of the model architectures, the lack of a standard makes interoperability with existing tools difficult as custom code needs to be written to mirror the post-processing steps.

Making models efficient

Models vary widely in terms of algorithmic efficiency, which will affect their adoption, especially in low-resource settings. While some challenges such as the CTC emphasize segmentation accuracy alone, others such as MMCSC evaluate efficiency as an explicit criterion, and the top performing models had good tradeoffs between accuracy and efficiency in runtime and memory usage. There are additional efforts in bringing model optimization tools to the bioimage community [58,59], as well as reducing runtime of bioimage analysis pipelines in general [59,60]. Efficient models, and model optimization tools will become increasingly important for training and inference tasks in local desktop and web-based tools [2,61], especially in contexts where moving data to the cloud is not viable or allowed.

Improving tool access and availability

Though local-first software and algorithms remain important to grow and maintain, cloud oriented tools and resources for bioimage analysis are becoming more prevalent and easier to use as demand for large-scale cloud-based workflows increases. In addition to providing large storage capacity and high performance hardware, cloud-based tools increase the availability and accessibility of models, and often move the technical complexity away from the end user.

Notebooks allow code, explanatory text and interactive elements to live together in a single package, providing alternatives or complements to libraries, documentation and GUI interfaces. ZeroCostDL4Mic [62] provides rigorously documented and annotated code notebooks with pre-written code which can be customized for specific workflows through the exposed settings. A major benefit to these notebooks is that they can be deployed either locally or on the Google Colab platform, which eases hardware requirements and allows running moderately sized workloads for free. Behind the scenes, a container is initialized in the cloud and the installation occurs via pre-configured installation scripts contained within the notebooks.

Beyond notebooks, other tools provide a larger degree of customization and control, albeit at the expense of additional complexity. The BioContainers project is an open-source and community-driven framework which provides cloud resources for defining, building, and distributing containers for biological tools [63]. The BioContainers Registry was developed with FAIR principles in mind, and provides both web and RESTful API interfaces to search for bioinformatics tools [64]. BIAFLOWS [65] is a community-driven, open-source web platform that allows deployment of and access to a wide variety of reproducible image analysis workflows. The platform provides a framework to import data, encapsulate workflows in container images, batch process data, visualize data, and assess performance using widely accepted benchmark metrics on a large collection of public datasets. BioImageIT [66] is a more recent, plugin oriented, workflow tool for data management and analysis. It has a unique emphasis on reconciling existing data management and data analysis tools, and although run locally has the ability to tap into remote data stores and job runners.

The BioImage Model Zoo [67] provides a community-driven repository for pre-trained deep learning models and promotes a standard model description format for describing metadata. Community partners can work with the BioImage Model Zoo to support execution of the models and include many common bioimage tools. In addition, model execution can be performed via the BioEngine application framework, on top of the ImJoy plugin framework [68], allowing inference both on the BioImage Model Zoo web application, and other web applications using the client ImJoy software. Behind the scenes, multiple containers are being run and managed with a container orchestration tool called Kubernetes.

For moderately more technical users, who are comfortable with using tools for deploying their own container orchestration workloads, there are some additional options. DeepCell Kiosk [69] is a cloud-native tool for dynamic scaling of image analysis workflows, utilizing Kubernetes orchestration similarly to the BioEngine inference engine. The tool is managed from several interfaces including a web portal and Fiji plugin. Distributed-Something [70] takes a script-based approach to scale and distribute arbitrary containerized jobs on AWS, automatically configuring the AWS infrastructure for container orchestration, monitoring, and data handling. It runs the work in a cost effective manner, and cleans up the infrastructure when the work has been completed.

Conclusion

The landscape of segmentation algorithms, enabling tools, workflow management systems, repositories, benchmarks, and challenges is constantly shifting. This very active landscape makes it all the more important to create community standards for reporting on methods and robust segmentation quality metrics, on which there has been recent guidance [35,71–73]. While there is still much work to do, the past two years have seen essential strides made in democratizing the use of advanced segmentation methods through user-friendly interfaces and improved documentation. Integrating tools and scaling up reproducible workflows fosters a more collaborative and robust ecosystem; these continuing trends will empower researchers from diverse backgrounds to collectively explore the intricate universe of single-cell biology, ultimately accelerating the pace of discovery and innovation in this vital field of study.