PyJAMAS: open-source, multimodal segmentation and analysis of microscopy images

Abstract

Summary

Our increasing ability to resolve fine details using light microscopy is matched by an increasing need to quantify images in order to detect and measure phenotypes. Despite their central role in cell biology, many image analysis tools require a financial investment, are released as proprietary software, or are implemented in languages not friendly for beginners, and thus are used as black boxes. To overcome these limitations, we have developed PyJAMAS, an open-source tool for image processing and analysis written in Python. PyJAMAS provides a variety of segmentation tools, including watershed and machine learning-based methods; takes advantage of Jupyter notebooks for the display and reproducibility of data analyses; and can be used through a cross-platform graphical user interface or as part of Python scripts via a comprehensive application programming interface.

Availability and implementation

PyJAMAS is open-source and available at https://bitbucket.org/rfg_lab/pyjamas.

Supplementary information

Supplementary data are available at Bioinformatics online.

1 Introduction

Image analysis has become a critical component of the cell biologist’s toolkit. As a consequence, many applications exist for the analysis of microscopy images (Aigouy et al., 2010; Arganda-Carreras et al., 2017; Carpenter et al., 2006; de Chaumont et al., 2012; Etournay et al., 2016; Farrell et al., 2017; Fernandez-Gonzalez and Zallen, 2011; Mashburn et al., 2012; Schindelin et al., 2012; Schneider et al., 2012; Sommer et al., 2011). However, these tools are often written in Java or C, which are rarely the first languages learned by beginner programmers. Thus, many microscopists apply image analysis tools as black boxes. This is aggravated when commercial applications are used (e.g. Matlab or Imaris), introducing proprietary code and adding a financial barrier to the adoption of the tools.

Python is a free, general-purpose programming language, broadly taught in introductory computer programming courses (Guo, 2014), and therefore it is familiar to a large community of users. In addition, Python has become the standard language in machine learning and data science. Python tools for microscopy image acquisition (Pinkard et al., 2021), visualization (Napari Contributors, 2019) and analysis (Aigouy et al., 2020; Lugagne et al., 2020; von Chamier et al., 2021) have recently emerged.

Here, we introduce PyJAMAS, an open-source Python application for the analysis of biological images. PyJAMAS is distributed under a GNU General Public License v3.0 or later. PyJAMAS combines classical segmentation methods with machine learning techniques, and can be used through a cross-platform graphical user interface or via an application programming interface (API) that can be invoked from Python scripts. PyJAMAS complements and contributes to the nascent ecosystem of Python-based tools for microscopy and bioimage analysis.

2 Materials and methods

2.1 Image input, representation and display

PyJAMAS can load images in multiple formats, including the multi-page TIFF files common in microscopy. PyJAMAS stores images as three-dimensional objects using Numpy, a package for numerical computation, and its multidimensional array structures, ndarrays (Harris et al., 2020). ndarray operations release the global interpreter lock that normally allows only one thread to access the Python interpreter. PyJAMAS takes advantage of the global interpreter lock release in ndarray operations by spawning new threads to conduct ndarray-intensive tasks, including watershed-based image segmentation and training and application of machine learning classifiers. This allows multiple threads to execute in parallel and conduct computationally expensive tasks efficiently. The three-dimensional (3D) images stored by PyJAMAS often represent two spatial dimensions and time (XYt), although analysis of three-dimensional stacks (XYZ) or multi-channel (colour) images (XYc) is also possible. Four-dimensional images can be analyzed by segmenting and saving object annotations in one 3D channel and opening the annotations for quantification in other channels. Images are displayed within a graphical user interface written using PyQt, the Python bindings for the Qt framework for cross-platform user interface development (Riverbank Computing) (Fig. 1, center).

Fig. 1.

PyJAMAS: biological image quantitation in Python. PyJAMAS provides multiple tools for image segmentation, including the watershed algorithm (illustrated here with the segmentation of epithelial cells of the Drosophila ectoderm), linear classifiers (logistic regression and support vector machines, demonstrated on the nuclei of heart progenitors during Drosophila cardiac development) and convolutional neural networks (U-Net architecture, shown as applied to cells in the Drosophila embryonic epidermis). PyJAMAS allows batch analysis of multiple images, and outputs Jupyter notebooks for quantitation and graph generation. PyJAMAS can be extended using plugins written in Python, and exposes an API that can be invoked from Python scripts. PyJAMAS is documented at pyjamas.readthedocs.io. An issue tracker (https://bitbucket.org/rfg_lab/pyjamas/issues) outlines current limitations and plans for further development

2.2 Image segmentation

Segmentation is the process of partitioning the pixels in an image into the objects that form the image. PyJAMAS includes both traditional segmentation methods (based on manual, semi-automated or fully automated approaches) and machine learning algorithms. The segmentation methods included in PyJAMAS have been benchmarked for their use on images from different light microscopy modalities (Ulman et al., 2017).

2.2.1 Fundamental methods

PyJAMAS provides several basic segmentation tools. These include manual object annotation using rectangles and polylines; semi-automated tracing using the LiveWire algorithm, a method that finds the brightest pixel path that follows the movement of the mouse (Fernandez-Gonzalez and Zallen, 2011) and can be used to correct the results of fully automated approaches; and automated segmentation and tracking using the watershed region growing algorithm (Beucher, 1992) (Fig. 1, bottom right) combined with particle image velocimetry to propagate watershed seeds in time or space, thus integrating segmentation and tracking (Wang et al., 2017).

2.2.2 Machine learning

PyJAMAS also includes machine learning classifiers for image segmentation. Linear classifiers use linear combinations of feature values to determine the category that an image belongs to (e.g. nucleus or background). PyJAMAS includes two types of linear classifiers: logistic regression and support vector machines—with the option of using a radial basis function kernel for non-linear classification problems—(Fig. 1, bottom) (Wang et al., 2017; Wang and Fernandez-Gonzalez, 2017). Training is based on the histogram of local gradient orientations of the images in the training set. In addition, PyJAMAS also implements a deep convolutional neural network architecture, U-Net (Falk et al., 2019), for large-scale, non-linear classification tasks (Fig. 1, bottom left). The inputs to the network are a set of images of the objects to be detected (cells, nuclei, etc.) and corresponding binary masks labeling the objects. The training process consists of minimizing a loss function that measures how far the network prediction is from the mask. The loss function is weighed with a map that has greater values at pixels in the proximity of object borders, thus ensuring that the network learns to separate touching objects.

U-Net training can be computationally expensive. PyJAMAS supports CUDA-enabled graphics processing units (GPUs) to train networks locally, or it can generate a Jupyter notebook for network training. The notebook can be executed locally or uploaded to hosted notebook servers, such as those in Google Colab, which provide remote access to GPUs for limited blocks of time to accelerate computation. The model trained remotely can be loaded back into PyJAMAS for application to new images.

Many different neural network architectures exist for image segmentation (Minaee et al., 2021). Users can request additional architectures through the issue tracker available in the PyJAMAS repository (https://bitbucket.org/rfg_lab/pyjamas/issues). Users can also implement new architectures and integrate them with PyJAMAS by submitting pull requests or plugins (see below).

2.3 Batch analysis and analysis notebooks

PyJAMAS can analyze and compare entire image datasets out-of-the-box, without requiring users to record macros or write code. The output of the batch analysis is a spreadsheet that contains morphological and pixel intensity features and their evolution over the third image dimension for all of the objects in the images. PyJAMAS also generates a Jupyter notebook that can be used to re-run the analysis, or to load the spreadsheet created by PyJAMAS and make custom changes to the analysis. The notebook generates various graphs, including time series, box plots and scatter plots of the different metrics (Fig. 1, top left).

2.4 The PyJAMAS API, plugins and documentation

A public API exposes the functionality of PyJAMAS for use in Python applications (Fig. 1, top and Supplementary Fig. S1). The PyJAMAS API is natively written in Python and can be combined with other Python packages to write plugins that extend PyJAMAS. We have documented PyJAMAS, the PyJAMAS API and the development of plugins, including examples, at pyjamas.readthedocs.io (Fig. 1, top right). We created the PyJAMAS documentation using sphinx and the autodoc extension, which ensures that changes to the API are automatically reflected in the documentation.

3 Conclusion

PyJAMAS is a flexible open-source tool for the segmentation and quantitation of microscopy images. PyJAMAS includes classic and machine-learning tools for image segmentation; fully automated, semi-automated and interactive approaches. Advanced users can invoke PyJAMAS from Python scripts, and they can extend PyJAMAS using plugins written in Python. We expect that the versatility of PyJAMAS, combined with the popularity and adaptability of Python and the increasing availability of Python-based tools for the analysis of biological samples, will empower cell biologists to take control of their imaging pipelines.

Funding

This work was supported by the Natural Sciences and Engineering Research Council of Canada [418438-13], the Canada Foundation for Innovation [30279], the Government of Ontario [ER14-10-170], the Canadian Institutes of Health Research [156279], the Translational Biology and Engineering Program of the Ted Rogers Centre for Heart Research., the University of Toronto XSeed Program and the GPU Seeding Program of NVIDIA Corporation. R.F.-G. is the Tier II Canada Research Chair in Quantitative Cell Biology and Morphogenesis.

Conflict of Interest: none declared.