The ImageJ ecosystem: Open‐source software for image visualization, processing, and analysis

Abstract

For decades, biologists have relied on software to visualize and interpret imaging data. As techniques for acquiring images increase in complexity, resulting in larger multidimensional datasets, imaging software must adapt. ImageJ is an open‐source image analysis software platform that has aided researchers with a variety of image analysis applications, driven mainly by engaged and collaborative user and developer communities. The close collaboration between programmers and users has resulted in adaptations to accommodate new challenges in image analysis that address the needs of ImageJ's diverse user base. ImageJ consists of many components, some relevant primarily for developers and a vast collection of user‐centric plugins. It is available in many forms, including the widely used Fiji distribution. We refer to this entire ImageJ codebase and community as the ImageJ ecosystem. Here we review the core features of this ecosystem and highlight how ImageJ has responded to imaging technology advancements with new plugins and tools in recent years. These plugins and tools have been developed to address user needs in several areas such as visualization, segmentation, and tracking of biological entities in large, complex datasets. Moreover, new capabilities for deep learning are being added to ImageJ, reflecting a shift in the bioimage analysis community towards exploiting artificial intelligence. These new tools have been facilitated by profound architectural changes to the ImageJ core brought about by the ImageJ2 project. Therefore, we also discuss the contributions of ImageJ2 to enhancing multidimensional image processing and interoperability in the ImageJ ecosystem.

Highlights

ImageJ is an open‐source image analysis software with a large, diverse user base. ImageJ has several components and distributions such that we refer to the entirety as the ImageJ ecosystem. Recent developments have adapted to the needs of users and the demands of increasingly large, complex biological datasets accompanying technological advancements in imaging. This review highlights several new tools developed in the ImageJ ecosystem to address needs for improved visualization and analysis of biological features.

1. INTRODUCTION

ImageJ is a widely‐used open‐source software that allows users to visualize, inspect, quantify, and validate scientific image data. ¹ Imaging‐based methods serve a crucial role in the life sciences and have undergone tremendous growth in the past decades. ² As novel imaging modalities emerge and datasets become more complex, having reproducible and reliable methods to interpret biological images is the core of image analysis, where “seeing is a matter of learning”. ² Image analysis allows users to extract information from images in a reproducible manner. For this to occur, the algorithms and parameters used must remain open and consistent. If any component of the algorithm is proprietary, researchers may be unable to reproduce their work or track changes with full transparency, which is imperative to the scientific process. ² To address this need, open source software is advantageous, and this is one of the main reasons for ImageJ's widespread use.

Novel imaging modalities offer enhanced resolution, specificity, and coverage and have contributed to many of the tremendous biological advancements over the past several decades. ³ , ⁴ , ⁵ , ⁶ , ⁷ Modern research therefore requires methods for the efficient and robust manipulation, interpretation, and visualization of such advanced, multidimensional imaging data. ⁸ This is important for a wide array of biological studies that include quantifying the proximity of fluorescent‐labeled proteins, ⁹ tracking cell fates over time, ¹⁰ automating cell counting, ¹¹ tracking invading cancer cells, ¹² collecting whole‐slide information, ¹³ quantifying and characterizing cells, such as microglia, within the brain, ¹⁴ or registering multiview light sheet fluorescence microscopy datasets to study development. ¹⁵ , ¹⁶ Image analysis also serves a crucial biomedical role for diagnostic interpretation. ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ As the prevalence of large multidimensional datasets continues to grow, the ability to manually take measurements not only becomes impractically time‐consuming, but the sensitivity, accuracy, objectivity, and reproducibility of doing so can become greatly inhibited. ²¹ , ²² Because of these challenges, we will highlight in this review current developments within the ImageJ ecosystem that address the handling of large, multidimensional datasets, including annotating and performing advanced image analysis techniques.

Historically, the predecessor to ImageJ, NIH Image, was developed by Wayne Rasband in 1987 at the National Institutes of Health. Rasband was enthusiastic about the concept of open code sharing on the electronic bulletin boards of the time and the possibilities offered by the release of the Apple Macintosh II with advanced graphics and support for the Pascal programming language. His goal was to allow third party developers to customize and create their own domain‐specific analysis routines ²³ and let users drive the applications of NIH Image. In 1995, when Sun Microsystems released the Java programming language that could run on any operating system, the transition of NIH Image to Java was initiated, and the first release of the succeeding platform, known as ImageJ, occurred. ²³

Figure 1 shows the evolution of the ImageJ ecosystem. ImageJ has a simple and single toolbar that opens when the program is running, and this toolbar has remained much the same since its early years. Next to this remarkable stability, ImageJ is also uniquely flexible. Its core feature from the very beginning has been, and still is, that it can be extended by users to fulfill the specific needs of their analyses. Thus, ImageJ—as designed in its first incarnation as NIH Image, as well as the first generation of the cross‐platform ImageJ, ²³ henceforth referred to as ImageJ1 to disambiguate it from ImageJ2—can adapt to changing times and technologies, and it continues to do so to this day.

FIGURE 1.

A diagram of the evolution of NIH Image to the current ImageJ ecosystem including release dates from 1987 until present. Fiji ²⁴ leverages the updated ImageJ2 framework, which contains elements from ImageJ1 via the optional ImageJ Legacy bridge that facilitates full backwards compatibility. Fiji comes prepackaged with many plugins and tools including (a,c) registration, (b) segmentation, and (d) measurement tools. Thumbnail images are courtesy of the Fiji community ²⁴ , ²⁵

Fiji (Fiji is Just ImageJ), a “batteries‐included” distribution of ImageJ (https://imagej.net/Welcome) bundling many plugins which facilitate scientific image analysis, was introduced in 2007 to ease installation and development of more complex plugins but also to be ready‐to‐use as‐is by biologists. ²⁴ Fiji developers introduced several key components such as an automatic update mechanism ²⁴ (https://imagej.net/Updater), a script editor (https://imagej.net/Script_Editor), and a more powerful image data model ²⁶ (https://imagej.net/ImgLib2). Concurrently, the ImageJ2 project sought to improve ImageJ's core architecture, introducing an improved plugin mechanism (https://imagej.net/SciJava_Common) enabling greater extensibility. ²⁷ The two projects joined forces, and now Fiji builds upon the ImageJ2 framework to which it contributed several important libraries. The marriage of Fiji and ImageJ2 did not mean divorce from ImageJ1; on the contrary, ImageJ2 provides backwards compatibility with ImageJ1 via the ImageJ Legacy bridge (https://imagej.net/ImageJ_Legacy), allowing existing ImageJ1 plugins to continue working as‐is in the new system. ImageJ is compatible with macOS, Linux, and Windows and comes in several “flavors”—or packages. The current ImageJ ecosystem includes the original ImageJ1, the updated ImageJ2, the Fiji distribution of ImageJ, and community plugins, shipped individually via Update Sites, ²⁸ which extend the functionality of ImageJ. More than 200 listed Update Sites (https://imagej.net/Update_Sites) in ImageJ's central registry provide a convenient vehicle for users to obtain the latest releases of novel ImageJ tools and specialized plugins. The Fiji distribution includes a broad collection of preloaded plugins available for scientific image analysis and is recommended for new users. ²⁹ Many advanced plugins (for a small selection, see Table I) rely on Fiji's modern distribution that includes accessible compilations, automatic updates, and third‐party libraries.

TABLE I.

Representative specialized plugins and libraries in ImageJ ecosystem

Plugin or library	Description	Representative contributors (limited to three)
BigDataViewer 30 (https://imagej.net/BigDataViewer)	Inspects large image files from local and remote data sources while maintaining the image metadata	Tobias Pietzsch
BigStitcher 31 (https://imagej.net/BigStitcher)	Software package that allows simple and efficient alignment of multitile and multiangle image datasets, for example acquired by light sheet, widefield or confocal microscopes	Stephan Preibisch, David Hörl
BigWarp 32 (https://imagej.net/BigWarp)	Tool for manual, interactive, landmark‐based deformable image alignment, using Java implemented thin plate splines to build a dense deformation model	John Bogovic, Stephan Saalfeld, Tobias Pietzsch
Bio‐Formats 33 (https://imagej.net/Bio‐Formats)	Standalone Java library for reading and writing life sciences image file formats capable of parsing both pixels and metadata for a large number of formats	Melissa Linkert, Curtis Rueden, Sébastien Besson
ClearVolume 34 (https://imagej.net/ClearVolume)	Real‐time, accelerated 3D‐volume rendering for multichannel image processing, inspection during acquisition	Florian Jug, Loic Royer, Martin Weigert,
CSBDeep 35 (https://imagej.net/CSBDeep)	Open‐source machine learning plugin compatible with Fiji, python, KNIME that can restore and segment fluorescence images	Uwe Schmidt, Deborah Schmidt, Martin Weigert
DeepImageJ 36 (https://deepimagej.github.io/deepimagej/)	User‐friendly plugin that enables the use of a variety of pretrained deep learning models in ImageJ and Fiji	Carlos García‐López‐de‐Haro, Estibaliz Gómez‐de‐Mariscal, Daniel Sage
Fijiyama 37 (https://imagej.net/Fijiyama)	3D registration tool for multimodal time‐lapse imaging	Romain Fernandez,Cédric Moisy
FLIMJ 38 , 39 (https://imagej.net/FLIMJ)	Fluorescence lifetime imaging microscopy analysis and image processing	Dasong Gao, Paul Barber, Curtis Rueden
ImageJ Ops 17 (https://imagej.net/Ops)	Framework for reusable image processing operations, ops extends Java's mantra of “write once, run anywhere” to image processing algorithms	Curtis Rueden, Christian Dietz, Alison Walter
ImageJ‐OpenCV (https://github.com/imagej/imagej‐opencv)	Library that enables OpenCV to be used from ImageJ commands and scripts and provides converters between ImgLib2 and OpenCV data structures	Gabriella Turek, Curtis Rueden
ImageJ‐TensorFlow (https://github.com/imagej/imagej‐tensorflow)	This project enables TensorFlow to be used from ImageJ commands and scripts	Curtis Rueden, Deborah Schmidt, Benjamin Wilhelm
ImageJ updater (https://imagej.net/Updater)	Keeps users up‐to‐date with all components of ImageJ, including both plugins and the core components (libraries) needed by those plugins	Johannes Schindelin, Curtis Rueden, Deborah Schmidt
ImgLib2 26 (https://imagej.net/ImgLib2)	General‐purpose, multidimensional image processing library and core to ImageJ2 and Fiji	Tobias Pietzsch, Curtis Rueden, Stephan Saalfeld,
Labkit (https://imagej.net/Labkit)	Advanced ImageJ2 plugin for image labeling and segmentation, featuring a user‐friendly interface to work on large datasets	Matthias Arzt
MaMuT 40 (https://imagej.net/MaMuT)	Annotation and multiviewing of big data, combining the power of BigDataViewer and TrackMate; requires manual or semi‐automated annotation	Jean‐Yves Tinevez, Tobias Pietzsch
Mastodon 41 (https://github.com/mastodon‐sc/mastodon)	Facilitates computational analysis within cellular, developmental, or stem cell biology to provide interactive inspection that is fast and responsive; semi‐automatic and fully automatic tracking data; extract numerical features to gather statistics	Tobias Pietzsch, Jean‐Yves Tinevez
SCIFIO42 (https://imagej.net/SCIFIO)	Flexible framework for SCientific image format input and output, that is, a library for reading and writing N‐dimensional image data	Curtis Rueden, Mark Hiner,
SciJava (https://imagej.net/SciJava)	Collaboration of projects providing software for scientific computing—An effort to cooperate and reuse code when feasible; includes ImgLib2, SCIFIO, bio‐formats, ImageJ2, or Fiji	Curtis Rueden, Johannes Schindelin, Mark Hiner
SciView 43 , 44 (https://imagej.net/SciView)	Visualize large 3D datasets, integrates ImageJ ops and ImageJ mesh to mesh data	Kyle Harrington, Ulrik Günther, Tobias Pietzsch
Script editor (https://imagej.net/Script_Editor)	Script editor and interpreter for SciJava scripting languages within ImageJ and Fiji	Johannes Schindelin, Curtis Rueden, Albert Cardona
Trainable Weka Segmentation 45 , 46 (https://imagej.net/Trainable_Weka_Segmentation)	Plugin that combines a collection of machine learning algorithms with a set of selected image features to produce pixel‐classifications	Ignacio Arganda‐Carreras, Johannes Schindelin, Matthias Arzt
TrakEM2 47 (https://imagej.net/TrakEM2)	Plugin for morphological data mining, three‐dimensional modeling and image stitching, registration, editing, and annotation	Albert Cardona, Stephan Saalfeld

The ImageJ ecosystem lends utility to a wide range of users with varying programming expertise: from the bench biologist with little‐to‐no programming experience taking advantage of the prepackaged plugins, to power users who customize their image analysis pipelines with scripts that automate entire workflows within ImageJ. One of the most unique and attractive elements of ImageJ is how easily customizable and adaptable it is even for novice users. Users with no formal programming experience can record actions using the Macro Recorder ²³ by performing them manually in ImageJ. From there, with the Script Editor, users can expand their code snippets into scripts that can be easily shared with others. On the software engineering side, the modular structure of ImageJ2 enables programmers to extend ImageJ in a wide variety of ways, including the ability to use components of ImageJ as building blocks for their own software tools and to build bridges with other platforms. ²³ , ²⁴

While ImageJ has been utilized predominantly by biologists working in a research setting, ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ it has also served users from diverse nonbiological research fields, including geologists ⁵² and astronomers. ⁵³ As of July 2020, a simple PubMed search of the term “ImageJ” yields 2,626 results that represent publications within biology, ¹¹ but also art, ⁵⁴ , ⁵⁵ dentistry, ²⁰ , ⁵⁶ medicine, ¹⁸ , ²¹ and geology. ⁵² Even historians have used ImageJ, for example, to reconstruct inner woodworm holes to study the interior state of precious medieval sculptures. ⁵⁵ Thus, the ImageJ ecosystem offers highly valuable, inclusive, and versatile image analysis capabilities across many disciplines.

Rather than being controlled by a single group of developers, development is community‐driven; users and developers interact to best optimize their code for a particular problem. The ImageJ wiki (https://imagej.net) serves as the main information repository for both users and developers. Additionally, the recently introduced Scientific Community Image Forum (https://forum.image.sc) serves as a discussion forum and a source of helpful advice for many open‐source toolkits, including ImageJ and Fiji. ⁵⁷ Virtually everything in the ImageJ ecosystem can be found in freely‐accessible GitHub repositories, facilitating contributions and collaborative development. Individuals may modify any particular component of the ImageJ ecosystem, be it core library or particular plugin, and publish these changes to GitHub where other users and developers review the new code, provide feedback, and incorporate useful additions into the ecosystem. ⁵⁸ The ImageJ wiki, the forum, and GitHub (https://github.com/imagej and https://github.com/fiji) are the three pillars on which the ImageJ ecosystem and its diverse, vibrant user and developer community rests. Together they allow for maintenance, growth, and adaptation of the ImageJ ecosystem to the changing image analysis needs of the scientific community.

This review emphasizes new and ongoing developments within the ImageJ ecosystem, and particularly within Fiji, that are useful for biologists working with advanced imaging techniques; they include machine learning with TensorFlow, ⁵⁹ the hosting of deep learning models with DeepImageJ, ³⁶ and even the training of TensorFlow‐based artificial neural networks from within Fiji with CSBDeep. ⁶⁰ , ⁶¹ We will also cover tasks such as tracking millions of cells in light sheet microscopy recordings of animal development with Mastodon ⁴¹ and MaMuT, ⁴⁰ and 3‐dimensional (3D) image data visualization with ClearVolume ³⁴ and SciView. ⁴² , ⁴³ These plugins were selected to highlight the diversity of the ImageJ ecosystem. In this review, we also aim to explain some of the latest advancements in the ImageJ ecosystem, revealing how this set of open source tools is adapting to the ever‐changing requirements in the fast‐paced field of bioimage analysis.

2. DISCUSSION

ImageJ's capabilities range from common tasks such as opening images of various formats, annotating and processing images, and executing simple workflows on images, to advanced projects involving visualizing and analyzing large image data and implementing machine learning algorithms. The following sections address common tools used in ImageJ and review frequently‐used image analysis techniques, as well as ways in which users can extend ImageJ and customize it through macros and scripting. We then highlight some of the current efforts in Fiji to handle large and complex datasets and the ongoing efforts to implement machine and deep learning approaches. Finally, we introduce the newer, more advanced features of ImageJ2, including the ImageJ Ops algorithm framework.

2.1. Opening and annotating images

While ImageJ is a useful image analysis tool for research in the life sciences, it also serves a broader audience as a basic image viewer and annotation tool. Bio‐Formats, included with the Fiji distribution of ImageJ, allows users to open images saved in a broad variety of file formats. ³³ This ability to open almost any image generated by a microscope often makes ImageJ the tool of choice for the “first‐look” at acquired image data. Brightness and contrast adjustment, image resizing, and application of lookup tables (LUTs) are common tasks that users frequently apply to freshly opened images. Next, the users may want to apply simple transformations, such as rotating the images or cropping, as well as adding various annotations to highlight a specific region of interest (ROI) or structures present in the images. Users may directly inspect pixel intensities, as well as the signal intensity within ROIs, by generating histograms. Most bioimage analysis tools offer a similar basic functionality, yet users often turn to ImageJ as it can open a particular format or because they are familiar with ImageJ's graphical user interface.

The ability to work with image stacks is one of the core ImageJ functionalities, deeply ingrained into the architecture of ImageJ1 and one of the factors in its longtime use. Stacks of images obtained at various depths for a specific field‐of‐view can be collapsed into a single image by a variety of approaches, most commonly by selecting the highest pixel value along the z direction of the stack, the so‐called Maximum Intensity Projection. This is just one example of the functionality of ImageJ in manipulating image stacks. It is particularly useful for fluorescence imaging of 3‐dimensional (3D) specimens to capture the fluorescence signal at various depths and focal planes.

For scientific publication and presentations, it is imperative to include information about the real size of the structures that are being displayed. The users of ImageJ can either import the size of pixels in microns from the image metadata or set the scale manually if this information is not available. This facilitates direct measurement of the size of structures present in the image. Additionally, users can use the pixel size data to place scale bars in the images for presentations or publications.

2.2. Common image analysis techniques: Segmentation, tracking, and registration

ImageJ provides users the ability to segment, ⁶² track particles, ⁶³ and register ⁶⁴ their datasets. Segmentation, that is, object detection/delineation, allows biologists to computationally separate certain regions in their images, such as the total area occupied by cells in brightfield microscopy images. ⁶² Particle tracking algorithms, that is, identifying and linking objects across multiple images over time, can, for example, assess mitochondrial transport movements in cortical and hippocampal neurons. ⁶³ Image registration, that is, transforming different sets of image data into one coordinate system, is crucial for generating high‐resolution stacks of thick 3D fluorescence microscopy data where the sample may be moved during acquisition to accommodate sectioning; one such example is serial imaging of the Drosophila brain where accurate stitching and registration is needed. ¹⁵ , ⁶⁴ , ⁶⁵

More advanced techniques and tools are available for segmentation, including Trainable WEKA Segmentation (https://imagej.net/Trainable_Weka_Segmentation), which is an ImageJ plugin that leverages machine learning provided by the open WEKA library to segment previously unseen image data. ⁴⁴ The Trainable WEKA Segmentation plugin allows users to train classifiers simply by “tracing” labels on selected pixels in the training images. ⁴⁶ The Fiji plugin Labkit (https://imagej.net/Labkit), also implementing pixel classification functionality via the WEKA library, additionally offers a user‐friendly and efficient way to label large 2D and 3D image datasets thanks to improvements in architecture and scalability in the BigDataViewer ³⁰ and the SciJava ⁶⁶ component collection.

Segmenting objects is the first step in tracking analysis, which allows researchers to follow the segmented objects' movements over time. ²⁹ , ⁶⁷ , ⁶⁸ Objects or structures of interest are separated from the background signal and then can be tracked frame‐to‐frame. Researchers are typically interested in tracking particular labelled biological entities, such as cells or sub‐cellular structures. TrackMate (https://imagej.net/TrackMate) is a Fiji plugin that facilitates manual, semi‐automated, or fully‐automated single particle tracking. ⁶⁹ This plugin can be utilized to track particles in 2D and 3D datasets and accommodates multichannel images. The user interface allows for separate execution of the image segmentation and particle‐linking steps. ²⁹ , ⁷⁰ Users are provided with algorithmic results for tracking, as well as quantitative results and plots. ²⁹ , ⁷⁰ TrackMate includes several tracking algorithms, such as LAP tracking, ⁶⁹ and experienced users can implement their own custom algorithms. The tracking data can be exported and analyzed in third party packages outside of ImageJ. The development of the Mastodon and MaMuT plugins, which allow for cell tracking and segmentation within large datasets, will be highlighted later in Section 2.4.

There are many advanced tools available in ImageJ for registration. TrakEM2 (https://imagej.net/TrakEM2) allows users to data mine their images based on morphological features and perform 3D modeling, stitching, registration, and annotation. ⁷¹ TrakEM2 was developed to segment neurons and reconstruct neuronal circuits by skeletonizing neuronal arbors based on nodes and directional edges that incorporate the degree of certainty in the continuity of edges. ⁷¹ Fijiyama (https://imagej.net/Fijiyama) is a 3D registration tool useful for multimodal time‐lapse imaging. ³⁷ , ⁷² When monitoring data changes over time, particularly if acquired on two different imaging systems, changes in position and orientation and registering different highlighted views of structures can be challenging if not impossible, yielding a compromised result. Fijiyama combines image stacks in a coherent manner that allows the user to explore their data using the ImageJ 3D Viewer. ³⁷ , ⁷³ Recent updates include support for registering 4D or 5D multichannel or multitime images, and future efforts will include new transformation models, as well as extended support for images where intensity may be modulated upon adjusting size. ⁷² The BigWarp tool (https://imagej.net/BigWarp) allows for interactive, manual landmark‐based image alignment. ³² The BigDataViewer ³⁰ (https://imagej.net/BigDataViewer) allows users to visualize and navigate their datasets, using landmark pair placement to display the effects of the warp. ³²

2.3. Scripting and plugins

Scripts allow users, even those with only basic programming knowledge, to automate repetitive image analysis tasks and package simple solutions into short, named programs that can be linked as shortcuts to the ImageJ toolbar or menu for quick and uncomplicated access. ⁷⁴ The most frequently used types of scripts in ImageJ are “macros”, a specific type of a script written in the ImageJ1 macro language. The ImageJ1 macro language was designed to be easy for new programmers to learn and debug, ⁷⁵ , ⁷⁶ but even users with no programming experience can use the Macro Recorder, which allows users to record a series of manually executed ImageJ commands directly into code ⁷⁶ (Figure 2). Such recorded sequences of events executed using the ImageJ GUI can then be adapted in the Script Editor into fully functional scripts ⁷⁵ , ⁷⁷ that can be saved and added to the ImageJ menu if desired. Detailed tutorials on how to record, implement, and publish ImageJ macros (https://imagej.net/Tutorials) are freely available on the ImageJ wiki. A testament to the popularity of this approach amongst ImageJ users are the thousands of available scripts listed on ImageJ1 website. ⁷⁶

FIGURE 2.

Example of how to generate simple code in ImageJ/Fiji. After opening (a) the Macro Recorder (Plugins › Macros › Record…), all subsequent manual commands are saved as code, in this case, the ImageJ1 macro language. In this example, (b) the “HeLa Cells” sample image available in Fiji was opened and channels split. Using the nuclei channel (C3), the image was filtered with a Gaussian Blur, and a mask was generated. A scale bar (10 μm) was also added. By calling the command Analyze Particles (Analyze › Analyze Particles…), nuclei were identified as regions of interest (ROIs) and were saved to (c) the ROI Manager and (d) a list of measurements appeared in the Results window. Measurements can be adjusted/selected via Set Measurements (Analyze › Set Measurements…). When a user clicks “Create” in the Macro Recorder, the generated code is added to the (e) Script Editor and, once there, can easily be edited, saved, and reused

In addition to the ImageJ1 macro language, Fiji introduced facilities to write scripts in the Script Editor in several popular scripting languages, including JavaScript, Clojure, BeanShell, Groovy, Python/Jython, Ruby/JRuby, R, and Scala. ⁷⁷ Not all available languages can be directly recorded via the Macro Recorder and therefore require intermediate programming skills. However, the programming capabilities in these established scripting languages are vastly more advanced compared to the ImageJ1 macro language. Since there are excellent online tutorials and templates (small script examples including all the syntactic requirements to start writing a specific solution) and many biologists have significant experience with scripting languages, this mechanism of generating workflows in the ImageJ ecosystem is increasingly prevalent.

2.4. Current efforts within ImageJ: Handling big data and specialized plugins for analysis

Challenges arise when opening large, multidimensional datasets using ImageJ1 (e.g., whole‐slide digital pathology images or stitched mosaics). Image datasets have gotten larger and larger, usually being multiple gigabytes, often reaching terabytes. Such data is difficult to open, cumbersome to view, annotate, or quantify using ImageJ's standard image window. Fiji's BigDataViewer (BDV) plugin has solved these problems for the ImageJ ecosystem by allowing users to navigate arbitrarily large image files from local and remote data sources. ³⁰ The key to BDV's performance on big image data is the use of hierarchical data structures (accompanied by suitable data formats such as HDF5 or N5) that store image data as nonoverlapping chunks (referred to as “blocks”) in a multiresolution pyramid. BDV requests and loads only the blocks that the user is currently interacting with and only at the appropriate resolution. Additionally, using ImgLib2's sophisticated caching infrastructure, BDV keeps track of recently used blocks to avoid loading the same data over and over again. BigDataViewer considers each imaging channel or acquisition angle from a multiparametric image as a separate source that users can manipulate independently. All this functionality relies on the powerful ImgLib2 library. ²⁶ After the original work on BDV, many additional functions have been added, with some still targeting visualization of large data and others concerning large data handling. Today, many research software developers in the ImageJ ecosystem use these functions, which were ultimately incorporated as core ImageJ components. The BigStitcher (https://imagej.net/BigStitcher) software package facilitates efficient alignment for multitile and multiangle imaging datasets acquired on light sheet, widefield, or confocal microscopes. ³¹ BigStitcher supports file sizes that can be as large as many terabytes of data, such as those often produced during light sheet microscopy acquisition. BigStitcher is integrated within BDV, and users can interactively display and process input images regardless of file size. ³¹ , ⁷⁸

The power of BDV is best demonstrated via tools such as Mastodon ⁴¹ and MaMuT ⁴⁰ that offer interactive segmentation and tracking of cells within large developmental bioimage datasets. In the context of studies of development, researchers are often interested in following many cells over time in 3D, quantifying their behaviors, and ultimately extracting developmental lineages through the division of cells as they build a tissue or entire organism over time. These tasks are often performed on multiterabyte 4D image acquisitions, and the challenges to display the data and run analysis on it in such contexts are formidable.

MaMuT (https://imagej.net/MaMuT) is an ImageJ plugin that allows efficient manual annotation of cellular lineages in the context of such big image datasets. ⁴⁰ It combines BDV's large data management capabilities with TrackMate's infrastructure for managing tracking results, described in Section 2.2. It also offers supervised semi‐automatic tracking of individual cells through the imaged volumes. In this way, it has been used to extract a complete developmental lineage of an appendage of the crustacean Parhyale hawaiensis, imaged with light sheet microscopy, answering long‐standing biological questions about appendage morphogenesis. ⁴⁰ In order to complete this work, it was necessary to isolate one appendage from the rest of the large, light sheet acquisition. MaMuT was unable to handle the many millions of cell detections in this complete dataset. To address such limitations, the original MaMuT authors are currently developing Mastodon (https://github.com/mastodon-sc/mastodon), a tool capable of dealing with developmental lineages that consist of tens of millions of cells. ⁴¹ Once finished, Mastodon will enable the analysis of automated tracking solutions generated by arbitrary tracking algorithms on large image data. As any such automated method will result in extremely large lineage trees that are full of errors, biologists will also be able to use Mastodon's highly optimized, ergonomic, and collaborative user interface to interactively correct such imperfect solutions. Figure 3 shows a glimpse of what Mastodon will be able to do by visualizing results of the automated tracking of nuclei in a 3D cartographic projection of a developing Tribolium castaneum, recorded by multiview light sheet microscopy.

FIGURE 3.

Using Mastodon to perform nuclei tracking in a 3D cartographic projection of a developing Tribolium castaneum using an image of the blastoderm shortly after its formation (left side: dorsal side; right side: ventral side). Each time‐point includes more than 500 tracked cells, visualized here in white fluorescence, amounting to more than a quarter of a million detections. The trajectory of each cell is represented by a line that fades from green to red backwards in time. The cartographic projection does not preserve lengths, and thus there is no uniform scale on the image. The cells on this image have a diameter that ranges from 6 μm (ventral) to 22 μm (dorsal). Images provided with the coordination of Drs. Jean‐Yves Tinevez (Institut Pasteur, Paris), Tobias Pietzsch and Vladimir Ulman, and courtesy of Dr. Akanksha Jain and Dr. Mette Handberg‐Thorsager, (Tomancak lab, MPI‐CBG, Dresden)

The challenging, large, and complex developmental lineaging data highlights the need for better, more performant image visualization tools in the ImageJ ecosystem. To that end, several recent updates within the ImageJ framework have been made or are currently under development to handle different analysis aspects of increasingly large datasets. To better visualize 3D datasets, the SciView plugin (https://imagej.net/SciView) is currently under development, with preview releases available through the ImageJ Updater. ⁴² , ⁴³ The SciView plugin integrates a combination of features from the more recent ImageJ2 libraries, including ImageJ Ops and ImageJ Mesh, allowing the user to interact with the image and mesh data. Figure 4 highlights an example of the SciView plugin for volume rendering and image segmentation.

FIGURE 4.

A SciView example visualization with synthetic data of a 3D image containing 20 hypothetical spherical regions of interest, representing artificial cells. The synthetic cell data is shown as a volume rendering. Regions of interest are shown as mesh‐based geometries generated by ImageJ Ops' implementation of the marching cubes algorithm. Color identifies nonoverlapping cells within the represented 100 x 100 x 100 pixels bounding box. Image courtesy of Dr. Kyle Harrington, Helmholtz Imaging Platform, Max Delbrueck Center for Molecular Medicine in the Helmholtz Association, Berlin, Germany and Dr. Ulrik Guenther, Center for Advanced Systems Understanding Görlitz/Center for Systems Biology, Dresden

The ultimate frontier is to make these types of tools, the advanced data visualization offered by SciView and the lineage tracing of MaMuT and Mastodon, work in real time during data acquisition. The ClearVolume plugin (https://image.net/ClearVolume) offers real‐time, GPU‐accelerated volume rendering for multichannel image processing, which is particularly useful for light sheet microscopy datasets. ³⁴ The Fiji integration of ClearVolume enables users to conveniently and very efficiently render and navigate through moderately‐sized, multichannel volumes and movies.

Once user data is compiled for visualization and navigation, signal quantification and measurement tools may be utilized for subsequent analyses, such as lifetime measurements. The fluorescence lifetime of a molecule is sensitive to the fluorophore's immediate environment, including pH, oxygen concentration, and proximity to other molecules exhibiting Förster resonance energy transfer. ⁷⁹ Fluorescence lifetime imaging microscopy (FLIM) and spectral lifetime imaging microscopy (SLIM) are two techniques that offer the ability to assess the state of a fluorescent molecule to gain insight into physiological changes such as metabolism. ⁸⁰ , ⁸¹ , ⁸²

The recently‐developed FLIMJ plugin (https://imagej.net/FLIMJ) reflects updated efforts to allow users easily to analyze FLIM data (Figure 5). ³⁸ FLIMJ accommodates single, double, or triple exponential fits with integrated noise models; it can fit individual pixels, entire images‐per‐pixel, or perform global analysis across the entire image. ³⁸ Fit methods include Levenberg–Marquardt Algorithm (LMA), Rapid Lifetime Determination (RLD), Bayes and others, and statistical models including Poisson, Gaussian, Maximum likelihood estimation (MLE), and others. Users are also able to select the image‐spatial binning size, thresholds, and the number of components. Batch processing can be performed to analyze datasets that include many FLIM images. ³⁸ , ³⁹ FLIMJ‐Ops plugs the FLIMlib library into ImageJ Ops (which will be covered in detail below), allowing users, via Jupyter notebooks, to invoke the Ops of FLIMJ‐Ops directly for processing.

FIGURE 5.

An example of the FLIMJ plugin. (a) The FLIMJ‐UI consists of six panels: two image panels, one fitting, one residual, one parameter, and one export panel. The fitting panel shows one pixel's intensity decay curve (yellow bubbles) and the fit‐curve (red). Fluorescence lifetime imaging analysis is then performed to analyze the (b) autofluorescence lifetime signal from mammalian HeLa cells as the input dataset. (c) The fitting of the same dataset can also be done from direct FLIMJ‐Ops functions. A minimalistic script can run any available fit‐function of choice. Image courtesy of Dr. Jenu Chacko, Laboratory for Optical and Computational Instrumentation, Center for Quantitative Cell Imaging, University of Wisconsin‐Madison

2.5. Machine learning in ImageJ

Machine learning applies artificial intelligence to train computer systems to automate processes and learn from experience to improve models without explicit programming. ⁸³ Deep learning, the sub‐field of machine learning that trains artificial neural networks (ANNs) on a body of labeled training data to solve various data analysis tasks, is currently one of the most potent techniques in the arena of artificial intelligence. ⁸³ Computer vision seeks to equip computers with high‐levels of understanding to automate tasks that human vision can perform. ⁸⁴ These techniques are often applied to very large, multivariable datasets that emphasize improving predictive accuracy, and now machine learning libraries are available in ImageJ via new plugins and libraries such as ImageJ‐OpenCV (https://github.com/imagej/imagej-opencv) and ImageJ‐TensorFlow ⁸⁵ (https://github.com/imagej/imagej-tensorflow).

2.5.1. OpenCV

OpenCV (https://imagej.net/OpenCV) is a library comprised of thousands of computer vision and machine learning algorithms. ⁸⁶ The OpenCV library supports image processing, feature detection, object detection, machine learning, and video analysis. Dominguez et al. created a package that combines ImageJ with OpenCV to leverage the robust computer vision algorithms of OpenCV with a simple ImageJ GUI for defining regions of interest. ²¹ The ImageJ‐OpenCV library helps to convert between ImgLib2 and OpenCV data using the SciJava framework. ⁸⁷

2.5.2. TensorFlow and ImageJ‐TensorFlow

Deep learning is used in various ways in the ImageJ ecosystem, enabled by developments such as the open‐source ImageJ‐TensorFlow library ⁸⁵ (https://github.com/imagej/imagej-tensorflow). ImageJ‐TensorFlow translates between ImageJ images (backed by ImgLib2) and TensorFlow tensors, ⁵⁹ thereby enabling ImageJ commands and scripts to utilize TensorFlow's data‐flow model that maps computations onto a broad scale of hardware platforms. This might include inference measurements made on all kinds of devices, including mobile devices, single machines, or large systems with thousands of graphics processing units (GPUs). ⁵⁹ A more in‐depth description of the computer workflow and models is discussed by Abadi et al. ⁵⁹

TensorFlow has been used in medicine to interface with DICOM images to classify normal versus Parkinson's disease groups, ⁸⁸ to detect early and advanced glaucoma on fundus photographs, ⁸⁹ and for segmenting boundaries on multiphoton images of skin cells. ⁹⁰ Most interestingly, the integration of TensorFlow via ImageJ‐TensorFlow not only enables the execution of pretrained deep learning models, but it can also enable the training of ANNs from within the ImageJ ecosystem, for example, within CSBDeep (see below).

2.5.3. DeepImageJ

In recent years, deep learning has proven its superiority to other approaches in tasks such as object detection and classification, image segmentation, face recognition, and many more. ⁹¹ Deep learning solutions in the microscopy image domain include image restoration with Content‐Aware Image Restoration (CARE) ³⁵ (https://imagej.net/CARE), denoising datasets with Noise2Void ⁹² (https://imagej.net/N2V), object and pattern recognition, and segmentation. ⁹³

DeepImageJ (https://deepimagej.github.io/deepimagej/) is a plugin that enables access to deep learning models from within Fiji or ImageJ (Figure 6). ³⁶ Users need no experience with deep learning but can simply download available deep learning models from a central repository and embed them directly into ImageJ or Fiji. ³⁶ The execution of the pretrained deep learning model makes use of the Java bindings of TensorFlow. ³⁶ Since many models require certain pre‐ and postprocessing of the image data, DeepImageJ also hosts and serves those in the form of ImageJ macros or scripts. Gómez‐de‐Mariscal and García‐López‐de‐Haro et al. show several examples of data processing using DeepImageJ, including recovering fluorescence isotropy from cells, ⁹⁴ virtually staining unlabeled cells with a hematoxylin and eosin filter with VirtualStain, ⁹⁵ and segmenting HeLa cells with the U‐Net feature. ⁹⁶

FIGURE 6.

An example of using DeepImageJ to segment HeLa cells in a differential interface contrast (DIC) microscopy image using a trained U‐Net. Panels (a–c) depict the ImageJ steps visualized by the user and (d–g) show the automatically executed steps defined by the model developer and executed inside of DeepImageJ. (a) The image in this figure is a DIC microscopy image of HeLa cells (scale bar =15 μm). (b) The DeepImageJ Run GUI allows the selection of a suitable model from the user's local machine and of any preprocessing and postprocessing ImageJ macros given within the selected trained model (in this case, “U‐Net HeLa Cell Segmentation”). DeepImageJ displays some important information about the model's configuration such as the patch size (input size admitted by the model's architecture) and the overlap size (size of the image borders that need to be discarded in the output image due to the size of the convolutions in the network). (c) The final result of the segmentation using a trained U‐Net is shown. (d) The DIC image intensity values were normalized (preprocessing step). (e) A dialog box displays the time and memory information about the U‐Net inference process. (f) The output of the U‐Net is shown, and (g) Watershed segmentation was used to obtain uniquely labeled cells (postprocessing). The raw DIC input image source is from the Cell Tracking Challenge (http://celltrackingchallenge.net/2d‐datasets/), captured by Dr. G. van Cappellen, Erasmus Medical Center, Rotterdam, The Netherlands. Figure courtesy of Dr. Daniel Sage, School of Engineering (STI), Institut de Microtechnique (IMT), and Laboratoire d'Imagerie Biomédicale (BIG), Ecole Polytechnique Fédérale de Lausanne

2.5.4. CSBDeep

CSBDeep (https://imagej.net/CSBDeep and https://csbdeep.bioimagecomputing.com/) is an open‐source library with extensive deep learning functionality in Python and Java (Fiji). Originally developed for CARE purposes, it is now also used in a plethora of other research and software projects that incorporates denoising tools like Noise2Void, ⁹² DenoiSeg ⁹⁷ (https://imagej.net/DenoiSeg) and others. ⁶¹ Initially, Weigert et al. showed how fluorescence microscopy images could be restored from images with 60 times fewer photons acquired, showing how resolution could be enhanced even after under‐sampling the images. ³⁵ Their work also demonstrated how to resolve granular structures smaller than the diffraction limit at 20‐times‐higher frame rates. ³⁵

Today, CSBDeep has become the basis for many other useful applications. ⁶¹ , ⁹² , ⁹⁷ , ⁹⁸ Additionally, the Fiji‐integrated Java parts of CSBDeep are now capable (via the use of ImageJ‐TensorFlow) to not only run saved deep learning models, but even train such models as Noise2Void and DenoiSeg. ⁹⁷ It is important to note here that all existing code is open and fully available to others. The intention of the developers of CSBDeep is to enable the wider developer community to deploy their deep learning solutions to the ImageJ ecosystem, which would be of great utility for all users.

2.5.5. Summary of plugin and library contributions in ImageJ ecosystem

From navigating large datasets to handling multiview and multidimensional imaging data to implementing machine learning and deep learning algorithms, ImageJ has responded to the feedback of researchers and risen to meet their data handling needs. Table I includes some key plugins and tools that we have highlighted in this review. All items in the table are available in Fiji and most are currently ImageJ‐compatible or plan to be. The following section introduces how changes to the ImageJ software framework via ImageJ2 enhance the handling of more complex datasets and facilitate interoperability (Table I).

2.6. ImageJ2 and ImageJ ops

Developments in modern microscopy and imaging are a driving force for change to the ImageJ framework. As multimodal and multidimensional microscopy have evolved, so too have the needs of ImageJ users. ImageJ2 (https://imagej.net/ImageJ2) is a major effort to redesign ImageJ's foundation to accommodate new imaging methods, including polarized light microscopy, ⁹⁹ light sheet microscopy, ¹⁰⁰ large stitched datasets, and fluorescence lifetime imaging microscopy. ³⁸ Because it was designed to reflect the needs of the user community and allow for collaborations to foster open development, ImageJ2 improves upon ImageJ's: extensibility, that is, its potential to be extended and enhanced with new features; modularity, that is, the extent to which its functionality is accessible as reusable building blocks rather than only a single monolithic application; interoperability, that is, its ability to be used in combination with other programs and workflows; and generality and scalability, that is, its capability to manage more complex data types and larger image file sizes. ²³ , ²⁷

ImageJ2 provides mechanisms for accessing ImageJ functionality from other software tools, including from Python (https://github.com/imagej/pyimagej) and Jupyter Notebook (https://github.com/imagej/imagej-notebook), from Node.js (https://github.com/imagej/imagej-node), from MATLAB (https://doi.org/10.1093/bioinformatics/btw681), from the command line (https://imagej.net/Scripting_Headless), and via web services (https://imagej.net/Server). Such interoperability allows for collaboration between bioimage analysis tools to facilitate improved automation, where the respective strengths of each software package may be effectively leveraged.

ImageJ2 is more than just a desktop application; it is a collection of reusable libraries and plugins built upon the foundational layer of SciJava (https://imagej.net/SciJava), an umbrella for software components which are broader than only image processing. ⁶⁶ The SciJava layer houses ImageJ's new plugin framework and application container (https://imagej.net/SciJava_Common) as well as the SciJava Script Editor (https://imagej.net/Script_Editor) and script language plugins. SciJava components also provide random‐access data input/output interfaces and plugins, expression parsing, common user interface elements, and native library utilities, to name a few examples.

ImageJ2 comes bundled with the ImageJ Updater to ensure that users have seamless access to the latest updates, as well as to new plugins when additional Update Sites are enabled. ²⁷ ImageJ2 has extensible support for scientific image file types thanks to the SCIFIO (Scientific Image Format Input and Output) library ⁴² and is able to handle N‐dimensional data through the integration of the ImgLib2 multidimensional image processing library that supports numeric and nonnumeric data types. ²⁶ The Bio‐Formats library, ³³ which is included with Fiji, plugs into SCIFIO, integrating its additional file formats more seamlessly into ImageJ.

The main library of ImageJ2 in terms of image processing algorithms is ImageJ Ops (https://imagej.net/Ops), which serves as an extensible framework for reusable operations. ²⁷ A primary goal of Ops is to make it easier and simpler for programmers to implement algorithms, such that they can be used as is from any SciJava‐compatible software project. ¹⁰¹ Like a function call in most programming languages, an operation is invoked by its name and arguments, but unlike a standard function call, Ops performs a “matching” process to determine the best function implementation to use based on the request. For example, for smaller kernel sizes, naïve convolution is faster than convolving via a Fast Fourier Transform (FFT), but FFT‐based convolution is a better choice as the kernels become larger. The Ops framework will pick the best approach based on the arguments given, in a manner enabling developers to extend ImageJ with additional implementations handling new cases or improving performance. For users, all routines available in ImageJ Ops are accessible in a consistent way from scripts, categorized by functionality into namespaces, deterministic in behavior, that is, always giving the same result for the same inputs, and usable without a graphical display for example, from the command line or on nodes of a compute cluster.

3. CONCLUSIONS

ImageJ has grown from a simple tool to analyze two dimensional images, into a widely utilized platform for modern biological image analysis. This growth is due to the tireless work of the creator of ImageJ1, Wayne Rasband, the contributions of the emergent and synergistic Fiji and ImageJ2 communities, and the vast contingent of users and developers adopting the ImageJ ecosystem for their work. It was developed with the intent to be adaptable by the users, for the users, and that tradition of help and accessibility continues through the ImageJ wiki, Scientific Community Image Forum, and GitHub repositories. Open‐source access to code repositories allows developers to adapt existing solutions to increasingly advanced image datasets, drawing from advances in the computer vision and machine learning fields. Cutting edge solutions are pushed through the ImageJ Updater to tens of thousands of users, who offer feedback directly to the developers at scale. As new ecosystems based on different programming paradigms and technologies emerge, unconstrained by decades of legacy code, the challenge for the ImageJ ecosystem is to remain relevant. This review highlights ongoing efforts within the ImageJ ecosystem to adapt to the emergent needs and challenges in modern biological image analysis. While the features and adaptability of the ImageJ infrastructure and the large numbers of users and developers invested in this software make it a powerful tool, the way forward will clearly be to continue building bridges to emerging platforms for the benefit of the larger scientific community.

AUTHOR CONTRIBUTIONS

Alexandra Schroeder: Writing‐original draft; writing‐review and editing. Ellen Dobson: Writing‐original draft; writing‐review and editing. Curtis Rueden: Writing‐original draft; writing‐review and editing. Pavel Tomancak: Writing‐original draft; writing‐review and editing. Florian Jug: Writing‐original draft; writing‐review and editing. Kevin Eliceiri: Conceptualization; resources; supervision; writing‐original draft; writing‐review and editing.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

Schroeder AB, Dobson ETA, Rueden CT, Tomancak P, Jug F, Eliceiri KW. The ImageJ ecosystem: Open‐source software for image visualization, processing, and analysis. Protein Science. 2021;30:234–249. 10.1002/pro.3993