ImageJ and CellProfiler: Complements in Open Source Bioimage Analysis

Abstract

ImageJ and CellProfiler have long been leading open source platforms in the field of bioimage analysis. ImageJ’s traditional strength is in single image processing and investigation, while CellProfiler is designed for building large-scale, modular analysis pipelines. Although many image analysis problems can be well-solved with one or the other, using these two platforms together in a single workflow can be powerful. Here we share two pipelines demonstrating mechanisms for productively and conveniently integrating ImageJ and CellProfiler for 1) studying cell morphology and migration via tracking and 2) advanced stitching techniques for handling large, tiled image sets to improve segmentation. No single platform can provide all the key and most efficient functionality needed for all studies. While both programs can be and are often used separately, these pipelines demonstrate the benefits of using them together for image analysis workflows. ImageJ and CellProfiler are both committed to interoperability between their platforms with ongoing development to improve how both are leveraged from the other.

Basic Protocol 1:

Studying Cell Morphology and Cell Migration in Timelapse Datasets using TrackMate (Fiji) and CellProfiler

Basic Protocol 2:

Creating Whole Plate Montages to Easily Assess Adaptability of Segmentation Parameters

INTRODUCTION:

ImageJ and CellProfiler are two of the most widely used open source image analysis platforms in science. Over the years, both projects have developed independently, cementing themselves in the field of image analysis. Cited in over 10,000 publications per year, ImageJ, as well as its plugins-included distribution known as Fiji, enable in-depth image investigations; both are ideally suited for interactive, single-image processing and are popular choices for both novice and experienced developers alike to add functionality (Schneider et al., 2012; Rueden et al., 2017; Schindelin et al., 2012). With over 10,000 publications to date, CellProfiler’s strength is in project-building, where complex pipelines of individual modules can be built to automate the processing of large experiments of images (Carpenter et al., 2006; Kamentsky et al., 2011; McQuin et al., 2018). The projects also differ in their respective languages, with ImageJ being Java-based and CellProfiler Python-based.

The ImageJ and CellProfiler teams have worked together for many years, not just on mutually useful software development, but also in developing the bioimage analysis community into its healthy and cooperative state today. For example, the teams advocate together for open source technologies, usability, and interoperability (Carpenter et al., 2012; Kamentsky et al., 2011). The two software projects joined forces to build the Scientific Community Image Forum (https://forum.image.sc/) (Rueden et al., 2019), an online resource for the worldwide scientific community to learn about image analysis from each other. Currently, 40 open source imaging software projects, termed ‘Community Partners’, use the Forum as their primary interaction with users. This Forum has increased communication among developers of various software packages and provided users access to a wide breadth of experts in various open source platforms. The Forum has blossomed into a solid community, with over 14,000 users worldwide, encouraging open science and reproducible research by advocating for open tools and their interoperability.

Recently, the ImageJ and CellProfiler teams at the University of Wisconsin-Madison and the Broad Institute have come together formally to pursue their common open source development and outreach goals by creating the Center for Open Bioimage Analysis (COBA) (https://openbioimageanalysis.org/) via a P41 grant from the National Institutes of Health National Institute of General Medical Sciences (NIH NIGMS). COBA’s mission is to serve the cell biology community by catalyzing the entire open source bioimage analysis software field to develop and implement cutting-edge methods, such as deep learning, making them more easily accessible. No single piece of software can handle the full diversity of bioimaging needs. Making tools interoperable and demonstrating how to link them together is therefore a priority for COBA.

Here, we aim to highlight the strengths and weaknesses of ImageJ and CellProfiler and showcase how these programs can be used in conjunction, and therefore more powerfully, to build effective analysis workflows. We also give two use cases that illustrate the complementary strengths of the two programs. The first protocol involves integrating the Fiji plugin TrackMate, for semi-automated cell tracking, into a CellProfiler pipeline for further analysis and measurements of cell morphology and migration in timelapse datasets. The second protocol involves using Fiji’s fast image manipulation and image exploration tools, specifically Hyperstack and Make Montage tools, to help verify that the segmentation results of a CellProfiler pipeline are robust across a number of different image phenotypes. In both cases, the strengths of each open source platform are used via a single workflow that best addresses researchers’ varying needs. We also describe future plans to improve upon these two projects’ interoperability. Overall, ImageJ and CellProfiler plan to continue their development as separate programs, while constructively and collaboratively improving researchers’ experiences and accessibility to leading image analysis techniques.

BASIC PROTOCOL 1

STUDYING CELL MORPHOLOGY AND CELL MIGRATION IN TIMELAPSE DATASETS USING TRACKMATE (FIJI) AND CELLPROFILER

Introductory paragraph:

This protocol uses Fiji and CellProfiler to track the movement of cells and additionally analyze the shape of the cells at each time point. Using this protocol it is possible to study the connection between variations in cell morphology and migration over time. The protocol consists of two main steps: tracking the cells using the TrackMate plugin in Fiji and then analyzing cell morphology using CellProfiler. With TrackMate it is possible to track the cells and also easily visually inspect and correct any errors in the tracking result (Tinevez et al., 2017). CellProfiler is powerful for detecting and quantifying cell morphology.

The workflow uses 2-channel timelapse videos as input: one channel contains a cytoplasmic marker (Figure 1a), the other channel labelled nuclei (Figure 1b). The positions of the nuclei over time are tracked using TrackMate. TrackMate returns the center of the tracked object (visualized as small dots in Figure 1c), but no information on the size and shape of a tracked object. Using the module “IdentifySecondaryObjects” in CellProfiler, we can use the position of the tracked nuclei as seeds for a ‘seeded’ watershed, which identifies the outline of the cells by starting at the seed positions and following the intensity profiles of the cytoplasmic marker image. We can then automatically extract a large number of size and shape measurements using CellProfiler and link them back to the tracking results from TrackMate.

Figure 1.

Protocol 1 uses 2-channel timelapse videos as input: one channel contains a cytoplasmic marker (a) and the other channel labelled nuclei (b). TrackMate finds positions of the nuclei (c) and tracks them over time, while CellProfiler can take the TrackMate positions as input and identify cell outlines (d); scale bar 25μm.

Materials:

• ❏ CellProfiler 4.0.7+ (cloned from https://github.com/CellProfiler/CellProfiler or downloaded from https://cellprofiler.org/releases)

• ❏ Current version of Fiji (downloaded from https://imagej.net/Fiji.html#Downloads)

• ❏ The following files are all freely available via Zenodo (https://zenodo.org/record/4317505#.X9OvOflKhPY):

  • Fiji IJMacro script: create_LabelledMasks.ijm

  • CellProfiler pipeline: TrackMate_CellProfiler.cppipe

  • Matlab script: Plot_per_cell.m

  • Saved, manually-curated TrackMate project: crop_1_60_ManualCuration.xml

  • Saved Spots Results Table of manually-curated TrackMate project: “Spots in tracks statistics.csv”

  • CellProfiler pipeline output: cp_output.zip

  • Example dataset: crop_1_60.tif (a subset of image data from (Shafqat-Abbasi et al., 2016))

  • Stack of labelled masks, output from TrackMate: crop_1_60_TrackMate_LabelledMask.tif

Protocol Steps:

Using TrackMate in Fiji: Preparations

• 1Load your data into Fiji by drag & drop or using “File -> Open”. The example dataset, crop_1_60.tif.

  For tracking we only need a time-lapse series of the fluorescently-labelled nuclei. However, it is easier to inspect the tracking result in a hyperstack that contains both channels (nuclei and cytoplasmic marker) over time. If the two markers are saved into two separate files, open both files in Fiji and combine them by running “Image -> Color -> Merge Channels”.

Track Cell Nuclei using TrackMate

• 2Start TrackMate: “Plugins -> Tracking -> TrackMate”.
• 3Double-check the calibration settings (Pixel width, Pixel height, Time interval) and correct if necessary. Click Next.
• 4Select the LoG detector.

  The LoG detector applies a Laplacian of Gaussian (LoG) filter before detecting local intensity maxima. The detector works best for bright round objects. Nuclei are not perfectly round, however, the LoG detector still tracks nuclei well in many cases. Errors in the detection, and therefore the tracking, can be expected in cell lines with many non-round nuclei. Using CellProfiler’s IdentifyPrimaryObjects module to identify and export identified nuclei may produce better results in such cases. See CellProfiler’s Help for that module for details.

• 5In the GUI interface, select the channel with nuclei for object segmentation. Enter an estimated blob diameter and a threshold. Click Preview for inspecting the settings. Settings for the example dataset: segmented channel = channel 1, estimated blob diameter = 20; threshold = 0.5.

  The estimated blob diameter should reflect the expected nuclei diameter. If the entered estimated blob diameter is set too low, the detector will find more than one detection per nucleus. If the blob diameter is too high, difficulties can arise when two cells are close to each other. You can estimate the diameter of a representative nucleus using the line tool in the main toolbar of Fiji. Setting the threshold is a balance between detecting background signals (threshold too low) and missing nuclei (threshold too high).

• 6Wait until nuclei are detected. Click Next.
• 7Initial thresholding; Click Next.

  The initial thresholding can be adapted, however in most cases this is not needed when the threshold in the LoG detector step was chosen properly.

• 8Wait for the HyperStack Displayer to load. Click Next.
• 9Set filters on spots. For the example data, it is not necessary to add a filter here. Click Next.

  Adding filters can help if there are too many erroneous spots detected. For adding a filter, press the green ‘plus’ symbol, and select a measure parameter to filter on in the drop-down menu.

• 10Select a tracker. Select the Simple LAP tracker.

  The Simple LAP tracker does not account for cell division.

• 11When choosing the Simple LAP tracker, the following parameters have to be set: Linking max distance, Gap-closing max distance, Gap-closing max frame gap. Settings for the example dataset: linking max distance = 25 microns, gap-closing max distance = 25 micron, gap-closing max frame gap = 2. Click Next.

  Linking max distance is the maximum distance the nuclei are expected to move from frame-to-frame. To define a good value, draw a line using the line tool in the main toolbar of Fiji and move one end of the line onto the center of one nucleus. Activate the next frame and drag the other end of the line to the new position of the nucleus. Then measure the length of the line using “Analyze -> Measure”.

  Tracks can have gaps, e.g. when cells leave the field-of-view and then return or when the detection fails in a frame. Gap-closing max distance is the maximum distance an object can move within a gap to still be considered belonging to the same track. Gap-closing max frame gap is the number of frames allowed for a gap.

• 12Wait for the Tracker to be executed. Click Next.
• 13Set filters on tracks. If needed, add filters by pressing the green ‘plus’ symbol. For the example data, no filters were used.

  The choice of track filters depends on the dataset. For example, in a dataset where all cells move, using Track displacement might help to filter out bright particles that stick to the surface and are detected. Number of gaps can be useful to filter out tracks of low detection quality. Be careful to not bias the data by filtering out tracks.

Save the Analysis and Export the Tracking Result

• 14Click Save to the analysis as an xml-file.

  It is important to save the xml-file for documentation of all tracking parameters. After saving, click Resume.

• 15Click Analysis. Save the table “Spots in tracks statistics” as a csv-file using “File -> Save As…”.

  Automatically, TrackMate will open 3 results windows: one window for measurements for each detected nuclei (spots), one window for the links from spot-to-spot, and one window for entire tracks. For further analysis in this protocol, only the file “Spots in tracks statistics.csv” is needed.

Tune and Manually Correct the Tracking Result

A good tracking result is essential for downstream analyses. Try to achieve good tracking results by adjusting the parameters for detection and track-linking (steps 6, 8, 10, 14). Read more about the parameters and settings of TrackMate online (https://imagej.net/TrackMate).

It is advisable to adjust the lookup-table of both channels (nuclei and cytoplasm) for best possible visual inspection of tracks. Do this using “Image -> Color -> Channels Tool”. Toggle between “Composite” and “Gray Scale” visualization when inspecting the tracking result.

Manual correction should be as avoided much as possible, since it is time-consuming and non-reproducible. In many cases, however, some manual corrections of the tracking results are unavoidable. TrackMate allows users to add and delete detections, and modify their position. Also, several tracks can be linked to one, and one track can be split into several individual tracks. Shortcuts support the user doing this efficiently. Read the documentation about how to manually curate tracking results (https://imagej.net/Manual_editing_of_tracks_using_TrackMate). When following this protocol for subsequent cell morphology analysis, the following corrections are crucial:

• Avoid gaps in tracks. Add a detection manually, and link it to a track to fill the gap.

• Dividing cells: the Simple LAP tracker gives one daughter cell the track-ID of the mother cell and the other a new track ID. This is useful because for this workflow it is not possible that two cells share the same track ID. Depending on the scope of the analysis, it can make sense to manually split the track after division such that both daughter cells receive a new track ID and are treated individually from the mother cell.

After correcting the tracks, don’t forget to save the xml-file under a new name and also click Analysis again to obtain and save a new “Spots in tracks statistics.csv” results file.

Only a few manual corrections had to be done in the example dataset using the parameters as specified in the protocol. The corrections consisted of closing gaps, adding undetected frames in the end of a track, and splitting dividing cells. Load the corrected xml-file, crop_1_60_ManualCuration.xml, and inspect the corrected tracking result for the example dataset. The corresponding results table listing the spots of the manually-corrected data is available as “Spots in tracks statistics.csv”.

Run create_LabelledMasks.ijm

The TrackMate results table “Spots in tracks statistics.csv” contains a list of all spots (here nuclei detections) and gives their x,y position at specific time points, t. It also lists to which track spots belong by listing the Track ID. The script, create_LabelledMasks.ijm, creates a new timelapse stack where for each time point. The detections are represented as dots with a gray value encoding the TrackID. The script is written in the IJ macro language.

• 16Open the timelapse file used for tracking and the “Spots in tracks” results table output from TrackMate.
• 17Open create_labelledMasks.ijm in Fiji by drag & drop. The script opens within the Script Editor. Click Run.
• 18Input dialog window. The macro needs to know the names of the timelapse file and the “Spots in tracks” results table. Adjust the defaults, if needed. Click OK.
• 19Select the directory where the macro will save the output. Click Select.
• 20The macro saves a stack of labelled masks into the chosen output directory. The name of this file is NameOfTimeLapseFile_TrackMate_LabelledMask.tif.

The labelled mask corresponding to the corrected tracking results of the example dataset is available online (https://zenodo.org/record/4317505#.X9OvOflKhPY).

Run CellProfiler

• 21Open CellProfiler and import the pipeline TrackMate_CellProfiler.cppipe by dragging to the part of the screen that says “Drop a pipeline file here” or via “File -> Import -> Pipeline from File…”.
• 22To save the output of the pipeline after running, set the Default Output Folder using “Output Settings -> Default Output Folder”
• 23Module Images: Drag and Drop the original timelapse file and the stack of labelled masks that were output from TrackMate into the files list. In the example dataset, these files are named crop_1_60.tif and crop_1_60_TrackMate_LabelledMask.tif.
• 24Module Metadata: Click “Extract metadata” and “Update”. Check the listed files; each row represents a frame of the original timelapse movie or a labeled mask from TrackMate.

  For the example dataset, there will be 180 rows in total. crop_1_60.tif consists of a total 120 rows representing the 60 time points, and for each time point “T”, there are two channels “C” (0, 1). crop_1_60_TrackMate_LabelledMask will be listed as 60 rows, reflecting each time point of the stack since the LabelledMask has only one channel.

• 25Module NamesAndTypes: Click “Update”. For each time point there should be 3 types of images, labelled as “cytoplasm_marker”, “nuclei_marker”, and “trackMate_result”. The pipeline assumes that the nuclei are in channel 1 and the cytoplasm marker is in channel 2. If you have a reversed order, change to “Metadata Does Have C matching 1” for the “nuclei_marker” images and “Metadata Does Have C matching 0” for the “cytoplasm_marker” images.
• 26Choose “Windows -> Hide All Windows on Run”.
• 27Run “Analyze Images”.
• 28Check the output in the Default Output Folder.

The output consists of a control image for each time point which shows the nuclei marker (magenta), the cell marker (green), the outlines of the cells as detected by CellProfiler (magenta), and the TrackID. Note that the CellProfiler TrackID = TrackMate TrackID+1. The pipeline also saves four .csv-files: cells.csv, Experiment.csv, Image.csv, and trackMate_detection.csv, with cells.csv containing all the important information for this workflow. Cells.csv lists many parameters quantifying the size and shape of the cells, as exported by the MeasureSizeShape module of CellProfiler. Read more about the details of the measurement in the module’s Help. It also contains the measurement “Intensity_MaxIntensity_trackMate_result_rescaled”, which reflects the CellProfiler TrackID (= TrackMate TrackID+1).

BASIC PROTOCOL 2

CREATING WHOLE PLATE MONTAGES TO EASILY ASSESS ADAPTABILITY OF SEGMENTATION PARAMETERS

Introductory paragraph:

CellProfiler is highly adaptable; researchers can create image analysis pipelines for large datasets and tune segmentation parameters to perform well across many different perturbations and phenotypes. There is a major challenge, however: for datasets containing large numbers of image phenotypes, it can be extremely time consuming to check segmentation parameters in enough fields to ensure that they reliably segment images of all phenotypes present in the set. This poses a particular challenge when some phenotypes are rare, and the screener does not know which treatment(s) are likely to produce these rare phenotypes. Missing a rare phenotype at the segmentation-checking-stage can lead to poor segmentation of this phenotype, potentially causing at least one of several undesirable outcomes: a) a treatment that did produce a real change, but the change is not captured in the segmentation and a true screening “hit” is identified as a “miss”, b) a poor segmentation that leads to the screener thinking a “hit” changes one aspect of the cell (size) when it truly changes another (intensity), causing the screeners to incorrectly pursue the wrong mechanism of action, c) a screener only later realizing that their “hits” are due to segmentation errors and needing to rerun dozens or hundreds of plates worth of segmentation and measurement results, costing both human and computer time. All of these are worth being avoided where possible, but in practice having a screener hand-check tens of thousands of segmentation results one-by-one is tedious and doing it quickly may cause them to miss the rare phenotypes for which they were so diligently searching.

It is possible to use a workflow that allows easy checking of segmentations en masse. It begins by running a CellProfiler pipeline that, after manually selecting segmentation parameters for a subset of sample images (Figure 2), is designed to export one segmentation image per well of a multiwell plate, followed by a Fiji script that automatically stitches these into a “pseudo-plate” image for easy review using Fiji’s Hyperstack and Make Montage tools (Figure 3). While this does not guarantee finding all rare phenotypes, since the screener is only visualizing the results of one field-per-well (and typically more than one field is acquired in each well), it creates a more acceptable tradeoff between the hands-on time required by the image analysis specialist and the likelihood that most rare phenotypes will be viewed and have their segmentation assessed. In theory, a CellProfiler-only version of this workflow can be created using the CellProfiler Tile module, but the CellProfiler-Fiji hybrid solution is orders of magnitude faster to execute and higher quality, as it allows more customized viewing of the output plate montages in the end; in Fiji, brightness and contrast can be adjusted for the whole plate, individual channels can be toggled on and off, the lookup tables of individual channels can be adjusted, and so on.

Figure 2.

An example for Protocol 2 of primary (left) and secondary (right) object segmentations of a particular site in CellProfiler. These images show reasonable segmentation. Image axes are pixels. Image resolution is .656 μm/pixel.

Figure 3.

For Protocol 2, a whole plate stitch (a) with area to be inset selected; scale bar 250μm. Inset of the whole plate stitch covering wells A01-C03 (b); scale bar 50μm. In both images, magenta = DAPI, green = Phalloidin + WGA, cyan = nuclear outlines, and white = cell outlines.

The CellProfiler section of the protocol here assumes that the dataset contains two channels, one a nuclear marker and one a whole-cell marker, but this approach, and the downstream Fiji script, is adaptable to any number of channels and/or objects; information on how to customize the pipeline to your exact needs is provided in the notes.

Materials:

• ❏ CellProfiler 4.0.7+ (cloned from https://github.com/CellProfiler/CellProfiler or downloaded from https://cellprofiler.org/releases)

• ❏ Current version of Fiji (downloaded from https://imagej.net/Fiji.html#Downloads)

• ❏ BBBC025_AssayDevelopment.cppipe (provided as Supplementary material)

• ❏ CellCounts.ipynb (provided as Supplementary material)

• ❏ Make_fiji_montages.py (provided as Supplementary material)

• ❏ Test image dataset, BBBC025 (https://bbbc.broadinstitute.org/BBBC025)

Protocol steps:

Configure image sets in CellProfiler

• 1Open CellProfiler and import the CellProfiler .cppipe pipeline file by dragging to the part of the screen that says “Drop a pipeline file here”.
• 2Load in images by dragging a folder of images to the part of the screen that says “Drop files and folders here”.
• 3Configure the Metadata module to extract metadata from the file and/or folder names; at a minimum, information about the plate name, well name, and site should be extracted for each image.

  This information is required to make sure the pipeline processes a single site for each well and that it correctly outputs the images into a separate folder for each plate.

• 4Configure the NamesAndType module to recognize the channels in your image set.

  NamesAndTypes is currently set up for two input channels (one corresponding to nuclei, and one corresponding to cells), but to customize to your own data you can use anywhere between one and hundreds of channels; see the module help for more information. NamesAndTypes is currently set up to use Site 1 from each well; if you would prefer a different site, you can change the “Metadata Does Have Site matching 1” line for each channel to use whichever site you like.

Perform initial assay development in CellProfiler

• 5Open CellProfiler’s Test Mode by clicking the “Start test mode” button at the bottom left corner of the program.

  CellProfiler can be run in two modes: TestMode, where one image is assessed at a time, step-by-step, interactively, and AnalysisMode, where all images are run sequentially. While you are testing your settings, you will want to use TestMode; once your configuration is complete, you will use AnalysisMode.

• 6Hit the “step” button twice to run the IdentifyPrimaryObjects and IdentifySecondaryObjects modules on your initial dataset to visualize your initial nucleus and cell segmentation, respectively.

  CellProfiler can be set up to analyze as many types of objects as you would like; in general, add more IdentifyPrimaryObjects modules to identify objects purely from a given image channel, and IdentifySecondaryObjects modules to identify objects based on a given channel plus a “seed object”. For example, one could identify nuclei in the DNA stain channel using IdentifyPrimaryObjects, then use IdentifySecondaryObjects to expand those nuclei to find cell borders in a channel where cell bodies are stained. See the in-program Help for each module (buttons marked with “?”), as well as the “Critical Parameters” section for links to more resources. If you do add/subtract/change objects in the pipeline, you will need to perform the optional step 11 in this protocol.

• 7If needed, adjust the parameters in IdentifyPrimaryObjects and/or IdentifySecondaryObjects to improve segmentation quality on your initial image set. Click “step” to execute any changes you make.

  The “?” button next to each setting gives information on what the setting does, as well as advice on adjusting it. See the “Critical Parameters” section for links to more comprehensive resources on adjusting these modules.

• 8Choose another image at random by going to the “Test” menu and selecting “Choose random image set”.

  If, instead of selecting an image set at random, you wish to navigate to a particular image, you may do so by going to the “Test” menu and then selecting “Choose Image Set”; a list of all of the loaded images and their metadata values will be populated. You can sort by whichever metadata value you like; select the image you want to test and then click “Ok” to load that image into TestMode.

• 9Repeat steps 6–8 until your segmentation parameters look suitable on many consecutive image sets. This may take many iterations. Reasonable segmentation examples are shown in Figure 2.

  Save your work intermittently by going to File -> SaveProject to avoid losing your work if a crash should occur.

Prepare CellProfiler to run assay development on plate(s) of data

• 10Exit Test Mode by clicking the “Exit Test Mode” button.
• 11(Optional) If needed, adjust your OverlayOutlines and SaveImages settings.

  This is necessary if you have adjusted the number of channels, names of channels, number of objects, or names of objects; generally, you need one OverlayOutlines module per object, and one SaveImages module for each image and object to be assessed. See the Troubleshooting section for more information if you have trouble with this step.

• 12Prevent the opening of any windows during the final run by selecting the “Windows” menu then selecting the option “Hide All Windows On Run”.

  This reduces processing time; you can keep individual windows open to monitor if you prefer.

• 13Select the “Output Settings” button and set the Default Output Folder where you want the output image files to be saved.

  You do not need to adjust the Default Input Folder in this pipeline. The Images module contains the information about image file location and there are no other inputs.

• 14Save the final copy of your project file, containing the image and pipeline information, by going to File -> Save Project.
• 15Click “Analyze Images” to analyze all images.

Create plate montages

• 16Open Fiji, and open the Fiji script, make_fiji_montages.py (provided as Supplementary material), by dragging-and-dropping the script onto the main Fiji toolbar.

  Alternately, to make this script permanently accessible from within Fiji, rename the script such that it starts with an underscore (“_”) and put it in your Fiji installtion’s “Plugins” folder; upon rebooting Fiji the script can be accessed in Fiji’s “Plugins” menu for as long as the file is in this location.

• 17Execute the script by clicking “Run”.

  If you are executing this script from the plugins menu, simply click on the name of the plugin in the plugins menu.

• 18Answer the script’s prompts to provide the folder where your CellProfiler-produced images were saved (i.e. the CellProfiler output folder that contains the folder of output images and the .csv files), as well as the number of columns, rows, and the scale (1x size the original images, 0.5x, etc) at which you want to save the output montage.
• 19Allow the script to execute. “Command finished” will appear in the Fiji bar when it has completed.

Review plate montage images

• 20When processing completes, you may open the finished montages in Fiji.
• 21Examine the montages (Figure 3) for all plates to check for appropriate segmentation in all wells. Note any plates and/or wells you think are particularly poor performing.

  To adjust the zoom of your montage, use the “Image -> Zoom” menu; to toggle colors on and off use the “Image -> Color -> ChannelsTool” menu-item-tool; to adjust the brightness of each channel use the “Image -> Adjust -> Color Balance” menu-item-tool; to adjust the displayed color of each channel, make sure that channel is selected in Fiji using the channel slider at the bottom of the montage and then select a new colormap in the “Image -> Lookup Tables” menu.

• 22When all plates have been examined, assess if you think the segmentation errors you have seen warrant adjustment of your segmentation pipeline. If so, repeat steps 5–21 as needed, using the notes you made in step 21 to guide your manual image selections as you run step 8.

  Be sure to check some images that worked well (plates and wells that were not on your list), as well as ones that worked poorly, to make sure you are not sacrificing segmentation of most images to fix things for only few, rare phenotypes. Achieving perfect segmentation is typically impossible, so your goal is simply to maximize the proportion of samples with suitably decent segmentation results.

• 23When you are satisfied with your segmentation parameters, you may finalize your CellProfiler pipeline to prepare it for full-scale analysis by a) removing the site filter in your NamesAndTypes module, b) removing all modules after the initial segmentation modules, c) adding any measurements you need for your phenotypes, and d) saving your project with a new name using the “File -> Save Project As” function in CellProfiler.

Sample Data

The sample pipeline and script are designed to be run on image set BBBC025 (https://bbbc.broadinstitute.org/BBBC025) from the Broad Bioimage Benchmark Collection (Ljosa et al., 2012). The pipeline minimally requires the Hoechst and PhGolgi channels from any single plate, but was optimized on plates 37983+38002; the additional channels of this experiment can easily be added as described in the CellProfiler sections of the protocol.

COMMENTARY

BACKGROUND INFORMATION:

ImageJ has been a mainstay in the image analysis community since its inception in 1987 by Wayne Rasband at the National Institutes of Health (NIH). Originally called NIH Image, it was developed with the intention to be easily and openly shared and extended via code contributed by the scientific community at large. Rasband set from the start the idea that this program would have third-party contributors that could customize and develop their own analysis components, making this a user-driven open source project. Sun Microsystems released a Java programming language that could run on any operating system in 1995, which would allow programmers to ‘write once, run anywhere’. This is what led to the transition of NIH Image, originally written in Pascal, to Java and the first release of ImageJ (Schneider et al., 2012).

The driving design behind NIH Image and ImageJ was to keep things simple, in particular the interface, which is minimalist and has remained essentially the same to this day. However, this interface ‘hides’ the true power and extensibility behind this platform. Because of the developer community orientation of ImageJ, it had to be made easily extensible by outsiders, as opposed to a centralized development model. Rasband achieved this through the use of macros and plugins. Macros are short, customizable scripts that allow researchers to automate their analysis workflows using existing features and commands within the software. Plugins are more advanced, modular software elements that extend the functionality of the software. Plugins were either integrated into ImageJ if Rasband saw their global utility or were available for download from third-party sites (Schneider et al., 2012). This decentralized development is what has led to one of ImageJ’s greatest strengths as an image analysis platform: the ImageJ community of users and developers.

ImageJ has, to this day, remained a community-driven open source project. The development of Fiji, a distribution of ImageJ primarily focused on biological-image analysis which began in 2007, allows for the formal sharing of new algorithms with end users through an integrated update system (Schindelin et al., 2012). ImageJ and Fiji have continued down the original path set out by Rasband years ago: to remain a ‘federation’ of decentralized developers driven by community needs. The core ImageJ team, which includes developer leadership by the Eliceiri laboratory at the University of Wisconsin - Madison and many collaborators around the world, does develop and improve the architecture of the entire ImageJ ecosystem (Rueden et al., 2017). However, its main goal is to drive independent learning and development as opposed to limiting analysis support and code production in-house.

ImageJ was developed with the self-taught coder at the bench in mind. It is oriented towards single-image processing and is quite customizable via scripting, etc. Users can add their own functionality as well, focusing on their own research priorities for analysis solutions. Therefore, the target audience for ImageJ remains the bench biologist who is able to find existing plugins or macros for a given task and even someone who is willing to put in the time and effort needed to script and expand the functionality of the program. It also targets users who need a platform for initial image viewing, editing, and processing. ImageJ is an ideal platform for efficient image inspection and testing/building potential analysis workflows. Running large-scale analysis is possible with ImageJ via integrations with KNIME, an analytics platform for visually constructing nonlinear workflow graphs whose nodes perform steps such as data mining, machine learning, image processing, and text analysis (Berthold et al., 2008; Dietz and Berthold, 2016; Dietz et al., 2020; Fillbrunn et al., 2017).

CellProfiler, another mainstay platform in open source image analysis, was started in 2003 by Anne E. Carpenter and Thouis (Ray) Jones in the Sabatini Laboratory at the Whitehead Institute for Biomedical Research and Golland laboratory at MIT’s Computer Science & Artificial Intelligence Lab. CellProfiler was originally born out of necessity, as driven by the work of Carpenter, a cell biologist who saw a need for user-friendly software for more advanced, large-scale image analysis (Carpenter, 2020). CellProfiler was initially released in December 2005, originally developed in MATLAB (Carpenter et al., 2006); it was re-written in Python and released as CellProfiler 2.0 in 2010 (Kamentsky et al., 2011). Support for volumetric analysis of 3D image stacks and optional deep learning modules was added in the release of version 3.0 in October 2017 (McQuin et al., 2018). The recent release of version 4.0 in September 2020 included improvements to speed, usability, and utility, including a migration to Python 3 (https://carpenterlab.broadinstitute.org/blog/cellprofiler-40-release-improvements-speed-utility-and-usability). The CellProfiler project is currently actively maintained by the Carpenter lab at the Broad Institute of Harvard and MIT.

Unlike ImageJ, CellProfiler was specifically designed for larger scale experiments and building modular, reproducible analysis pipelines. Its modules’ clear inputs and outputs can be easily pipelined into extensive analysis workflows. Most of CellProfiler’s user base is unfamiliar with code, mainly biologists who aim for user-friendly, ‘point and click’ solutions for their analysis needs. Users rely on the CellProfiler team in the Carpenter lab for their analysis and development expertise; their top-down design results in heavy curation of CellProfiler features and extensive, consistent documentation that readily orients beginners to making their own analysis pipelines without a need for coding.

ImageJ and CellProfiler differ not only in their audiences and development style and organization, but also in technical features, with some algorithms and strategies being only available in one or the other, along with a language divide (Java versus Python), making it non trivial to exchange code directly. Enabling ImageJ and CellProfiler to interoperate began with a bridge to run ImageJ macros in the context of CellProfiler pipelines in CellProfiler 2.2.0 (Kamentsky et al., 2011), a functionality that has since been deprecated over the years due to maintenance burden. Because of their now formal commitment as a collaborative organization via COBA, ImageJ and CellProfiler have renewed their dedication to creating and maintaining bridges between the two platforms. As a result, a new CellProfiler module is under development in CellProfiler 4.0.8 (RunImageJMacro), which allows users to pass images and variables to an ImageJ macro from a CellProfiler pipeline and receive images back from ImageJ that they can use in the rest of their CellProfiler pipeline. This bridge is just the beginning of more advanced interoperability to come.

CRITICAL PARAMETERS:

Protocol 1

The critical parameters in TrackMate are the initial parameters for spot detection (step 5) and the linking parameters (step 11). The critical parameters in CellProfiler are the parameters of IdentifySecondaryObjects (step 7), since they define the detection of the cell borders.

Image data has to be suitable for tracking and segmentation: the nuclei- and cell-markers should have a high signal intensity compared to the background, but signals should not be saturated. High cell densities should be avoided, and the frame rate has to be adjusted such that the cells are still easy to link from frame-to-frame (i.e. minimal cell displacement from frame-to-frame).

Protocol 2

Segmentation accuracy is driven by adjusting the suite of settings in IdentifyPrimaryObjects and/or IdentifySecondaryObjects in parallel (protocols steps 6–8). The full description of how to adjust these parameters to any dataset the user may want is outside the scope of this manuscript; the CellProfiler manual (https://cellprofiler-manual.s3.amazonaws.com/CellProfiler-4.0.7/modules/objectprocessing.html) contains an explanation of all parameters and our written (https://github.com/CellProfiler/tutorials) and video (https://www.youtube.com/playlist?list=PLXSm9cHbSZBBy7JkChB32_e3lURUcT3RL) tutorials also cover these concepts in detail. It is not typically possible to find parameters that work perfectly on every cell in every image; the goal is to find parameters that treat all images fairly and work well for the vast majority of cells in each image. See our blog post (https://carpenterlab.broadinstitute.org/blog/when-to-say-good-enough) on assessing image quality and when a pipeline is “good enough”.

When running the Fiji script, adjusting the scale parameter in the script is essential to the usefulness of the stitched montages. At too small of a scale, the montage may be so downsampled that segmentation quality cannot be accurately assessed; at too large a scale, the montage may be too large for the user to open. It is reasonable to start at the default-provided parameter of 1, and adjust empirically based on the size of your original image, the number of channels, and the size of the objects whose segmentation you wish to assess.

TROUBLESHOOTING:

General

When troubleshooting integrated ImageJ and CellProfiler protocols such as those presented in this manuscript, the best location to discuss hurdles and gain potential solutions is the Scientific Community Image Forum (https://forum.image.sc/). Due to the variability and uniqueness of such integrated protocols, specific questions can be posted on the forum to gain insight from experts within the entire open source community, including the authors of this paper, who regularly frequent the forum.

Protocol 1

When observing cells over time and extracting measurements, there are typically two types of errors: errors in the tracks and errors in how the cells are segmented or outlined. Here we describe how to detect errors and what effect they may have on the final result. There are two straightforward ways to detect both segmentation and tracking errors: 1) look for outliers in the final result and 2) browse through and visually inspect the final output. The output from CellProfiler, including outlines of cytoplasms and TrackID-numbers from the tracking, can be easily re-opened and interactively viewed in Fiji, as described below. If outliers are detected in the final results, the same approach can be used, focusing on the cells and time frames corresponding to the outliers.

• 29. Run “File -> Import -> Image Sequence…” and select the first image of the series in the output folder. Fiji loads the output images into one stack.

• 30. rowse through the images and inspect whether outlines of cytoplasms have been sufficiently well-detected. If there are too many incorrect outlines, go back to CellProfiler and adjust the settings for identifying the cytoplasm in IdentifySecondaryObjects.

  For detecting tracking errors, focus on the TrackID number while browsing through the stack. If the number disappears in a frame, the track contains a gap. There is a problem in linking the nuclei if the number changes from one frame to another or if a number appears on top of another nucleus. Go back to TrackMate in Fiji to adjust settings to try to reduce tracking errors.

Changing the Settings of IdentifySecondaryObjects

• 31. To visually examine the result on a given time frame, click Start Test Mode and use “Test -> Choose Image Set” in the top menu to choose one of the time frames in which you detected incorrect segmentations. Note that the time frames start with Frame = 0 for the first frame in CellProfiler, but t=1 in Fiji.

• 32. Click on the grayed-out eye symbol next to IdentifySecondaryObjects to open the eye.

• 33. Click Run.

• 34. In the Module IdentifySecondaryObjects, check which settings work best for the test image.

• 35. Click Step, and inspect the output of IdentifySecondaryObjects.

• 36. Load other time frames of the stack using “Test -> Choose Image Set” to check whether the settings are generally applicable to the stack. Click Run to run the pipeline on the chosen test image.

• 37. Exit Test Mode, and re-run the analysis using Analyze Images.

Re-loading the tracking result in TrackMate and correcting for tracking errors

• 38. Run “Plugins -> Tracking -> Load a TrackMate file” to open the .xml-file saved. TrackMate will open the TrackMateGUI already at the step “Display option” and also the time-lapse movie used for tracking.

• 39. Correct the tracking result as described in this protocol under ‘Tune and Manually Correct the Tracking Result’.

• 40. Save the new .xml-file under a new name using the “Save” button of the TrackMate GUI.

• 41. Click Analysis, and save the results table “Spots in tracks statistics.csv”.

• 42. Re-run create_LabelledMasks.ijm as described under the section ‘Run create_LabelledMasks.ijm’ in this protocol.

Effect of errors

The effect of errors while tracking depends on the goal of the analysis. If the goal is to monitor an overall effect of e.g. adding a perturbant to the cell culture, mixing up two tracks may have little effect on the final result. However, if the actual behaviour of individual cells in a heterogeneous population is crucial, care should be taken to correct the TrackMate output.

Gaps in tracks can result in segmentation errors if the cell containing the tracking-gap is close to another one.

The CellProfiler pipeline may over- or under-estimate the shape, size, and intensity measurements of individual cells as a result of segmentation errors. This may happen if the cytoplasms become very extended and fragmented, meaning that long protrusions may partially fall below the intensity threshold in the segmentation step. To keep track of the potential effect of segmentation errors it may be a good idea to add a step to the pipeline where the total number and area of cytoplasmic objects that are above the intensity threshold are measured. Comparing this number to the total cell area will give a hint on cell fragmentation in large-scale experiments.

Protocol 2

Users should take care that when adjusting the number or names of channels used and/or the number or names of objects whose segmentation is to be tested that they adjust any necessary modules downstream of their changes. Modules that need to be adjusted will have a red “X” next to them, which will tell you the error if you hover over it; modules that do not need to be adjusted will have green checks next to them.

As a rule of thumb, the CellProfiler pipeline should include one SaveImages module for every raw image the user wants to see in their segmentation montage and one Identify(Primary/Secondary/Tertiary)Objects + one OverlayOutlines + one SaveImages module for each object whose segmentation needs to be examined. Users may include more modules as they like, such as an ExportToSpreadsheet module to export object counts or measurement modules to assess object properties, such as size or intensity, for troubleshooting. In order to correctly batch the data by plate, it is important that each SaveImages module uses the “Default Output Folder SubFolder” option, with the subfolder set to the plate metadata so that the Fiji script has the structure it requires to run.

The Fiji script assumes that all rows and columns are full and that the total number of images present must be divisible by the stated number of rows*columns; it will otherwise refuse to stitch. Users should therefore be careful that all the CellProfiler output data is complete before executing the Fiji script and that no other image files are present in the plate subfolders. No other subfolders (i.e. data subfolders) should be present in the folder containing the plate subfolders, but files not in subfolders are fine and will be ignored.

If the actual plate layout is such that not all rows and/or columns contain the same number of wells, the user can choose to load into CellProfiler only those rows/columns which do form a complete rectangle or can fill in missing data by duplicating and renaming existing image data to stand in for data from missing wells.

STATISTICAL ANALYSIS:

General

Data produced by the described protocols can also be exported via Windows Excel, Google Sheets, R, KNIME, etc. for further processing and analysis.; use the ExportToSpreadsheet module of CellProfiler (or ExportToDatabase if you prefer that format).

Protocol 1

Statistical analysis depends on the research question. Typical parameters for analyzing cell motility are mean square displacement, analysis of the turning angles between frames, and distribution of turning angles (Beltman et al., 2009). CellProfiler outputs a csv file, cells.csv, with a large number of measurements per cell and time point. It does not automatically output velocity, but as x and y coordinates are provided for each time frame, this velocity can easily be calculated. Using a short Matlab script, plot_per_cell.m, we calculated velocity and created a scatter plot of velocity, form factor, and area, where each dot represents a specific cell at a given time point (FIgure 4a). From this plot, we can see that cells with small area and small form factor have lower velocity while larger cells tend to have higher form factor and higher velocity. This could be statistically verified by measuring the coefficient of determination (R2) between e.g. per cell mean area and mean velocity. Next, we also plotted area per cell over time (FIgure 4b). This plot is useful for determining tracking and cell segmentation quality, as smooth variations over time should be expected. These types of plots are also useful when e.g. visualizing and quantifying changes over time, such as the effect of a perturbant added at a specific time point. For statistical analysis, mean and standard deviation in per cell change over time may be useful.

Figure 4.

The per-cell and per-time-frame measurements output by CellProfiler for Protocol 1 can be processed in many ways. A scatter plot (a) of velocity, form factor, and area, where each dot represents a specific cell at a given time point, shows the relationship between shape and motion, while (b) shows variation in area per cell over time. CellProfiler also provides cell IDs and outlines (c); scale bar 25μm.

UNDERSTANDING RESULTS:

Protocol 1

As described in the statistical analysis and Figure 4, there are a number of different ways the output data can be used. The cells.csv format is a simple comma-separated data file, where each row represents a cell in a given time point. The feature measurements extracted by CellProfiler are named in the top of each column of the data file, and many of them are self explanatory. For detailed descriptions, see the manual for the corresponding measurement moules in CellProfiler. As mentioned above, it is important to notice that it is the “Intensity_MaxIntensity_trackMate_result_rescaled” that reflects the TrackID from TrackMate. The corresponding ID numbers are shown on top of the individual cells as shown in Figure 4c.

Protocol 2

It is very difficult to assess segmentation quantitatively without laborious creation of a ground truth; however, in our sample datasets with our preferred segmentation pipeline we achieved a mean +/− standard deviation of 109 +/− 62.5 cells per image, distributed as per Figure 5. These data were analyzed in a Jupyter Notebook, CellCounts.ipynb, using pandas 1.1.4 and seaborn 0.11.0 (Kluyver et al., 2016; McKinney, 2010; Reback et al., 2020; Waskom and team, 2020), which can also be used to reproduce results of your own segmentation parameters.

Figure 5.

The cell counts from each image exported by CellProfiler’s ExportToSpreadsheet module for Protocol 2, as broken down by plate.

TIME CONSIDERATIONS:

Protocol 1

Processing of the example dataset takes only few seconds (detection and linking) in TrackMate and 2–3min in CellProfiler (system: Intel(R) Core(TM) i7–6700HQ CPU @ 2.60GHz, 2601 Mhz, 4 Core(s), 8 Logical Processor(s), 32 GB RAM). Manually inspecting the tracking results and correcting them can take up to several hours, and the time investment has to be considered. The decision of whether the time for manually correcting the tracking results is acceptable depends on both the dataset and the research question.

Protocol 2

The amount of time needed to complete the CellProfiler section will depend on the number of images to be run, the number of objects per image, and the number of CPUs available on which to run; the 768 fields of view from 2 sample plates of BBBC025, which contain on average 100–200 cells per image, took approximately 25 minutes to complete in CellProfiler 4.0.7 utilizing 4 of the 8 cores of a MacBook Pro (Intel Core i7 3.1 GHz, 16GB of RAM). A cluster can be utilized to speed processing of very large datasets; see this manual page (https://cellprofiler-manual.s3.amazonaws.com/CellProfiler-4.0.7/help/other_batch.html) and this GitHub page (https://github.com/CellProfiler/CellProfiler/wiki/Adapting-CellProfiler-to-a-LIMS-environment) for more information on running CellProfiler on a cluster.

The Fiji section will likewise somewhat depend on the size of the images and the number of channels the user would like to assess; for the sample data, which had a 1080×1080 pixel image size, 4 input channels (two image channels and two outline overlay channels), and 384 wells, execution took 2–3 minutes per plate on the same machine as the CellProfiler assessment.

FUTURE DIRECTIONS:

There is great value in using ImageJ and CellProfiler programs separately. However, we hope we have conveyed the strength in using them together in combined image analysis workflows to best harness their individual strengths. ImageJ and CellProfiler complement each other quite well, and given their commitment via COBA, there remain opportunities to improve existing bridges between them. One of the major hurdles has been the divide in languages for development between Java and Python. However, PyImageJ, a recently developed Python wrapper for ImageJ (https://github.com/imagej/pyimagej), allows for the mixing and matching of image processing routines from ImageJ, including plugins and scripts, with Python-side image processing (e.g. scikit-image, ITK, and OpenCV). Therefore, PyImageJ will allow for even more convenient use of ImageJ and CellProfiler platforms in the same workflow, providing a strong step towards improved, long-term interoperability.