Workflow and metrics for image quality control in large-scale high-content screens

Abstract

Automated microscopes have enabled the unprecedented collection of images at a rate that precludes visual inspection. Automated image analysis is required to identify interesting samples and extract quantitative information for high content screening (HCS). However, researchers are impeded by the lack of metrics and software tools to identify image-based aberrations that pollute data, limiting an experiment's quality. We have developed and validated approaches to identify those image acquisition artifacts that prevent optimal extraction of knowledge from high-throughput microscopy experiments. We have implemented these as a versatile, open-source toolbox of algorithms and metrics readily usable by biologists to improve data quality in a wide variety of biological experiments.

Introduction

The advent of high-content screening (HCS) has had a tremendous impact on drug discovery and genome analysis. Sophisticated algorithms are integral to HCS, easing the process of automatically scoring images for phenotypes without visual inspection. The use of automated microscopy and image analysis has increased screening speed, objectivity and statistical power by multiplexing the extraction of complex quantitative information from each cell.

However, these advances have also generated the need for robust metrics and workflows for quality control (QC) of HCS images, typically to reduce the number of images that are spuriously scored as screening hits. Anomalies in image quality can pollute high-throughput microscopy data and in some cases even preclude certain types of experiments. Analysis of these datasets requires the researcher to explore and visualize gigabytes of image and extracted feature information; simple spreadsheet programs (e.g. Microsoft Excel) are inadequate for the task. Commercial automated systems do not package QC metrics and visualizations as a cohesive toolset readily accessible to users; researchers may resort to custom software solutions¹ or to visual exploration of hits to exclude obvious artifacts. Solutions are needed that package together data metrics and visualizations in a user-friendly software tool compatible with any automated microscope, to enable automatic detection of image aberrations that would otherwise corrupt the data and diminish the experimental results.

In this paper, we characterize several QC metrics for common artifacts that confound high-throughput imaging experiments. Our approach to HCS QC has developed over many years of image-based screens for complex phenotypes and is implemented in open-source software.

Methods and Materials

HCS microscopy images were acquired using an ImageXpress Micro automated cellular imaging system (Molecular Devices). 384-well plates containing U2OS cells stained with Hoechst 33342 and Alexa Fluor 594 phalloidin markers were imaged with an exposure of 15 and 1000 ms for Hoechst and phalloidin respectively, at 20× magnification and 2× binning. For each site, the optimal focus was found using laser auto-focusing on the Hoechst channel. The automated microscope was then programmed to collect a z-stack of 32 images (z = 0 at the optimal focal plane, 16 images above the focal plane, 16 below) with 2 μm between slices. For additional validation, we used images from a large phenotypic screen obtained with the protocol above, along with the addition of a Mitotracker mitochondrial stain and the acquisition of 4 sites per well. Even though focusing was performed automatically, two plates contained extensive focus artifacts and were discarded from the screen; for these plates, each image was manually annotated as in- or out-of-focus.

Synthetic HCS cellular images (25 per test condition) were created using SIMCEP simulation software² run in MATLAB (Mathworks, Inc). The cells were simulated for a given cell count with a clustering probability of 25% and a CCD noise variance of 0.0001. The synthetic image dimensions were the same as those of the microscopy images, and the simulated nuclei and cell areas were matched to the corresponding average areas obtained from an HCS plate. Increasing focus blur was simulated by applying Gaussian filters of increasing size to the images.

Metrics directed towards two types of artifacts were examined: focal blurring and image saturation. The various focus metrics were evaluated using the multi-well microscopy image z-stacks and the artificially-blurred synthetic images with the following approach:

• Computing QC thresholds on a training image set: An expert (one of the co-authors, MAB) defined ground truth by selecting the z-offset corresponding to the boundary between acceptable in-focus images and unacceptable out-of-focus images, as determined visually. A training set of images spanning all z-offsets was created from a random selection of wells, and the optimal threshold for each focus metric was chosen based on this training set. True positives (TP) and false positives (FP) were defined as images correctly or incorrectly detected as in-focus according to ground truth, respectively; a false negative (FN) is an image incorrectly detected as out of-focus. For each metric, the per-image F-score was calculated for each bin, defined as

  $$\text{F-score}=2prl(p+r)$$ (1)

  where precision (p) and recall (r) are defined as

  $$p=\text{TP}∕\left(\text{TP+FP}\right)$$ (2)

  $$r=\text{TP}∕\left(\text{TP+FN}\right)$$ (3)

  Because the F-score is a measure of detection accuracy, we define the QC threshold for a metric as the optimal boundary between in- and out-of-focus images; it is computed as the value corresponding to the maximum F-score.

• Evaluating metric performance on a test image set: The F-scores were calculated for each metric from the remaining wells using the QC thresholds established above.

A similar approach was taken with the saturation artifact metrics, with the exception that ground truth comprised a set of artifact-laden images manually selected from five plates from another HCS microscopy experiment. Training and test sets were created from a random selection of images from the ground truth set and from the remaining artifact-free images respectively.

Most quality control metric computation and cellular analysis was performed using a pipeline (a serial set of image analysis algorithms) constructed using version 2.0 of CellProfiler, a cell image analysis software package³. A MetaXpress (Molecular Devices) journal/macro was used to retrieve and export the instrument-acquired per-image signal/noise ratio. Exploration of image data was performed with CellProfiler Analyst. The open-source software packages CellProfiler and CellProfiler Analyst were created and are maintained by the Imaging Platform at the Broad Institute; both packages are freely downloadable from http://www.cellprofiler.org.

Results and Discussion

Characterization of QC Metrics

Common artifacts that confound image analysis algorithms include out-of-focus images, debris, image overexposure, fluorophore saturation and uneven field illumination, among others. Because these anomalies affect intensity, morphological, and textural measurements in a variety of ways, we have found that a single QC metric that captures all types of artifacts is not feasible. Therefore, multiple metrics were developed and applied tailored to the specific artifact. Here, we characterize metrics commonly used by our group.

QC Metrics for Out-of-Focus Images

The most common and problematic HCS artifacts are out-of-focus images acquired despite automated microscope autofocusing. Relative focus metrics have been studied extensively for optimal focal plane selection⁴, in which the goal is to select the best focal plane among many images of the same field of view. However, these methods are not directly applicable in the context of HCS QC, in which an absolute focus metric is needed to identify the focus quality of single images from thousands of fields of view.

Because image focus, signal/noise, and cell density commonly fluctuate across an HCS experiment, it is useful to test the candidate QC metric against similarly varying images to ensure reliability across a broad range of experiments⁵. In addition to informal testing in dozens of HCS experiments, we validated and characterized several candidate focus metrics using two sets of benchmark images (Figure 1A):

• We created a set of synthetic HCS images of cell populations with ground truth segmentation. We then blurred the images artificially and quantified the performance of metrics with respect to the amount of blur and the cell count per image. As expected, we observed that a typical quantitative measure such as cell count degrades with increasing blur (Figure 1B, left).

• Because synthetic cell images are unlikely to fully represent the actual experimental variation typical of an experiment, we also tested microscopy images of cell samples from a 384-well HCS plate. We imaged multiple fields of view per well, and multiple focal planes per field above and below the optimal focal plane. Applying larger blurring filters in the synthetic images is considered to be equivalent to increasing the distance from the optimal focal plane in the microscopy images, and both will be referred to in terms of F. Again, quantitative measures such as cell count (Figure 1B, right) and DNA content distribution (Figure 1C) of the cell population degrade with increasing focal blur.

We assessed the performance of four previously proposed metrics with respect to variations in focal blur F and cell count C. Our criteria for an optimal QC metric are (a) insensitivity to cell count C, since C typically varies widely in HCS; and (b) detection accuracy in the presence of focal blur F. The metrics are as follows:

• The Mean/STD is the ratio of the average image intensity to the standard deviation of the image intensity (or, inverse coefficient of variation). It is provided as a QC measure in the commercial MetaXpress system (defined as the Signal-to-Noise Ratio). The Mean/STD value decreases as C decreases or F increases (Figure 2A),

• The Focus Score is a normalized measure of the intensity variance within an image⁶ and has been recommended as the optimal method for autofocusing purposes⁴. Its value also decreases as C decreases or F increases (Figure 2B).

• The Image Correlation is a measure of the image spatial intensity distribution computed across sub-regions of an image⁷ and its value increases as F increases (Figure 2C). For a given spatial scale, the Image Correlation is insensitive to C.

• The Power Log-Log Slope (PLLS) evaluates the slope of the power spectrum density of the pixel intensities on a log-log scale⁸. The PLLS is relatively insensitive to C and decreases as F increases because high-frequency image components are lost as an image is blurred (Figure 2D).

Using F-score as a measure of ability to accurately distinguish in-focus from out-of-focus images for the two stains in both synthetic and microscopy images (Figure 2E,F), the highest-performing metric is the PLLS, despite some sensitivity at very low C. The Image Correlation metric performs well in terms of moderate F-score and insensitivity to C. However, selecting an appropriate spatial scale is important for the metric's performance because differences in the spatial scale capture various features: moderate scales capture the blurring of intracellular features better than small scales and larger scales are more likely to reflect intercellular confluence than focal blur. The Mean/STD and Focus Score metrics both have a mild dependence on C but their large variance for a given amount of focal blur makes them less reliable for HCS QC focus assessment.

Figure 1.

Analysis of focus metrics using synthetic and microscopy HCS image data. (A) Representative images as a function of focus blur amount F and cell count C. (B) Cell count accuracy deteriorates with increasing focal blur. (C) DNA content accuracy decreases with increasing focal blur for Hoechst microscopy images.

Figure 2.

(A–D) Metric performance for synthetic nuclei and microscopy Hoechst images. For the synthetic data, plots show specific cell counts, while quartiles of the cell count distribution are plotted for the microscopy data. Image Correlation values are shown for three spatial scales as indicated, where the number specifies the spatial scale in pixels for the synthetic images and in microns for the microscopy images. Errorbars indicate standard deviation. PLLS: Power Log-Log Slope. (E, F) Table of candidate QC metrics for (E) synthetic and (F) microscopy images, rank-ordered by F-score.

QC Metrics for Saturation Artifacts

Another screening irregularity often encountered relates to saturation artifacts: inappropriately bright regions of images, which can be caused by debris contamination, reagents aggregates, or inappropriate exposure/gain settings (Figure 3A). Saturated regions preclude accurate intensity measurements and can confuse cell identification even in unaffected image regions in extreme cases.

Figure 3.

(A) Examples of saturation artifacts. (B) Listing of candidate QC metrics for the microscopy Hoechst stains used, rank-ordered by F-score. STD: Standard deviation, MAD: Median absolute deviation.

To assess saturation artifacts, we typically use the Percent Maximal (PM) metric to compute the percentage of pixels at the maximum image intensity value; high PM values can identify saturated images as candidates for exclusion⁹. Other statistical measures related to image intensity have also been used for detecting saturation artifacts in microarray image data¹⁰.

Because saturation artifacts do not fall into well-defined archetypes and varies greatly in appearance, we relied upon our informal experience in dozens of HCS experiments as well as a data set of artifacts (40 images) that occurred by chance in five HCS plates. We quantified the artifact detection accuracy using the F-score to compare eight metrics: PM, the total, mean, median, standard deviation, maximum and median absolute deviation (MAD) of the image intensity, and the Otsu image threshold. Three forms of Otsu thresholding were evaluated: the standard two-class method and the three-class method with the middle class assigned to either the foreground or background. Our analysis indicates that PM detects saturation artifacts more robustly than the other candidate metrics (Figure 3B).

Design of the QC Workflow

Optimal software tailored to QC for image-based experiments requires linking numerical data back to the raw image data, flexible statistical analyses, and the ability to operate within the hierarchical data structure inherent in image-based data (i.e., an experiment is comprised of samples, a sample is tested in replicate wells, the wells contain a number of images, and each image contains hundreds of cells, each with hundreds of features measured).

A QC workflow should perform, at minimum, the following tasks:

• Calculation of a suite of metrics that represent the variety of the artifact types that screeners are likely to encounter during an experiment.

• Provision of tools for analysis and exploration of calculated metrics, in order to assist researchers unfamiliar with a particular data analysis task.

• Interaction with the QC metric data using interactive visualization tools promoting the direct coupling of the metrics to the image data, allowing the researcher to examine images and identify acceptable QC thresholds (i.e., boundaries between acceptable and unacceptable images).

• Gating of an image subset based on the selected QC thresholds, such that images within the gate are deemed acceptable for further analysis.

• Application of the gates to the images to obtain a list of acceptable images for export as input to a downstream pre-processing or analysis workflow.

• Saving the applicable plots and gates representing user decisions during a QC session for later use, either across multiple sessions for a single experiment or across multiple experiments.

In the course of work with a wide variety of image-based assays, we have gradually developed a workflow to accomplish these tasks.

Calculation of QC metrics in CellProfiler

We implemented the metrics described above in CellProfiler, allowing analysis modules to be assembled into an automated pipeline. The Measure Image Quality module includes the PLLS, Focus Score and Percent Maximal (PM), the Measure Texture module measures the Image Correlation at multiple spatial scales and the Measure Image Intensity module evaluates the mean, median, standard deviation, minimum and maximum image intensities. As described above, we recommend PLLS to detect blur and PM to detect saturation artifacts. For a small number of images, the measurements extracted by the QC pipeline may be viewed in Excel or a similar spreadsheet program. However, for the large number of images typically generated by an HCS experiment, measurements are best deposited into a database using the Export To Database module. In either case, the QC pipeline with the selected metrics can be saved for reuse in a later experiment. We have provided an example QC pipeline as Supplementary material, for use with a set of HCS images available for download at http://www.cellprofiler.org/published_pipelines.shtml.

In most cases, we run a CellProfiler QC pipeline to process all channels of all images in an experiment, followed by an illumination correction pipeline and finally an analysis pipeline to quantify the phenotype of interest. If artifacts are absent or negligible, the first two pipelines can be run simultaneously. If this is not the case, the results from the QC pipeline are used to assess which images should be omitted from the illumination correction and analysis pipelines.

Evaluation of QC metrics in CellProfiler Analyst

We use CellProfiler Analyst for the interactive exploration and analysis of the resulting multidimensional data¹¹. In the following example, we describe an image quality workflow using CellProfiler Analyst where selected QC metrics are visualized and inspected and QC thresholds are applied to a subset of plates from an HCS phenotypic screen in which the image data contained extensive focusing artifacts.

When the screen involves less than a few dozen plates or the user wants to obtain a qualitative estimation of the imaging quality, the screener can select these plates for review using the PlateViewer tool (Figure 4A&B). This tool requires that the database of measurements contains plate, row and column identifiers, which can be extracted automatically in CellProfiler. The PlateViewer may be used to visualize any of the computed metrics in a familiar plate layout, with each well color-coded to represent an aggregate per-well measurement. For example, examining the image PM reveals a well with a high mean per-well value (well N17, Figure 4A). The PlateViewer can also display image thumbnails (Figure 4B); viewing these thumbnails may be useful for directly visualizing obvious image artifacts. Additionally, the images acquired for a selected well can be displayed as a montage; in this case, viewing the four images from well N17 shows that an image containing saturated debris is responsible for the aberrant QC measurement value.

Figure 4.

Screenshots of the PlateViewer and Histogram tools in CellProfiler Analyst. (A) PlateViewer displays the plate layout for a single 384-well HCS plate; per-well averages of the Percent Maximal (PM) QC measures for the Hoechst channel are shown in color. Numerical data is shown by hovering the pointer over the well. (B) Plate layout showing image thumbnails. Clicking on a well brings up an image montage of all images available for the well, allowing visual confirmation of an artifact detected by the PM metric. (C) Histogram of PLLS from the Hoechst channel, showing gating (dotted line) and menu options. (D) Representative images from the numbered clusters in (C). Panels were cropped from the original images for visibility. (E) The list of images corresponding to the in-focus images gated in (C). (F) HCS screening example in which focus artifacts strongly affect quantitative measurement of DNA content. Two peaks are expected, as seen in the in-focus (blue) data: the left peak is 2N DNA content and the right peak is 4N DNA content. In the out-of-focus images (green), DNA content is not accurately quantified. (G) Using a well-chosen QC threshold improves the overall result. Note that the red line almost completely covers the green line, showing that the QC results are virtually indistinguishable to that of manual annotation of the in-focus images.

For screens involving a large number of plates, visual inspection for QC assessment is not recommended and a more quantitative approach is required. We typically begin our assessment using the Histogram or ScatterPlot tool for visualization of image-level measurements, proceeding to individual images for manual review as needed. The approach is to plot the metric appropriate to the specific QC problem and then identify QC thresholds that exclude unacceptable images. Figure 4C shows a histogram of Hoechst PLLS values, revealing a bimodal distribution, in contrast to the expected unimodal distribution if all images were in-focus. An example image from each peak is shown in Figure 4D, revealing marked differences in focus.

In some instances, an image may contain an unforeseen artifact not captured by the metrics described here or the recommended QC metrics may be confounded by the phenotype of interest. In both cases, it is not straightforward to select a single QC threshold for a given metric. Machine learning provides a powerful means to allow the computer to identify unusable images based on a few examples¹². CellProfiler Analyst contains a machine learning tool that has been validated for phenotypic scoring at the per-cell level¹³. This same framework can be extended for per-image machine learning-based QC scoring by referring CellProfiler Analyst to a table of image measurements.

Applying QC thresholds to exclude images

The final workflow task is to apply the desired QC metrics and thresholds to exclude unacceptable images from further analysis. For example, in Figure 4C, a screener using the Histogram tool would notice a distinct clustering between in- and out-of-focus images, and would like to remove the out-of-focus images from further analysis. This can be accomplished in CellProfiler Analyst by manually delineating a horizontal sub-section of the plot containing the sub-population of acceptable images (a procedure known as “gating”). Because the more positive PLLS values correspond to in-focus images, we manually select a threshold that falls at the trough between the two peaks (Figure 4C). CellProfiler Analyst allows for the creation of multiple gates, which is useful if a combination of metrics are needed to produce the final QC result. The screener may label a new gate with a descriptive name (e.g., “InFocus”) for future reference. The same can be done using the Scatter Plot and Density Plot visualization tools. Once a gate is defined, it may be used to filter any of the measurements visualized by the tools in CellProfiler Analyst. Using the automatic assessment of blurriness, we found that the chosen PLLS threshold of –2.27 resulted in the exclusion of 910 (29.7%) of 3064 images as blurry, while 2154 images (70.3%) were deemed in-focus. Comparing this result to the manual annotations of this same image set, 81 (8.9%) of the 910 “blurry” images were false positives and in fact in-focus, and 28 (1.3%) of the 2154 “in-focus” images were false negatives and in fact were manually annotated as blurry. This yielded an F-score of 0.975. By excluding blurry images based on the automated QC analysis, we can see improvement in a quantitative measure heavily affected by blurring such as DNA content (Figure 4F,G).

It is often sufficient to mark acceptable images with a new “flag” column in the experiment's database to assist in downstream data analysis. For illumination correction calculations, however, aberrant images must be excluded and the illumination correction pipeline must be re-run because illumination correction is not robust against poor image quality¹⁴. Typically, it is also desirable to exclude the images from the phenotypic analysis pipeline. To assist in these goals, one can produce a list of images filtered by any or all of the gates in CellProfiler Analyst; this list consists of the file and path names of the accepted images along with links to the original images for inspection (Figure 4E). This list can then be exported as a file of comma-separated values (CSV) for use in a CellProfiler (re-)analysis pipeline; this pipeline would contain the Load Data module which loads the contents of the CSV file. Alternately, the full list of images can be exported, but with binary QC flags for each image. In this case, the analysis pipeline will also contain a Flag Image module to filter images based on the flags.

Saving QC gates and workspaces

The choice of suitable QC metrics and their appropriate thresholds is typically determined empirically, depending on the context of the assay and taking into account the expected variation between experiments. Once a set of measures and thresholds have been found to be reliable, the user may save the gates as part of a configuration file that can later be loaded by CellProfiler Analyst, and/or as a workflow file in order to apply the same tools, metrics and gates to a new experiment. The workflow file saves the information necessary to recreate all the display tools and settings that were open during a prior QC review session. In this way, a user can easily reproduce and display a set of consistently-used measurements for each new screen. We have provided an example configuration (or “properties”) file and a workflow file as Supplementary material; these files will load and display some of the more commonly used metrics from a database of measurements and the image set mentioned above available from http://www.cellprofiler.org/published_pipelines.shtml.

Conclusions

We characterized several QC metrics to exclude low-quality images in an HCS experiment. Our goal in this work was to provide a suite of recommended HCS QC metrics to remove aberrant images from consideration, all in the context of an open-source workflow.

All candidate QC measures are computed by the Measure Image Quality module and included as part of a QC pipeline in CellProfiler. Once the QC measures have been made, CellProfiler Analyst is used to interact with the data using a variety of visualization tools. Gating is used to create custom thresholds for each QC metric of interest, and the gates are combined to yield the final list of acceptable images for downstream processing. All pipelines, workflows and gates can be saved for use in later experiments. By integrating these data metrics and visualizations in a user-friendly software tool, biologists can automatically detect image aberrations that would otherwise corrupt the data and diminish the quality of their experimental results.