Deep Learning Image Analysis of High-Throughput Toxicology Assay Images

Abstract

High-throughput chemical screening approaches often employ microscopy to capture photomicrographs from multi-well cell culture plates, generating thousands of images that require time-consuming human analysis. To automate this subjective and time-consuming manual process, we have developed a method that uses deep learning to automatically classify digital assay images. We have trained a convolutional neural network (CNN) to perform binary and multi-class classification. The binary classifier binned assay images into healthy (comparable to untreated controls) and altered (not comparable to untreated-control) classes with >98% accuracy; the multi-class classifier assigned “Healthy,” “Intermediate” and “Altered” labels to assay images with >95% accuracy. Our dataset comprised high-resolution assay images from primary human hepatocytes and undifferentiated (proliferating) and differentiated 2D cultures of HepaRG cells. In this study we have focused on testing and fine-tuning various CNN architectures, including ResNet 34, 50 and 101. To visualize regions in the images that the CNN model used for classification, we employed Class Activation Maps (CAM). This allowed us to better understand the inner workings of the neural network and led to additional optimizations of the algorithm. The results indicate a strong correspondence between dosage and classifier-predicted scores, suggesting that these scores might be useful in further characterizing benchmark dose. Together, these results clearly demonstrate that deep-learning based automated image classification of cell morphology changes upon chemical-induced stress can yield highly accurate and reproducible assessments of cytotoxicity across a variety of cell types.

Introduction

The interpretation of cellular morphology changes, in context with fundamental knowledge of anatomy and physiology, serves as a pillar of clinical diagnostic judgement. In the fields of toxicology and pharmacology, imaging is likewise increasingly recognized as an important component of in vitro screening. Imaging facilitates the characterization of complex cellular biology dynamics and allows researchers to assess integrated aspects of biological response to environmental chemical exposures^{1, 2}. However, only recently have high throughput imaging systems become widely available to capture photomicrographs and document cell morphology over time from each well of multi-well assay plates (e.g., 96- and 384-well plates). While these systems now have sufficient capacity to efficiently generate thousands of images, the results still require time-consuming and subjective human interpretations. Therefore, successfully incorporating these technologies into a high-throughput pipeline for toxicity screening is dependent on automating the interpretation and classification of large numbers of rapidly generated images.

Deep convolutional neural networks have revolutionized the field of computer vision³ and these methods have so far enabled unprecedented advances across an incredibly diverse range of applications, including automated facial recognition^{4, 5}, autonomous vehicle⁶, astronomy⁷, and agriculture⁸. Unsurprisingly, these advances have also increasingly found applications in biomedicine and cell biology⁹. For example, Waisman et al.¹⁰ utilized convolutional neural networks to process images of light microscopy slides and correctly identify differentiating stem cells. The image-based results were as accurate as standard transcription-based approaches for detection but were much more rapid and cost effective. Similarly, Jimenez-Carretero¹¹, et al. used deep learning for automated detection of chemical toxicity via image assays. Their approach, which operates on microscopy images of fluorescently stained cell nuclei, was able to accurately detect toxicity across a wide spectrum of toxicity mechanisms including from drugs having modes of action different than those that the model was originally trained on.

Here we describe a deep-learning method, suitable for usage in the context of a high-throughput toxicity screening pipeline, that uses a convolutional neural network (CNN) to identify images exhibiting morphological changes consistent with cellular stress in response to chemical exposure. We demonstrate the applicability of this approach across an array of chemical exposures that includes a recent comprehensive investigation of 24 reference compounds with known clinical associations to human liver injury and drug interactions using in vitro hepatocyte models in tandem with high throughput transcriptomics¹². We demonstrate that in multiple hepatocyte cell types (HepaRG and primary human hepatocytes (PHH)) at varied states of differentiation (proliferating and 2D-differentiated) this approach has the capacity to efficiently and objectively identify images exhibiting signs of cellular stress with high accuracy (>95%). The method works on lightonly microscopy with no fluorescence staining required. Furthermore, the approach is resilient to noisy data sets, and has the added advantage that it can identify specific regions of the image that are ambiguous, even when toxic effects are not uniform across the cell culture well. Because the results indicate near 100% recall on the extreme image classes “Healthy” and “Altered,” integration of such a method into a high throughput toxicity screening pipeline would permit human experts to focus on the challenging intermediate cases, while allowing the majority of the less ambiguous images to be classified automatically by machine. We expect that such an approach could significantly reduce subjectivity while simultaneously greatly increasing throughput. In addition, identification of treatment-induced morphological changes from the live-cell image data would bypass the need for additional biochemical assays and could potentially provide cell-specific phenotypic signatures associated with stress response.

Methods

Cell Culture and chemical exposure

Cell culture images of HepaRG cells (Lonza, Catalog: NSHPRG) were captured from two culture modes, (1) 2D differentiated (2D-Diff) and (2) proliferating as described by Ramaiahgari et al. [12]. Here, 2D-Diff cells were initiated using cryopreserved HepaRG at a seeding density of approximately 20,000 cells per well onto collagen(type-I)-coated 384-well plates (Corning, Catalog: 356667) and maintained for 10 days in culture for complete differentiation into cholangiocyte- and hepatocyte-like cell populations prior to chemical exposures. For

HepaRG proliferating/undifferentiated cultures, 2,000 cells per well were seeded 3 days prior to chemical exposure. Both culture modes were cultured using Williams E medium (WEM) (ThermoFisher, Catalog: A1217601) supplemented with plating additive (Lonza, Catalog: MHTAP, 35.4 mL of MHTAP to 500 mL of WEM), and 24 hours later were exchanged with William’s E medium supplemented with maintenance additive (Lonza, Catalog: MHPIT; 37.5 mL of MHPIT to 500 mL WEM) for the duration of cultures.

Images of primary human hepatocytes (PHHs) in culture were captured from cryopreserved sandwich cultures of primary human hepatocytes (HUM4080) procured from Triangle Research Laboratories (now Lonza, Morrisville, NC) from a female Caucasian donor, 47 years of age. For HUM4080 PHHs, sandwich cultures were prepared in 96-well BioCoat plates (collagen Type I-coated) by plating PHHs (per the manufacturer’s protocols) at a seeding density of ~50,000 cells/well, then allowed to attach for ~5 hours in a humidified incubator at 5% CO₂ and 37^oC. PHHs were then overlaid with Matrigel (0.35 mg/mL) to form sandwich cultures in ice-cold William’s E medium (WEM) containing ITS+, GlutaMAX, 1% Pen/Strep, 15 mM HEPES buffer, and 100 nM dexamethasone (well-established to effectively model human liver enzyme induction of major cytochromes P450 CYP1A2, CYP2B6, and CYP3A4 in vitro). Cultures were maintained for ~4 days with daily renewal of cell culture medium & test compounds.

Chemical exposures across the image training and validation sets reflect a diverse number of individual studies that include both published^{12, 13} and previously unpublished images (Ferguson et al., GBE in preparation). In general, chemical exposures were initiated with addition of 500X or 1000X stock concentrations of compounds prepared in DMSO, culture medium, ethanol, acetonitrile or water and serial diluted in 96-well plates at half-log or third-log spacing. From these stocks, 2 µL of each compound was transferred to a separate 96-well plate containing hepatocyte maintenance medium (e.g., Lonza, Catalog: MHPIT for HepaRG cultures, William’s E Medium with serum-free supplements as described in Hewitt et al.¹³) using calibrated electronic pipettors or a Vialflo semi-automated pipettor (Integra Biosciences, Catalog: 6031). Exposure solutions were subsequently mixed thoroughly, and 50–100 µL were transferred to hepatocyte culture plates. The exposure periods varied from 24 to 96 h. In most examples, for the final exposure time point, images were captured prior to collection of spent culture media for LDH assays, in situ metabolism assays or gene expression assays. A complete list of exposure compounds is available in supplementary file S1.

Imaging

Photomicrographs of HepaRG cells and primary human hepatocytes (PHH) were acquired using the Incucyte imaging system (Essen Bioscience, Ann Arbor, MI). Microwell plates were loaded into the imager and live images were captured using 10X magnification to cover more than half of the surface area of the microplate well. The Incucyte imager has built-in settings for plate dimensions and autofocus settings based on manufacturer plate type which automatically capture and store live single timepoint image files. Example images are shown in supplementary Figure S9.

Cytotoxicity and cell viability assessments

Cell cytotoxicity was assessed by measuring lactate dehydrogenase (LDH) release from damaged cell membranes. Briefly, 20 µL of the exposure media from each culture well were collected after 96 hours post exposure and LDH release was measured using CytoTox-ONE reagent following the manufacturer’s protocol (Promega, Catalog: G7890). LDH graphs are provided in the supplementary file S2. Cell viability assay data as shown in Figure 1 measured ATP depletion with the CellTiter-Glo assay system (Promega, Madison WI) at the conclusion of chemical exposures, and followed the manufacturer’s protocol. Both LDH leakage and ATP depletion assays were measured on a ClarioStar plate reader system (BMG LabTech, Cary NC) using established laboratory procedures.

Figure 1.

Heatmap views of an example HepaRG 2D-Differentiated annotated plate. The top table shows cell viability assay data (Relative light units reflecting ATP depletion from the CellTiter-Glo assay system, Promega, Madison WI). Shades of red formatting indicate substantial loss (lower 25^th percentile) of cell viability in comparison to blue formatted cultures (upper 75^th percentile viabilities) that include untreated vehicle control culture wells. The center table shows the annotations of cell morphology images by human experts, where 1 indicates ‘Altered’ morphologies with profound changes observed and consistent with substantial loss/death of cells, 0 indicates cell cultures that appeared Healthy and indistinguishable from vehicle control well morphologies, and 0.5 was assigned as Intermediate for images with clear changes in hepatocyte morphologies in portions of culture wells. Intermediate phenotypes typically reside in transitional exposure ranges of emerging cellular stress or focal cell death that has not fully manifested into profound loss of cell viability. The bottom table shows the independent computational model predictions for each culture image. The cells outlined with black rectangles show the disagreements between human annotations and model predictions.

Deep Learning for image classification

For model training, photomicrographs from confluent PHH cultures and proliferating/confluent HepaRG were annotated into 3 classes: “Healthy”, “Intermediate”, and “Altered.” Human experts reviewed individual images and graded morphologies in context with concentration-response progressions over multiple replicate wells to establish broader context for manual interpretations. Manual ‘truth’ morphology classifications involved initial training and harmonization of photomicrograph image interpretations to assess specific features of cellular stress/death with hepatocyte monolayers that included: presence/absence of cells on monolayer, cobblestone-like topologies, defined cell-cell interactions boundaries, cytosolic clarity, cellular size/shape, nuclear size/shape/brightness, swollen organelles or lipid droplets. While graded and classified individually, each image was interpreted in context with its respective vehicle control images (multiple replicates). ‘Healthy’ classified images were indistinguishable from vehicle control images. ‘Intermediate’ images displayed more subtle or localized changes in cellular morphology consistent with cellular stress that frequently were observed in transitional portions of concentration-response progressions to higher exposure levels, or randomly were observed in a small percentage of culture wells prior to chemical exposures that persisted through the duration of exposures. ‘Altered’ images were clearly distinct from vehicle control wells, and generally were observed at the highest chemical exposure levels with examples including total cell loss, large regions of apparently dead cells still tethered to the collagen (I) basement matrix, or larger regions of cellular stress (e.g., granular cytosol, loss of distinct cell-cell junctions, altered nucleus size/shape/brightness), cell shrinkage, or cell loss.

Data preparation

The original, high-resolution images were 1392×1036 pixels in size. It is challenging to use images of this size as direct input into a convolutional neural network (CNN) network because this would result in millions of parameters and excessive time and memory to train the model. Instead, we split the images into a 4×4 grid where each tile was of size 348×259. Each tile was then processed independently of other tiles belonging to the same image. Besides making the image sizes manageable, splitting the images into 16 tiles also provided other advantages: it allowed us to generate more training data (16 times more), and enabled the model to predict specific regions of the image that are problematic, even when toxic effects are not uniform across a plate (Figure 2). In the HepaRG proliferation image dataset, some of the images had a watermark in the bottom right corner of the image. To avoid any bias, we excluded the 16^th tile and the corresponding watermark from training. All the tiles from the same image were kept together in the same dataset, i.e. either the training or validation group.

Figure 2.

Example of Healthy HepaRG 2D Differentiated image subdivided into grid, with predictions made separately for individual tiles. The different border colors represent different predictions. Green represents Healthy and Red represents Altered.

To generate additional data, and to introduce some noise into the dataset for a more robust training outcome, we performed data augmentation using functions available in the fast.ai (v1) library¹⁴. Augmentation is a technique used to expand a dataset by introducing variability in the augmented images, without generating/collecting more data. This is a standard approach used frequently in deep-learning and image classification to make the most of available data and to generate additional training instances such as rotated, zoomed, clipped, and inverted versions of an image that can help the model to better generalize its predictions to unseen data. Furthermore, certain types of augmentation are also thought to perform smoothing over sparse datasets and can be effective, along with other techniques such as drop-out and regularization, at preventing overfitting. After augmentation, images were cropped to have square dimensions and were then scaled to 224 × 224 pixels. Augmentation increased the available training data by 8-fold. Pixel intensities of images were then normalized so that the pixel values for each of the three channels (red, green and blue) had the same mean and standard deviation in each image. Normalization of the images ensures that each input parameter(pixel) has a similar data distribution, which makes the convergence faster while training the model.

CNN Networks and transfer learning

Prepared images were used to train a ResNet-50¹⁵ based convolutional neural network designed to classify image segments according cell colony health. ResNet-50 is a convolutional neural network that has been previously trained on more than a million images from the ImageNet database¹⁶. The network has an image input size of 224 × 224 pixels. The network is 50 layers deep and can classify images into 1,000 predefined generic object categories, such as “keyboard,” “mouse,” “pencil”, “person”, “man”, “woman” and various animals. As a result, the lower layers of this network contain feature-rich representations for a wide range of images. It has been shown that early layers of deep networks trained on ImageNet tend to encode basic shapes, edges and textures whereas deeper layers encode progressively more image-specific features¹⁷. It was our hope that by fine-tuning the deeper layers of a pretrained ResNet-50 network on a new target image dataset, “transfer learning” would facilitate the repurposing of these early layers and thereby lessen the required training data needed to achieve good performance.

We used discriminative learning rates and gradual unfreezing to retrain the model using the prepared assay images. First, the learning rate tool in fast.ai was employed to plot learning rate versus loss (cross entropy) and to identify the optimal learning rates to use with the Adam optimizer. We used smaller learning rates in early layers to permit the weights in those layers to change more slowly than those from later layers: 1e-4 for the first layers, 1e-3 for the middle layers and 1e-2 for the last layers. In the first three training epochs, only the final layers were first trained, with all prior layers frozen. Subsequently, all layers were unfrozen and trained for an additional 3 or 4 epochs (Supplementary Figure S10).

Initially, we trained a binary classification network to predict the two extreme classes in each dataset (Healthy, Altered), setting the Intermediate images aside. Then, we separately trained a second multi-class network to include all three categories (Healthy, Intermediate, Altered). After observing suboptimal performance on the Intermediate class, we combined the Intermediate and Altered images to train a third and final binary classification model.

Alternative CNN architectures were also tested during the preliminary analysis, including ResNet34, ResNet101, DenseNet¹⁸, and SeNet¹⁹. We observed that all of these provided very similar accuracy, with ResNet50 slightly better than the rest (data not shown). Neural network training was performed using a GPU machine configured with 4 CPUs, 32 Gb RAM, and one Tesla K40c GPU. Computing was done on Ubuntu (v.16.04) in a python environment configured for deep learning. Model training was performed using fastai (v.1) and pytorch.

Gradient-weighted Class Activation Maps (Grad-CAM) for network visualization

We used Gradient-weighted Class Activation Maps (GRAD-CAM) to visually inspect the regions of the images that the trained model is “looking at” to make its classification decisions. The activations at the last convolutional layer of the model contain information regarding the different features in the image used for classification. The gradient of the predicted class with respect to these activations provides information about which features lead to the final prediction of the image²⁰.

Results

Image Datasets

After cell culture, chemical exposure, image capture and manual annotation of the resulting images according to the procedures described in sections 2.1 – 2.4, we obtained 3 independent datasets consisting of human-labeled images from multiple cell types. The contents of the resulting datasets are described below.

Primary Human Hepatocytes (PHH)

The PHH dataset contained a total of 282 high resolution (1392 × 1036) images. These were annotated according to two classes, “Healthy” (137 images) and “Altered” (145 images). Unlike the remaining datasets, this dataset is nearly balanced with roughly equal numbers of images in the two classes.

Proliferating HepaRG

The proliferating HepaRG (“Proliferating”) dataset consists of images corresponding to a set of 24 reference chemicals, each with 10 concentrations and 9 replicates (90 images per chemical)¹². Four images were excluded from the analysis as they displayed significant abnormalities (e.g., humidity droplet condensation) and were flagged as having significant “artifacts” during manual curation. In total, the proliferating dataset contained 2,156 high resolution (1392 × 1036) images (Table 1a). Images were labeled Healthy (1,778 images), Altered (234 images) and Intermediate (111 images).

Table 1:

Input Image dataset

a) Raw Image Counts by Label
Categories	Primary Human Hepatocytes		HepaRG Proliferating		HepaRG 2D-Diff
Healthy	137		1,778		1,897
Intermediate	n/a		111		120
Altered	145		234		140
Total:	282		2,156		2,160
b) Each image was split into 16 tiles of smaller size so it can be fed into CNN. The amount of data after splitting into training and validation set.
Categories	Primary Human Hepatocytes		HepaRG Proliferating		HepaRG 2D-Diff
	Training	Validation	Training	Validation	Training	Validation
Healthy	1,744	448	22,720	5,728	24,192	6,160
Intermediate	n/a	n/a	1,472	304	1,552	368
Altered	1,856	464	2,960	784	1,712	528
Total:	3,600	912	27,152	6,816	27,456	7,056

2D-Differentiated HepaRG

Like the proliferating HepaRG dataset, the 2D-Differentiated HepaRG (“2D-Diff”) dataset also used images corresponding to the same set of 24 reference chemicals, each with 10 concentrations and 9 replicates (90 images per chemical)¹² for a total of 2,160 images (Table 1a). Images were labeled Healthy (1,897 images), Altered (140 images) and Intermediate (120 images).

Image Classification

Binary Classification

The first dataset that was imaged and annotated was the PHH dataset. For this dataset, annotators classified images into two categories: “Healthy” and “Altered.” Accordingly, we used the available data to train and evaluate a binary CNN classifier to distinguish between these two classes. All subsequent HepaRG datasets also had a third annotation category “Intermediate,” which was used to describe images where there was some morphological evidence of distress, but where the observable changes were subtle and less obvious than in the Altered class. For this first round of experiments, we excluded these Intermediate images in the HepaRG datasets.

We randomly divided each dataset into a training and validation set according to an 80/20 split (Table 1b). Using the data in the training split, we trained a binary classifier to differentiate between Healthy and Altered images. As described previously, each image was subdivided into 16 tiles which were input into the network independently. The trained classifier was then evaluated on the independent validation split. An image was classified as Altered if the majority of its tiles were classified as Altered; otherwise the image was classified as Healthy. Supplementary Figures S10a-c show the training curves for the PHH, HepaRG proliferation and 2D-Diff datasets.

In all three of the datasets, more than 98% of the images were correctly classified in the independent validation set; in one of the datasets, 100% of the images were correctly classified (Table 2). For the HepaRG 2D-Diff dataset, 3 out of 33 of the Altered images were mis-classified as Healthy, which lowered the F1 score (harmonic mean of recall and precision) to 93.75%. All other datasets exhibited F1 scores of at least 98%. Taken together, these excellent classification results on the simplified datasets indicate that clear cell morphology changes between Healthy and Altered cells are observable via light microscopy with detectable differences that can easily be learned using a convolutional neural network.

Table 2:

Binary classifier results

a) Confusion Matrix of binary classification on Primary Human Hepatocyte dataset.
		Predicted Label
		Healthy	Altered
True Label	Healthy	28	0
	Altered	1	28
b) Confusion Matrix of binary classification on HepaRG Proliferating dataset.
		Predicted Label
		Healthy	Altered
True Label	Healthy	358	0
	Altered	0	49
c) Confusion Matrix of binary classification on HepaRG 2D-Diff dataset.
		Predicted Label
		Healthy	Altered
True Label	Healthy	384	1
	Altered	3	30
d) Binary classifier Result
Metric	Dataset	Healthy	Altered
Recall	Primary Human Hepatocyte	100%	96.55%
	HepaRG Proliferating	100.0%	100.0%
	HepaRG 2D-Diff	99.74%	90.91%
Precision	Primary Human Hepatocyte	96.55%	100.0%
	HepaRG Proliferating	100.0%	100.0%
	HepaRG 2D-Diff	99.22%	97.77%
F1	Primary Human Hepatocyte	98.25%	98.25%
	HepaRG Proliferating	100.0%	100.0%
	HepaRG 2D-Diff	99.48%	93.75%
Accuracy	Primary Human Hepatocyte	98.25%
	HepaRG Proliferating	100.0%
	HepaRG 2D-Diff	99.04%

Multiclass Classification

Although results from this first set of classification were encouraging, eliminating the Intermediate images simplified the problem substantially by excluding the images that are most difficult to accurately classify (even by humans). In a second set of experiments, we tested the applicability of our approach using the full datasets, this time including the Intermediate images. We modified our models to perform multiclass classification, outputting all three labels rather than only the extremes. Supplementary Figures S10d-e show the training curves for the HepaRG proliferation and 2D-Diff datasets. The 80/20 training and validation split was performed as described above and results are shown in Tables 3. For the Healthy images, both recall and precision were excellent, with all F1 scores greater than 95%. For the Altered images, performance remained excellent (100% F1 score) in the HepaRG Proliferating dataset but dropped slightly in the HepaRG 2D-Diff (86.96%). Performance on the Intermediate class was not as good with F1 scores of 82.35% and 48.65% for the HepaRG Proliferating and 2D-Diff (Table 3).

Table 3:

Multiclass classifier Results

a) Confusion matrix of multi class classification on HepaRG Proliferating dataset.
		Predicted Label
		Healthy	Abnormal/Intermediate	Altered
True Label	Healthy	356	1	0
	Intermediate	5	14	0
	Altered	0	0	49
b) Confusion matrix of multi class classification on HepaRG 2D-Diff dataset.
		Predicted Label
		Healthy	Abnormal/Intermediate	Altered
True Label	Healthy	382	3	1
	Intermediate	9	9	5
	Altered	1	2	30
c) Multiclass classifier Results
Metric	Dataset	Healthy	Intermediate	Altered
Recall	HepaRG Proliferating	99.61%	73.68%	100.0%
	HepaRG 2D-Diff	98.96%	39.13%	90.91%
Precision	HepaRG Proliferating	98.10%	93.30%	100.0%
	HepaRG 2D-Diff	97.45%	64.29%	83.33%
F1	HepaRG Proliferating	98.85%	82.35%	100.0%
	HepaRG 2D-Diff	98.20%	48.65%	86.96%
Accuracy	HepaRG Proliferating	98.16%
	HepaRG 2D-Diff	95.21%

Figure 1 shows the image health scores for a single 96-well plate from the HepaRG 2D-Diff dataset. Each cell in the three heatmaps corresponds to a single image and the layout corresponds to the plate layout. In general, the dosage is decreasing from left to right and the two leftmost and two right most columns are untreated. The top heat map shows CellTiter-Glo scores for the images. Below this are annotator scores (1=Altered, .5=Intermediate, 0=Healthy) for the same plate. The bottom heatmap shows scores output by the CNN for the image. Notably, all of the healthy cells (green) called by the annotator are also called as healthy by the CNN (100% recall). Recall is similarly quite high for the Altered class (23/24 = 96%). In general, the patterns are roughly consistent between the machine and human annotator, with the majority of the disagreements occurring on cells the human annotator has labeled as “Intermediate.” In particular, the Pearson correlation between scores from the human annotator and the machine was 0.966 on this plate. Conversely, the CellTiter-Glo scores (high scores indicate healthy cells) exhibited a strong inverse correlation, as expected, with both the human annotator (−0.828) and with the machine (−0.831).

Final Classifier

Although results of the simple binary and multiclass classifiers are promising, there are still a substantial number of misclassifications of the Intermediate class. For example, 9 out of 23 Intermediate images in the HepaRG 2D-Diff dataset were misclassified as Healthy and 5 were misclassified as Altered. We would prefer Intermediate images to be correctly classified or else, if that is not possible, to be misclassified as Altered. Since the majority of the images in the datasets are Healthy, we could then use an automated method to predict Healthy images with high confidence and allow human experts to examine the remaining images. Therefore, we modified our approach by combining the Intermediate and Altered Images into a single composite “Not Healthy” class and trained a new binary classifier. Furthermore, to make the classifier more sensitive to the Not Healthy class, we made the classification threshold for this class less stringent: if any one of the titles in an image was classified as Not Healthy, the entire image was classified as Not Healthy.

The results are shown in Tables 4. Recall for the combined Not Healthy class is >= 95% in both HepaRG datasets, while precision for this class is somewhat reduced. Using this approach, we could use the automated classifier to classify images and to confidently filter out the majority of the Healthy ones with high precision. Human experts could then examine the remaining images to provide the correct classification. Because the Not Healthy images are much less frequent, this approach could greatly reduce the workload of human toxicologists while still achieving excellent performance on all three classes. For example, in both datasets shown in Table 4a, this method would reduce the number of images needing human attention to 16.4% (70/426) of the total for the HepaRG proliferating dataset and 14.5% of the total (64/442) for the HepaRG 2D-Diff dataset.

Table 4:

Results from Binary model trained on Healthy and Intermediate + Altered (Classifying image as Altered if any of the tile is predicted as Altered)

a) Confusion Matrix of binary classification on HepaRG Proliferating dataset.
		Predicted Label
		Healthy	Not Healthy
True Label	Healthy	355	3
	Not Healthy	1	67
b) Confusion Matrix of binary classification on HepaRG 2D-Diff dataset.
		Predicted Label
		Healthy	Not Healthy
True Label	Healthy	376	10
	Not Healthy	2	54
c) Binary classifier Results
Metric	Dataset	Healthy	Not Healthy
Recall	HepaRG Proliferating	99.16%	98.53%
	HepaRG 2D-Diff	97.41%	96.43%
Precision	HepaRG Proliferating	99.72%	95.71%
	HepaRG 2D-Diff	99.47%	84.38%
F1	HepaRG Proliferating	99.44%	97.10%
	HepaRG 2D-Diff	98.43%	90.00%
Accuracy	HepaRG Proliferating	99.06%
	HepaRG 2D-Diff	97.29%

GRAD-CAM

We applied the GRAD-CAM method to the binary model to visualize the regions of the image which contribute the most towards the final classification results (Figure 3). Although, this approach cannot fully explain the detailed functioning of a neural network classifier, it can help to provide a bit of insight regarding the types of image details the method is “looking at” when it classifies an image. For example, for the Healthy PHH images (Figure 3a), the model seems to focus on healthy cells and their nuclei; for the Altered PHH images, the classifier seems to be focusing on the accumulation of dead or disrupted cells. In the HepaRG 2D-Diff images, the Healthy images appear to have an increased focus on healthy regions of the image with a “cobblestone” texture, whereas, for the Altered class, attention tends to rest on dead and empty regions in the image.

Figure 3.

GRAD-CAM: feature map activations from the 17 convolution blocks for both a healthy and altered assay tile of a) Primary Human Hepatocyte, b) HepaRG Proliferating and c) HepaRG 2D-Differentiated dataset. The highlighted areas show the regions used by model to predict the class.

Classification Dose-Response Curves

The classifiers we have trained are designed to output a probability score between 0 and 1 representing the model’s confidence that a given image belongs to the Altered or Not Healthy class. Supplementary Figure S11 shows a plot of these scores for the HepaRG Proliferating dataset as a function of the dosage for 12 chemicals. The plots clearly indicate a strong correspondence between dosage and predicted scores, with human labeled Altered images tending to have scores close to 1, Healthy images with scores close to 0, and Intermediate images often falling in between. Furthermore, for the compounds exhibiting a toxic response, the mean prediction score tends to be a monotonically increasing function of dose.

Discussion

Here we have demonstrated that deep learning classification is capable of accurately discriminating between Healthy and Altered monolayers of chemical-exposed hepatic cells in bright field photomicrographs. Our conclusions are derived from experiments conducted on 3 different cell types from 3 distinct datasets comprising more than 4,500 human-annotated images. Our work demonstrates the feasibility of integrating automated image analysis processes into our existing high-throughput toxicology pipeline. In practice, the datasets that we will acquire during application of these methods are expected to be highly unbalanced with the majority of images falling into the Healthy class (between 75–90% of the data in our datasets). The convolutional neural networks that we have constructed exhibit high recall and precision for the majority Healthy class in all 3 datasets. By eliminating the need for manual examination of the Healthy images, the automated classifiers would allow expert toxicologists to focus on the analysis of a smaller number of challenging Intermediate and Altered images. Furthermore, classification of the minority classes is likely to improve with additional data and therefore the model could be periodically retrained to incorporate newly annotated images over time.

When examining a set of images, a cell biologist normally has access to the plate layout as well as additional metrics such as biochemical assays. Automated image analysis results, which do not necessarily integrate this external information, provide an independent, unbiased score for each image in isolation, similar to metabolic measurements arising from each individual well. Therefore, automated image analysis also has the capacity to serve as an independent data stream available to experienced cell biologists when interpreting the results of an experiment. In the datasets we examined, automated image scores had high correlation with ATP utilization scores as well as human-assigned cell morphology health scores. Especially for images falling into the Intermediate category, there is some inherent ambiguity to the human-assigned labels, and cell biologists can sometimes modify their score assignments when other factors (e.g. replicate samples) are taken into account, including the opinions of other toxicologists or shifting perceptions over time of the severity of morphological change. Having an independent, automated image classifier has the potential to improve consistency of image scoring in future experiments. One general limitation of image-classification approaches based on convolutional neural networks is that the inner workings of these models are opaque. Methods such as GRAD-CAM have the potential to partially ameliorate this condition, but in general it is often necessary to treat neural network models as a “black box.” For this reason, we do not intend to use these methods to replace human experts; rather, it is our hope that these methods can serve as an additional tool that allows experts to work more efficiently and with more time spent analyzing the most interesting and unusual images.

For the set of chemicals examined, our model output demonstrates a clear relationship to dosage (Supplementary Figure S11) with Altered scores increasing monotonically as a function of concentration for the toxic chemicals. These results are consistent with well-established cell viability and cytotoxicity observations and suggest that it should be possible, in principal, to use these scores to compute an image morphology-based benchmark dose (BMD). The resulting benchmark doses could then be integrated with BMDs calculated via other data streams such as gene expression or toxicology assay endpoints, to better elucidate the physiological impact of a potential toxicant. However, accurate computation of such a benchmark dose will require novel statistical methods and is therefore outside the scope of the current study. We have begun work on several possible methods to achieve BMD modeling using image scores and intend to publish our findings in a future manuscript.

To date, the Tox21 program has screened thousands of chemicals for individual molecular targets²¹ towards the ultimate goals of reducing, refining, and replacing the use of animals in toxicology testing. Understanding the causal relationships between molecular pathway interactions and corresponding morphological changes is central to enhancing the utility of these screening systems. So far, ultra-high throughput screening platforms have provided a wealth of knowledge that quantitatively links chemicals to individual molecular targets and pathways. However, it can be challenging to translate identified molecular-level responses to later stages of biological response that include adverse outcome pathway key events and apical outcomes. This can be addressed in part by utilizing imaging systems to capture cell morphology images from individual assay wells to elucidate cellular phenotypes that anchor translational predictions of human health effects. Recently, the Tox21 program has begun to enhance its capacity for high throughput and high content imaging, and already we are routinely generating thousands of photomicrographs. Utilizing deep learning methods has the potential to improve the interpretive power derived from these images, reduce the dependence on human-based subjective interpretations, and accelerate the pace of translational research.

In summary, these results strongly suggest that automated classification of cellular images has the potential to powerfully augment high-throughput toxicology endeavors. We are currently in the process of integrating these methods into a pipeline for image analysis and are performing additional work to further extend our methods to cover additional cell types. We anticipate that as the development of novel imaging machinery and deep-learning methods continues to evolve at a rapid rate, automated image analysis will begin to play an increasingly important role in the application and practice of toxicology and pathology.

Supplementary Material

Supplemental materials

Supplementary Data S1. The list of compounds used in primary human hepatocyte and HepaRG datasets

Supplementary Data S2. The LDH leakage data of HepaRG differentiated cultures (2D-DIFF) and proliferating (PROLIF).

Supplementary Data S3. GRAD CAM visualization of Primary Human Hepatocytes Healthy image

Supplementary Data S4. GRAD CAM visualization of Primary Human Hepatocytes Altered image

Supplementary Data S5. GRAD CAM visualization of HepaRG Proliferating Healthy image

Supplementary Data S6. GRAD CAM visualization of HepaRG Proliferating Altered image

Supplementary Data S7. GRAD CAM visualization of HepaRG 2D-Differentiated Healthy image

Supplementary Data S8. GRAD CAM visualization of HepaRG 2D-Differentiated Altered image

Supplementary Figure S9. Example images from three datasets.

Supplementary Figure S10. Error loss and accuracy rate per epoch

Supplementary Figure S11. Predicted probability of “Altered” class label for 12 chemicals as a function of exposure log concentration for the HepaRG proliferated dataset