Automated human induced pluripotent stem cell colony segmentation for use in cell culture automation applications

Abstract

Human induced pluripotent stem cells (hiPSCs) have demonstrated great promise for a variety of applications that include cell therapy and regenerative medicine. Production of clinical grade hiPSCs requires reproducible manufacturing methods with stringent quality-controls such as those provided by image-controlled robotic processing systems. In this paper we present an automated image analysis method for identifying and picking hiPSC colonies for clonal expansion using the CellX^™ robotic cell processing system. This method couples a light weight deep learning segmentation approach based on the U-Net architecture to automatically segment the hiPSC colonies in full field of view (FOV) high resolution phase contrast images with a standardized approach for suggesting pick locations. The utility of this method is demonstrated using images and data obtained from the CellX^™ system where clinical grade hiPSCs were reprogrammed, clonally expanded, and differentiated into retinal organoids for use in treatment of patients with inherited retinal degenerative blindness.

Introduction

Human induced pluripotent stem cells (hiPSCs) have the potential to differentiate into almost any cell type [1]. They have an unlimited capacity for self-renewal, can be generated through reprogramming of a variety of human cell types, and are not fraught with the ethical concerns associated with using human embryonic stem cells. hiPSCs and hiPSC-derived cells and tissues have been used effectively in disease modeling, drug discovery, cell therapy, and regenerative medicine [1–3]. A variety of methods are available for reprogramming somatic cells into hiPSCs for use in various applications [4–6]. However, the process of reprogramming requires several steps performed over a number of weeks that are time consuming, must be performed in a sterile highly-controlled environment, and require constant visual monitoring of the status of the cells during the process. Many decisions about the treatment of the cells during this process are based on the subjective review of the cells by experts in stem cell biology. This approach can result in high variability in outcomes and is difficult to adapt to large scale parallel production, that will be required for those attempting to evaluate genotype/phenotype correlations on a patient specific basis or generate clinical grade products intended for autologous cell therapy. Automated technologies are currently being developed to address the various needs of hiPSC manufacturing [7–11], many of which will be driven by the image analysis of stem cell cultures. Several image analysis methods have already been proposed to identify and classify iPSCs in culture [12–17]. However, many of these methods cannot be readily expanded to large scale image analysis in real time and their utility in identifying the hiPSCs that are most likely to result in high quality cell products downstream has yet to be investigated.

The goal of this research was to develop a machine vision method to control an automated robotic system (CellX^™ , CellX Technologies Inc., Cleveland Ohio) for imaging and passaging hiPSC colonies during the reprogramming phase of hiPSC generation [18]. A deep learning segmentation approach based on the U-Net architecture [19] was developed to automatically segment the hiPSC colonies in high resolution large field of view (FOV) images of hiPSC cultures and an algorithm for suggesting robotic pick locations within these segmented colonies is presented.

Methods

Study sample

Dermal fibroblasts isolated from skin biopsies were obtained from 15 patients with inherited retinal degenerative blindness [18]. Cells were reprogrammed using CytoTune2, a non-integrating Sendai viral reprogramming kit, as previously reported [18]. After transduction cells were passaged onto a 6 well laminin 521 coated culture dish at 10,000 cells per well. Cultures were fed three times per week for approximately 3 weeks until colonies reached sufficient size (~1.5–3 mm in diameter) for passaging and clonal expansion. hiPSC cultures were picked at 25–30 days post Sendai viral transduction. Aspiration height, rate, and volume were set to minimize the fluidic shear force on the cells. This resulted in an annular pick that required overlapping pick points to transfer the largest quantity of cells from a colony [18]. The amount of material obtained from 4-6 overlapping picks was determined to be sufficient for subsequent clonal expansion.

Robotic system

The CellX^™ system (Cell X Technologies Inc., Cleveland Ohio), used in this study, was equipped with an Olympus IX-83 inverted microscope (Olympus Corp., Tokyo Japan), a Retiga R3 monochrome CCD camera (Q Imaging,Teledyne Photometrics, Tuscon Arizona) for imaging [7,8], and a specially-designed motion control system for well plate and fluidic tip positioning. Both imaging and fluidic operations such as colony passaging can be performed using this system.

Image acquisition

Full FOV images (15360 × 14800 pixels) of the non-meniscal region of hiPSC colonies in 6-well plates were obtained prior to passaging individual colonies into 12-well plates. Phase contrast (PC) images of live hiPSC colonies from a total of 44 wells were collected using a 4X objective (pixel size = 0.001135 mm) and subsampled by a factor of two for automated colony segmentation and picking analysis (image size = 7680 × 7400 pixels, pixel size=0.00227 mm) (Fig. 1A). The hiPSCs generated using the CellX^™ system were determined to be pluripotent via score card analysis and genetically stable via karyotyping. They also gave rise to retinal organoids indistinguishable from manually generated hiPSCs [18].

Fig. 1.

A. Full FOV of phase-contrast image of hiPSC colonies in a single well 6-well cell culture. This image was acquired using CellX^™ system equipped with an Olympus IX83 with 4X objective and subsampled by a factor of two (image size = 7680 × 7400 pixels, pixel size=0.00227 mm) at the end of the reprogramming phase just prior to passaging. The imaging was restricted to the non-meniscal region of the cell culture. B. The full FOV image with manually defined colonies (yellow outlines) and the automatically segmented colonies (green outlines).

U-Net segmentation training data preparation

A single full FOV image from ten different patient cell lines were used to create the training/validation data set for the deep learning U-Net segmentation model. hiPSC colony masks were manually outlined (ImageJ, Cintiq Pro) for visual review and correction by an expert in hiPSC biology. Differentiated, fibroblast, and other cell types were not included in the annotation mask (Fig. 2). The PC image and the corresponding annotation mask were then cropped into 512 × 512 patches starting in the upper left-hand corner of the image extending across the entire full FOV. Image patches that were within the non-meniscal region of the well (17.6 mm diameter) and had segmented colonies were used to develop the deep learning model. The image patches were restricted to the non-meniscal region because colonies within the meniscal region are typically out of focus. This resulted in a total of 538 patches being used for model development.

Fig. 2.

Example of manual annotation mask where only hiPSC cells in a colony were outlined. This image (image size = 500 × 575 pixels, pixel size=0.00227 mm) was cropped from a full FOV image acquired using CellX^™ system equipped with an Olympus IX83, 4X objective, and subsampled by a factor of two.

U-Net model development and colony segmentation

A four-level U-Net model was built using TensorFlow (v2.11, python 3.7) with 16 convolutional filters in the initial level that doubled on each lower level, with a 0.1 to 0.3 dropout in each block and ELU activation functions. This model is similar to that proposed in Franco-Barranco et al. [20] and Matuszewski and Sintorn [21] for electron microscopy images of mitochondria. Binary cross-entropy was used for the loss function, Adam optimization for model training, and accuracy for the evaluation metric. Ten percent of the patches were used for validation while building the model. A batch size of 16 with a total number of training iterations of 42 epochs resulted in the best fit model with an accuracy of 95.8% and Intersection-over-Union (IoU) of 0.83. The model was built in 5.91 min on a desktop computer running Intel Xeon with NVIDIA RTX A4500 graphic card using a single GPU.

The U-Net based segmentation model was then applied to all 34 images obtained from the ten patient cell lines used to create the U-Net model and an additional ten images from five patient cell lines not used in the original model development. The greatest variability in hiPSC colony images comes from the differences observed between patient cell lines as opposed to between wells within a single cell line. Therefore, we added these ten images to observe how the segmentation would perform on cell lines that were not used in the original training set. Full FOV image segmentation was performed by creating two sets of 512 × 512 image patches for each image; one based on a grid starting in the upper left corner of the image and one shifted by 128 pixels in both X- and Y- directions. This helps to reduce the edge effects resulting from segmenting the smaller 512 × 512 patches. The model outputs from these two sets of patches are then logically combined to create the original full FOV segmented image. Segmented structures less than 0.015 mm² in size were removed from the image as debris.

Segmentation validation

The full FOV image segmentations were then evaluated using the dice-coefficient (DICE). DICE was calculated for any colonies with a minimum of 50% of the area within the well mask within the non-meniscal region. The dice coefficient is a metric that measures the amount of overlap between manual annotations (ground truth) and those determined from the U-Net segmentation model (predicted).

$$\text{DICE}=\text{2TP}/(\text{2TP+FP+FN})$$ (1)

Where TP=true positive, FP=false positive, and FN=false negative.

DICE ranges from 0 to 1 with 1 being exact overlap and 0 being no overlap. DICE was calculated using the EvaluateSegmentation software developed by Taha and Hanbury [22].

The colonies based on manual and automatic segmentation images were labeled using a connected components algorithm [23]. The number of colonies automatically segmented correctly (TP), number colonies missed (FN), and the number of additional colonies segmented (FP) for each image were counted. The mean area (mm²) of the colonies correctly segmented using the automated method was compared to the mean area of the corresponding colony segmented using the manual method. The mean areas of the missed (FN) and extra colonies (FP) were also calculated.

Automated pick locations

Automated segmentation of colonies provides the basis for controlling the pick locations in an automated cell culturing system such as the CellX^™. Based on our review of the 34 images, where a reviewer subjectively chose pick locations in hiPSC colonies using the CellX^™ system for passaging and culture expansion, we designed a configurable picking algorithm to identify single or multiple picks within each segmented colony. The size of a pick was based on the outer diameter (0.73 mm) of the 250 μl needle syringe used for fluid aspiration in the CellX^™.

For colonies with single picks, the centroid of the pick is placed at the location of the maximum Euclidean distance in the colony. For colonies with multiple picks, the centroids of the automated picks are placed along isolines located within each colony (Fig. 3). These locations are controlled by two adjustable parameters; the starting distance from the edge of the colony for the first isoline (Edge Offset) and the distance between picks determined from a multiplication factor of the needle diameter (Pick Factor). The Pick Distance is calculated as follows:

$$\text{Pick}\text{Distance}=\text{Pick}\text{Factor}*\text{needle}\text{diameter}$$ (2)

0 < Pick Factor < 1: overlapped adjacent picks

Pick Factor >= 1: non-overlapped adjacent picks

Fig. 3.

Schematic illustrating automated pick locations based on Pick Factor and Edge overlap.

The isolines are calculated by thresholding the Euclidean Distance map of the colonies based on the Edge Offset and Pick Distance. The centroids of the picks are placed along the ordered point isolines based on the Pick Distance. Each potential pick location on the isoline is checked against existing picks to make sure that it does not violate the amount of overlap defined by the Pick Factor.

Evaluation of automated pick algorithm

We evaluated 160 picks performed by an expert observer on the 34 images from the 10 patient cell lines. The manual colony segmentations were used for this analysis. The analysis included the calculation of the number of picks per colony, size of colonies based on the number of picks, and the distance between multiple picks in a colony. We then evaluated the performance of the automated picking algorithm using the automated segmentations from the 34 images from the 10 patient cell lines. The Pick Factor and Edge Offset for the automated picks was based on the analysis of the manual picks by the expert reviewer in these images.

Results

An example of a full FOV PC image that has been automatically segmented using the U-Net model is shown in Fig. 1B. The yellow and green outlines in this image represent the manually and automatically segmented colonies, respectively. Overall, the colony segmentation appeared consistent between the manually and automatically segmented images. The time required for processing a full FOV image currently takes 30 s.

Segmentation validation

The mean (SD) DICE for the 34 images captured from the original ten cell lines was 0.84 (SD 0.08). The DICE ranged from 0.68–0.96, with two images that had a DICE value less than 0.70 (B1336_1A, B483_2B). For these images we observed falsely segmented groups of cells surrounded by fibroblasts that lacked a distinct outer boundary (Fig. 4). The mean (SD) DICE for the ten images captured from the 5 additional cell lines was 0.81 (0.08). The DICE ranged from 0.65–0.91, with one image less than 0.70 (B-1309_2B). These DICE values were close to that observed for the images from the original 10 cell lines.

Fig. 4.

Phase contrast images cropped from a full FOV image acquired using CellX^™ system equipped with an Olympus IX83, 4X objective, and subsampled by a factor of two (image size = 400 × 400 pixels, pixel size = 0.00227 mm) A. partially reprogrammed cells surrounded by fibroblasts that lacked a distinct outer boundary, B. corresponding outline (green) of this region from the automated segmentation, C. hiPSC colony with distinct outer boundary indicated by red arrow, and D. manual (yellow) and automated (green) outline of the hiPSC colony shown in C.

The colony counts for manually and automatically segmented images for the 34 images based on the original 10 cell lines are presented in Table 1. The average number of colonies in these images was 19 with a minimum of 3 and maximum of 59. The automated algorithm identified 96% of the manually traced colonies correctly but resulted in an additional 4 false positive colonies being segmented per image on average.

Table 1.

Colony counts manual and automated from images obtained from original 10 cell lines (n=34).

	Total Number Colonies (Manual)	Total Number Colonies (Automated)	Correctly Segmented Colonies (TP)	Missed Colonies (FN)	Extra Colonies (FP)
Mean	19	23	18	1	4
SD	14	15	13	2	4
Minimum	3	3	3	0	0
Maximum	59	67	57	8	15

The size of the manually and automatically segmented colonies are listed in Table 2. The colonies segmented using the U-Net model were 85% the size of the manually outlined colonies. However, upon visual inspection of each image we observed that in most cases the automated segmentation provided a better and more consistent segmentation of hiPSC colonies than the manual segmentation (Fig. 5). This can be attributed to the difficulty and variability observed in manually outlining structures in images. The size of the missed colonies was 0.24 mm² that is 15% of mean colony size and the size of the extra colonies was 0.15 mm² that is 10% of mean colony size.

Table 2.

Colony sizes manual and automated from images obtained from original 10 cell lines (n=34).

	Size (mm2) Segmented Colonies (Manual)	Size (mm2) correctly segmented Colonies (Automated)	Size (mm2) Missed Colonies	Size (mm2) Extra Colonies
Mean	1.29	1.14	0.25	0.24
SD	3.53	3.26	0.26	0.34
Median	0.43	0.37	0.33	0.13
Minimum	0.08	0.07	0.08	0.07
Maximum	56.24	53.29	1.40	3.52

Fig. 5.

Phase contrast images cropped from a full FOV image acquired using CellX^™ system equipped with an Olympus IX83, 4X objective, and subsampled by a factor of two (image size = 540 × 700 pixels, pixel size = 0.00227 mm) A. hiPSC colony, and B. manual (yellow) and automated (green) segmentations. Red arrows point to regions where manual outlining included cells with a more variable morphology than the hiPSCs.

The colony counts for manually and automatically segmented images for the 10 images based on the 5 additional cell lines are presented in Table 3. The total number of colonies based for the manual and automated segmented images are similar to that observed for the original 10 cells lines. This was also observed for the correctly segmented, missed, and extra colonies. The size of the manually and automatically segmented colonies for the 5 additional cell lines are listed in (Table 4). The colony sizes for these 5 cell lines were within the range of colony size observed in images from the original 10 cell lines.

Table 3.

Colony counts manual and automated from images obtained from 5 new cell lines (n=10).

	Total Number Colonies (Manual)	Total Number Colonies (Automated)	Correctly Segmented Colonies (TP)	Missed Colonies (FN)	Extra Colonies (FP)
Mean	19	21	17	2	4
SD	12	12	11	0	4
Minimum	9	9	9	1	0
Maximum	48	47	45	3	11

Table 4.

Colony sizes manual and automated from images obtained from 5 new cell lines (n=10).

	Size (mm2) Segmented Colonies (Manual)	Size (mm2) correctly segmented Colonies (Automated)	Size (mm2) Missed Colonies	Size (mm2) Extra Colonies
Mean	0.46	0.42	0.16	0.21
SD	0.43	0.45	0.11	0.16
Median	0.31	0.27	0.13	0.14
Minimum	0.07	0.07	0.07	0.07
Maximum	2.7	2.68	0.48	0.75

Evaluation of automated pick algorithm

The colonies chosen by the reviewer for ‘picking’ ranged in size between 0.08 and 56.24 mm² with a mean and median size of 2.90 and 1.22 mm², respectively. There were 59 colonies with single picks and 58 colonies with multiple picks that covered the entire colony. The remaining 43 colonies had multiple picks that did not cover the entire colony (Table 5).

Table 5.

Colony size based on reviewer-chosen picks.

Number Picks	1	2	3	4	5	6	>=7
Mean	0.43	0.93	1.74	2.23	5.57	3.77	9.33
Median	0.37	0.89	1.43	1.97	3.95	4.15	4.04
SD	0.23	0.36	0.80	0.85	4.64	2.52	14.17
Minimum	0.08	0.48	1.15	1.33	1.30	1.40	2.27
Maximum	1.15	2.11	4.12	4.29	21.91	10.01	56.24
Count	59	24	14	9	19	11	24

Thirty-four of the colonies (57%) were less than the size of single pick (i.e., 0.42 mm²). Single picked colonies were observed primarily in one cell line, B-483, where only small colonies (mean (SD) size = 0.33(0.27) mm) were present. There was a range in pick distance based on the number of picks per colony (Fig. 6) that tended to increase as the number of picks increased from two to five picks/colony. The distance between multiple picks in a single colony ranged from 0.22 to 0.89 mm with a mean distance of 0.59 (SD = .14) mm. This corresponds to a mean Pick Factor of 0.83.

Fig. 6.

Distance between Reviewer-chosen multiple picks as a function of the number picks per colony.

We chose to evaluate the automated picking algorithm using the mean Pick Factor calculated from the reviewer-chosen picks and an Edge Offset of 0. Setting the Edge Offset to 0 ensures that none of the picks will extend beyond the boundaries of the segmented colony. For this evaluation, it was not expected that the computer-chosen pick locations would match the reviewer-chosen pick locations. In fact, that would not be desirable in that we are trying to standardize the process such that there is less variability in the overall process. Thus, we evaluated the number of multi-pick colonies (>=4 picks/colony) available for passaging and culture expansion. This number of overlapping picks was determined to be sufficient for subsequent clonal expansion in these samples [18]. A comparison of the size and number of colonies that supported four or more picks is presented in Table 6 for both the reviewer- and computer-chosen pick locations. The number of wells (Nwells) analyzed per cell line varied from one to six and the B483 cell line did not have large enough colonies to support four or more picks. The number of computer-chosen multi-pick colonies was similar to that observed for the reviewer-chosen multi-pick colonies with the exception of the B755 cell line. This was due to smaller narrow colonies that would not support multiple picks within the colony boundaries with the Edge Offset at 0. An example of the reviewer- and computer-chosen picks for single and multiple picks are presented in Fig. 7.

Table 6.

Mean Colony size (mm²) for colonies >= 4 pick locations and number colonies with >= 4 pick locations.

	B216	B482	B483	B539	B755	B1336	B1342	B1427	B1433	B1447
Nwells	3	4	4	2	5	4	6	1	2	3
Reviewer-Chosen
Mean	5.39	7.20		18.02	3.17	11.90	2.92	5.46	2.55	12.23
Median	5.49	5.14		5.80	2.53	11.90	3.25	5.75	2.48	4.45
SD	1.86	6.62		25.51	1.63	14.16	1.15	2.54	0.89	18.16
Minimum	2.68	1.75		4.26	1.97	1.89	1.30	2.17	1.33	2.39
Maximum	7.32	25.35		56.24	5.02	21.91	3.86	8.09	4.34	48.90
Sum	26.95	79.20		72.10	9.52	23.80	11.67	43.72	50.91	73.40
Count	5	11	0	4	3	2	4	8	20	6
Computer-Chosen
Mean	7.54	6.28		14.69		23.00	5.00	5.83	3.52	13.60
Median	6.42	4.27		4.44		23.00	5.00	6.63	3.37	4.60
SD	2.49	4.24		21.63		25.46	1.41	2.15	1.04	18.73
Minimum	4.65	2.76		3.85		5.00	4.00	3.05	2.35	2.83
Maximum	11.17	18.62		53.29		41.00	6.00	7.98	5.44	46.83
Sum	75.43	81.68		73.44		46.00	10.00	40.81	52.82	68.00
Count	10	13	0	5	0	2	2	7	15	5

Fig. 7.

Phase contrast images cropped from a full FOV image acquired using CellX^™ system equipped with an Olympus IX83, 4X objective, and subsampled by a factor of two (pixel size = 0.00227 mm). The automatically segmented hiPSC colony are outlined in yellow, the Reviewer-chosen picks in green, and the automated picks in light blue. Three different size colonies from different patient lines and wells are shown here. A. single pick colony (image size = 600 × 500), B. small multi-pick colony (image size = 600 × 815), and C. large multi-pick colony (image size = 1300 × 1430).

Discussion

Our overall goal was to develop a computer-based approach for automatically segmenting hiPSC colonies and identifying pick locations that could be used to replate hiPSC colonies at the end of the reprogramming phase using the robotic CellX^™ system. We have developed an automated hiPSC colony segmentation based on a modified U-Net deep learning approach and coupled that with a configurable picking algorithm based on the tip diameter of the CellX^™ system. The deep learning CNN model used for colony segmentation was based on a light weight U-Net model with 16 convolution layers and a limited number of training samples. The mean DICE for 34 full FOV phase contrast images of hiPSC colonies was 0.84 demonstrating that there was good agreement between the predicted and manually traced colonies. We then applied this segmentation to 10 full FOV images from five new cell lines that were not used in the original training set. These DICE values were close to that observed for the images from the original 10 cell lines (mean=0.81). These are both good outcomes when one considers a) the variability observed in manual tracing biomedical images and b) that upon close visual inspection we observed automatically segmented colony boundaries that were more consistent than those manually outlined. We observed false positive segmentation where partially reprogrammed cells were observed within a dense population of fibroblasts. These segmented regions exhibited similar morphology to hiPSC colonies but lacked a defining boundary observed in hiPSC colonies and generally were small in size (0.15 mm²). We also determined that adding more images from the same patient cell line or using data augmentation, such as adding images with blur, did not improve our overall segmentation results. These light weight U-Net architecture have previously been evaluated for other imaging applications and found to be stable, less prone to over-fitting, and gave similar results to models developed with larger number of hyperparameters [20,21].

Other groups have proposed multi-class segmentations that include additional cell types such as single hiPSCs, differentiated, and dead cells [17,24]. We chose to develop our segmentation algorithm using a single class that was required for identifying hiPSC colonies for picking. Developing a single-class model reduced the number of annotations and images required for model development and simplified the segmentation model so that it can be run in real time during an automated scanning and picking procedure. We have also determined that this same model can be trained for different stages of hiPSC morphology during the reprogramming phase of cell processing, such as the weeding stage that requires mechanical removal of confluent fibroblasts to allow hiPSC colonies to grow.

We have also presented an automated picking algorithm that can be controlled by two user-defined adjustable parameters. These parameters may be fixed for a given application but they allow us to adjust the picking locations based on the ultimate goal of new applications. We presented an example where the parameters were set to match how operators of the CellX^™ system chose locations subjectively and tested our approach based on the subjective picking locations from the 34 images that were automatically segmented using the U-Net segmentation model. The algorithm was able to identify four or more multi-pick locations in several of the colonies in each well similar to that observed by a subjective operator of the system. The approach standardizes the picking process and the Edge Offset can be adjusted to ensure that unwanted cell types are not included in the picks. The automated picking algorithm is currently driven by the size of the colonies. However, as we begin to track the fate of individual colonies we may be able to identify morphological subtypes in the reprogramming phase that are desirable for improved clonal expansion and organoid development. This is the first step in fully automating a laborious task for cell colony manufacturing.

Conclusions

A deep learning segmentation approach based on the U-Net architecture [19] was developed to automatically segment the hiPSC colonies in high resolution large FOV phase contrast images of hiPSC cultures, as well as an algorithm for placement of automated pick locations within these segmented colonies. This work provides a basis for investigating hiPSC morphology that is observed in early phase reprogramming images and correlating these features to the quality of the final downstream cell products. Identifying morphometric features that correlate with downstream quality is significant with regards to the cost in time and money required to develop clinical grade cell-based products that can take several months to develop.