Deep Learning for Chondrocyte Identification in Automated Histological Analysis of Articular Cartilage

Abstract

Background:

Histology-based methods are commonly used in osteoarthritis (OA) research because they provide detailed information about cartilage health at the cellular and tissue level. Computer-based cartilage scoring systems have previously been developed using standard image analysis techniques to give more objective and reliable evaluations of OA severity. The goal of this work was to develop a deep learning-based method to segment chondrocytes from histological images of cartilage and validate the resulting method via comparison with human segmentation.

Methods:

The U-Net approach was adapted for the task of chondrocyte segmentation. A training dataset consisting of 235 images and a validation set consisting of 25 images in which individual chondrocytes had been manually segmented, were used for training the U-Net. Chondrocyte count, detection accuracy, and boundary segmentation of the trained U-Net was evaluated by comparing its results with those of human observers.

Results:

The U-Net chondrocyte counts were not significantly different (p = 0.361 in a paired t-test) than the algorithm trainer counts (Pearson correlation coefficient = 0.92). The five expert observers had good agreement on chondrocyte counts (intraclass correlation coefficient = 0.868), however the resulting U-Net counted a significantly fewer chondrocytes than the average of those expert observers (p < 0.001 in a paired t-test). Chondrocytes were accurately detected by the U-Net (F1 scores = 0.86, 0.90, with respect to the selected expert observer and algorithm trainer). Segmentation accuracy was also high (IOU = 0.828) relative to the algorithm trainer.

Conclusions:

This work developed a method for chondrocyte segmentation from histological images of arthritic cartilage using a deep learning approach. The resulting method detected chondrocytes and delineated them with high accuracy. The method will continue to be improved through expansion to detect more complex cellular features representative of OA such as cell cloning.

Clinical Relevance:

The imaging tool developed in this work can be integrated into an automated cartilage health scoring system and helps provide a robust, objective and reliable assessment of OA severity in cartilage.

Introduction

Osteoarthritis (OA) is a joint disease associated with painful degenerative changes in articular cartilage and the underlying subchondral bone. Patients typically present to physicians with end-stage OA after the cartilage has completely eroded and significant bony changes have taken place. Therefore, a large amount of OA-related research is focused on intervening earlier in the natural history of the disease, while cartilage is still present. Detailed information about cartilage health at the cellular and tissue level for this research is often obtained via histology-based methods. The histological characteristics of early OA include loss of the integrity of the articular surface and loss of proteoglycan (PG) content, together which can result in the disruption of the extra cellular matrix (ECM). In this environment, chondrocytes initially become more active, replicating and producing more PG in an attempt to stabilize the ECM, but later dying within the compromised ECM.¹ The severity of early OA is often characterized by scoring schemes based on these histological features.

One of the earliest and most widely implemented scoring schemes is the Mankin score, which independently evaluates articular surface integrity, cellularity, PG depletion and tidemark integrity, and then sums the sub-scores of each feature to obtain a single value to represent OA severity.² There have been a number of other similar scoring systems patterned on the Mankin score, including the OARSI, O’Driscoll, ICRS, and Knutsen scoring systems,^3-5 and further species-specific adaptations of several of these individual scoring schemes. Many of these systems were developed to formalize the extent to which the tissue is affected by the severity of degeneration, and they utilize a “stage” and “grade” approach then multiply these numbers to obtain a degeneration “score”.⁶ Despite the variation in scoring schemes, a common feature to all histological scoring schemes is the necessity to assess cartilage cellularity and deviations from normal in order to obtain this OA severity score.

Conventional application of any scoring scheme and thus the assessment of cellularity is performed by an experienced individual observing the specimen under a microscope and subjectively assigning a score. This approach usually results in high intra- and inter-observer variability.⁷ Therefore, more objective and reliable computer-based scoring systems are highly desirable. To develop such algorithms, methodologies to automatically detect chondrocytes, articular cartilage boundaries, etc. using image analysis techniques must be developed. Identifying an object in a histological image requires it to visually appear different from the surrounding structures so that programs can be developed to extract it from the surrounding image. There are many different techniques to achieve this identification, including thresholding, in which an object is identified by having a brightness or a color pattern that meets a specific cutoff value; edge detection using a variety of gradient-based filters to identify abrupt changes in color or brightness; watershed-based methods which separate each object based on local minimums on object boundaries;⁸ and blob detection which utilizes a shape-detection method to identify objects of interest.⁹ The challenge of using any of these techniques to identify chondrocytes in histological images of arthritic cartilage is that there is a huge variety in the appearance of cartilage associated with the disease as well as the histological processing and staining itself (Figure 1).

Figure 1.

Common variations in chondrocyte appearance in normal and osteoarthritic cartilage. OA results in loss of red PG staining (upper right and lower left) and cloning (lower left). Thick or torn sections result in deformed or indistinct chondrocytes (lower right). High PG intensities can also partially obscure cells present only in the slide-side half of the tissue section (lower left). Composing a single analysis algorithm to address this level of variation has proven challenging.

Modern artificial intelligence-type approaches have been brought to bear on challenging image analysis tasks similar to identifying individual cells within images. “Deep learning” is one such technique that has achieved enormous success in advanced image analysis applications. The use of convolutional neural networks (CNN or ConvNets), a typical deep learning model designed to process image data, has performed excellently in several computer vision applications including object detection and semantic segmentation.^{10, 11} In contrast to conventional segmentation methods, in which the computer is explicitly instructed to execute a series of programmer-selected steps to perform a segmentation task, a CNN model can “learn” to do the task, essentially developing an internal series of steps needed to perform the segmentation from a set of raw input images and the associated “correct” segmentations. To do this, the CNN iteratively outputs a segmentation image (prediction), which is computationally compared with the gold standard human segmentation for the image (ground truth). The model is programmed to learn from this comparison by optimizing the internal routine in-use to output a prediction that is progressively closer to the ground truth. This process is referred to as training. It has been shown that a CNN can not only learn to identify lower level image features like color and edges, but also higher level image features such as histograms of oriented gradients.¹² This ability to identify complicated features in image representation allows the CNN to outperform traditional image processing methods. We hypothesized that an appropriately trained CNN model could achieve excellent chondrocyte segmentation accuracy. The goal of this work was to adapt and train the U-Net, a CNN model with providing excellent cell segmentation in other contexts,¹³ to perform chondrocyte segmentation and then to compare the automated results to the gold standard of human segmentations.

Methods

General Approach and U-Net Architecture

The U-Net is a symmetrical CNN model consisting of a contracting path followed by an expansive path (Figure 2).¹³ Beginning at the contracting path, the histology image is convoluted using different learnable filters to extract different features (such as colors and edges) at each location in the image. This operation results in an intermediate, multi-channel image with each channel representing the detection of a certain feature using the associated learning filter. Next, the “max-pooling” layer down-samples the intermediate image with respect to its height and width in order to obtain the most representative feature combination in each local region. This process (convolution and max-pooling) is repeated to extract higher level features, e.g. different appearances of nuclei. At the deepest level, the intermediary image containing the highest level features enters the expansive path. It becomes up-sampled and concatenated with the corresponding intermediate image that was generated along the contracting path. The objective is to localize critical information from high-level features to a known location in the histology image space. The expansive path terminates with an image of the same size and resolution as the input histology image. A loss function is then used to quantify the difference between the model prediction segmentation image and the ground truth segmentation image. Therefore, selection of an appropriate loss function for comparison with the ground truth image will depend on the nature of the image being analyzed.

Figure 2.

U-Net architecture:¹³ the left side of the “U” shape represents the contracting path, while the right side represents the expansive path.

Training the algorithm also requires two different input datasets, both of which are selected to encompass the range of appearance of chondrocytes in the cartilage. Training is conducted by providing the algorithm with each pair of images/ground truth segmentations that are in the designated “training set”. The corresponding loss is then minimized over multiple iterations of the algorithm in which the convolution filters are modified. To prevent overfitting to the training set, after every training iteration (called an epoch), the model is required to predict on a fully separate set of images/ground truth segmentations, termed the validation set. This ensures that the model will remain accurate for yet unseen images. The model is saved when the lowest validation loss is achieved.

U-Net Adaption and Training for Chondrocyte Segmentation

In this study, the U-Net framework was adapted for analysis of 256 x 256 color (RGB) histology images. Each convolutional layer of the U-Net utilized 3 x 3 filters followed by a batch normalization layer and rectified linear unit activation function.¹⁴ A batch normalization layer was also added after the concatenation layer (bottom of the U) along the expansive path. At the final output layer, sigmoid activation was applied to result in an output image with values between 0 and 1, which represents the probability value range of a given pixel being a cell (1) or not in a cell (0). A binary cross entropy loss function was utilized to compute the probability difference between the U-Net prediction image and the ground truth segmentation image.¹⁵ For each pixel, the difference between the prediction value and the ground truth value (loss) at that pixel location is computed, weighted, and normalized by the sum of weights over the entire image to form the final loss function to be minimized by optimization. The weight is taken from a weight map, which is a precomputed constant from the ground truth segmentation (Figure 3). Higher weight values force the associated feature to be learned with greater emphasis. In this work, the weight value for each pixel in the weight map was the inverse of the frequency of the pixel’s ground truth class (cell or not cell) appearing in the full ground truth map. This approach forces the model to preferentially learn cells. Similarly, the values of the pixels along the borders of very closely adjacent cells were weighted higher to improve border detection the border between very closely adjacent cells (Figure 3).

Figure 3.

Example histology image of chondrocytes (top left) with the associated ground truth segmentation (bottom left). The U-Net is trained using a series of these image/segmentation pairs. The segmentation mask is overlaid on the histology image on the top right. A weight map (bottom right) assigning loss weights for each pixel is precomputed from the ground truth segmentation. Warmer colors represent higher weight value assigned to the loss of the corresponding pixel, which forces the U-Net to learn preferentially in those higher-weighted regions.

To train the algorithm, safranin-o/fast green-stained histological sections of rabbit cartilage with varying degrees of arthritic changes induced by ACL transection¹⁶ and medial meniscus destabilization¹⁷ were used. Histological sections were digitized using a stage scanner microscope (Olympus VS110, Olympus America Inc., Center Valley, PA, USA) at a resolution of 322.25 nm/ pixel. Training and validation image (256 x 256 pixels) sets of 235 and 25 images, respectively, were selected to encompass chondrocytes with different sizes, shapes, appearances, and zones of origin. The chondrocytes in each image in both sets were manually segmented by a single individual leading algorithmic development (hereafter referred to as the “algorithm trainer”) using MATLAB (R2018b, The Mathworks, Natick MA). To increase the number of images in the training set in order to reduce model overfitting, data augmentation was utilized to increase the number of images in the training data set without requiring additional manual segmentation. The data augmentation included image rotation, mirroring, and brightness adjustment to reproduce the variability of cartilage histology in practice.

The U-Net model was implemented using the open source deep learning library Keras with TensorFlow backend (https://keras.io/). An NVIDIA Tesla K80 GPU was used for model training. The model was randomly initialized according to the work by Glorot and Bengio,¹⁸ and dropout layers were applied at the expansive path to further prevent overfitting.¹⁹ Due to the GPU memory limitation, a batch size of 1 pair of training images at a time was used. The model was trained for 60 epochs (training iteration + validation check) using an Adam optimizer (learning rate = 5*10-4, momentum = 0.99).²⁰

Model Evaluation

The trained U-Net was evaluated in terms of performance in the following three related tasks: chondrocyte counting, chondrocyte detection, and chondrocyte boundary segmentation. As the goal of this work was to improve upon manual segmentation and assessment of chondrocytes, data from the U-Net were compared with manual cell segmentations performed by five expert observers with experience identifying chondrocytes in histological, immunohistochemical and fluorescent confocal microscopic images. This comparison was performed on a totally separate set of 30 images (testing set). Five of the 30 images in the set were repeated to assess intra-observer variability in the segmentation process. Image selection, format and segmentation were performed as described above. Manual segmentation of this additional testing set was also performed by the algorithm trainer and compared with U-Net’s performance. Intra-class correlation coefficient (ICC) among the expert observers was calculated to quantify inter-observer variability in chondrocyte counting. An ICC value greater than 0.8 was considered good agreement between the observers.²¹ For each of the five observers, a Pearson’s correlation coefficient was computed between the replicate segmentations of the repeated images in the data set. Similarly, Pearson’s correlation coefficients were computed between the U-Net predictions and the average of the expert observers and between the U-Net predictions and the algorithm trainer.

Chondrocyte detection accuracy was evaluated using an F1 score.²² Given a pair of segmentations (prediction and manual segmentation), the F1 score is calculated as the harmonic mean of precision (P) and recall (R) using the following equations:

where N_tp, N_fp, and N_fn denote the number of true positives (TP), false positives (FP), and false negatives (FN), respectively.²² A U-Net predicted cell covering more than 40 percent of the manually segmented cell was considered a true positive; otherwise it was considered a false positive. A false negative was defined as less than 40 percent of a manually segmented cell being intersected by a U-Net-predicted cell. Precision or recall values close to 1.0 indicate very few FP or FN, respectively. A F1 score of 1.0 thus represents perfect detection. The F1 score was computed relative to the algorithm trainer and relative to a representative expert observer.

To evaluate the accuracy of the U-Net chondrocyte segmentation, an intersection over union (IOU) metric (Figure 4) was computed between the U-Net segmentations and the algorithm trainer segmentations.¹³ A perfect U-Net segmentation is identical to the ground truth segmentation, which gives an IOU value of 1.0. No overlap of cells between the two segmentations yields a value of zero. Object-wise IOU was calculated for each individual chondrocyte in each of the 35 test images, and a total IOU value was computed by summing the chondrocyte size-weighted IOU of all chondrocytes.

Figure 4.

A schematic illustration of intersection over union (IOU) is shown on the left. IOU is calculated based on the overlapping (intersection) and the total (union) area of a given cell pair. On the right are two example segmentation comparisons overlaid on the original image yellow contours represent the manual segmentation by the algorithm trainer, and transparent blue mask represents U-Net segmentation. An overall IOU > 0.95 is achieved in these two examples, indicating excellent chondrocyte segmentation.

Results

U-Net training with the parameters specified above required approximately 90 minutes. Both the training loss and validation loss started from infinity and decreased below 0.2 after 20 epochs. The model was saved when it achieved the lowest validation loss (0.152) (Figure 5).

Figure 5.

The output from the training process is plotted, illustrating the minimization of both training loss and validation loss. Training loss decreases steadily through iterations while validation loss generally decreases in a fluctuating manner. The minimum validation loss occurred at the 44th epoch and the model was saved.

There was good inter-observer agreement (ICC = 0.868) in cell counting among the five expert observers. Intra-rater agreement was excellent with Pearson’s correlation coefficients of 0.984, 0.979, 0.994, 0.922, and 0.974. Despite reasonable correlation between the U-Net and the expert observer chondrocyte counts (0.786, 0.771, 0.75, 0.85, and 0.858), the U-Net predictions of chondrocyte counts were on average 24% (±15%) lower than the average of the expert observer counts (Table 1). This difference was statistically significant (p < 0.001 in a paired t-test). In contrast, the Pearson’s correlation coefficient between the trained U-Net model and the algorithm trainer was higher (0.92), and there were no statistically significant differences in the number of chondrocytes counted (p = 0.361 in a paired t-test). Chondrocytes were accurately detected, with a majority of true positives and very few false detection (both FP and FN) (Figure 6). F1 scores (for accuracy of cell detection) of 0.86 and 0.90 were achieved for the selected expert observer and algorithm trainer, respectively. Segmentation accuracy relative to the algorithm trainer was also high (IOU = 0.828). Generally, chondrocytes with an elongated shape (as would be found in superficial zone) were less accurately detected (Figure 7). The U-Net was also less accurate in identification of small cells, out of plane cells, or in torn cartilage with a “fuzzy” appearance.

Table 1.

Cell Counts for Each Individual Observer and the Associated U-Net Counts

	Expert Observer				Expert		Algorithm
	1	2	3	4	5	Observer Average	U-Net	Trainer
Image 1	66	72	65	73	70	69	58	59
Image 2	21	25	22	22	23	23	18	20
Image 3	35	44	35	48	35	39	27	27
Image 4	45	41	40	48	44	44	41	37
Image 5	42	42	41	44	43	42	26	30
Image 6	28	42	22	29	27	30	27	22
Image 7	45	51	38	47	40	44	32	31
Image 8	57	60	54	59	64	59	56	53
Image 9	40	43	39	37	40	40	31	32
Image 10	67	70	65	37	68	61	40	46
Image 11	32	44	29	33	32	34	33	31
Image 12	51	48	46	39	39	45	23	26
Image 13	36	45	35	36	33	37	34	32
Image 14	34	34	29	30	28	31	19	25
Image 15	42	52	33	32	36	39	25	25
Image 16	41	45	37	45	43	42	36	36
Image 17	67	69	60	65	67	66	55	53
Image 18	63	55	51	60	59	58	53	47
Image 19	19	30	18	22	21	22	19	18
Image 20	76	75	72	71	68	72	50	52
Image 21	52	48	49	42	45	47	30	35
Image 22	26	40	26	25	26	29	24	21
Image 23	44	50	43	46	42	45	35	36
Image 24	54	56	53	46	49	52	37	46
Image 25	37	40	37	37	41	38	32	31
Image 26	35	55	30	35	37	38	32	23
Image 27	34	58	34	36	36	40	30	30
Image 28	56	58	48	58	49	54	48	43
Image 29	41	51	35	37	36	40	27	28
Image 30	42	45	43	35	38	41	12	25
Image 31	43	49	41	44	41	44	26	33
Image 32	27	46	25	27	28	31	27	22
Image 33	60	67	57	60	57	60	56	54
Image 34	95	102	93	87	83	92	58	73
Image 35	48	51	47	49	43	48	23	26

Figure 6.

Chondrocyte detection by U-Net compared with the manual segmentations of two different human observers. The images analyzed with the trained U-Net and evaluated by the human observers are shown in the top row. The resulting U-Net segmentations are compared with one of the expert observers (middle row) and with the algorithm trainer (bottom row). The middle and bottom rows are colorized manual segmentation maps with colors indicating detection accuracy.

Figure 7.

An example of relatively poor performance of our trained U-Net is shown. Several chondrocytes from superficial zone are missed, as indicated by their blue color in the middle image. Segmentation accuracy was also reduced in the superficial zone as indicated with a lower IOU shown in the right image (zoomed in view of red boxed area).

Discussion

Histopathological features encode important information about cartilage pathology and are often interpreted by human observers to evaluate cartilage health. Computer-based, automated evaluation of cartilage has been limited by challenges associated with using standard image analysis routines on the wide variety of chondrocyte appearances in osteoarthritic cartilage.²³ As cellularity (cell density, organization, etc), is a key feature of different stages of OA severity, inaccurate cell counts limit the utility of some computational analysis algorithms. In this study, a state-of-the-art deep learning method to improve automated chondrocyte segmentation was developed and evaluated. A specific CNN architecture termed the U-Net was trained using normal and arthritic rabbit cartilage samples and compared to gold-standard chondrocyte segmentations. The trained U-Net model was able to automatically and objectively identify cells accurately compared to several human observers. Furthermore, the chondrocyte boundaries identified by the trained U-Net model were very similar to those traced by human observers, especially for chondrocytes that were not immediately adjacent to the articular surface.

In contrast to traditional image processing methods which require explicit computer programming to teach the computer to detect cells from low-level features, the U-Net can detect and segment cells by learning multiple high-level features such as cell components (cytoplasm and nucleus) and cellular organization. This approach assists with separating cells that are immediately adjacent to or even overlapping each other, while remaining robust to many artifacts of histological processing. The U-Net has previously been adapted to other cell segmentation tasks where different cell types are the objects of the interest with excellent results;²⁴ however, to the best of our knowledge, this work is among the first to use the U-Net approach to identify chondrocytes with a goal of automatically and objectively assessing OA severity.

There are several limitations to this work that should be considered. First, the results of the correlation analysis indicated that our trained U-Net correlated better with the segmentations of the algorithm trainer than with those of the expert observers. This makes intuitive sense given that the U-Net was trained on the segmentation pattern of that algorithm trainer. Training the U-Net on a set of images representing expert consensus segmentation may result in a slightly differently trained model that is able to more accurately replicate expert observer cell counts. The U-Net was trained and tested on rabbit cartilage, and it is not yet clear if a single model will be accurate for evaluating cartilage of different species, or if separate algorithms will need to be trained for the variety species used to model OA. It is still challenging for the model to detect and segment chondrocytes that reside in the superficial zone. Some of those cells are very small, elongated in shape, and have homogeneous color patterns, which is a combination of features that in deeper zones is related to artifact or out of plane cells. Further training of the U-Net that includes more images of cells from the superficial zone with this appearance and more images of true non-cells from the deeper zones could improve performance of the U-Net prediction in a wider variety of cartilage locations. Finally, only a single probability threshold was utilized in this work, and it is possible that a systematic evaluation of threshold values could be performed to identify an optimal value for cell retention in the U-Net images.

In this project, state-of-the-art deep learning has applied a version of the U-Net method to detect and segment chondrocytes from histologic images of arthritic cartilage. This is a first attempt to bring machine learning to the critical histological data analysis nodes within the field of orthopedic research. Future work will include improving the U-Net’s performance to agree with expert opinion, as well as expanding the application to detect more complex cellularity features representative of OA such as cell cloning and more complex histological artifacts such as those associated with fraying or torn tissue. The resulting chondrocyte segmentation tool will be integrated into an automated Mankin scoring system²³ and used more generally as a robust, objective technique for chondrocyte identification in image processing.