Label-Efficient Deep Color Deconvolution of Brightfield Multiplex IHC Images

Abstract

Brightfield Multiplex Immunohistochemistry (mIHC) provides simultaneous labeling of multiple protein biomarkers in the same tissue section. It enables the exploration of spatial relationships between the inflammatory microenvironment and tumor cells, and to uncover how tumor cell morphology relates to cancer biomarker expression. Color deconvolution is required to analyze and quantify the different cell phenotype populations present as indicated by the biomarkers. However, this becomes a challenging task as the number of multiplexed stains increase. In this work, we present self-supervised and semi-supervised approaches to mIHC color deconvolution. Our proposed methods are based on deep convolutional autoencoders and learn using innovative reconstruction losses inspired by physics. We show how we can integrate weak annotations and the abundant unlabeled data available to train a model to reliably unmix the multiplexed stains and generate stain segmentation maps. We demonstrate the effectiveness of our proposed methods through experiments on mIHC dataset of 7-plexed IHC images.

I. Introduction

Immunohistochemistry (IHC) utilizes antibodies, coupled with sensitive reagents to detect the presence of specific proteins in tissue sections. While IHC is limited to a single biomarker, multiplexed immunohistochemistry (mIHC) extends IHC by allowing simultaneous staining of multiple biomarkers within the same tissue section. Using antibodies with differently colored stains (chromogens) for different biomarkers, mIHC images can give more advanced characterization of the tumor microenvironment (TME). With such images, we can capture the density of different cell phenotype populations, their spatial relationships, and their colocalization. All these characteristics can then be correlated with disease diagnosis, prognosis and/or survival and thus learning effective therapeutic strategies [1]–[11].

While traditional IHC only stains one biomarker per slide, mIHC is a multiplex imaging technique that enables the simultaneous staining of up to 10 targets per slide [6], [12]–[16]. This allows much more enriched analysis of of the tumor microenvironment (TME). Moreover, without the need for any specialized scanners or software, brightfield mIHC becomes an attractive multiplex imaging approach for large scale analysis [6], [12], [16]. Analyzing the resulting mIHC images, however, is not trivial. A key step is to unmix the stains, i.e., recovering the stain concentration maps of all biomarkers from a single mIHC image, so that we can identify locations of different biomarkers (see Fig. 1 and Fig. 2). However, this is an ill-posed inverse problem. When the number of biomarkers is larger than two, it requires solving an underdetermined linear equation system, and thus has infinitely many solutions. In this work, we tackle the challenging problem of automatic color deconvolution for mIHC, i.e., the decomposition of the image into its constituent stains and generating stain segmentation maps [13], [16], [17].

Fig. 1.

A sample brightfield multiplex IHC image (left) and the stain segmentation map (right) produced by our proposed method.

Fig. 2.

Illustration of the stain concentration maps and the inference pipeline. First, the model generates stain concentration maps. Then these maps are processed by a thresholding operation to keep only the regions of interest in each stain. Finally, at each pixel, the stain with the highest concentration is selected as the predicted pixel label resulting in the final stain segmentation map.

Traditionally, the problem is solved by color deconvolution, in which the original image is decomposed into separate stain concentration maps. Based on these pixel-wise concentration maps, one decides which stain is dominant at each pixel, constituting a multi-labeled semantic mask called stain segmentation map See Fig. 2 for an illustration.

Several unsupervised color deconvolution methods [18]–[21] have been applied to the analysis of traditional IHC images. The extension to mIHC with a larger number of stains is non-trivial. It involves solving an underdetermined linear equation system, which can have many undesired solutions. To address this issue, in [22], [23], a lasso regression model is proposed to find solutions with sparse concentration maps. The results, however, are not satisfying, mostly because these methods treat different pixels independently, failing to consider the contextual information. Some methods opt for implicit deconvolution for downstream tasks [24], [25], i.e., decomposing an image into multiple channels that are not guaranteed to correspond to different biomarkers. A weakly supervised deep learning approach was proposed in [26], in which the deconvolution method is supervised with dot annotations specifying centers of cells of different biomarkers. However, the model’s performance is highly dependent on a large amount of manual annotation of cell centers, and thus is extremely time consuming in practice.

In this paper, we propose a novel annotation-free, self-supervised deep learning method for multiplexed color deconvolution that can scale to a larger number of stains without the need for time-consuming manual annotations of cells. Similar to traditional color deconvolution methods, the only supervision we need are hand-picked color samples of each biomarker. Given a multiplex image and the reference stain colors, the model learns to generate a concentration map for each stain such that their combination recovers the mIHC image. The concentration maps represent the strength of the stains’ pigments in the mIHC image which allows us to derive a stain segmentation map of the corresponding biomarkers. Our method uses a convolutional neural network and thus it can properly leverage the contextual information rather than treating each pixel independently.

Our model learns to predict the concentrations of the individual stains so that their combination recovers the mIHC image. Nevertheless, with a large number of stains and without training annotations, the solution is not unique. In other words, different combinations of the stains can recover the same color. Using incorrect combinations leads to incorrect stain segmentation maps of the biomarkers. To address this challenge, we introduce novel constraints to limit the solution space. More specifically, we propose a novel Inverse Reconstruction Loss to help the network learn high quality stain concentration maps using only the reference stain colors. We quantitatively and empirically demonstrate the power of our method; even without annotations, our method can generate high quality concentration maps and stain segmentation maps that are comparable to a fully-supervised method.

However, due to the variation in staining and how cells can express multiple biomarkers simultaneously, there remain cases where experts’ supervision is required to improve the performance. To this end, we propose a semi-supervised approach. This approach takes advantage of the vast amounts of unlabeled data, and meanwhile, learns from experts through a relatively small set of weak annotations. Our method performs competitively and sometimes even superior than a fully-supervised approach since it can further learn from the variations in the vast amount of unlabeled images available. Moreover, our proposed self-supervised method can be used as a pre-training step to generate initial annotations. In this way, the pathology expert only needs to fix the wrong predictions rather than annotating from scratch. The evaluation of the self-supervised method predictions reveal a competitive performance compared to full supervision. This proves its reliability for initializing the training data annotations, thus speeding up the annotation and reducing its cost. The resulting annotations can be used to train a semi-supervised model from scratch or refine the pretrained model. In summary, our contributions are:

• An annotation-free self-supervised color deconvolution method. To the best of our knowledge, our proposed method is the first to scale to more than 4-plex staining with competitive performance to supervised solutions.

• To avoid wrong solutions in the ill-posed color deconvolution problem, we propose a novel Inverse Reconstruction Loss to regulate the model.

• We extend our self-supervised solution to the semi-supervised setting, by incorporating losses for annotated images and we study the best way to combine supervised and self-supervised losses.

• We combine our semi-supervised approach with a state-of-the-art (SoTA) ensemble approach for multiplex IHC. The result is a new SoTA stain unmixing solution.

Experiments show that our method can effectively scale to 7-plex brightfield multiplex IHC images. Meanwhile, the semi-supervised solution allows us to take advantage of expert knowledge while minimizing annotation cost and achieving new state-of-the-art (SoTA) performance.

This work is a substantial extension of our previous preliminary work [27]. The conference paper only focuses on the self-supervised setting (the first and second contributions). We expand the original work by extending the self-supervised method to a semi-supervised solution, thus creating a powerful algorithm that can annotate multiplex IHC images both efficiently and accurately. The algorithmic extension adopts supervised loss by a prior supervised method [26]. However, a direct incorporation of the supervised loss is not effective. Through a thorough empirical evaluation, we propose a combination of the self-supervised loss terms and supervised loss to maximally exploit the limited labeled data.

II. Related Work

Color deconvolution is a classical problem that has been studied extensively on IHC images that usually have one protein biomarker in addition to a counter stain. To solve this problem, most methods use the optical density (OD) color representation. In this representation, the color appearing in the IHC image is a linear combination of the concentrations of the underlying stains. The set of linear equations in the optical density space ties the reference stain vectors to the stain concentration maps [21]. See Section III. Following [21], different methods for IHC color deconvolution have been proposed [19], [20], [28]–[31].

Staining technology has been advancing rapidly. Multiplex staining now enables simultaneous staining with more than 2 biomarkers in the same tissue section. Brighfield mIHC allows multiple simultaneous biomarkers [6], [12]–[16] to be visualized in the same image. With the increasing number of multiplexed stains, the color deconvolution problem becomes more challenging and the methods developed for IHC data do not naturally extend to the mIHC domain. One reason for this is that the optical-density-based system of linear equations becomes underdetermined and numerous solutions can satisfy the system of equations. An extension of [21] was proposed in [22], [23] to support more than 2 biomarkers by using lasso regression to enforce a sparsity prior. However, with more than 4 biomarkers, the sparsity prior fails to reach good solutions, mainly because it treats all pixels independently and thus fails to consider contextual information surrounding each pixel.

Deep learning has been used to solve this problem. In [26] a convolutional autoencoder is used to estimate the concentration maps in a supervised fashion. An ensemble approach combining the autoencoder and UNet was proposed to leverage strengths of both architectures [13]. However, these supervised methods still require a large amount of annotations, which can be prohibitive in practice. For completeness, we also refer to methods that directly incorporate color deconvolution into downstream tasks [24], [25], e.g., tissue segmentation. These methods, however, do not have supervision, and thus cannot guarantee to capture different stain concentration maps.

III. Preliminary

In this section we describe the relationship of stains and their concentrations under Beer Lambert law, which is the basis for most color deconvolution methods, including ours.

Linear Stain Mixing under Beer Lambert Law

Optical Density (OD) measures the absorbance of light as it passes through a medium. The optical density at a pixel $i$, denoted as $y_i$, is the negative log of its normalized RGB values, ${\left[{\mathit{r}}_i,{\mathit{g}}_i,{\mathit{b}}_i\right]}^{\top }\in {[0,1]}^3$:

$${\mathit{y}}_i={\left[-\log \left({\mathit{r}}_i\right),-\log \left({\mathit{g}}_i\right),-\log \left({\mathit{b}}_i\right)\right]}^{\top }.$$ (1)

Beer Lambert Law (BL) stipulates that the stains and their concentrations are linearly related in the OD space. Assume a given set of $m$ stain vectors whose OD vectors are represented by the matrix $\mathit{S}=\left[{\mathit{s}}_1,{\mathit{s}}_2,\cdots ,{\mathit{s}}_m\right]\in {ℝ}^{3\times m}$, where ${\mathit{s}}_j\in {ℝ}^3$ is the OD vector of the $j^{th}$ stain. By Beer Lambert Law (BL) [32], for any pixel $i$, the optical density ${\mathit{y}}_i$ and its stain concentration vector ${\mathit{c}}_i$ satisfy a linear relationship, i.e., ${\mathit{y}}_i=\mathit{S}\times {\mathit{c}}_i$, where ${\mathit{c}}_i\in {ℝ}^m$ denotes the concentrations of the different stains at pixel $i$. Generalizing to an image with $n$ pixels, we get:

$$\mathit{Y}=\left[{\mathit{y}}_1,\cdots ,{\mathit{y}}_n\right]=\mathit{S}\times \mathit{C}$$ (2)

in which $\mathit{Y}\in {ℝ}^{3\times n}$ is the optical density matrix of all the pixels in the image and $\mathit{C}=\left[{\mathit{c}}_1,\cdots ,{\mathit{c}}_n\right]\in {ℝ}^{m\times n}$ is the corresponding stain concentration matrix.

Solving the Linear Equation. The stain mixing was first modeled by [21] using Beer Lambert Law. Given the reference stains matrix $\mathit{S}$, the stain concentrations can be recovered by solving the linear equation system, Eq. (2). In practice, stain color vectors are obtained by sampling representative pixels of each stain and the background tissue. The background color vector prevents the background tissue from getting erroneously computed as a mixture of other stains. For higher than 2-plex staining i.e., when the stain matrix $S$ consists of $m>3$ stain vectors (2 stains and the background), the linear equation system is underdetermined and does not have a unique solution. To address this challenge, different mIHC color deconvolution methods are proposed in the literature.

IV. Method

In Section IV-A, we describe our proposed self-supervised color deconvolution method, InverseAE. Next, in Section IV-B, we show how to expand the method to the semi-supervised setting, so that we can train with both labeled and unlabeled data to achieve SoTA performance with only a small fraction of labeled data.

A. InverseAE for Self-Supervised Multiplexed Color Deconvolution

In this work we propose InverseAE, a self-supervised deep-learning color deconvolution method (first published in our preliminary conference paper [27]). We assume that at each pixel there is a dominant stain which should be assigned to that pixel in the final stain segmentation maps. Hence, our proposed method is designed so that it results in meaningful stain concentration maps from which the dominant stain/biomarker at each pixel can be inferred directly. It only requires the set of multiplex stained images and the set of reference stains’ color vectors.

We use an autoencoder to generate the estimated stain concentration maps $\widehat{\mathit{C}}$ for an input image $\mathit{Y}$, based on the given stain vectors $\mathit{S}$. The autoencoder is trained with the reconstruction loss: ${\mathcal{L}}_{re\mathit{c}on}(\mathit{Y},\widehat{\mathit{C}})=MSE(\mathit{Y},\mathit{S}\times \widehat{\mathit{C}})$ that is aimed to solve Eq. (2), where MSE stands for the mean squared error function. The loss compares the image $\mathit{Y}$ with the reconstructed image from the estimated concentration maps $\mathit{S}\times \widehat{\mathit{C}}$ using the mean squared error. When the number of stain vectors $m>3$, Eq. (2) becomes an under-determined system of equations with possibly infinite number of solutions. Indeed, in practice, when we train the autoencoder with only the reconstruction loss, the model often converges to suboptimal solutions. To this end, we propose the Inverse Reconstruction Loss to impose additional constraints.

The Inverse Reconstruction Loss implements a regularized training mechanism by promoting a solution, i.e., an estimated concentration map $\widehat{\mathit{C}}$, that can reconstruct both the image and its inverse. In other words, the same solution should also reconstruct the inverted image, using the inverted stain color vectors; see the illustration in Fig. 3.

Fig. 3.

The proposed inversion regulation. The model generates stain concentration maps. Following Beer Lambert law (BL), these maps together with the reference stain vectors (top row) are used to reconstruct the input image. Similarly using the same concentration maps together with the inverted reference stain vectors (bottom row) are used to reconstruct the inverted input image. A reconstruction loss is applied on both the reconstructed image and inverted image. In this way, the inversion regulation pushes towards solution concentration maps that satisfy both the image and its inverse.

Formally, an inverted image is the image formed from the pixels’ complement color vectors. For any pixel with normalized RGB color vector ${[\mathit{r},\mathit{g},\mathit{b}]}^{\top }$, $0\le \mathit{r},\mathit{g},\mathit{b}\le 1$, its inverted color vector is ${[1-\mathit{r},1-\mathit{g},1-\mathit{b}]}^{\top }$, and its inverted optical density is

$${\mathit{y}}_{inverse}={[-\log (1-\mathit{r}),-\log (1-\mathit{g}),-\log (1-\mathit{b})]}^{\top }$$ (3)

Similarly, ${\mathit{S}}_{inverse}$ is the inverted optical density for the stain matrix optical density $\mathit{S}$. To regularize the solution of Eq. (2), we add the inverted equations into the system.

$$\text{For each pixel:}{\mathit{y}}_{inverse}={\mathit{S}}_{inverse}\times \mathit{c},$$ (4)

$$\text{For the image:}{\mathit{Y}}_{inverse}={\mathit{S}}_{inverse}\times \mathit{C},$$ (5)

The inverse reconstruction regularization loss is defined as:

$${\mathcal{L}}_{inverse}(\mathit{Y},\widehat{\mathit{C}})=MSE\left({\mathit{Y}}_{inverse},{\mathit{S}}_{inverse}\times \widehat{\mathit{C}}\right).$$ (6)

We use superscript $i$ to refer to the $i$’s sample. The overall loss function for all unlabeled samples, ${\mathcal{D}}_U$, is:

$$\sum _{i\in {\mathcal{D}}_U}\left\{{\mathcal{L}}_{recon}\left({\mathit{Y}}^i,{\widehat{\mathit{C}}}^i\right)+{\mathcal{L}}_{inverse}\left({\mathit{Y}}^i,{\widehat{\mathit{C}}}^i\right)\right\}.$$ (7)

The model is trained to generate concentration maps $\widehat{\mathit{C}}$. When combined with the stain matrix $\mathit{S}$, $\widehat{\mathit{C}}$ reconstructs the input image $\mathit{Y}$. Meanwhile, when combined with the inverted stain matrix ${\mathit{S}}_{inverse}$, $\widehat{\mathit{C}}$ reconstructs the inverted image ${\mathit{Y}}_{inverse}$. We show in the experiments how this regularization pushes the prediction towards good solutions, that is, at any location, the stain corresponding to the observed dominant color gets the highest concentration. This allows us to infer high quality stain segmentation maps and achieve evaluation scores close to the supervised solution. We next describe the rationale behind the inversion regulation.

Rationale.

This choice of inverse reconstruction as a regulator is not random. A suitable regulator requires two properties: (I) a solution exists that satisfies both the original image and the regulator, (II) the regulator adds constraints that limit the solution space by eliminating suboptimal solutions and promoting desired solutions. These properties motivate our choice of the inversion reconstruction.

First, there is a special relationship between a color and its complement (or inverse). When an object absorbs light of a particular color, we perceive the object as having the complementary color, and vice versa. Indeed, the relationship can be stated quantitatively. It can be shown that a color and its inverse share the same saturation in the HSL color space. Saturation measures the departure of a hue from white [33] and thus correlates with the amount of pigment present, i.e., its concentration. These observations suggest that a color and its inverse can share a same concentration vector, with regard to the original stains and the inverse stains, respectively. In other words, the same $\mathit{c}$ can satisfy both $\mathit{y}=\mathit{S}\times \mathit{c}$ and ${\mathit{y}}_{inverse}={\mathit{S}}_{inverse}\times \mathit{c}$. This is the desired property I.

Second, in the HSL color space, a color and its inverse have contrasting luminosity. Luminosity is defined as the average of the minimum and maximum of the red, green, and blue color components. It can be shown that when the luminosity of a color is $L$, the luminosity of its inverse is $1-L$. Similarly, we define the absorbance power of a color as the average of the minimum and maximum of its red, green, and blue light absorbance. Then, by definition, its luminosity is inversely correlated with its absorbance power. Consequently, since a color and its inverse have contrasting luminosity, they also have contrasting absorbance powers. This property allows inversion regulation to eliminate a large portion of the undesired solution space and promote good sparse solutions (property II). We next explain this further.

BL allows a color to be made up of multiple colors added together weighted by their concentrations. Consequently, in the OD space, a pixel’s color can be composed by the addition of multiple stains -other than the true stain present at that pixel- as long as their red, green, and blue OD (absorbance) values sum up to the color’s red, green, and blue OD values, respectively. This is an undesired solution. However, this solution will not hold in the inverse space as long as no stain is a function of another by a multiplicative factor. We can show this with a simple example involving the white/background stain. We can show that in the original space, a white stain can be added to the unmixing solution with an arbitrary concentration. However, this is not permitted in the inverse unmixing solution. A stain vector representing white or light colored background tissue will have very low absorbance. By BL, it can be added to any pixel without significantly modifying the pixel’s OD. On the other hand, in the inverse space, the inverse color will have high absorbance. Hence, it will significantly change the OD of any pixel it is added to blindly. Thus, the inverse regulation helps to avoid many such undesired solutions and promote good solutions.

The above discussion justifies how the inversion regulation brings special properties that are essential to converge to good solutions, and hence, produces high accuracy stain segmentation maps.

B. Semi-Supervised Multiplexed Color Deconvolution

While the proposed annotation-free, self-supervised method fully exploits the unlabeled images, it cannot reach ideal performance without supervised training. In this section, we discuss how to extend the method to semi-supervised setting. That is by adding a small portion of labeled images, (i.e. with limited human input) we can achieve satisfying color deconvolution performance. As will be shown in the experiments section, our proposed approach Semi-Unmix-AE empowers the model to match the performance of a fully-supervised model with much fewer annotated images.

To efficiently acquire human annotation, we use dot annotations, i.e., dots marking the centers of cells. Each dot also has a class/stain label associated with it. This form of annotation presents weak supervision to avoid the burden of manual segmentation labels. To use the dot annotations for pixel-wise supervision, we expand them to neighboring pixels with similar staining using the superpixel algorithm SLIC [34]. Each dot label extends to the superpixel segment it lies within. The whole superpixel is then marked as the stain of the dot. This results in a pixel-wise pseudo segmentation map, in which each pixel is labeled with one of the stains, or the background. See Fig. 4 for an illustration.

Fig. 4.

Transforming dot annotations into pseudo segmentation labels. The multiplex image is converted into superpixels. The superpixels that intersect with dot annotations are assigned the corresponding dot labels and form the pseudo segmentation labels used in training.

During training, for each labeled image, we use the pseudo segmentation map to supervise the predicted concentration maps. The loss, called label consistency loss, is also a reconstruction loss. At each pixel, the label stain from the pseudo segmentation map (Fig. 4 e) is used to reconstruct the pixel color. The label consistency loss minimizes the distance between that reconstructed image and the input image. In other words, the label consistency loss trains the model to generate concentration maps where the label stain at each location has the majority concentration, such that, by itself, it can reconstruct the multiplex image pixel to a large degree. Denote by ${\ell }_i$ the one hot vector of the label stain at pixel $i$, and let $L=\left[{\ell }_1,\cdots ,{\ell }_n\right]$. The label consistency loss is

$${\mathcal{L}}_{label}(\mathit{Y},\widehat{\mathit{C}})=\text{MSE}(\mathit{Y},\mathit{S}\times (\mathit{L}\odot \widehat{\mathit{C}})),$$ (8)

in which “$\odot$” is the Hadamard product

Our total training loss is the sum of the reconstruction loss ${\mathcal{L}}_{recon}$, the inverse reconstruction loss ${\mathcal{L}}_{inverse}$, and the label consistency loss ${\mathcal{L}}_{label}$. The reconstruction loss is applied to all images, labeled or unlabeled. The inverse reconstruction loss is applied to all unlabeled images, ${\mathcal{D}}_U$. The label consistency loss is applied to all labeled images, ${\mathcal{D}}_L$. Formally,

$$\begin{array}{l}{\mathcal{L}}_{semi}\left({\mathcal{D}}_U,{\mathcal{D}}_L\right)=\sum _{i\in {\mathcal{D}}_U\cup {\mathcal{D}}_L}{\mathcal{L}}_{recon}\left({\mathit{Y}}^i,{\widehat{\mathit{C}}}^i\right)\\ +\sum _{i\in {\mathcal{D}}_U}{\mathcal{L}}_{inverse}\left({\mathit{Y}}^i,{\widehat{\mathit{C}}}^i\right)+\sum _{i\in {\mathcal{D}}_L}{\mathcal{L}}_{label}\left({\mathit{Y}}^i,{\widehat{\mathit{C}}}^i\right).\end{array}$$ (9)

With this total loss, the network learns to generate solutions where the label stain has the highest concentration at the corresponding pixel. Fig. 5 shows the model architecture and training. It should be noted that while the inverse reconstruction loss can be applied to both labeled and unlabeled images, we choose to apply it only on unlabeled data. We hypothesize that because the labels serve to rectify how the model perceives and differentiates the stains, as a result, applying both the inverse reconstruction loss and the label consistency loss on the same data can result in conflict during training. Thus, it can potentially reduce the model’s ability to differentiate between the stains. We explore this in more details in the ablation studies.

Fig. 5.

Semi-supervised model architecture and training. The model’s output is conditioned on the input mIHC image and the target reference stain. It is trained with a reconstruction loss. Additionally, the inverse reconstruction loss is applied on unlabeled images (lower branch) and label consistency loss is applied on labeled images (top branch).

V. Experiments

We evaluate our methods on labeled and unlabeled data to show the significance of the proposed self-supervised method as well as the semi-supervised solution. To do so, we evaluate both the quality of the derived stain segmentation maps as well as the predicted stain concentration maps. We compare against a variety of baselines. The results show that our proposed self-supervised method has a competitive performance compared to SoTA supervised solutions, moreover, our semi-supervised approach is the new SoTA by boosting the performance of the previous SoTA ensemble stain unmixing method [13]. Furthermore, we present a set of ablation studies to show the efficacy of our proposed solutions.

A. Experimental Setup

We experiment with 7-plex mIHC stained pancreatic cancer tissue. There are 7 biomarkers. They include the immune cell biomarkers: CD3 (yellow), CD4 (teal), CD8 (purple), CD16 (black), CD20 (red), a tumor biomarker (brown), in addition to hematoxylin as a nuclear counterstain (blue). The dot annotations used in training and evaluation are manually set by a pathologist. Each dot marks the center of a cell and is assigned a single stain label, which is the stain for the biomarker identifying that cell type and is presumably the stain with the maximal concentration in the cell.

Annotated Training Data.

The training set consists of 60 annotated patches from 6 whole slide images. The patches has size 1200×1920 at 40x. 40 Patches from 4 cases are used for training and the remaining 20 patches from the other 2 cases are used for validation. For each deep learning method, the training is repeated 3 times and we report the mean and standard deviation. Each time a different pair of cases are used in the validation set and the other 4 are used in training. The training images are resized to 20x and cut up into 400 × 400 overlapping patches, resulting in a total of 240 labeled training patches.

Unlabeled Training Data.

For semi-supervised training, additional patches without annotation are extracted from 7 more cases. 60 patches of size 400 × 400 at 20x are randomly selected from each case, resulting in a total of 420 additional patches.

Models Compared.

We compare our proposed methods: (i) InverseAE: the self-supervised model. It is trained on the set of 240 labeled images but without using the labels. (ii) InverseAE+: similar to InverseAE and is trained with 240 images as before, plus the additionally collected unlabeled training data. This model is comparable to the semi-supervised model Semi-Unmix-AE (mentioned next) since it is trained on the same set of images. (iii) Semi-Unmix-AE: the proposed semi-supervised method. It is trained using both the annotated and the additional unlabeled training data described previously. Table VI shows the models’ architecture. These methods are compared against:

• Unsupervised baselines: (i) LassoReg: the lasso regression method described in [22] which is designed to work on images with 4+ multiplex staining. We use L1 norm weight $\lambda ={10}^{-3}$. (ii) PseudoInv: an extension of [21] for color deconvolution that solves the equation $C=S^{-1}\times Y$ for more than 3 stain vectors using pseudo inverse of $S$, (iii) Nearest Neighbor: a simple baseline that assigns to each pixel the biomarker with the shortest color distance.

• Weakly supervised baselines: (i) ColorAE [26]: the weakly supervised method. It is trained on the 240 annotated patches. (ii) UNet: a deep learning model based on UNet model architecture [36]. It is a supervised method. Here, the output is not stain concentration maps but rather likelihood maps. For each pixel, the output is the probability that the pixel belongs to each stain class. During training, the superpixel annotation maps are used as the ground truth labels.

• Ensemble methods: (i) Union anchor AE [13]: SoTA stain unmixing method. It combines the predictions of both UNet and ColorAE, taking advantage of both cross entropy loss and image reconstruction losses. (ii) Union anchor Semi-AE: replaces ColorAE in the previous ensemble with Semi-Unmix-AE. It proves to be the new SoTA.

TABLE VI.

Model architecture of InverseAE and Semi-Unmix-AE.

Encoder	Decoder
Conv2D (1×1, 128 channels)	ConvTranspose2d (3×3, 16 channels)
ReLU	Conv2D (3×3, 32 channels)
Conv2D (3×3, 64 channels)	ReLU
MaxPool2d	ConvTranspose2d (3×3, 32 channels)
ReLU	Conv2D (3×3, 64 channels)
Conv2D (3×3, 32 channels)	ReLU
MaxPool2d	Conv2D (3×3, 128 channels)
ReLU	ReLU
Conv2D (3×3, 16 channels)	Conv2D (3×3, <num. stains> channels)
ReLU

Generating Stain Segmentation Maps.

To compute the stain segmentation maps (Fig. 2), we assume a single dominant stain at each pixel. During inference, each stain concentration map is thresholded using hysteresis thresholding and then an argmax function is applied to select the stain with the maximal concentration at each location. In this way, the resulting segmentation map has a single stain/biomarker assigned at each pixel that reflects the predicted dominant stain at each location. The hysteresis thresholding ensures that for each stain we only get regions where there is high confidence in the presence of that stain and then expand it to cover the larger cell area. The threshold parameters are fixed for each stain and are empirically selected on a small held out set. For the supervised and semi-supervised methods, there is more confidence in the predicted stain concentration maps, Fig. 9. As a result, we can directly apply the argmax function. In the case of UNet, the stain with the highest predicted likelihood is selected at each pixel.

Fig. 9.

Comparing predicted stain concentration maps by InverseAE and Semi-UnmixAE. The semi-supervised results better reflect the visible stain and show more confidence in the prediction compared to the unsupervised results.

B. Evaluation Metrics

Evaluation with Dot Annotations.

We test the model on 96 patches of size 1200 × 1920 from 42 test cases. These test patches have manual dot annotation for each biomarker where the annotation marks the approximate centers of the cells. We use this test data to evaluate the model’s accuracy in capturing the per-pixel dominant stains. We generate stain segmentation maps from each models predictions. The F-score for each biomarker/cell type is computed as follows: a true positive is a predicted segment that intersects with an annotation dot, a false negative is a segment that doesn’t intersect with any annotation dot, a false negative is an annotation dot that does not intersect with any predicted segment. This allows us to assess the quality of the predicted concentration maps for direct downstream analysis. The results are shown in Table I.

TABLE I.

F-score of stain segmentations with respect to dot annotations on 96 patches from 42 cases. The models are grouped into self/unsupervised, weakly/semi-supervised, and ensemble models. Best result Within each group is indicated in bold and best over all models is underlined.

Method	CD3	CD4	CD8	CD16	CD20	Tumor	Mean
PseudoInv	0.158	0.208	0.441	0.267	0.265	0.403	0.290
Nearest Neighbor	0.223	0.168	0.734	0.277	0.106	0.357	0.311
LassoReg	0.456	0.501	0.801	0.050	0.223	0.389	0.404
InverseAE (Ours)	0.596 ± 0.036	0.526 ± 0.029	0.799 ± 0.014	0.671 ± 0.028	0.317 ± 0.015	0.591 ± 0.029	0.583 ± 0.005
InverseAE+ (Ours)	0.581 ± 0.018	0.555 ± 0.052	0.793 ± 0.010	0.684 ± 0.011	0.321 ± 0.018	0.572 ± 0.053	0.585 ± 0.005
UNet	0.579 ± 0.060	0.541 ± 0.035	0.820 ± 0.025	0.712 ± 0.032	0.348 ± 0.145	0.566 ± 0.118	0.595 ± 0.059
ColorAE	0.645 ± 0.006	0.583 ± 0.035	0.837 ± 0.006	0.740 ± 0.020	0.404 ± 0.036	0.563 ± 0.044	0.629 ± 0.013
Semi-Unmix-AE (Ours)	0.665 ± 0.019	0.597 ± 0.014	0.824 ± 0.006	0.734 ± 0.017	0.378 ± 0.008	0.659 ± 0.01	0.643 ± 0.007
Union-anchor-AE	0.707 ± 0.003	0.633 ± 0.007	0.871 ± 0.011	0.810 ± 0.020	0.436 ± 0.033	0.707 ± 0.020	0.694 ± 0.010
Union-anchor-Semi-AE (Ours)	0.701 ± 0.013	0.633 ± 0.037	0.874 ± 0.010	0.815 ± 0.022	0.452 ± 0.051	0.736 ± 0.052	0.702± 0.018

Comparison with Pseudo-Ground Truth Concentration Maps.

We wanted to assess the quality of the concentration maps generated by the trained models, however, there is no ground truth for this problem. So in this evaluation, we evaluate our deconvolution result using single stained IHC images. These single stained IHC images are classic and relatively simple; they only have hemaoxilin staining for the nuclei and one other biomarker of interest. On such single stain slides, we use traditional deconvolution methods (Vahadan [20] and Bayesian K-SVD [18]) to extract the stain concentration map. The assumption is that on such single stain samples, the deconvolution results are reasonably reliable using IHC color deconvolution methods and can be used as pseudo-“ground truth”. Our models however, will treat these images as multiplex-IHC images, which allows us evaluate their deconvolution performance. We compare the concentration maps generated by our multiplex models to those generated by [20] and [18] using structure similarity index (SSim). The results are in Table IV. All the mean scores are above 0.8 which indicate a reliable performance by all models. In our experiments, Bayesian-K-SVD seems to fail to deconvolve almost all of the CD16 images. This is why the corresponding column is left blank in Table IV.

TABLE IV.

Evaluation of Concentration Maps Using Structure Similarity Score on Single Stain Images.

Reference	Vahadan [20]
Method	CD3	CD4	CD8	CD16	CD20	Tumor	Mean
ColorAE	0.903 ± 0.004	0.886 ± 0.012	0.913 ± 0.006	0.621 ± 0.060	0.853 ± 0.001	0.844 ± 0.003	0.837 ± 0.013
InverseAE	0.934 ± 0.000	0.885 ± 0.012	0.881 ± 0.016	0.682 ± 0.001	0.867 ± 0.001	0.811 ± 0.009	0.843 ± 0.002
Semi-Unmix-AE	0.917 ± 0.001	0.866 ± 0.009	0.873 ± 0.002	0.715 ± 0.001	0.857 ± 0.003	0.848 ± 0.001	0.846 ± 0.002
Reference	Bayesian K-SVD [18]
Method	CD3	CD4	CD8	CD16	CD20	Tumor	Mean
ColorAE	0.902 ± 0.007	0.885 ± 0.002	0.927 ± 0.006	-	0.887 ± 0.002	0.856 ± 0.001	0.892 ± 0.002
InverseAE	0.932 ± 0.001	0.860 ± 0.015	0.898 ± 0.004	-	0.891 ± 0.001	0.831 ± 0.004	0.882 ± 0.003
Semi-Unmix-AE	0.916 ± 0.001	0.844 ± 0.007	0.895 ± 0.002	-	0.892 ± 0.000	.854 ± 0.001	0.880 ± 0.001

Model Complexity.

The proposed method uses a lightweight model with around 200K trainable parameters. The experiments were run on RTX 8000 GPU. The model processes a single test image of size 900x600 in around 1.5 sec and the training time per epoch is around 94.7 sec/epoch.

C. Results of the Self-Supervised Model InverseAE.

F-score Evaluation. Table I shows the F-score results of comparing the prediction against the ground truth dot annotations. Our method performs superior in the self/unsupervised setting. The baselines Nearest Neighbor and LassoReg are slightly behind the InverseAE, while PseudoInv fails to bring meaningful concentration maps. While InverseAE performs better than UNet, as expected, it does not surpass the supervised models ColorAE and Union Anchor AE ensemble, however its performance is competitive. Adding more unlabeled training data in InverseAE+ does not yield significant improvement.

SSim Evaluation.

Table IV shows the SSim results on the single stained slides. All the scores are above 0.8 except for the black macrophages CD16 which can be challenging as it is often non-uniform and messy. See Fig. 6 (B and F).

Fig. 6.

Samples stain segmentation results, comparing InverseAE and Semi-UnmixAE (our proposed unsupervised and semi-superviseds solutions, respectively) against the pseudo segmentation labels created from dot annotations and ColorAE (the supervised method). In B and D, when the tumor brown staining is light the unsupervised method (InverseAE) can miss it, while the semi-supervised method (Semi-UnmixAE) best captures the whole brown region even with light staining. In E, some of the teal stain (CD4) is missed from InverseAE and ColorAE predictions but the semi-supervised model can capture them. Also the very dark brown is mistakenly predicted as black stain (CD16) while correctly predicted in supervised models. In (A), InverseAE mistakes the black stain (CD16) for brown (tumor), and purple (CD8) for red (CD20) stain. On the other hand, Semi-UnmixAE captures teal stain (CD4) missed by both InverseAE and ColorAE methods.

Ablation Study: InverseAE Design Choices.

We present ablation studies on the design choices for our self-supervised model InverseAE. Table II presents the F-score results for the 96 annotated patches using variations from our method.

TABLE II.

Ablation study on the self-supervised model design InverseAE, evaluated with F-score of stain segmentations with respect to dot annotations on 96 patches from 42 cases.

Inverse Reg.	Conditional AE	L1 Sparsity	Inv. Aug.	CD3	CD4	CD8	CD16	CD20	Tumor	Mean
	✓			0.235	0.307	0.409	0.121	0.326	0.147	0.257
	✓	✓		0.215	0.228	0.337	0.000	0.300	0.000	0.180
	✓		✓	0.407	0.411	0.716	0.000	0.117	0.000	0.275
✓				0.482	0.501	0.753	0.324	0.274	0.220	0.426
✓	✓			0.644	0.530	0.813	0.632	0.327	0.556	0.584

First, the results show the importance of the inverse reconstruction loss. Without it (rows 1–3), the predicted maps are not representative of the stains visually present in the mIHC image. This is clear from the very low F-scores. Second, replacing the inverse reconstruction loss with a L1 sparsity constraint (row 2) is not sufficient to get representative concentration maps. We notice sparsity in the prediction, nevertheless the prediction is not correlated with the observed stains. We find the dominating stains in the prediction are mostly CD3’s yellow and CD8’s purple, while the tumor (brown) and the CD16 (black) biomarkers are missing, as evidenced by their zero F-score in the table. Third, instead of the inverse regulation loss, we use the inverted images and stains as data augmentation (row 3). We observe that we get slightly improved performance compared to the L1 sparsity constraint (row 2), however it also suffers from missing biomarkers in the predicted concentration maps. Finally, without the conditional autoencoder (row 4) i.e., predicting all the stains at once without target stains conditional input, the scores are lower and the model does not generalize well. The conditional input prevents the model from overfitting and allows the model to better capture the target stains in the image.

D. Results of the Semi-supervised Method

F-score Evaluation.

We evaluate our semi-supervised method and report the results against the dot annotations in Table I. Semi-Unmix-AE achieves the best F-score among supervised and unsupervised methods. This proves how the inverse reconstruction loss can take advantage of the variations offered by including the unlabeled data to boost the performance further.

Ensemble Models.

The best performing models in Table I are the ensemble models (last 2 rows). We experiment with Replacing ColorAE in Union Anchor AE [13] with Semi-Unmix-AE. We call this model Union anchor Semi-AE (Table I, last row). Utilizing Semi-Unmix-AE boosts the performance further and results in a new SoTA model. The results support our claim of the value of the proposed semi-supervised learning approach.

SSim Evaluation.

The quality of the predictions are further assessed with the SSim scores. Semi-Unmix-AE achieves best scores, especially for the more challenging stains: Brown (Tumor) and Black (CD16 Macrophages), where performance improvement is more significant. It also holds the highest mean SSim across all stains.

Ablation Study: Amount of labeled data in semi-supervised training.

We evaluate the tradeoff of the amount of labeled and unlabeled training data. We train 5 different models on the same data as described before, except that for the labeled images, each model only trains with the labels of a subset of these images and the rest are treated as unlabeled. We compare with the models trained with 10%, 30%, 50%, and 70% of the annotated images, Table III. If we inspect the mean F-score in table, we observe the 10% model performance is very close to the self-supervised model InverseAE. The 30% and 50% models’ performances are very close to the weakly supervised model Semi-Unmix-AE. The 70% and 100% models’ performances surpasses both InverseAE and ColorAE. This confirms our hypothesis that the two losses complement each other. See the plot in Fig. 7.

TABLE III.

Ablation study on semi-supervised model using different subsets on annotated training data. Evaluated with F-score of stain segmentations with respect to dot annotations on 96 patches from 42 cases.

Method	CD3	CD4	CD8	CD16	CD20	Tumor	Mean
Semi-Unmix-AE (10%)	0.622	0.591	0.827	0.620	0.357	0.505	0.587
Semi-Unmix-AE (30%)	0.647	0.623	0.834	0.671	0.348	0.577	0.617
Semi-Unmix-AE (50%)	0.672	0.616	0.815	0.674	0.310	0.622	0.618
Semi-Unmix-AE (70%)	0.671	0.614	0.834	0.712	0.406	0.652	0.648
Semi-Unmix-AE (100%)	0.686	0.616	0.831	0.710	0.385	0.654	0.647

Fig. 7.

The Plot of supervised vs semi-supervised stain segmentation F-score when training with different percent subsets of data annotations. Both models are trained with a subset the annotated data. The semi-supervised model additionally takes advantage of unlabeled data. The semi-supervised model F-scores are on an increasing trajectory with the increase in annotations, while the F-score of the supervised solution remains within a range because of the limited annotations.

Ablation Study: Weight of inverse reconstruction loss applied on labeled data.

We explore how to best model the training in the semi-supervised model. From Eq. 9, we apply ${\mathcal{L}}_{inverse}$ on the unlabeled training data only. To reach this design decision we conduct an ablation study on the weight $\lambda$ given to ${\mathcal{L}}_{inverse}$ when applied to labeled data. In Table V, we compare models trained with $\lambda \in \{1,0.75,0.5,0.25,0\}$. We find the best performance is when $\lambda =0$ and worst is when $\lambda =1$. As $\lambda$ is increased from 0 to 1 the performance deteriorates. We believe that this is caused by conflict between the supervised and self-supervised losses when applied on the same image. The supervision is supposed to guide the model towards solutions that fix issues that would be otherwise be incorrect given only the self-supervised losses. Looking at the mean column, The performance monotonically decreases as $\lambda$ increases. These results support our model design decisions.

TABLE V.

Ablation study on weight of inverse reconstruction loss applied on labeled data when training Semi-Unmix-AE.

$\lambda$	CD3	CD4	CD8	CD16	CD20	Tumor	Mean
1	0.619	0.085	0.832	0.676	0.012	0.534	0.460
0.75	0.719	0.561	0.832	0.675	0.249	0.536	0.595
0.5	0.702	0.453	0.822	0.754	0.238	0.580	0.592
0.25	0.731	0.609	0.841	0.730	0.074	0.588	0.596
0	0.686	0.616	0.831	0.710	0.385	0.654	0.647

Ablation Study: Sensitivity to Variation in Pseudo Segmentation Labels.

We evaluate the impact of variations in the pseudo segmentation labels generation algorithm. In Table VII, we show the effect of varying the compactness parameter in the SLIC algorithm and using SAM to generate the segmentation labels. Fig. 10 shows the difference in the stain pseudo segmentation labels obtained with the different variations. We evaluate the supervised methods, i.e. UNet, ColorAE, and Semi-Unmix-AE and observe that for each configuration we get similar relative performance across the 3 models. The best performance uses SLIC with compactness=10, which is the configuration we use in our experiments. Moreover, Semi-Unmix-AE is the least sensitive to variation in pseudo labels, which supports the effectiveness of the unsupervised loss component.

TABLE VII.

Evaluation of models trained with different pseudo segmentation labels.

	CD3	CD4	CD8	CD16	CD20 I	Tumor	Mean
Method	SLIC [34], Compactness=0.1
UNet	0.563	0.509	0.795	0.753	0.452	0.093	0.528
ColorAE	0.633	0.439	0.765	0.762	0.349	0.632	0.597
Semi-Unmix-AE	0.674	0.508	0.822	0.752	0.317	0.659	0.622
	SLIC [34], Compactness=1
UNet	0.621	0.484	0.810	0.738	0.478	0.103	0.539
ColorAE	0.657	0.496	0.813	0.748	0.333	0.581	0.605
Semi-Unmix-AE	0.653	0.514	0.825	0.747	0.376	0.655	0.628
	SLIC [34], Compactness=10
UNet	0.496	0.496	0.835	0.699	0.144	0.405	0.513
ColorAE	0.639	0.585	0.829	0.728	0.353	0.573	0.618
Semi-Unmix-AE	0.686	0.616	0.831	0.710	0.385	0.654	0.647
	SLIC [34], Compactness=25
UNet	0.633	0.528	0.801	0.704	0.433	0.108	0.535
ColorAE	0.652	0.419	0.832	0.703	0.313	0.579	0.583
Semi-Unmix-AE	0.683	0.546	0.793	0.746	0.357	0.614	0.623
	SLIC [34], Compactness=100
UNet	0.527	0.467	0.718	0.585	0.378	0.092	0.461
ColorAE	0.690	0.410	0.769	0.759	0.218	0.622	0.578
Semi-Unmix-AE	0.666	0.481	0.816	0.736	0.374	0.620	0.615
	SAM [35]
UNet	0.557	0.516	0.733	0.582	0.405	0.094	0.481
ColorAE	0.651	0.505	0.831	0.765	0.373	0.538	0.610
Semi-Unmix-AE	0.702	0.527	0.822	0.757	0.406	0.601	0.636

Fig. 10.

Comparing the pseudo stain segmentation labels obtained with different configurations of SLIC [34] and SAM [35]. In our experiments we use SLIC with compactness=10.

It is worth mentioning that there are different strategies to generate the segmentation labels with SAM. We explored the following:

• Using all the positive and negative labels for each stain, i.e. using the stain dot annotations as positive input coordinates and other stains as negative coordinates.

• using only positive labels for each stain, i.e. submit all the stain dot annotations as positive input coordinates and no negative input.

• using only positive labels for each stain, submitting one dot coordinates at a time and get the results for each one separately.

We found the last strategy provided the most reasonable pseudo ground truth segmentation maps which we used in the training.

Ablation Study: Variation in Model Architecture.

The architecture of the model is shown in Table VI. In the design of the auto encoder, we take into consideration that it is easy to overfit on the color values in the input. To that end we choose a moderate depth and width for the network to avoid overfitting and at the same time be able to capture the essential features. We show the effect of increasing the depth and width of the autoencoder. The configurations studied are in Table IX. In Configurations B and C, we experiment with increasing the width of the network. The number of channels in each layer are doubled or quadrupled, respectively. In Configuration D, we increase the depth of the network with an additional block in each of the encoder and decoder. In Configuration E, we both double the number of channels and increase the depth. We evaluate the different architecture on the test set and show the F-scores in Table VIII. We observe that the performance drops with either increase in width (Conf B and C) or depth (Conf D) and fails completely with an increase in both (Conf E). This suggests that the model overfits and its capability to generalize to unseen data is reduced, thus supporting our choice of model architecture.

TABLE IX.

Model Configurations in Ablation study on architecture design choices. ReLU activation layers are omitted for brevity. ConvT is Transposed Convolution. Conv3–128 is a 2D convolution with 3×3 kernel and outputs 128 channels. $\mathit{n}$ is number of stains or biomarkers.

A Proposed	B Inc. Width-1	C Inc. Width-2	D Inc. Depth-1	E Inc. Depth-2
Encoder
Conv1–128	Conv1–256	Conv1–512	Conv1–128	Conv1–256
Conv3–64	Conv3–128	Conv3–128	Conv3–64	Conv3–128
MaxPool	MaxPool	MaxPool	MaxPool	MaxPool
Conv3–32	Conv3–64	Conv3–128	Conv3–32	Conv3–64
MaxPool	MaxPool	MaxPool	MaxPool	MaxPool
Conv3–16	Conv3–32	Conv3–64	Conv3–16	Conv3–32
			MaxPool	MaxPool
			Conv3–8	Conv3–16
Decoder
ConvT3–16	ConvT3–32	ConvT3–64	ConvT3–8	ConvT3–16
Conv3–32	Conv3–64	Conv3–128	Conv3–16	Conv3–32
ConvT3–32	ConvT3–64	ConvT3–128	ConvT3–16	ConvT3–32
Conv3–64	Conv3–128	Conv3–256	Conv3–32	Conv3–64
Conv3–128	Conv3–256	Conv3–512	ConvT3–32	ConvT3–64
Conv3-n	Conv3-n	Conv3-n	Conv3–64	Conv3–128
			Conv3–128	Conv3–256
			Conv3-n	Conv3-n

TABLE VIII.

Ablation Study on Model Architecture. The detailed architecture of the models is in Table IX

Architecture	CD3	CD4	CD8	CD16	CD20	Tumor	Mean
A (Proposed)	0.686	0.616	0.831	0.71	0.385	0.654	0.647
B (Inc. Width-1)	0.635	0.488	0.81	0.744	0.309	0.565	0.592
C (Inc. Width-2)	0.568	0.559	0.791	0.747	0.255	0.572	0.582
D (Inc. Depth-1)	0.649	0.486	0.794	0.726	0.305	0.562	0.587
E (Inc. Depth-2)	0.158	0.127	0.249	0.282	0.03	0.59	0.239

Qualitative Results.

Sample results from the different methods are visualized in Fig. 8 and Fig. 6. From Fig. 8, we observe that across all evaluated method the segmentation maps from InverseAE, ColorAE, and Semi-Unmix-AE are the most consistent with the true stains seen in the sample patches. Comparing those three methods more closely in Fig. 6, we see that InverseAE can sometimes miss light staining or mistake colors with closely overlapping spectrum. Semi-UnmixAE can make use of additional unlabeled data to avoid errors made by InverseAE and ColorAE. Fig. 9 shows the raw concentration maps generated by InverseAE and Semi-UnmixAE. The samples reflect more confidence in Semi-UnmixAE predictions at the true stain regions which is the reason behind the higher quality stain segmentation maps.

Fig. 8.

Sample stain segmentation results from the different methods.

VI. Conclusion

With the continuous increase in the number of simultaneous staining in brightfield multiplex IHC and its technological and economical accessibility advantages, it remains challenging in practice to decompose the resulting multiplex images into the individual stains for analysis. In this work, we have tackled the problem of color deconvolution in multiplex IHC images with relatively large number of stains. We propose both self-supervised and semi-supervised methods. Our proposed methods require little effort from pathologists and the results can be directly used to obtain stain segmentation maps that capture the underlying dominant stain at each location. Thus transforming brightfield multiplex IHC into a practical multiplex staining option for research and analysis.