Society for Immunotherapy of Cancer: updates and best practices for multiplex immunohistochemistry (IHC) and immunofluorescence (IF) image analysis and data sharing

Abstract

Objectives

Multiplex immunohistochemistry and immunofluorescence (mIHC/IF) are emerging technologies that can be used to help define complex immunophenotypes in tissue, quantify immune cell subsets, and assess the spatial arrangement of marker expression. mIHC/IF assays require concerted efforts to optimize and validate the multiplex staining protocols prior to their application on slides. The best practice guidelines for staining and validation of mIHC/IF assays across platforms were previously published by this task force. The current effort represents a complementary manuscript for mIHC/IF analysis focused on the associated image analysis and data management.

Methods

The Society for Immunotherapy of Cancer convened a task force of pathologists and laboratory leaders from academic centers as well as experts from pharmaceutical and diagnostic companies to develop best practice guidelines for the quantitative image analysis of mIHC/IF output and data management considerations.

Results

Best-practice approaches for image acquisition, color deconvolution and spectral unmixing, tissue and cell segmentation, phenotyping, and algorithm verification are reviewed. Additional quality control (QC) measures such as batch-to-batch correction and QC for assembled images are also discussed. Recommendations for sharing raw outputs, processed results, key analysis programs and source code, and representative photomicrographs from mIHC/IF assays are included. Lastly, multi-institutional harmonization efforts are described.

Conclusions

mIHC/IF technologies are maturing and are routinely included in research studies and moving towards clinical use. Guidelines for how to perform and standardize image analysis on mIHC/IF-stained slides will likely contribute to more comparable results across laboratories and pave the way for clinical implementation. A checklist encompassing these two-part guidelines for the generation of robust data from quantitative mIHC/IF assays will be provided in a third publication from this task force. While the current effort is mainly focused on best practices for characterizing the tumor microenvironment, these principles are broadly applicable to any mIHC/IF assay and associated image analysis.

Introduction

The need to better understand the immune response to cancer is a key challenge in the field of immuno-oncology (IO). Closely related is the need to define predictive tissue-based biomarkers of response to available immunotherapies or new drugs under development.1,3 Some of the most promising, emerging biomarkers for IO use emerging multiplex immunohistochemistry/immunofluorescence (mIHC/IF) technologies to characterize the spatial relationship between multiple cell types, the location of cells within compartments of the tumor microenvironment (TME), for example, tumor versus stroma,⁴ and complex coexpression-defined immunophenotypes, including immunoactive marker expression. Specifically, a meta-analysis that compared mIF/IHC assays to programmed death-ligand 1 (PD-L1) IHC, interferon-gamma-related gene signatures, and mutational density for predicting response to anti-PD-(L)1 therapies, showed that mIF/IHC assays had an area under the summary receiver operating characteristic curve on the order of 0.8, while the other modalities had an area under the curves (AUCs) of∼0.65–0.7.

While there are no established guidelines, a validated AUC of 0.8 or above is in keeping with a potential companion diagnostic and may warrant consideration for a biomarker-driven clinical trial.⁵ Some examples of mIF/IHC biomarkers that have been shown to predict therapeutic responses with AUCs in this range include: Quantifying the proportion of intratumoral CD8+CD39+ cells⁶ or the density of CD8+FoxP3+ T cells in patients with non-small cell lung carcinoma;⁷ assessing the density of programmed cell death protein-1 (PD-1)+to PD-L1+cells within a certain proximity in patients with Merkel cell carcinoma;⁸ and a combinatorial biomarker using CD8+FoxP3+PD-1^low/mid+and CD163+PD-L1− cell densities in patients with advanced melanoma.⁹ These technologies have also been used to describe and categorize the TME of different tumor types into geographic immune contexts, also known as “immunotypes” that may be used to further inform our understanding of immune escape mechanisms and predict therapeutic responses.^{4 10}

Many analysis strategies for mIHC/IF provide single-cell resolution, while others are analyzed by pixel-based approaches that rely on regional similarities to infer coexpression.¹¹ Regardless, the resultant rich and complex mIHC/IF slide data typically require specialized computer algorithms for exploration. A comprehensive overview of different mIF/IHC technologies is provided in “The Society for Immunotherapy of Cancer statement on best practices for multiplex immunohistochemistry (IHC) and immunofluorescence (IF) staining and validation”, and a smaller table for quick reference is also provided here in table 1.

Table 1. Summary of current multiplex IHC/IF technologies (abbreviated from the Society for Immunotherapy of Cancer statement on best practices for multiplex immunohistochemistry (IHC) and immunofluorescence (IF) staining and validation)¹¹.

	Light microscopy		Fluorescence microscopy
	Multiplex IHC	MICSSS	Multiplex IF	DSP	Tissue-based mass spectrometry
Basic description	Simultaneous/sequential application of immunostaining without removal of the previous marker.	Iterative cycles of immunostaining, scanning, removal of chromogenic enzyme substrate and blocking previous primary antibody.	Iterative cycles of immunostaining using cyclical stain/stripping, TSA amplification, or DNA barcodes.	1º antibodies bound to UV cleavable fluorescent DNA tags. A numerical value is generated that corresponds to the # of antibodies bound.	Mass spectrometry imaging of primary antibodies tagged with elemental mass reporters.
# markers per section	3–5	10+	5–8 for TSA-based, 30–60 for non-TSA-based, cyclical staining approaches.	40–50	40
Imaging area	Whole slide	Whole slide	Up to the whole slide	ROI=0.28 mm2 (larger areas may be imaged by tiling ROIs).	ROI=1.0 mm2 (larger areas may be imaged by tiling ROIs).

DSPDigital Spatial ProfilingMICSSSmultiplexed immunohistochemical consecutive staining on single slideROIregion of interestTSAtyramide signal amplificationUVultraviolet

That manuscript also includes a discussion of the optimization and validation of immunolabeling mIHC/IF assay conditions (eg, antibody clone, antigen retrieval) that are critical to achieving the required performance criteria of individual markers have been discussed. Importantly, the digital image processing pipeline for mIHC/IF assays must also be validated and optimized, with quality assurance (QA) and quality controls (QC) applied to all steps from image acquisition and processing through final data output.

This current, companion manuscript describes a general framework for typical image acquisition and analysis workflows, associated QA/QC considerations, interinstitutional harmonization strategies, and data-sharing models. The aim is to define a basic understanding of “responsible use” by investigators to facilitate robust data generation and associated minimum reporting elements. Specifically, we believe the first step is to ensure that investigators are actually reporting key details of how they designed and performed their analytic steps (eg, cell segmentation, phenotyping, batch correction). Broad, detailed reporting of methodology will enable anticipated future analyses of which approaches provide the most robust results for a given research question.

Image acquisition

The first step in digital pathology analysis is to acquire an image of part or all of a slide for analysis. figure 1. Successful image analysis and reliable resultant outputs are significantly dependent on a priori conditions, such as separable color combinations used and scanning procedures (eg, calibrated scanners, well-focused images, reduced tiling effects). For image acquisition, a protocol is typically designed in the application’s software, detailing the microscope objective and exposure time for each filter set. The appropriate per-pixel resolution for acquisition, as well as whether a whole slide (ie, the entire tissue section on a slide) or a specific region of interest (ROI) is imaged, depends on the specific research question, method, and technology used. For example, the multiplexed immunohistochemical consecutive staining on single slide (MICSSS) method typically acquires whole slide images using a bright field scanner.¹² In contrast, for some of the mIF technologies, specific ROIs rather than the entire slide are sometimes acquired.

Figure 1. Overview slide scan of an mIF stained slide showing different HPF/ROI sampling strategies for image acquisition and subsequent analysis. (A) A 10× resolution overview slide scan from an advanced stage melanoma specimen taken by a Vectra multispectral microscope. The background lymph node and TIL appears as light blue–green surrounding the periphery of the tumor nodule. (B) Representative photomicrograph generated in Python showing HPF/ROI selection targeting “hot spots” with high CD8+T cell density, located here at the tumor-stromal interface. These HPFs can then be acquired (often at higher resolution) and studied in detail. (C) Representative photomicrograph showing sampling of the TME with fields deliberately chosen across both the central and more peripheral regions of the tumor nodule as well as areas with and without a prominent immune cell infiltrate. The aim of this sampling strategy is to capture the potential heterogeneity of the TME. (D) An example of a high resolution HPF taken with a Vectra multispectral microscope that can be used for further downstream analysis. The displayed HPF was generated in Python using an unmixed component image exported from inForm. HPF, high-power field; PD-1, programmed cell death protein-1; PD-L1, programmed death-ligand 1; ROI, region of interest; TIL, tumor-infiltrating lymphocyte; TME, tumor microenvironment.

To date, ROI selection has varied significantly between studies and is typically user-dependent. It is also dependent on the size of the tissue being scanned, for example, core biopsies versus large resection specimens. Previous studies have generally sampled a minimum of five high-power fields (HPFs), ranging from around 0.33–0.64 mm² each.¹³ Some studies have deliberately sampled regions based on morphological features or immune cell densities (ie, “hotspots” and “coldspots”),14,17 while others have assessed immune cells in both the tumor core and tumor invasive margin.¹⁸ If a marker or phenotype is rare or extremely heterogeneous, then extended sampling strategies may be required.⁹ In these instances, the diagnostic value of ROI selection compared with whole slide acquisition should first be preliminarily explored before deployment on the complete test cohort, including ideally using test, validation, and external validation cohorts.^{9 19 20} To ensure reproducibility across studies, it is essential that investigators describe their approach to ROI selection—the number of ROI analyzed per specimen, how the ROI was chosen, and whether there are criteria for ROI inclusion/exclusion at the analysis phase.

Whole slide imaging, although more demanding in terms of time and computation, can be valuable where marker or tissue heterogeneity is high or when complex cell populations are being assessed across multiple large regions. With technological advances, the analysis of whole-slide tissue is becoming a common practice, as opposed to selected HPFs. Further, whole slide imaging can facilitate semiautomated or automated hotspot detection.921,23 Whole-slide imaging followed by automated ROI detection has also been shown to improve the signal-to-noise ratio for certain mIF assays, resulting in improved predictive value.^{9 24}This approach also reduces potential HPF/ROI selection bias, which will be of benefit in standardizing outputs across studies and institutions as these technologies move towards clinical use.

Color deconvolution and spectral unmixing

For both mIHC and mIF, color deconvolution and spectral unmixing, respectively, are essential for an accurate assignment of marker expression. This process has a pronounced impact on the downstream steps of cell segmentation, phenotyping and scoring.

Singleplex chromogenic IHC uses a single primary antibody to detect and visualize the presence of a target protein in tissues or cells, producing a visible color reaction. When multiple markers are used, color deconvolution of the resultant color vectors from red, green, blue (RGB) images can be used to identify the specific contribution of individual stains.²⁵ For example, color deconvolution algorithms can extract hematoxylin and separate chromogen channels as 8-bit images, figure 2A.^{25 26} This results in a separate chromogenic channel for each immunostain, where each chromogen is assigned a unique color code that can be used to generate pseudocolored images of the tissue, figure 2B. It is important to note that color deconvolution techniques assume a linear relation between stain concentration and absorbance.²⁷ This is not always the case—for example, after enzymatic chromogen signal saturation, the ratio of target to signal is no longer linear.

Figure 2. mIHC/IF images contain multiple color vectors that can be separated. (A) mIHC image separated into its three individual color vectors using color deconvolution. Most deconvolution methods are derived from Ruifrok and Johnston,²⁵ and several open-source tools are available for this purpose. In this case, the image was deconvoled with the automatic color vector estimation tool in the QuPath image analysis platform. Additionally, mIHC images may also be transformed into pseudofluorescent representations, as demonstrated in the bottom panel, which was generated from the images in the top panel. (B) mIF image from a Vectra multispectral microscope of a 6-marker panel plus DAPI. The individual signals are separated by a combination of band-pass filters and unmixing algorithms that capitalize on the emission spectra of each individual fluorophore for isolation. In this example, inForm was used to unmix the individual signals and generate the displayed composite and single channel images. DAPI, 4’6-diamidino-2-phenylindole; IF, immunofluorescence; mIHC, multiplex immunohistochemistry; PD-1, programmed cell death protein-1; PD-L1, programmed death-ligand 1.

In IF, tissue samples or cell culture preparations are stained with fluorescently labeled antibodies that target specific proteins, with each antibody labeled with a different fluorophore emitting a unique wavelength of light when excited appropriately. The multiple signals are separated with optical filters that allow the selective transmission of light at certain wavelengths while blocking or attenuating light at other wavelengths, figure 2C. Their spectral size can vary from dozens to hundreds of nanometers and is typically selected based on factors such as the spectral properties of the fluorophores, the degree of spectral overlap between different fluorophores, the protein target expression levels, and the desired sensitivity and specificity of detection. If two or more fluorophore signals are captured by the same filter(s), it becomes necessary to unmix the signals.

Spectral unmixing is a mathematical algorithm that separates the multiple fluorophore signatures into individual channels. It is typically facilitated by the generation of spectral libraries. These libraries are the emission spectra of each individual fluorophore, captured using singleplex-stained tissue samples. The spectral libraries are used as inputs to an unmixing algorithm that determines the contributions of individual fluorophores from composite signals. The most commonly used spectral unmixing algorithm is non-negative matrix factorization, which finds the non-negative abundance coefficients for each fluorophore to reconstruct the acquired signal spectra.²⁸ Some systems also remove the spectral contribution from tissue autofluorescence, which is the intrinsic natural fluorescence emitted by endogenous molecules within tissues when they are excited by light. Autofluorescence can lead to increased background noise and can be removed using control samples that lack specific fluorophores but contain the same tissue background. Some more recent efforts include strategies for unsupervised spectral unmixing or unmixing without reference spectra of RGB, mIF, and image cytometry signals.29,31

Tissue and cell segmentation

Separated colors and spectra are used as input into image analysis software for the detection of candidate objects, which include tissue, figure 3A, tissue compartments, (tumor vs stroma) and cells or cell compartments, figure 3B, among other things. The identification of tissue compartments and individual cell boundaries is often referred to as tissue segmentation and cell segmentation, respectively. This detection can be based on image segmentation methods (eg, intensity thresholding, edge detection) and morphology-related image processing operations (may take into account features such as signal intensity, texture, or morphology-related parameters). For example, algorithms could use staining features, for example, a pan-cytokeratin immunostain, to highlight epithelium and distinguish it from the stroma.

Figure 3. Tissue segmentation, cell segmentation, and pixel-based analysis. (A) Representative photomicrograph illustrating tissue segmentation, that is, selection, of the entire tissue area on the slide (red line). This example of tissue segmentation was performed using the Simple Detection tool in the open-source QuPath image analysis platform. (B) An image from a melanoma biopsy stained with DAPI (blue) and a cocktail of CD44/CD45/ATPase to identify cell membranes (white stain).³⁴ The image was taken using a Vectra multispectral microscope, unmixed with inForm, and the image for display was rendered using Python. The component layers were then passed to the pretrained Mesmer algorithm,³⁵ and the resultant cell segmentation is shown on the right (red lines indicate boundaries of individual cell membranes). (C) Non-segmented mIF image divided into patches of pixels that can be studied for expression patterns across numerous pixels within the patches. The arrow points to a representative patch, displayed as a heat map of pixel confidence values generated by passing the image patches into the NaroNet algorithm.⁴⁴ Confidence reflects the impact of each individual pixel on the machine learning model’s prediction of patient outcome. DAPI, 4’6-diamidino-2-phenylindole; mIF, multiplex immunofluorescence.

Morphological features should be confirmed by a pathologist to prevent potential tissue segmentation errors that can have a significant impact on further analysis, since even small mistakes during this process could result in downstream effects for subsequent steps (see Algorithm verification section below). For example, the quantification of the expression of a given marker by an individual cell or specific subcellular compartment is highly dependent on accurate cell segmentation. Another use of tissue classifiers is the removal of artifacts, such as tissue folds or tears, dust on the slide, and surgical margin colors or pigments that obscure tissues, by classifying them as a separate category with this methodology and ultimately removing them from the analyzed data.

Single-cell segmentation offers a number of specific advantages. Cell segmentation allows for detailed assessment of the cartesian positioning of individual cells, enabling calculation of cell densities in user-defined samples or tissue areas/compartments and spatial cell-cell interactions, as well as subcellular localizations of protein and/or mRNA expression patterns (nuclear vs cytoplasm vs membrane). Unimarker or multimarker colocalization can be used to identify cell phenotypes and measure the levels of functional markers within specific cell populations (eg, Ki-67, GZB).^{32 33} Cell segmentation algorithms typically rely heavily on the detection of the individual cell nuclei. Hematoxylin and 4’6-diamidino-2-phenylindole (DAPI) are commonly used for detecting nuclei in IHC and IF, respectively. Following nuclear detection, basic algorithms will expand the nuclear borders by a selected diameter or until they reach the border of an adjacent cell, in order to mimic the plasma membrane. In doing so, this also allows for the provision of a cytoplasmic compartment for analysis.

Cell segmentation algorithms based solely on nuclear detection are heavily dependent on both cellular and extracellular uniformity. However, many tissues are composed of cells of diverse sizes and lineages, and cells may be in different planes as tissue is sectioned for slide generation. Ongoing efforts to improve accurate cell membrane detection include the development of robust pan-membrane cell stains to aid image analysis algorithms (as shown in figure 2A), as well as the utilization of new AI-based algorithms.34,37

Cell segmentation algorithms may segment all cell types in the image at the same time, for example, tumors, macrophages, and lymphocytes, or may be trained to segment a single cell morphology at a time. One recent study showed that the larger cells or those that are irregular or dendritic in shape (tumors and macrophages) were often significantly over-segmented when all cell types were processed simultaneously. Improved cell segmentation accuracy was achieved when each cell type was segmented independently.⁹ Such a process can be labor intensive and does not always readily extrapolate across data sets without additional training. Cell segmentation of tissues with high lymphocyte densities, for example, lymph nodes, where cells are packed very tightly together, can also be error-prone, and diligent testing of different settings might be required for adequate cell segmentation. Cells where the nucleus is not in the plane of the section also present a segmentation challenge.

Due to some of the challenges with segmenting individual cells as well as some specific morphologic or diagnostic scenarios that benefit from pixel-based strategies, some investigators are forgoing cell segmentation and instead are using a pixel-based approach for analysis, table 2.

Table 2. Comparison between cell segmentation versus pixel-based approaches to signal quantification.

Cell segmentation	Pixel-based analyses
Advantage: characterization of biology in situ at single cell levelDisadvantage: can be labor-intensive to achieve accurate single-cell segmentation.	Advantage: does not require time-intensive and potentially error-prone process of cell segmentation.Disadvantage: by definition lacks single-cell resolution and thus potential biological insights are not as specific.
May be of specific benefit when addressing: Expression of a marker by an individual cell, that is, determining phenotypes. Expression detection in a subcellular compartment, for example, cytoplasm, nucleus, membrane, for mIF/IHC protein and/or mRNA in situ hybridization technologies. Characterizing spatial cell–cell interactions. Assessment of cell densities. Generating H-scores of marker expression.	May be of specific benefit when addressing: Marker-based tissue classification or “molecular compartmentalization”. Unusually shaped cells providing segmentation challenges—dendritic or spindle cells, or those where nuclei are not in plane of section. Assays that rely on a surface area measurement of marker expression. Patches of pixels are often used as input for AI approaches. Tissue regions lacking cellular detail, for example, tears or folds in tissue, or necrosis.

AIartificial intelligenceIHCimmunohistochemistrymIFmultiplex immunofluorescencemRNAmessenger RNA

These approaches treat each pixel as a unit of analysis and extract information based on the presence and intensity of chromogenic/fluorescent staining, figure 3C.^{38 39} This strategy has also been applied to high-parameter tissue imaging using imaging mass cytometry output.⁴⁰ Clustering, colocalization, or spatial relationships between different markers can then be assessed for geographical patterns and associations with other clinical and pathologic parameters. A version of this approach is used for “molecular compartmentalization” methods such as automated quantitative analysis and NanoString GeoMx.^{41 42} In these approaches, a tissue mask is first defined by one marker, and quantitative analysis of protein expression within that mask (rather than feature extraction algorithms and/or cell segmentation) is used to ascertain the coexpression of other markers. This latter approach may be thought of as a data selection strategy, for example, the aforementioned use of a pan-cytokeratin pixel mask could be used to help delineate tumor versus stroma. Unsupervised deep learning approaches have also been applied to individual pixel-based imaging data.⁴³ Using technologies such as NaroNet, it is possible to study patches of pixels in specific arrangements to help identify discrete, repeating geographies that associated with patient outcomes,⁴⁴ figure 3D. These pixel arrangements then typically need to be characterized using cell segmentation strategies capable of single-cell resolution to help define the underlying identified biology.

Many commonly used image analysis software packages such as HALO, inForm, or plugins for QuPath, among others, are capable of performing cell segmentation as well as determining the number of pixels that are positive above a given threshold in an annotated area. An outstanding question in the field is whether diagnostic or predictive biomarker performance differs between pixel-based and cell segmentation-based strategies. It is also important to note that these approaches are not necessarily mutually exclusive. Cell segmentation algorithms are typically trained on an individual project basis and can require extensive manual training to achieve accurate machine-learning algorithm development. In contrast, pixel-based approaches are typically faster to implement since they do not require extended manual machine learning algorithm training. However, the interpretability of the underlying biology that manifests that observed spatial pixel pattern is limited. As such, a combination of pixel-based (for fast discovery and/or exclusion of tissue artifacts) and cell-based (for interpretability) methods could be of benefit. Another example of combining these approaches is to use a pixel-based strategy overlying segmented cells to detect membrane marker polarity to aid in inferring the functional engagement of immune cells.⁴⁵ In the future, deep learning strategies may also help alleviate the manual burden currently associated with training accurate cell segmentation algorithms.⁴⁶

Phenotyping approaches

Phenotyping refers to the classification of cell types as a function of features such as color or spectral intensity, cell shape, or a combination thereof, and typically follows cell segmentation. The determination of whether cells are “positive” for a given feature may be achieved using either thresholding approaches or machine learning classifiers. It is recommended that a description of the phenotyping method used (a threshold value, machine learning training, other) is detailed in the methods of studies reporting mIHC/IF data. The expression intensity of a given marker beyond simply determining whether a cell is positive versus negative for a given marker, may also be of value when determining a phenotype, for example, PD-1^neg, PD-1^low, PD-1^mid, and PD-1^high phenotypes indicate different functional states of lymphocytes and have also been shown to have biomarker value.^{9 47 48}

Conventional thresholding

Thresholding is the simplest and fastest method to classify cells based on a single feature. Each segmented cell bears various intensity features, for example, color or spectral expression intensity for each subcellular compartment (membranous, cytoplasmic, nuclear, and total cell area), and thresholding uses a cut-off value for feature data to identify a cell population that is considered “positive”. It may also be used in instances where cell segmentation is not performed and individual pixels are quantified instead.³⁸ The threshold for positivity is typically determined manually by the analyst using visual verification within the software package. Although visual thresholding is widely used, intuitive and commonly accessible, it is subjective and can vary across samples and users.

Machine learning classifiers

There are two main types of machine learning methods—supervised and unsupervised—both of which may be used to aid in phenotyping cells in digital pathology images. The most common approach to phenotyping involves supervised learning, which is a machine learning process where the algorithm is trained on labeled data, figure 4. In this case, images are manually annotated for a given phenotype, using visual inputs of cell morphology and marker expression. The representative image gallery is then used to generate labels on new cells or cases that the machine had not seen before. Examples of mathematical methods that are employed by supervised algorithms include random forests, boosted decision trees, multidimensional regression, k-nearest neighbor, and support vector machines.

Figure 4. Flowchart illustrating the iterative approach typical for development of a phenotyping algorithm. Representative photomicrographs from a Vectra multispectral microscope showing visualizations of algorithm performance. Typically phenotyping or cell classification algorithms are initially developed on a small subset of data using interactive software, for example, the algorithm for the displayed images was generated in inForm. Once trained, the algorithm can be applied to a larger data set, and visual verification should be performed on this larger subset. In the visual verification example on the right, from a non-small cell lung biopsy and generated in MATLAB using the QC module from Merge a Single Sample (MaSS):⁹ the pathologist is presented with a selection of HPFs where algorithm-detected objects are already marked with overlays. The different colored points represent each identified phenotype. The pathologist then evaluates whether the phenotype classification meets an acceptable criteria. If the algorithm fails, adjustments are made to address any missed objects and/or false detections. The revised algorithm is then reprocessed to generate a corrected score which is compared with the original algorithm output. Following final algorithm deployment, identified cells and structures, for example, boundaries, are described by Cartesian coordinates, which form the basis of the output report for data analysis. The photomicrograph in the bottom left, from an advanced melanoma and generated in MATLAB, displays a representation of this final output with red lines showing cell membranes and small dots overlaid to represent the phenotype algorithm output. PD-1, programmed cell death protein-1; PD-L1, programmed death-ligand 1; QC, quality control.

Unsupervised learning, on the other hand, involves using unlabeled data. The objective is for the algorithm to discover patterns, structures, or relationships within the pathology data without any explicit guidance on what to look for. In the context of computational pathology, it can be used to identify subtypes or clusters of similar tissue samples, cell types, or morphologies without prior knowledge of their diagnoses. This can be achieved with k-Means and hidden Markov models. Neural network/deep learning algorithms using weak or minimal supervision have also been described.⁴⁹

Machine learning classifiers and thresholding methods can also be used in combination. For example, PD-L1+tumor cells could be identified by combining morphologic features (irregularly shaped, large, and hyperchromatic nuclei) detected using machine learning, and thresholding (PD-L1 expression above a given intensity). In the future deep-learning approaches may provide additional solutions, including potentially simultaneous segmentation and phenotyping.⁴⁶

Tissue segmentation, cell segmentation, and phenotyping algorithm verification

Digital pathology image analysis tools applied to multiplexed assays can provide quantitative results, once the algorithm has been optimized for the application of interest.⁵⁰ The tissue segmentation, cell segmentation, and phenotyping algorithms all require strong QC of the output data, and best practice guidelines are currently scarce. The pathologist/trained specialist remains the gold standard, and as such, visual verification of the algorithm’s performance is critical at each of these steps to ensure consistent and accurate performance across samples, figure 4. Typically, the user is presented with a selection of fields-of-view (FOVs) where the objects detected by the algorithm are already marked with overlays. The user then adds any missing objects and/or edits false detections to the algorithm’s detection result. The algorithm is processed again to create a new corrected model, which can then be further iterated on, depending on performance. It is important to include enough input data during training to represent the anticipated variation across cases. This typically requires training and algorithm verification across a number of distinct specimens. It is recommended that key choices regarding image analysis algorithm parameters, for example, whether a nuclear-detection algorithm was used, and details such as maximum cell size detected, are reported for the machine learning-based approaches. This could even take the form of a screenshot showing the fundamental choices made during image analysis algorithm development, which could be included in supplemental materials. In the case of data sets where the raw images themselves are released, it is suggested that the thresholds that the investigators used for determining positive versus negative signals for each marker are also shared, so others may readily replicate the results.

Algorithm performance may be quantitatively assessed in a number of different ways throughout the image analysis process. For example, with regard to phenotyping, the pathologist/trained specialist can count cells of interest in selected FOV and compare them to results provided by the algorithm. The measurement of concordance between the user’s score(s) and the algorithm score(s) can be assessed using both the slope and the correlation coefficients, such as Pearson’s correlation and Lin’s concordance correlation coefficient. Together with sensitivity and specificity, this information provides guidance on the possible improvement range for the algorithm, ultimately meeting predetermined acceptance criteria. Ideally, an algorithm performance verification task should be performed by more than one pathologist/trained specialist to account for inter-user variation. In one report describing a novel cell segmentation algorithm, performance was assessed by showing algorithmic and human-determined results to trained pathologists and determining quantitatively whether there was a visually-appreciated difference in performance.³⁵ Metrics such as the F1-score/Dice coefficient for pixel/cell/object detection can be used to quantify an overlap between the human’s reference annotations and the algorithm’s output.^{34 51} Such visual validation strategies for algorithm performance are not readily applicable to technologies such as GeoMx, and other orthogonal analytic validation approaches are of value.⁴²

In theory, it would be ideal to train an image analysis algorithm and use it in a completely automated fashion for all slides in a project or even multiple different projects, that is, a stable-over-time, “locked” algorithm. However, when there are multiple slides in a project with different preanalytical variables and/or slides stained in multiple batches, there may be differences in staining color and fluorophore intensity between slides. In the absence of effective preprocessing steps for color/intensity normalization, using the same threshold(s) across these conditions might decrease accuracy. Well-trained machine learning-based supervised classifiers are sometimes able to overcome this limitation, since they use additional features beyond thresholding for scoring. In either approach, the algorithm performance may “fail” visual inspection or the otherwise defined acceptance criteria for a subset of specimens, thus, the question of how to best handle these cases remains. One option is to count those specimens in another way: either manually by a pathologist/trained specialist or by using a computer-assisted approach with more human involvement. If such an alternate counting approach is taken, it is important to disclose the number of cases this was used for and to show that the specimens are not skewed between study groups to guard against any potential bias, for example, the specimens that failed and had to be recounted using an alternate approach included those from both “responders” and “non-responders” to therapy. Another option is to simply exclude the cases where the algorithm failed from the final analysis. This is the most conservative approach, and while it may suffice for research studies, it is suboptimal as mIHC/IF technologies move toward clinical care. Going forward, algorithms that account and adjust for preanalytical sample variations as well as approaches that normalize for batch-to-batch variation are anticipated to improve and standardize data output quality.52,54

Annotations

Once images are processed, it is possible to select certain regions of the images for inclusion in the analysis and/or for exclusion, such as tissue tears, figure 5. This is typically achieved through manual annotation tools, which facilitate the drawing of different tissue regions, topographic boundaries, figure 5, etc, or help highlight a specific anatomic compartment, for example, portal tracts or tertiary lymphoid structures (TLS). The strategy for annotation should be predefined and consistent across all specimens in a given cohort. Having a single user perform the annotations or at least inspect them all for consistency is recommended. Algorithms that automatically mask tissue tears and folds or other artifacts, for example, formalin deposition in tissue, or which can select features of interest, such as the aforementioned TLS, have been described and may facilitate throughput.⁵⁵

Figure 5. Annotation tools may be used to help select regions of the TME for either inclusion or exclusion in subsequent analysis. (A) The blue line outlines a tumor nodule, in this example the annotation was generated using HALO. Image analysis tools enable the expansion of this annotation at predetermined increments, for example, 50 µm, facilitating a standardized and reproducible approach to characterizing the peritumoral zone. (B) Regions may also be excluded from analysis. In this representative photomicrograph of a skin biopsy stained with a mIF assay and imaged with a Vectra multispectral microscope, invasive melanoma tumor nodules have been annotated (green line) using HALO. The yellow line delineates the TME regions included in the final analysis, deliberately excluding any in situ melanoma in the overlying epidermis, which was the a priori study design. mIF, multiplex immunofluorescence; TME, tumor microenvironment.

Batch-to-batch correction

As previously discussed, measurements of individual marker expressions vary from batch to batch, necessitating normalization across batches. This may be due to variable staining performance of instruments between runs, changes in reagents over time, recalibration of instruments, etc.⁵⁶ It is important that new image libraries are generated after scanner maintenance to ensure comparative performance before versus after maintenance. Additionally, bulb change information should be collected and included in the meta-data for any study. An additional consideration is the comparison of results across multiple imaging devices, and an approach to aligning illumination intensity and spectral sensitivity between multispectral microscopes has now been described.⁵⁷

Batch-to-batch corrections to address staining and instrument variations may be performed at multiple points in the workflow, depending on the image analysis strategy used. Ideally, the imagery would be corrected for stain intensity variation prior to cell segmentation and subsequent phenotyping; however, it is recognized that for those using off-the-shelf image analysis suites, it is often not possible to perform batch corrections until after segmentation and phenotyping have been performed.

It is possible to use whole slide sections of control tissues, for example, tonsil, for biomarker-positive and biomarker-negative controls. A minimum of 10 different control tissues have been recommended for robust batch effect correction.⁵⁸ Tissue microarrays (TMAs), including both tissues and engineered control preparations such as cell line transfectants or beads, are especially well suited to this task as they provide the ability to use multiple different types of tissue, allowing for averaging over the variations expected between different tissue types. Serial slices of TMAs provide near-identical groups of control tissues that may be stained and imaged along with each batch of slides analyzed. Normalization may then be achieved by quantifying differences in marker expression intensity between batches, facilitating more accurate assessments of intensity as an output metric. One study showed that across nine mIF staining batches, the coefficient of variation of expression intensity for PD-1 and PD-L1 was∼20%. When the signals across batches were normalized using the TMA controls, this variance was reduced by 50%.⁹

Whole slide image QC

As previously noted, when most mIHC/IF images are acquired, they are acquired in smaller HPF regions that must be tiled together, especially when performing whole slide analysis. Some algorithms use microscope-stamped image coordinates to arrange the images together. However, small mechanical deviations, typically caused by an internal motor, can lead to “jitter” in the HPF locations. Such “jitter” should be accounted for in the form of image alignment. Most algorithms for registration use a form of cross-correlation in overlapping regions to find the optimal image coordinates.^{9 59 60}

Other forms of image correction facilitating the generation of seamless, whole-slide images from tiled HPFs with consistent illumination include flat-fielding, lens correction, and exposure time corrections. Additional descriptions of these phenomena, their potential impact on mIF data, and the development of corrective factors can be found by Berry et al.⁹ Some platforms do not already include algorithms for image correction. In those cases, open-source modules for image correction are available.⁶¹

Image registration

Image registration is a key component of MICSSS, tissue-based cyclic immunofluorescence (t-CyCIF), and other “stain and strip” cyclic approaches to mIHC/IF which require multiple rounds of staining and imaging a single slide.1262,64 It can also be used to facilitate the analysis of serial slide sections from a sample—with either different mIHC/IF panels performed on sequential slides, or with the same mIHC/IF panel performed on sequential slides. These may facilitate increased “plex” of an assay, by performing different panels on two sequential slides or even three dimensional reconstruction of a tumor, respectively. Most image registration methods perform a grid search to maximize a similarity measurement. Some measurements are more robust for particular registration experiments and are detailed in the subsequent sections. Moreover, for some experiments regional registration (either manual or automatic) may be sufficient; while for others it may be necessary to use a more local registration, typically made up of a random grid of control points.

Repeatedly imaging the same slide stained with a consistent IHC/IF method

“Stain and strip” platforms require stripping and antigen retrieval before every cycle of immunostaining. Each slide is also scanned after immunostaining, and placing slides into scanner racks with small variations changes the coordinates of the tissue between images. This introduces tissue warping and shifts at the microscopic level on each marker image, potentially establishing slightly different total cell numbers for the same ROI. The alignment of images with minimal tissue shifts and warping can be achieved with algorithms like scale-invariant feature transform,⁶⁵ speeded-up robust features, Oriented FAST, and Rotated BRIEF, which detect large, local histologic features such as normal ducts or tumor profiles for registration. ROIs annotating the same tissue zone for multiple markers are exported as individual images from whole slide images, and they are either registered automatically or manually. The Fiji TrakEM2 plugin⁶⁶ is one example of a useful tool for this purpose, and can register images automatically or based on selected landmarks. Other more robust algorithms can be applied and may register images using multiple random control points across the entire slide.⁵⁹Algorithms for this kind of registration, typically need less robust registration criteria such as the mean square difference or cross-correlation to fix all cells to single coordinates for different markers on a single slide.

Registering images from adjacent slides slide stained with a consistent IHC/IF method

Serial slide sections taken from a single block may also be stained to increase the “plex” of IHC/IF assays. Serial slide registration is typically performed by putting the two slides into a single coordinate system using an algorithm based on a cross-correlation function minimization, some of which also include deformable or elastic registration to account for image differences (figure 6A). Some examples of publicly available algorithms for serial section registration include Warpy and Ashlar, among others.^{59 67} Composite images can be analyzed in image analysis software with similar principles to those used for individual images (figure 6B)³⁸

Figure 6. Serial section images can be registered and analyzed as though the stains were performed in mIHC/IF on a single slide. (A) IHC for three different markers was performed on sequential slides, which were scanned using a Hamamatsu NanoZoomer digital slide scanner. The images were then registered to each other in a Z-stack using HALO. Image courtesy of Dr Nicolas Giraldo. (B) The individual images can then be merged into a multicomponent image for further analysis using mIHC/IF tools. The MATLAB-generated photomicrograph overlay shows cell geometries colored by phenotype, with dots indicating PD-1 (blue) or PD-L1 (green). Dot size represents PD-1 and PD-L1 expression levels, and lines indicate PD-1+and PD-L1+cells within 20 µm of each other in this proximity analysis. IF, immunofluorescence; mIHC, multiplex immunohistochemistry; PD-1, programmed cell death protein-1; PD-L1, programmed death-ligand 1.

Multimodal registration

Multimodal registration, such as combining mIF and mIHC images acquired on the same or adjacent slides, or incorporating spatial transcriptomics, poses additional challenges as compared with registration of the same types of images. The registration approach must account for image rotation and deformation to register the images and must synthesize different image types into the same coordinate system. These challenges make algorithms based on cross-correlation less reliable; instead, registration algorithms based on mutual information, such as elastix,⁶⁸ Warpy,⁶⁹ or Whole-Slide Imaging, Mutual Information Registration³⁴ can be employed.

Quality metrics for image registration

A lack of ground truth has hampered the generation of quality metrics for image registration. Currently, the reference for image alignment relies largely on a qualitative assessment by pathologists. One approach to allow for a direct comparison between registration methods involves the identification of tissue boundaries by a boundary identification tool, and then using those boundaries to evaluate the maximization of overlap generated by different registration approaches.³⁴ Another approach could be to evaluate the minimization of overlap of registered images on smaller image regions.⁵⁹ Common examples of overlap measurements used for assessment in this way include the Jaccard index, the Dice similarity coefficient, and the Hausdorff distance.

Proportion, density, neighborhoods, and other spatial analyses

When assessing shifts or differences in cell populations, it is crucial to consider both the proportion and density of cells, as these metrics provide complementary insights into the underlying biology. The choice of analysis should reflect the most unbiased representation of the data. Additionally, spatial dependence is an important factor that can significantly impact results. For instance, while changes in average cell–cell distances can be informative and correlate with clinical outcomes, they often fail to account for variations in cell density or proportions within a sample. To accurately account for spatial dependence, data should be normalized against both cell density and proportion.

Spatial statistics is a well-established field with roots in disciplines like economics, public health, astronomy, and metrology, all of which offer valuable methods for image analysis. Summers, et al, provided an insightful perspective of the diverse applications of spatial statistics in biological image analysis.⁷⁰ Broadly, spatial clustering and autocorrelation statistics, such as Moran’s I and Geary’s C, offer a more robust analysis than simple distance-based measures by inherently accounting for spatial dependence.^{71 72}

Identifying spatially coherent regions or neighborhoods is a key focus in multiplex mIF and mIHC image analysis. The rapid growth of spatial transcriptomics has further driven the development of clustering techniques which are also applicable in this space. Most statistical methods start by identifying spatially variable genes or phenotypes using global versions of the spatial autocorrelation statistics,^{73 74} with neighbors typically defined through N-nearest neighbor algorithms or a maximum distance threshold between cells. Additionally, these statistics can be used to estimate local autocorrelation for specific phenotypes, providing more localized insights.⁷⁵ Some approaches apply a grid over the cells and compare these grids rather than individual cells directly.⁷⁶ Neighbors can then be compared across or between samples to estimate enrichment, clustering, or spatial variance.77,79

More advanced methods employ permutation testing to identify statistically significant spatial phenotypes, comparing neighboring regions to a null hypothesis of spatial independence or randomness.^{80 81} Others, such as SPARK and SpatialDE, use generalized linear spatial models to model cell expressions—either directly or normalized—offering greater efficiency for larger data sets by comparing modeled distributions.^{82 83} Recent efforts have focused on integrating spatially defined phenotypes with histological data and gene expression to cluster neighborhoods or spatial domains.84,86 However, many of these methods are still computationally intensive and may require significant optimization for large-scale, whole-slide analyses.

Data and code sharing

Once produced in a validated and reproducible way, the ability to share data is critical. Some consensus on the data structure and related data management requirements that permit effective data sharing is, therefore, obviously needed.

Readouts from mIHC/IF assays include tabulated results of two-dimensional (X, Y) coordinates of pixels and/or cells showing the spatial distribution and intensity of expression for each marker across the imaged portion of the slide. Additional features can also be extracted and reported, such as the shape and texture of cells/objects. For cells that are irregularly shaped, some reporting strategies may not fully describe cell contours, potentially impacting the computed location of the cell membrane when calculating distance measurements. However, various methods are suitable for different applications, so standardization is not indicated here, rather caution regarding clarity when communicating methods used. When image analysis tools are used to segment cells, the tables may also include the coordinates of the centroid of individual cells. This information may be used to calculate outputs such as cell density (eg, # lymphocytes/mm²), marker expression intensity (eg, low vs high expression), inter-object distances, percentages of object types in certain tumor areas (eg, % lymphocytes in tumor stroma, % tumor area with macrophage marker expression), etc.^{87 88} These algorithms may be in-house coded via scripts (Python, Julia, R, MATLAB), notebooks (Jupyter, Google Colab) or analyzed with publicly available toolboxes like MCMICRO or CytoMAP.^{76 89}

Best practices for data sharing include reporting of the raw data from the image files prior to image analysis. This is often in the form of a .csv tabular file or raw image data in czi, tiff, or qptiff formats. A data dictionary should also be provided that defines all abbreviations. This allows other research groups to then explore the data to not only verify results but to test different hypotheses. This practice also aligns mIHC/IF assays with the new National Institutes of Health (NIH) guidelines for data reporting as of July 1, 2023. Additional key outputs after image analysis that are critical to the conclusion of the paper should also be reported, for example, if a paper is reporting PD-1 to PD-L1 distance metrics as a proposed biomarker, those distances should be displayed in data tables on a per-patient basis.

In addition to sharing data, it is important to share the code required to reproduce the analysis results. The code and its updates should be readable, with clear documentation of inputs and outputs. Ideally, anyone should be able to download the code and either run on the data provided to exactly reproduce the plots in the paper, or run on their own data, processed into appropriate input formats, to produce equivalent plots. The use of versioning software (Git and SVN) and digital repositories (GitHub, Code Ocean, and Zenodo, among others) can be used to streamline code and data sharing, including digital object identifier (DOI) issuance.

Representative photomicrographs are also recommended for inclusion to allow the reader the opportunity to visually verify critical results, for example, key or unusual phenotypes. To date, the resultant raw mIHC/IF images themselves have not been routinely shared in publications using this technology. One major barrier to this is the infrastructure required to host large files, which may be several gigabytes (GB) in size for each whole-slide image. Some individual laboratories have been hosting their own slide scans, for example, https://datasets.deepcell.org/. It may also be possible to host images on Synapse (www.synapse.org)⁹⁰ or a similar public repository, such as the National Cancer Institute’s (NCI’s) imaging data commons (https://datacommons.cancer.gov/repository/imaging-data-commons which facilitate archiving and DOI generation. Web-browser architectures for visualizing mIHC/IF data may also assist with this task.⁹¹ Irrespective of whether they are posted publicly, we recommend maintaining the raw images as they come off the instrument in a permanent location along with any associated metadata. Going forward, it will be of benefit to have multiple public data repositories that can host large volumes of mIHC/IF images. While these data are more complex than raw sequencing data, mIHC/IF images and associated raw data output share similar needs with other fields of research that increasingly rely on the analysis of so-called “big data”. A number of resources are available that describe key features of big data management, including the important principles of findability, accessibility, interoperability, and reusability.⁹² When sharing data publicly, it is also important to ensure patient anonymization to ensure privacy.

Current multi-institutional harmonization efforts

A number of multi-institutional efforts are underway to harmonize mIHC/IF. One harmonization effort performed across six sites on the same mIF platform showed strong inter-site and intrasite concordance of a 6-plex assay performed on tumor tissue for measures requiring single-cell resolution, for example, density of specific immune cell subsets, coexpression metrics (%PD-L1 on immune cells, etc), and proximity (PD-1 to PD-L1 distance).⁵⁶ A second multi-institutional effort was conducted by the NCI-designated Cancer Immune Monitoring and Analysis Centers, which are tasked with providing standardized biomarker assays for NIH-sponsored clinical trials.⁹³ After harmonizing antibody clones used in a 5-plex assay, they found a strong agreement for immune cell densities characterized by each of the five individual markers in their panel. Notably, this effort compared cell density results generated from MCISSS and brightfield scanning to mIF staining and scanning with spectral unmixing technology. These two distinct multi-institutional efforts had some similar features and findings worth noting: (1) Image analysis was highly coordinated across sites or performed at one site, and did not use a prospectively-developed, locked-down algorithm; (2) The highest concordances were observed when quantifying parameters related to lymphocytes. Lower concordances were observed when quantifying macrophages and other cell types with irregular morphologies; (3) Both studies used either TMAs as input or select HPFs. Similar studies using whole slide images have yet to be completed.

The expectation of these processes is not to achieve perfect agreement between institutions, but rather to quantify inter-institutional variability following harmonization efforts. The results are then considered a benchmark for achievable precision for a specified biomarker assay that can, in turn, be used to establish rational guidelines for interpreting the results of the test when it is applied to clinical samples.

Future directions

The sections in this manuscript described best practices for processing multiplex images, with a focus on techniques for minimizing systematic errors. Crucial next steps for the field of mIHC/IF image analysis and data sharing include harmonization across platforms where possible, continued standardization, and an improved understanding and quantitation of residual error and its impact. This will help pave the way to broad clinical implementation of these promising technologies.

Both alignment across platforms and reproducibility across different platforms are critical but different goals for the field. Alignment on staining sensitivity can be aided by bridging results from different platforms using the same reference material/samples. Further, if validation is performed to the “gold-standard” of singleplex chromogenic IHC, it ensures a similar minimum threshold of detection across platforms and can facilitate clinical deployment.¹³ Similar “minimum standards” of different steps in the image acquisition, processing, and analysis pipeline are the next step, and the development of a checklist, currently underway through the Society for Immunotherapy of Cancer, may allow for improved interplatform alignment and more consistent comparisons and data generation. Efforts such as the NIH Human Biomolecular Atlas Program (https://commonfund.nih.gov/hubmap) will also undoubtedly help drive the continued development of a common, community standard for multiplex data generation and formatting.⁹⁴

In addition, the field needs to go beyond the development of techniques for minimizing systematic errors and evaluate how much uncertainty remains after corrections have been made to better understand how those uncertainties can affect the ultimate analysis, for example, predictions of patient outcomes. If a large fraction of tumor cells are over-segmented, that would impact tumor-cell-dependent outputs, but may have relatively little effect on an analysis that focuses only on lymphocytes. Gaining an understanding of these uncertainties would allow results to be reported with detailed potential errors, strengthening scientific reporting. Additionally, identifying dominant uncertainties can help pinpoint calibrations that need further work to improve future analyses. This can inform methods development, both in the lab and in data analysis.

Guidelines such as those described here are a requisite step for moving multiplexed image analysis systems and technologies into a clinical environment. While the regulatory environment involves many considerations beyond the scope of this manuscript, the considerations of shareability and comparability are germane for diagnostic testing applied to patient samples in a College of American Pathologists accredited, Clinical Laboratory Improvement Amendments certified, and/or US Food and Drug Administration regulated setting. There are also non-trivial logistical and financial considerations when considering incorporating mIHC/IF assays into hospital workflows, including potential laboratory information management system/electronic medical record inclusion of images and questions surrounding payer reimbursement.⁹⁵

Conclusions

The tissue biomarker community faces a number of challenges in order to optimize the value of the rich, quantitative data that is rapidly being generated. The comparability of mIHC/IF data produced by individual labs and their reproducibility is a critical issue, since producing the type of rich data sets required to truly understand the TME and the immune response to cancer will require contributions by multiple institutions. Nonetheless, models such as Digital Imaging and Communications in Medicine and NCI Data Commons are available that illustrate the evolution of performance standards, validation requirements, and harmonization practices needed to ensure the quality and comparability of tissue biomarker data. Here we describe guidelines for the acquisition and processing of digital images from mIHC/IF slides, including approaches to algorithm verification, image QC, quantification of registration accuracy, and batch-to-batch correction, among others. We also describe key considerations for what data should be shared, and how, for appropriate publication of this mIHC/IF assay results.