Spatial analyses of immune cell infiltration in cancer: current methods and future directions: A report of the International Immuno-Oncology Biomarker Working Group on Breast Cancer

Abstract

Modern histologic imaging platforms coupled with machine learning methods have provided new opportunities to map the spatial distribution of immune cells in the tumor microenvironment. However, there exists no standardized method for describing or analyzing spatial immune cell data, and most reported spatial analyses are rudimentary. In this review, we provide an overview of two approaches for reporting and analyzing spatial data (raster versus vector-based). We then provide a compendium of spatial immune cell metrics that have been reported in the literature, summarizing prognostic associations in the context of a variety of cancers. We conclude by discussing two well-described clinical biomarkers, the breast cancer stromal tumor infiltrating lymphocytes score and the colon cancer Immunoscore, and describe investigative opportunities to improve clinical utility of these spatial biomarkers.

Introduction

With advances in technology, it is now possible to visualize the spatial interface between tumors and host immunity on the level of individual cells, thus generating vast spatial datasets that may contain clinically important prognostic and predictive information. However, this capability is accompanied by the challenge of developing rigorous and reproducible statistical methods for reporting and interpreting spatial immune cell (IC) data. Currently there is no standardized approach; most published analyses only scratch the surface of the rich information hidden within the spatial orientation of ICs. Here, we summarize published spatial IC analysis methods and future directions. The review is organized into three sections. First, we summarize the status of histologic imaging technology, including machine learning (ML) approaches to identify IC cancer cell spatial locations and their attributes, such as cellular phenotype. We also introduce the basic data structure, terminology, and analytic methods for spatial datasets. Second, we summarize published research in the field, providing examples of different types of spatial metrics across categories including IC density, heterogeneity, clustering, colocalization, interfacing, and tissue segmentation. Finally, we conclude by discussing the spatial analytic methods of two leading IC prognostic biomarkers, the breast cancer stromal tumor infiltrating lymphocytes (sTIL) score [1] and the colon cancer Immunoscore [2], and propose how advanced spatial analysis methods could be applied to create new spatial biomarkers.

Data acquisition: technologies, ML, and spatial data structure

Here, we provide a summary of imaging technologies, methods of acquiring spatial data from images, and the organizational structure of spatial data.

Imaging technologies

Modern imaging technologies have permitted the acquisition of spatial datasets of ICs in cancer. The most common platforms employ high-resolution microscopy to evaluate the expression of multiple cell markers from a single tumor slide. A shared feature of these platforms (which include multiplex immunohistochemistry [mIHC] and immunofluorescence [mIF]) is the use of multiple antibodies that bind to and label specific cellular proteins, to visualize various cell types and identify their precise two-dimensional geographic location in relation to cancer cells. Analysis of multiple markers on a single slide ensures that spatial relationships of cell types are preserved, whereas single-marker analyses on sequential slides may be less accurate for characterizing spatial relationships between cell types. Great care must be taken to standardize and validate the assay workflow, which includes steps such as antibody/reagent selection, antibody staining order, sample storage, processing, and staining, and quality control steps employing positive and negative controls to ensure consistency. Best practices are summarized in an expert consensus statement by the Society for Immunotherapy of Cancer (SITC) [3]. In addition to mIF and mIHC, nascent technologies permit extensive characterization of protein or RNA differential expression on the level of individual cells, or within spatially defined subregions of the tumor. Such technological platforms include mass ion beam imaging, imaging mass cytometry, digital spatial profiling, or spatial single-cell RNA sequencing, which are summarized elsewhere [3–5]. Of note, older platforms, such as single-stain chromogenic IHC or H&E can still provide a wealth of spatial information, which can be leveraged using imaging software such as QuPath [6,7].

ML to extract spatial IC data from images

A shared and crucial step across all imaging platforms is to extract spatial data from raw images. High-resolution microscopy images are large datasets comprised of many pixel locations, each with associated intensity or color/hue values. Automated methods using ML are helpful in extracting spatial features from the imaging data that summarize the spatial distribution of cell and tissue types. Additionally, computational platforms are used to spectrally unmix multiplexed imaging data to accurately identify the spatial coordinates of the different biomarkers. Using various ML-based approaches, images can be used to generate data features such as tissue type (i.e. stroma, necrosis, fat, microvessels, cancer cell nest, blank/acellular) [8], region-specific features (such as the invasive margin [IM] of the tumor), and cells (annotated by location and phenotype). ML methods can be either fully automated, such as with the Definiens imaging analysis platform for the colon cancer Immunoscore [2] (Developer XD, Definiens, Munich, Germany), or they can depend on user supervision to refine the classification of tissues or cells, as with the mIF InForm software package (Akoya Biosciences, Marlborough, MA, USA) [9,10]. A separate summary statement has been published that describes ML and computational methods for acquiring spatial IC data [11].

Structure and analysis of spatial IC datasets

Before the advent of these technologies, spatial IC data were reported descriptively, visually, or in the form of whole-slide summary metrics, such as cell count, intensity, density, or other textural features. These metrics are poor at characterizing heterogeneity of ICs across the span of the tumor. It is now possible to quantify IC densities precisely across multiple geographic subregions of the tumor. In the context of mIF/mlHC, this is often conducted across high-power microscopy fields (HPFs), determined by the field of view of the microscope lens [3,12]. When ICs are interrogated across multiple subregions, characteristics of subregions within a tumor can be compared. With these data, additional information, such as ‘hotspot zones’ or inter-region variability can be reported (described further below).

When a tumor is repeatedly sampled across HPFs, it is important to consider the underlying geographic interdependencies across the various HPFs. Summary statistics, such as mean IC density across multiple HPFs, can be confounded if sampling of subregions is non-uniform and/or if an insufficient proportion of the tumor is sampled. To build robust analytic pipelines using HPFs, it will be important to determine the number of HPFs required to accurately characterize the tumor and to ensure that the sampling distribution is not spatially biased toward certain tissue architectures. It was recently shown in the context of a breast cancer cytokine-based immunotherapy trial that at least 15 HPFs per patient were required to overcome the potential confounding effect of spatial heterogeneity and accurately characterize changes in IC density related to the cytokine treatment [12], This trial also illustrated how statistical models, such as hierarchical linear modeling, could be leveraged to better capture intratumoral heterogeneity and estimate IC density [12]. In a melanoma cohort it was demonstrated how HPFs could be tiled to cover the entirety of the tumor space and overlapped by 20% to permit correction of errors on the periphery of HPF images due to illumination and lens effects [13]. An alternative to tiling HPFs is to utilize whole-slide imaging platforms, such as the mIF Polaris (Akoya Biosciences); however, these platforms are still in development and the imaging can be time-consuming.

To fully utilize spatial data, observations must be annotated by their exact spatial/geographic position, so that both spatial interdependencies and heterogeneity can be fully explored. There are two distinct approaches for analyzing spatial data, raster analysis and vector analysis, each of which can be used in different contexts of spatial IC analysis (illustrated in Figure 1A–D). Vector-based datasets can be converted to raster-based and vice versa. However, when converting raster to vector, the resolution depends on the size of the raster cells.

Figure 1.

Raster versus vector spatial data structure. (A) An example of a TNBC specimen imaged by H&E, with high-resolution multi-color images obtained using mIF (Vectra platform). High-resolution images are used to obtain cell coordinates and phenotypes. (B) In raster-based spatial analysis, the tumor is divided into small subregions (usually by a rectangular grid) and spatial metrics are calculated independently across each subregion, allowing for analysis of spatial metrics such as average cell count, deviation/skewness, and hotspot analysis. (C and D) In vector-based spatial analysis, cells are annotated by their phenotype, (X,Y) geographic location, and other attributes, such as PD-L1 expression. These data can then be analyzed using statistical software to calculate a variety of metrics.

With raster-based datasets, spatial analyses are accomplished by partitioning the evaluable space by a grid of subregions or raster cells, each annotated by their geographic location. For each raster cell, measurements pertaining to IC infiltration (such as IC density, programmed cell death ligand 1 [PD-L1] intensity, percentage tumor/stroma) are individually estimated, thus generating a mosaic of estimations across the entirety of the tumor space (Figure 1B). This method allows for the identification of hotspot zones, investigation of spatial relationships between ICs and locoregional anatomic structures, and evaluation of the heterogeneity of IC characteristics.

One important factor affecting raster analyses is the partitioning scale/grid size: if the raster cell size is too large, the resolution is poor for detecting locoregional variations (such as hotspots or variations in IC density); whereas if the raster cell size is too small, estimates of IC density become volatile due to inadequate sampling space within each cell. In a recent breast cancer prognostic evaluation, raster cell size affected the overall prognostic performance of IC hotspot analysis [14]. There are several proposed methods for optimizing raster cell size. One is to select a size that reflects underlying biological considerations, such as the span of cell-cell paracrine signaling [15]. An alternative to rectangular grids is provided by Voronoi tessellation [6], which can be used to subdivide the tumor space into uneven polygons that maximize grouping of cells with their nearest neighbors, and could be more flexible to adapt to underlying anatomic structures [7,16]. Superpixels is another method that segments tumors into raster cells according to similarities in adjacent pixel colors or other features [17,18]. Finally, approaches can be implemented that analyze tumors on different scales (multiscaling) or to optimize raster cell size as a hyperparameter, as it is effectively changing the resolution and accuracy of the statistical modeling [19].

Vector-based datasets are comprised of a set of unique observations corresponding to each visualized cell across the tumor space, with measured attributes such as cell type (cancer versus immune versus stromal cell) or protein expression (such as PD-L1 quantitative intensity by mIF). Each observation/cell is spatially annotated by its point location (x, y coordinates in two-dimensional space; Figure 1C,D) [20]. Locations of anatomic boundaries within the tumor, such as the IM or cancer cell/stromal interface, are defined as a series of (x,y) spatial points, that when connected linearly, create either a line or a polygon in two-dimensional space (Figure 1C,D). With these multi-featured vector datasets, software packages (such as Python or Simple Features sf package in R [20]) can efficiently calculate distances to the IM for each IC, calculate locoregional IC densities or PD-L1 expression, or label ICs according to the location within different tissue compartments (such as intraepithelial versus stromal). Another benefit of vector datasets is the ability to convert the data into detailed spatial maps that graphically illustrate the interrelationships between cancer cells and ICs (illustrated in Figure 1C).

Analysis of spatial data: overview of published research

Here we provide an overview of spatial IC analyses that have been conducted across a variety of cancer contexts and summarize their potential clinical prognostic and/or predictive utility. When possible, we describe these metrics using the terminology and framework of raster versus vector spatial analyses. We also describe the metrics as either global (i.e. summarized across all spatial subunits) or local (i.e. metrics that are reported repeatedly across spatial subunits). These metrics are summarized in Table 1.

Table 1.

Examples of spatial metrics and definitions.

Metric/references	Local versus global*	Raster versus vector	Description
Count/density metrics
Cell count [21–23]	Either	Raster	Count of cells within a given analysis area
Cell density [21–23]	Either	Raster	Count of cells, normalized for area
Density ratio [24–26]	Either	Raster	Ratio of two cellular densities
Radial density [27]	Either	Vector	Density of cells within a given radius of a given geographic location
Kernel density [28]	Local	Vector	Density of cells in an area, smoothed for densities in surrounding area
Heterogeneity metrics
Density variance [29]	Global	Raster	Variation in densities across subregions of a tumor
Coefficient of variation [30]	Global	Raster	Variance, normalized for overall cell density
Quartile coefficient of dispersion [30]	Global	raster	Alternate measure of heterogeneity, less influenced by outlier regions
Skewness [31]	Global	Raster	Global measure of degree of outliers in a sample
Shannon entropy [6]	Global	Raster	Global measure of heterogeneity/randomness across subregions
Clustering metrics
Quadrat analysis [6]	Global	Raster	Variance/mean ratio of densities across subregions, scores >1 indicate clustering
Moran’s I statistic [20]	Global	Raster	Strength of correlation of cell densities with those of neighboring subregions
Ripley’s K function [32]	Global	Vector	Metric providing clustering strength at different cell–cell distances
Getis-Ord general G-statistic [20]	Global	Raster	Generalized metric summarizing degree of clustering across neighboring subregions
Hotspot enumeration [33]	Global	Raster	Quantification of numbers of hotspots in a sample
Hotspot fractional area [33]	Global	Raster	Proportion of tumor that meets hotspot criteria
Hotspot density [2,13,29,34]	Global	Raster	Average density within hotspot regions (various definitions of hotspots can be used)
Getis-Ord local Gi [20]	Local	Raster	Hypothesis test to evaluate presence/absence of hotspot (p value for each subregion)
Local Moran’s l [20]	Local	Raster	Application of Moran’s I statistic to identify subregions likely to be hotspots
Cluster morphometric analysis [6,35]	Local	Vector	Evaluation of shape and other geometric characteristics of IC clusters
Colocalization metrics
Morisita-Horn Index [7,16]	Global	Raster	Global measure of degree of localization of two cell species
G-cross function AUC [36]	Global	Vector	Estimation of likelihood of two cell types to be adjacent, across various distances
Nearest neighbor [27]	Global	Vector	Average nearest pairwise distance of two cell types
Interface metrics
Band density [6,37]	Global	Either	Cell densities measured at intervals of distances from IM
Band density ratio [37]	Global	Either	Ratio of band densities (measure of dependency of cell density with IM)
Fractal dimension [7,34,38]	Global	Vector	Measure of complexity of tumor/stromal border
Density gradient [39]	Global	Either	Rate of change of cell density, by distance from tumor IM
Segmentation metrics
Intratumoral density ratio [28]	Global	Raster	Ratio of cellular densities, comparing stromal versus intraepithelial
Area ratios [20]	Global	Vector	Ratio of stromal area to intraepithelial area (or other areas of interest)

^*Local refers to analyses conducted across various tumor subregions; global refers to analyses that summarize the whole tumor.

IC counts/densities

The most basic spatial metric is IC count or density. Cell count is defined as the number of detected cells within a given spatial unit (i.e. HPF), whereas density is the cell count normalized by the two-dimensional area of the spatial unit. Cell counts can be influenced dramatically by tissue composition, which may be an admixture of cancer cells, non-cancerous tissue (stroma, adipocytes, vessels/lymphatics), non-viable tissue (necrosis or artifacts), or empty space. Cell densities are preferred over counts when comparing IC infiltration across regions with a non-uniform evaluable tumor area. Densities of certain cell types have been shown across multiple studies to be prognostic of outcome or predictive of response to immunotherapy or chemotherapy. For example, CD8⁺ cell density within either the tumor compartment or the IM was shown to predict melanoma response to treatment with anti-PD-1 (pembrolizumab) [21]. In early-stage triple-negative breast cancer (TNBC), the TIL density within the stromal compartment in an H&E slide is prognostic of outcome [40], predictive of neoadjuvant chemotherapy response [22], and predictive of anti-PD-1 response [23]. Interestingly, in a study on TNBC correlating TIL density assessed by AI in different tissue compartments (intratumoral, within the tumor stroma, at the tumor periphery, etc.) with patient outcome, it was shown that a similar prognostic value could be observed independent of the measurement region, indicating that for these tumors, TIL assessment may be less sensitive to variations between methods [41].

Once cell counts or densities are determined, derivative metrics, such as ratios, can be calculated. One example is the PD-L1 combined positive score (CPS), which uses single-color IHC labeling of PD-L1 to identify patients who may benefit from anti-PD-1/PD-L1 immunotherapy. The CPS is a ratio calculated as the PD-L1⁺ cell count (ICs or cancer cells) within the tumor region, divided by cancer cell count, on a single slide [24–26]. The PD-L1 CPS can predict the response to PD-1 immunotherapy (pembrolizumab) in some disease contexts (such as with chemotherapy in first-line metastatic TNBC) [25], but fails to predict the response in other disease contexts (such as in stage II/III TNBC) [42]. One downside of the CPS method is that the score is capped at 100 and, therefore, has a more limited dynamic range for describing PD-L1 expression compared with other methods. Another downside is that the inclusion of cancer cells in the denominator makes the score particularly sensitive to variations in the ratio of cancer cells and other cells. In another example, a high ratio of cytotoxic CD8⁺ T cells to CD68⁺ macrophages within primary melanoma was found to be associated with distant recurrence-free survival [43].

IC densities and ratios can also be analyzed across tumor subregions as a local metric, using either raster-based or vector-based methods. As used in the colon cancer Immunoscore (discussed below), the Developer XD software (Definiens) can be used to estimate IC density across rectangular raster cells and to generate novel metrics, such as hotspot density, defined as the mean IC density across the three raster grid cells with the highest-ranked IC density. The hotspot density was shown to be prognostic in patients with metastatic colon cancer [2,29]. With vector datasets, IC densities across anatomically defined subregions (such as stromal versus intraepithelial) can be efficiently calculated using statistical software (such as the sf or sp packages in R) [39].

Kernel density smoothing or estimation is another statistical method of estimating IC density across spatial subregions of a tumor. In kernel density estimation, vector datasets are used to estimate IC density as the sum of kernel functions centered in the IC locations. With this approach, a smoothed estimation of IC density can be calculated for each (x,y) coordinate within the tissue. This method was employed to create topographical maps of lymphocyte densities across breast tumors [28]. The method was also used to characterize lymphocytes according to the density of cancer cells within their locoregional environment. Lymphocytes were categorized into three spatial subtypes: intratumoral, adjacent to tumor, and distant to tumor [28]. The ratio of intratumoral lymphocytes to tumor cells, designated the intratumoral lymphocyte ratio, was prognostic of clinical outcome, independent of other clinical factors.

Metrics of spatial heterogeneity

Cell counts, densities, and ratios define IC patterns based upon the central tendency of immune infiltrates within a given tumor region. However, the underlying spatial distribution of IC across tumor subregions may provide additional prognostic/predictive information. Using raster analysis, once IC densities are estimated across raster cells/subregions of the tumor, spatial heterogeneity summary metrics can be calculated. One common metric is the standard deviation (SD), which indicates how much subregion IC densities differ from the mean IC density. In an analysis of metastatic colon cancer specimens, the SD of CD8⁺ cell density was shown to associate with inferior disease-free survival and overall survival [29]. SDs are influenced heavily by the underlying mean IC density (with densely infiltrated tumors having a greater tendency to exhibit higher SDs). An alternative statistic is the coefficient of variation (CV), which is the ratio of the SD to the mean. These statistics may also be influenced by the size of the subregion.

Another metric of heterogeneity is skewness, which is a measure of the asymmetry of distributions and can be calculated as the difference between the mean and the median, normalized by the SD (non-parametric skew). A positive skewness value indicates the occurrence of high-value outliers. In an analysis of tissue microarrays of 998 early-stage breast cancer specimens, high skewness of the lymphocyte/cancer cell ratio (indicating the presence of subregions with unusually high ratios) was prognostic for improved disease-free survival in HER2-positive breast cancer [31]. Shannon’s entropy metric provides a measure of randomness across the tumor subregions, with higher scores indicating a higher degree of heterogeneity of IC density. In a recent analysis of TNBC specimens, Shannon’s entropy was shown to be higher in tumor subregions adjacent to the IM of the tumor [6]. Other useful measures of species richness borrowed from the ecology field may be useful for describing heterogeneity, including the Gini Simpson, inverse Simpson, and hill numbers.

Cluster/hotspot analysis

Cluster analysis aims to define IC location along a spectrum that varies from clustered to dispersed. Cell type ‘clusters’ indicate that the cells of interest are more likely to be colocalized with one another, whereas ‘dispersed’ indicates that cells are repelling one another. Following this paradigm, statistical testing can be employed to evaluate whether the ICs are clustered or uniformly dispersed versus the alternative hypothesis that ICs are spatially random (i.e. neither clustered nor uniformly dispersed). Because IC clusters may reflect underlying biological immune engagement, there is tremendous interest to identify tumor subregions with abundant IC density (‘hotspots’) versus areas with IC paucity (‘coldspots’).

Numerous hotspots/clusters analysis approaches have been reported in the literature. Visual characterization of H&E breast cancer TIL hotspots and coldspots (as binary ‘absent or present’ variables) were shown to predict progression-free survival, independent of clinical risk factors, in a cohort from the Cancer Genome Atlas and Carolina Breast Cancer Study [44]. A commonly adopted quantitative method for analyzing hotspots is to rasterize the tumor area, rank-order the subregions by IC density, and define hotspots as either a fixed number, fixed percentile, or a fixed SD cut-off of highest-density subregions (Figure 2A) [13,29,34]. In a series of 53 melanoma patients treated with pembrolizumab, it was concluded that a hotspot definition of IC densities of 30% of the highest-ranking HPFs resulted in the best prediction of the clinical response to pembrolizumab and outperformed conventional assessments of mean IC density [13]. In another study of metastatic colon cancer using the Definiens platform, hotspot density was defined as the density within the three most densely infiltrated raster cells within the tumor area [29]. Using this definition, patients with high hotspot density tumors had improved survival.

Figure 2.

Illustration of various hotspot metrics. Various methods of calculating IC hotspots have been described in the literature, and include methods based upon rank-ordering of IC density across subregions, or based upon inferential testing. (A) Rank-order-based approaches, which define hotspots as either ‘top 3’ (the three most densely infiltrated subregions) or ‘top 30%’ (the top 30% most densely infiltrated subregions). (B) The Getis-Ord Gi* method, which uses inferential statistical testing to estimate p values indicating the likelihood of each subregion being a hotspot or a coldspot. The Getis-Ord test statistic follows a normal distribution and can be thought of as a measure of local IC density in neighboring subregions, relative to overall IC density. IC, immune cell; TC, tumor cell.

One limitation of rank-order hotspot analysis is that it can be influenced by tumor size: larger tumors that have more extensive spatial sampling will have a higher inherent likelihood of having hotspots with greater IC density. This observation was noted in the colon cancer hotspot analysis [45]. Another limitation is that it rests upon the presumption that all tumors have hotspots, whereas it is biologically possible that IC hotspots occur at different frequencies across tumors. As an alternative, the Getis-Ord Gi* is a probabilistic approach for defining hotspots and coldspots within a tumor (Figure 2B). The approach involves defining neighbors for each tumor subregion (i.e. usually defined as immediately adjacent subregions) and conducting a formal hypothesis test using these data to estimate a likelihood that any given neighborhood is a hotspot, given the totality of the spatial data. With this approach, not all tumors will have hotspots, and derivative global metrics describing the relative proportion of hotspots (as a fraction of total tumor area) can be calculated. In a breast cancer dataset, fractional area of cancer/IC colocalization hotspots were found to be independently prognostic for clinical outcome in estrogen receptor-negative breast cancer [14] and estrogen receptor-positive breast cancer [33]. This metric was only weakly correlated with lymphocyte density, indicating that the two metrics may be useful in conjunction for stratifying tumors by immune response. Figure 2 illustrates the top 3, top 30%, and Getis-Ord Gi* approaches for defining hotspots in a breast cancer specimen. In this example, the mean IC hotspot density varies substantially depending on the method (top 3: 3.6 × l0⁻³/μm²; top 30%: 1.6 × 10⁻⁴/μm²; Getis-Ord p < 0.1: 2.2 × 10⁻⁴/μm²; Getis-Ord p < 0.05: 1.8 × 10⁻⁴/μm²).

Once clusters are defined, they can be further characterized according to features such as size (e.g. number of cells, area, maximum diameter of perimeter), shape (e.g. convexity, circularity, eccentricity, dimensionality of the perimeter), or IC phenotype/composition [6]. Clusters could also be characterized by their spatial relationships to other neighboring clusters while using different distance metrics like Euclidean norm, correlation, or nearest neighbors [35]. In a study of lung cancer specimens, cluster analysis was used in conjunction with supervised ML to generate a spatial score called the spaTIL score, which was shown to be prognostic of recurrence-free survival in non-small cell lung cancer (NSCLC) [35].

There are numerous hotspot/clustering metrics employed in the literature (some listed in Table 1), highlighting the lack of standardization in the field. Efforts should be made by investigators to justify their selection of hotspot method and to conduct sensitivity analyses to evaluate the impact of the hotspot method on their conclusions.

Metrics of cellular colocalization

Vector analysis of spatial IC data allows for efficient calculation of pairwise distances between individual cells, which is particularly useful for characterizing cell–cell interactions. One metric for describing colocalization of cells is the nearest neighbor distance metric. The nearest neighbor distance is calculated for each IC in the specimen and is defined as the shortest pairwise Euclidean distance between two cell types of interest. Similarly, with vector datasets, one can calculate the locoregional density of cells found within a specified radius of each cell. This method was employed to explore the relationship between clinical outcome and T-regulatory (Treg) suppressor cells in oral squamous cell carcinoma [27]. The authors posited that the density of FoxP3⁺ T-cells within a 30-μm radius of CD8⁺ effector cells (a biologically plausible distance for cell–cell interactions) could serve as a metric for Treg suppressor engagement with effector T-cells. They found that high Treg/CD8⁺ radial densities were associated with inferior overall survival, whereas the simple density ratio of Tregs to CD8 cells was not prognostic.

Using the G and cross-G function method, with vector datasets one can calculate the likelihood that ICs colocalize with themselves or other cell types at specified distances [36]. The likelihoods are plotted as a curve, with the x axis representing distance and the y axis representing the cumulative probability density of the nearest IC being within that distance (Figure 3A–D). This provides a graphical representation of colocalization (blue line) compared with what would be expected by random chance (calculated as a Poisson-distributed homogenous point process, red line). For example, in Figure 3C, for cancer cells the red line exceeds the blue line between distances of 0 and 13 pixels, suggesting that cancer cells tend to repel each other at these distances, whereas at distances >13 pixels, the cancer cells tend to group together. In Figure 3D, the red line exceeds the blue line across all distances, suggesting that cancer cells and ICs do not colocalize. The area under the curve (AUC) can be calculated to provide a global metric of colocalization of two cell types within a certain proximity range (e.g. 50 pixels, as illustrated in Figure 3D). With this approach, in a NSCLC cohort, AUC scores of cross-G functions between Treg or CD8⁺ ICs to cancer cells were prognostic for survival [36]. As a raster-based alternative to the G function, Ripley’s K function quantifies colocalization across specified distances by comparing IC densities across neighboring raster cells. In an analysis of H&E breast cancer specimens, extremes in the K function for stromal cells were associated with improved prognosis [32].

Figure 3.

G and cross-G function for describing cellular colocalization. (A) The (X,Y) locations of ICs in relation to cancer cells of an early-stage breast cancer specimen. Colocalization of cells at specified distances can be illustrated using (B and C) the G function (colocalization of the same cell type) and (D) the cross-G function (colocalization of two distinct cell types). The blue lines illustrate the observed colocalization patterns of the sample, whereas the red lines illustrate the expected colocalization under the assumption of randomness/homogeneous point pattern. In (D), the AUC is illustrated in green and is used to provide a global metric of colocalization of two cell types within a certain proximity range (<50 pixels in this example). TC, tumor cell.

Another proposed colocalization metric borrowed from the ecology literature is the Morisita-Horn index. The method is raster-based and generates a scaled score between 0 and 1 based on the calculated ratios of cell densities for two cell types within each raster cell [7], with scores close to 1 indicating colocalization and scores close to 0 indicating spatial independence of the two cell types. In a breast cancer cohort, higher Morisita-Horn indices between ICs and cancer cells were prognostic for disease-free survival, particularly in HER2-positive breast cancer. This metric outperformed IC density or hotspot analyses [16]. Numerous alternative methods for quantifying colocalization patterns have been reported, again highlighting the lack of standardization in the field and the need for investigators to justify their selection of test [6].

Using multiplex assays, recurrent structures of complex cell–cell spatial interactions can be uncovered within the tumor microenvironment. Using a spatial database of 458 breast tumors, hierarchical clustering methods were used to uncover 10 cell–cell interaction profiles, defined by unique clustering patterns of mixtures of cell types [46]. Examples of profiles include a tertiary lymphoid structure-like profile and a regulatory-cell profile that included Tregs combined with cells with upregulated checkpoint proteins such as PD-1. Classification of tumors according to the presence/absence of these structures was found to predict long-term survival [46]. In another study, a graph neural-network ML approach was used to analyze patterns of cell–cell distances and predict long-term survival in a cohort of gastric carcinomas [47].

Segmentation and interface metrics

Segmentation of a tumor into biologically distinct tissue compartments (such as epithelial nest versus stroma) is a crucial component of spatial IC analysis. The segmentation process can be either manual (e.g. via visual pathologist inspection, as exemplified in the breast cancer sTIL score, or by using pan-cytokeratin stains to identify epithelial populations) [1,48], semi-automated using supervised ML (e.g. using the InForm package for mIF or QuPath) [9,12,49], or fully automated with novel ML platforms [8,50]. In the breast cancer sTIL score, ICs are separately scored across stromal versus intraepithelial compartments, providing a powerful prognostic tool (described below) [1,40].

In many epithelial cancers, such as invasive ductal carcinoma of the breast, the epithelial/stromal boundary is well-defined and can be further characterized according to the distributions of cellular phenotypes at various distances from the boundary or by the geometric shape of the boundary. Using multiplex staining it has been shown that certain cell phenotypes (such as myofibroblasts) are enriched at the stromal boundary of breast cancers, whereas other phenotypes (including B-cells) are depleted [46]. With fractal dimension analysis, the geometric complexity of the interface is quantified (with a simple line receiving the lowest score and jagged interfaces receiving higher scores) [7,34]. The epithelial/stromal fractal dimension may reflect differences in underlying tumor biology. For example, a high fractal dimension in breast cancer is associated with higher cancer cell proliferation indices (measured by Ki67 labeling) [38]. Because a high fractal dimension results in increased stroma/cancer cell surface area, it has been proposed that the fractal dimension could modulate suppressive immune reactions between stromal fibroblasts that reside at the interface [34]. Supporting this hypothesis, the fractal dimension of NSCLC specimens was higher in immune cold regions compared with immune hot regions [34]. Fractal dimension analysis of fibers of the extracellular matrix, and other features of matrix fibers, can be quantified with image analysis software, and may contain prognostic information [51].

In breast, colon, and other cancers, ICs congregate in greater density at the IM of the tumor relative to the tumor center [2,37] and thus the margin could influence overall IC density within a specimen. This is especially relevant for the formation of tertiary lymphoid structures that form IC ‘hotspots’ found preferentially near the IM and provide relevant insights into activation of the immune system [52,53]. Biologically, differences at the IM could be related to physical impediments such as desmoplastic/fibrous stroma or interstitial pressure, versus chemical barriers related to secreted factors such as chemokines [37,54]. The IM is most commonly identified by pathologist visual annotation [6,37]; however, ML approaches are also being developed to automate this process [2,8]. In spatial analyses, the IM is often defined as a subregion of tissue spanning a fixed distance from the invasive edge. The colon cancer Immunoscore uses 360 μm, but other cut-offs have been proposed, including 500 or 100 μm [6,37,55]. Once these regions are defined, metrics can be calculated, such as the IM cell density [2] or ratios of densities [37]. These metrics can also be used to categorize tumors according to the degree of infiltration at the IM tumor center (i.e. ‘immune desert’, which describes low IC density at both, versus ‘immune excluded’, which describes low IC density in the tumor center but higher IC density at the IM surrounding the tumor). Using vector datasets, more sophisticated metrics, such as density gradients, can be calculated [39]. One proposed method is to partition the tumor area into bands defined by Euclidean distance from the IM and to estimate IC densities and 95% CIs of density across each distance range [6,56].

Case discussions: the breast cancer sTIL score and colon cancer Immunoscore

Here, from a spatial data perspective, we summarize two leading IC clinical biomarkers, breast cancer sTIL score and the colon cancer Immunoscore, and describe future enhancements that could be achieved by applying advanced spatial analysis techniques, such as those described above.

sTIL score in breast cancer

The breast cancer sTIL score is a visual estimation of the mean proportion of stromal area occupied by lymphocytes from a single H&E slide. Based upon early observations of the prognostic significance of sTILs [22], the International Immuno-Oncology Working Group (TIL working group or TIL-WG) was created to standardize and validate a method of scoring sTILs [1], which has since been shown in multiple datasets to be prognostic [40,57,58] and predictive of chemotherapy ± immunotherapy response [23,59]. These efforts have contributed to the endorsement of the sTIL score for clinical use in early-stage breast cancer by the St. Gallen International Consensus Guidelines [60]. Because the TIL-WG sTIL score relies on visual estimation, it exhibits only modest interobserver scoring concordance [61]. The RING studies demonstrated a modest concordance score (intraclass correlation of 0.70 by untrained pathologists [95% CI 0.62–0.78]), which was improved to 0.89 (0.85–0.92) with the incorporation of training, additional safeguards including visual controls/scales, and the requirement for sub-sampling or the evaluation of at least three regions of tumor. An intraclass correlation greater than 0.8 is generally considered to be sufficient for adoption in clinical practices. Many of these safeguards are addressed in a continuing medical education (CME)-accredited FDA course on TILs available on the TIL-WG website [62]. A root-cause analysis identified the following recurring contributors to discordance: intratumoral heterogeneity of sTIL density, poor-quality staining, misclassification of cell type or cell segment, and insufficient spatial sampling (in biopsies with minimal evaluable tumor area).

Ongoing efforts are underway to evaluate whether novel imaging platforms or spatial analyses could produce spatial metrics that perform similarly or superior to the H&E sTIL score for clinical prognosis/prediction (illustrated and summarized in Figure 4) [3,11,63–65]. These can be summarized across several discrete steps of the sTILs scoring process: (1) segmentation; (2) subregion sampling and density estimation; and (3) calculation of an overall sTIL score. In the segmentation step, a tumor must be accurately separated into stromal and non-stromal compartments, and cells must be accurately categorized as lymphocytes versus other cells. This step was identified as a frequent source of discordance in the RING studies, with pathologists erroneously categorizing intraepithelial TILs as sTILs, or categorizing neutrophils, histocytes, and apoptotic cells as sTILs [66]. Proposed methods for improving segmentation involve the use of automated image analysis/ML methods that can more accurately classify tissues and cell types (summarized in [11]) or the use of mIF/mIHC to enable more precise segmentation through the use of cytokeratin staining of cancer cell-containing regions, as well as multiple cellular markers to discern lymphocytes from non-lymphocytes [12,63].

Figure 4.

sTILs score and Immunoscore: methodology and opportunities for spatial applications. (A) Segmentation step: for the sTILs score, the intraepithelial versus stromal tumor compartment are visually determined by a pathologist, whereas for Immunoscore, the IM versus tumor center is determined using an automated ML platform. (B) Sampling and density estimation steps: for the sTIL score, several representative subregions are visually selected and sTIL counts are estimated across each subregion, whereas for Immunoscore, the entirety of the tumor area is divided by a rectangular raster grid, and CD3⁺ and CD8⁺ ICs are counted for each raster cell. (C) Calculation step: for the sTIL score, the arithmetic mean of sTIL densities for each subregion is calculated, whereas for Immunoscore, the arithmetic mean of cohort-level percentile scores across the four cellular compartments is calculated (CD3⁺ IM, CD3⁺ TC, CD8⁺ IM, CD8⁺ TC). (D) Advanced ML, histologic imaging, and spatial analytic approaches can be applied to the sTIL score and Immunoscore to potentially improve predictive/prognostic utility.

In the subregion sampling step, pathologists must sample enough subregions of the tumor space to reflect the entirety of the specimen. In the RING studies, interobserver concordance improved substantially when pathologists were explicitly required to evaluate at least three tumor subregions [66], thus raising the hypothesis that pathologists, when unassisted, may not visually assess enough areas of the tumor. In this regard, automated imaging and/or multispectral imaging would provide a clear advantage over H&E visual sTIL estimation. ML-based software platforms could permit efficient sampling across the entirety of the tumor slide or even provide whole-slide analysis. However, further work is necessary to optimize the accuracy of these platforms and to evaluate their prognostic/predictive utility in comparison with the standard visual TIL-WG method [11]. Alternatively, high-resolution multispectral imaging permits sampling of many HPFs across a specimen. In a recent evaluation of mIF in breast cancer, using Monte Carlo simulation studies, it was shown that at least 15 HPFs per patient would be required to adequately characterize changes in sTILs in the context of an immunotherapy clinical trial [12]. Before mIF can be used clinically, significant progress must be made to standardize the mIF/mIHC workflow across institutions and validate the assay [3,67].

When TIL densities are estimated for each subregion, pathologists must accurately visually estimate the proportion of stromal area occupied by sTILs within a given subregion. This was a frequent cause of interobserver discordance. However, it was improved in the second RING study, when pathologists were provided standardized images of various sTIL scores for visual comparison [66]. Nonetheless, this step can be improved tremendously with automated analysis and/or mIF, which has adequate resolution to localize individual TILs, their precise geometric location in relation to the stromal compartment, and to compute the overall stromal area for each subregion.

The last step is calculation of overall sTIL density, which is where spatial data analysis can provide the largest gains for accuracy and precision. In the RING studies, pathologists were instructed to report the arithmetic mean of sTIL scores across the various tumor subregions. However, from a spatial analytic perspective, this method relies on a simplistic assumption that IC density is homogenous, when in fact biologically the IC density can vary tremendously across various subregions of the tumor, and can be influenced by spatial interdependencies (i.e. spatial autocorrelation) as well as anatomic features (such as the IM, which exhibits inherently higher IC density relative to the tumor center). Using vector-based datasets that include (x, y) coordinates of ICs as well as anatomic landmarks (such as the IM), spatial regression methods (such as inhomogeneous Poisson process models) [20] could be used to generate adjusted sTIL scores that may more accurately reflect the intrinsic immunogenicity of the tumor. With inhomogeneous Poisson process models, IC densities are estimated repeatedly across the tumor sample (using kernel density smoothing, radial densities, or other methods). Furthermore, for each observation, spatial covariates such as proximity to the IM can be estimated and included as parameters in a regression model, thus allowing for a spatially adjusted estimation of IC density. Other variables of interest could also be included in the model, such as local PD-L1 protein expression or treatment exposure [20].

Whole-slide TIL density metrics have also been shown to be prognostic in melanoma, and there are ongoing efforts to standardize and automate scoring. Recently, automated TIL density estimation using ML classifiers (NN192 or QuIP TIL CNN) has been shown across multiple datasets to improve prognostication compared with the historical Clark grading system for TILs [68–71]. A detailed analysis of the potential pitfalls of ML assessment is provided in a companion article [72].

The colon cancer Immunoscore

Immunoscore was developed as a prognostic biomarker in the context of early-stage colon cancer. It differs from the breast cancer sTIL score in several regards. First, it is based upon quantification of T-cell populations using IHC staining of CD3 and CD8. The selection of these markers was based upon expert consensus, genomic studies illustrating the prognostic value of Thl-type gene signatures (that would correlate with CD3 and CDS infiltration and initially also on T-cell activation reflected by granzyme B expression) [73], and the relative stability and quality of staining of these antigens. Second, whereas the H&E sTIL score aims to characterize the mean cell density across the entirety of the tumor stroma, Immunoscore addresses spatial heterogeneity by segmenting and quantifying IC densities within two compartments, the tumor center and the IM, with the margin defined as a region surrounding the border between normal cells and cancerous cells. Analysis of these two distinct compartments was shown in a large cohort to better predict colon cancer outcomes, compared with single-region analysis. Third, Immunoscore relies on image analysis software (Immunoscore Analyzer, HalioDx), and the coding and detailed methods are not available publicly.

Extensive collaborative efforts by a SITC consortium have been undertaken to clinically validate Immunoscore as a prognostic biomarker for colon cancer [2]. In this study, Immunoscore was found to be prognostic across a training set, a validation set, and an external validation set. Consistency of staining was established through the use of a biomarker reference center that disseminated an optimized immunostaining protocol and control specimens. Cut-offs for high versus low Immunoscore were defined using the training set. Cut-offs of 25% and 70% were selected to create a three-tier Immunoscore (low versus intermediate versus high). A cut-off of 25% was selected for a two-tier Immunoscore (low versus high). Using this methodology, Immunoscore was proven across training and two validation cohorts to be highly prognostic of disease-free and overall survival, independent of other clinical factors. Recently, Immunoscore was also shown to have prognostic significance in the context of metastatic colorectal carcinoma [29].

Reframed using spatial metrics, the Immunoscore method includes the following steps: (1) segmentation of the tumor into IM versus tumor center; (2) rasterization of each compartment and estimation of IC densities in each raster cell; (3) calculation of mean IC densities for each compartment and calculation of a composite Immunoscore. Like the breast cancer sTIL score, Immunoscore could be enhanced by leveraging novel spatial analytic techniques across each of these steps. In the segmentation step, tumor regions are segmented into one of two tissue types, IM or tumor center. This is conducted by defining an IM buffer zone as 360 pm spanning each side of the border separating tumor from normal tissue. This approach is reasonable, albeit arbitrarily defined, but there exist alternative methods for adjusting spatial heterogeneity in relation to the IM. Across multiple tumor types, ICs have been shown to exhibit a density gradient according to distance from the margin. To capture the data inherent to the density gradient, IC and IM spatial locations could be collected in vector dataset format, allowing for computation of densities across regions spanning various distances from the margin. These densities could then be assessed for prognostic significance using regression modeling. One caveat of Immunoscore is that the IM may not be adequately sampled in every biopsy.

In the rasterization step, the tumor is divided into rectangular subregions. Each subregion is labeled as margin or center, and the IC densities are estimated within each subregion. Because raster grid size or shape (e.g. rectangular, hexagonal, Voronoi polygon) has been previously shown to influence overall IC estimation and prognostic outcome, in the context of the SITC Immunoscore validation project [2], it would be of interest to use this dataset to conduct a sensitivity analysis that illustrates the impact of rasterization method on Immunoscore. These data may be helpful for improving the performance of Immunoscore but may also enhance other investigations in the field of spatial IC analysis.

In the calculation step, raster cell IC densities are used to calculate the sample’s overall Immunoscore. First, for each cell type (CD3 and CD8) and compartment (IM and tumor center), the mean IC density across all raster cells is computed. The mean is then converted into a percentile score by comparing against scores across the entire cohort of samples. Finally, the percentiles across the four cell types (CD3/margin, CD3/center, CD8/margin, CD8/center) are averaged, creating a single Immunoscore for each patient, which is prognostic of survival. Using the clinically annotated SITC Immunoscore dataset, alternative spatial metrics could be developed and tested for clinical utility. Examples would be to explore the prognostic significance of colocalization metrics of ICs with tumor cells (as summarized above), to evaluate the significance of the shape of the IM (i.e. fractal dimension, as described above), or to evaluate the prognostic impact of IC hotspots/coldspots (as described above). A ML approach could be adopted to identify a unique combination of spatial metrics that confers the optimal prognostic utility.

Conclusion

Modern histologic imaging combined with ML has provided the opportunity to evaluate not only the overall IC density of tumors, but also the degree of heterogeneity of IC infiltration, the absence/presence of IC clustering, patterns of cell–cell colocalization, the complexity of the tumor/stromal interface, and the gradient of IC density in relation to the IM of the tumor. Two complementary analytic approaches, raster-based and vector-based, can be leveraged to generate spatial biomarkers that offer a more fine-grained picture of tissue architecture. An important future step is to amalgamate spatial and transcriptomic data using next-generation spatial sequencing technologies, which provides us with the challenging opportunity to analyze all cell types of a tissue in their spatial context. Although most of the available technologies focus on analysis of two-dimensional images, in the future, strategies may be developed to analyze TIL patterns in three-dimensional space. Additionally, sustainable strategies for cohort generation, data sharing, and dimensionality reduction methods to decrease data storage requirements are becoming more crucial than ever.

In this nascent field, a key deficiency is the lack of an investigative path to establish that novel spatial biomarkers can improve clinical prediction/prognosis beyond that of existing biomarkers. With the advent of cloud computing and with increasing emphasis on data sharing by regulatory agencies and editorial boards, in the future it may be possible to leverage shared databases that contain digitized imaging files and outcomes data from pivotal clinical trial datasets. Such databases could provide necessary standardization, quality control, and sample sizes to permit the validation of clinical utility of a novel spatial biomarker. Efforts should be made by industry, consortia, investigators, and other agencies to contribute to these databases. Oversight by a committee may be required to ensure that the data are used efficiently, without bias, and without compromising patient privacy. An additional validation step for a novel biomarker is external quality assessment, whereby multiple laboratories replicate and confirm findings. In the context of mIF/mIHC-based biomarkers, external validation might be achieved via multi-institutional consortia sponsored by the National Institutes of Health, SITC, and other organizations.