Label-free deep profiling of the tumor microenvironment

Abstract

Label-free nonlinear microscopy enables non-perturbative visualization of structural and metabolic contrast within living cells in their native tissue microenvironment. Here a computational pipeline was developed to provide a quantitative view of the microenvironmental architecture within the cancerous tissue from label-free nonlinear microscopy images. To enable single-cell and single-extracellular vesicle (EV) analysis, individual cells, including tumor cells and various types of stromal cells, and EVs were segmented by a multiclass pixelwise segmentation neural network and subsequently analyzed for their metabolic status and molecular structure in the context of the local cellular neighborhood. By comparing cancer tissue with normal tissue, extensive tissue re-organization and formation of a patterned cell-EV neighborhood was observed in the tumor microenvironment. The proposed analytical pipeline is expected to be useful in a wide range of biomedical tasks that benefits from single-cell, single-EV, and cell-to-EV analysis.

Introduction

Tumor cells, cancer-associated stromal cells, and extracellular vesicles (EVs) orchestrate their influence on cancer progression with regionally distinct structural, molecular, and metabolic changes (1-6). Direct microscopic analysis of cells and EVs in living tissue is imperative for understanding the spatial and functional heterogeneity or landscape of the tumor microenvironment, which has profound biological and clinical implications. Many advances have been made in the area of microscopy thanks to the rapid development of markers and labels (7-11). However, marker-based analyses are fundamentally limited by the complex tissue distribution of the markers, unknown disturbance of physiological functions, the requirement of prior knowledge about the targeted molecule or receptor, unavoidable artifacts of non-specific false-positive binding, and the limited number of fluorescence labels that can be used simultaneously (12,13).

Label-free nonlinear optical microscopy, which produces high-resolution images with rich functional and structural information based on intrinsic molecular contrast, has demonstrated strong potential to overcome these problems by generating a broader array of volumetric signals from tissue structures and molecular composition (14-20). Here we use two-photon autofluorescence (2PF) and three-photon autofluorescence (3PF) to target the metabolic coenzymes FAD and NAD(P)H, respectively, which are the primary electron acceptor and donor in oxidative phosphorylation (21). Metabolic imaging of the relative concentration of these two coenzymes reflect the activity of the mitochondrial electron transport chain, and can shed light on metabolic changes including cellular metabolic rate and vascular oxygen supply (17,21,22). Second Harmonic Generation (SHG) occurs for non-centrosymmetric molecules, resulting in its high sensitivity and specificity to collagen fibers (23). Third Harmonic Generation (THG) occurs for interfaces with optical heterogeneity, such as sharp differences in refractive index, dispersion, resulting its sensitivity to a variety of structural interfaces, including adipocytes, leukocytes, microvesicles (24). Using a tailored optical fiber source with unique wavelengths (1080-1140 nm), pulse shaping, and repetition rate (10 MHz), the metabolic signatures of the tissue via two-photon fluorescence (2PF) of FAD and three-photon fluorescence (3PF) of NAD(P)H (25-29), and the structural properties (lipid-water interface, collagen fibers) via THG (optical heterogeneity) and SHG (non-centrosymmetric structures) (24), can captured with precise temporal and spatial co-registration (27,30). The previously developed simultaneous label-free autofluorescence multi-harmonic (SLAM) microscopy (See more in Methods) provided a useful platform for simultaneously visualizing and quantifying various cell types, EVs, and the extracellular matrix in living tissue based on their intrinsic, metabolic, and structural optical signatures.

With the rich molecular information provided by label-free nonlinear microscopy, a systematic and comprehensive spatial analysis of the individual cells and EVs promises to provide a unique view of the complex and heterogeneous tumor microenvironment. Computational tools that can capture the spatial diversity and molecular alterations of cancer cells, stroma cells, and EVs in their authentic environment will accelerate the rate of biological discovery as well as the translation of this knowledge into new clinical biomarkers. Here, we developed a computational pipeline that analyzes the major cell types and extracellular vesicles in the authentic tumor microenvironment from label-free nonlinear microscopy. Identification of the major components of the tumor microenvironment, including tumor cells, stromal components (fibroblasts, endothelial cells, lymphocytes, red blood cells, adipocytes) (31), and EVs, was performed by a multiclass pixel-level deep neural network (DNN)-based segmentation model (32,33). By comparing cancer tissue with normal tissue, extensive tissue re-organization and formation of a patterned cell-EV neighborhood was observed in the tumor microenvironment. To infuse the metabolic and structural information into the spatial analysis, the morphological features extracted from the segmentation maps were overlaid with the optical signatures, which is a combination of molecular structural contrast and metabolic contrast. The computational pipeline developed here allows for deep profiling of cells and EVs within the original microenvironmental context in a single-cell/EV, label-free, and multilevel fashion, and will advance investigations and our understanding of the spatial heterogeneity of the tumor microenvironment. In addition, due to the label-free nature and the streamlined analytical capacity, the proposed computational pipeline can potentially serve as a complementary real-time analysis tool for point-of-procedure diagnosis (34), providing new insights into disease and therapy monitoring at the point-of-care through metabolic mapping of the tumor microenvironment (16,35).

Methods and Materials

Virtual histology image datasets

Laboratory-based virtual histology images were collected using a custom-built benchtop multimodal multiphoton microscope capable of performing simultaneous label-free autofluorescence multiharmonic (SLAM) microscopy intravital in normal and tumor-bearing rats (27,30). The system setup and acquisition parameters largely follow our previous work (27). The excitation beam uses 1110 ± 30 nm pulses generated from a fiber supercontinuum. For detection, THG channels uses a band-pass filter of 370 ± 10 nm, 3PF 450 ± 30 nm, SHG 555 ± 15 nm, and 2PF 610 ± 30 nm. For this pilot study, three rats were imaged for the control group (normal/healthy) and three rats were imaged for the experimental group (cancer). From these rat models, images were acquired with a field-of-view of 1 mm x 1 mm (4 million pixels) from 7 different imaging sites in the experimental group and from 6 different imaging sites in the control group, at an optical resolution of 700 nm and a pixel size of 500 nm. Animal procedures were conducted in accordance with a protocol approved by the Institutional Animal Care and Use Committee at the University of Illinois at Urbana-Champaign. To induce mammary tumors in the female Wistar-Furth rats (Harlan, IN), NMU (N-Nitroso-N-methylurea) (Sigma, St. Louis, MO) diluted in distilled water (12.5 mg per mL) was injected intraperitoneally at a concentration of 55 mg per kg into the left side of the abdomen when the animals (Fisher 344, Harlan, Indianapolis, IN) were 7 weeks old. One week later, the same amount of NMU was injected intraperitoneally into the right side of the abdomen. After approximately 12 weeks of age, when mammary tumors became palpable, rats were prepared for in vivo imaging. Surgery to expose the primary tumor and its neighboring mammary tissue was performed under isoflurane anesthesia. During the imaging sessions, each rat was anesthetized with 1% isoflurane mixed with O₂, at a flow rate of 1 L per min, and every effort was made to minimize suffering. Physiological temperature was maintained by a heating blanket. Imaging duration was kept under 3 hours to avoid complications of long-term anesthesia. Rats were euthanized immediately after imaging. Histological slides were obtained after imaging session, which were used as our gold standard for correlating and recognizing cells and structures of the tissue.

Training the DNN

The input to the algorithm was the raw four-channel virtual slides (multiphoton images) together with the multiclass pixelwise labelling (negative, tumor cells, fibroblasts, endothelial cells, lymphocytes, EV-shedding stroma cells, red blood cells, adipocytes, and EVs). All input images were converted to 8-bit representation and saved directly to the GPU memory to facilitate access for the training process later. During each iteration, a mini-batch of 32 sets of 128 x 128 x 4 tiles was randomly cropped and augmented (flipped and rotated) from randomly chosen virtual slides. The loss function was defined as the cross entropy between the ground truth (manual labels obtained by correlation with histology and domain expertise) and the predicted probability of each class. We followed the modified U-Net (32) architecture by Yang et al. (33), where ResNet bottleneck design with identify shortcuts and batch normalization are applied on top of the standard U-Net (32) (Supplementary Figure S1). An Adam optimizer was used for iteration steps, with a learning rate of 0.0005, first momentum coefficient of 0.9, second momentum coefficient of 0.999, and epsilon of 1x10⁻⁸, which were adopted from previous segmentation work (36). Similar to H&E slides, our images have high resolution (500 nm as pixel size) and large field-of-view (1 mm x 1 mm). Since there are many cells in each mosaicked image, we selected sub-patches from the mosaicked image to perform labeling. More specifically, 226 image patches (128 by 128 pixels) were used for training, 16 image patches (128 by 128 pixels) from were used for validation, and 22 image patches (128 by 128 pixels) from were used for testing (results shown in Supplementary Table S1). The main challenge of using DNN for cell segmentation is the limited training data. As anti-rat antibodies specific to different immune cells are challenging to obtain and apply, we resorted to different approaches for validation of different cell types. Spatial alignment and correlation between label-free multiphoton images and second-round marker-based images can be difficult and was performed with care and labor. For example, in the case of stromal cell labeling in normal breast tissue, as dipping and immersion is required for acridine orange to work on thick tissues, we relied on the geometry of the tissue and collagen fiber structures to maintain the same spatial location, depth, and orientation when the tissue was placed back onto the imaging stage for the second-time visualization (after acridine orange labeling). A more robust method is needed in the future for a larger-scale study to fully unleash the power of label-free intravital imaging.

Visualization

The rendering of SLAM images follows our previous work (27). Without image preprocessing, the raw four-channel image/video was loaded onto FIJI (National Institutes of Health) to apply pseudo-color maps to merge the contrast. The same color maps were used consistently throughout the study, namely magenta (hot) for THG, cyan (hot) for 3PAF, green for SHG, and yellow (hot) for 2PAF. These maps were then loaded into MATLAB to facilitate automatic generation of pseudo colored images. For each set of SLAM image, the segmentation mask was first created as a stack of image with multiclass labels, where each image corresponds to the probability map of a certain class. Then we assign different colors to different labels and overlay the different labels onto one image, which generate the multiclass segmentation mask shown in Fig. 3 (middle panel). To have a better correlation of the original image and the segmentation mask, the segmentation mask was added to the original image at with a third of intensity, which generates the overlaid image in Fig. 3 (right panel).

Figure 3. Segmentation results of tumor cells, fibroblasts, vascular endothelial cells, lymphocytes, EV-shedding stroma cells, red blood cells, adipocytes, and EVs from representative label-free images of A) control and B) cancer animals (27).

Scale bar: 200 μm. Red blood cells were labelled and segmented in the form of a blood flow/strip because it was challenging to distinguish them at the single-cell level in the static image, especially for blood flow within the vessels.

Feature quantification

A total of 8 features were chosen among the features extracted from the spatial map and multimodal signatures of the images. Morphological features include the diameter, symmetry, area, perimeter, roundness, and irregularity of the shape. Optical features include the intensity of 2PF and 3PF, representing the relative level of FAD and NAD(P)H, respectively, and the THG being generated from the lipid layers/structures of each EV (optical heterogeneity at the aqueous-lipid interface). The co-occurrence value describes the probability of the co-occurrence between two cell types (37). A high probability score (or a high degree of positive association between A and B) corresponds to a high probability of co-occurrence of cell type A and cell type B. The probability is estimated by computing the frequency of cell type B as the nearest neighbor in the presence of cell type A (as the center/index cell of niche) divided by the expected frequency of the cell type B. This metric was adapted from Goltsev et al.’s work on labeled cell analysis of the mouse splenic architecture (37) to quantify the cellular interaction strength.

Statistical Analysis

To describe the discrepancies of the properties between individual cells (Supplementary Table S2), the 95% confidence interval of the average feature value was computed using bootstrap with 10,000 bootstrap samples for each feature. A two-sample Student’s t-test was performed to compare the co-occurrence probability of cell pairs between the cancer samples (n=7) and normal samples (n=6) (Supplementary Figure S2A-C). As the most frequent cells in cancer tissue (e.g. tumor cells, EV-shedding stromal cells) are absent (or scarce) in the normal tissue, the statistical significance of the high association between two cell types within cancer tissue (e.g. endothelial cells and EV-shedding stromal cells) was tested against other cell pairs (e.g. the normally highly-associated pairs -- endothelial cells and fibroblast) within the same set of cancer tissue (n=7) using the paired Student’s t-test (Supplementary Figure S2D,E). To investigate the correlation between the optical signature of EVs and their distance to different cell types, Pearson Linear correlation coefficient was computed, and the corresponding P-value was computed for testing a non-zero Pearson’s correlation using a Student’s t distribution (EV number = 10,620).

Data availability

The data and code to perform the analyses have been deposited in OSF (https://osf.io/tgprd/?view_only=22262b9c2a394af8a41327ff553d3823) and on GitHub (https://github.com/Biophotonics-COMI/Segmentation-SLAM). The authors provide these for joint collaborative investigations with other research groups.

Results

Identification of cells and EVs in label-free multiphoton microscopy

Similar to other cell-centered microscopic analyses, the first step of quantitative analysis for label-free nonlinear microscopy is to identify biological structures of interest and segment the multimodal images into meaningful regions (Fig. 1). Tumor cells, stroma cells, and EVs are of primary interest for their vital roles in carcinogenesis and metastasis. By correlating with histological images and antibody-labeled images, a wide array of key cell types and structures stand out from the label-free multiphoton images due to their distinct endogenous fluorescence, unique morphology, and consistent spatial contextual patterns in the tumor microenvironment (27-30). Inspired by previous work on artificial labelling of label-free cell/slide imaging (38,39), here we investigated the possibility of artificial labeling of label-free intravital imaging in a more complex and richer tumor microenvironment. As demonstrated previously, a deep neural network (DNN) is a proper candidate for this task due to its ease of integrating multi-dimensional spatial cues in the absence of explicit physical models. In this case, the DNN directly learns from data about the implicit rules that explore optical signatures, morphological cues, and contextual relationships for every target of interest.

Figure 1. Flowchart of this framework.

Proposed computational pipeline for label-free profiling of the tumor microenvironment. Label-free nonlinear microscopy will be first converted to segmentation masks of various cell types and EVs through a DNN-based model. Subsequently, the characteristics of the cells and EVs can be profiled with the spatial interaction mapping, optical signature analysis, and morphometric measurements, thus enabling label-free deep profiling of the tumor microenvironment.

The main challenge arises from the difficulty of obtaining ground truth for label-free intravital imaging or thick tissue imaging. We have resorted to different validation approaches for different targets of interest within the label-free multiphoton images over a series of studies (27-30). For cells and structures are have distinct morphological features such as tumor cells, vascular endothelial cells, and red blood cells, we relied on histology for morphological correlation. Comparisons between representative SLAM microscopy images and corresponding H&E-stained tissue sections are shown in Fig. 2A-C. The high degree of similarity in size, morphology, and spatial distribution provides morphological evidence for the identity of tumor cells, vascular endothelial cells, and red blood cells in the images. For cells that cannot be reliably recognized from conventional histology, such as leukocytes, we used surface protein markers (Fig. 2D). Without exogenous contrast, lymphocytes are visualized as THG-visible hollow spheres (6-8 μm in size) due to the distribution the intrinsic lipid granules within the cell bodies, which give rise to phase-matching THG-signals(40). With the injection of a CD45-based marker (Monoclonal antibody OX1, eBioscience, for leukocytes), lymphocytes are expected to exhibit the color of the tag (cyan channel in this example). The correlation between the CD45 images and THG images provides guidance to identify lymphocytes.

Figure 2. Identification of cells and EVs by comparing label-free (SLAM) images with previously established marker-based methods.

(A) Images of tightly packed, round-shaped, 10-micron-sized tumor cells, (B) aligned and elongated vascular endothelial cells, and (C) stream-like flowing red blood cells in a vessel by SLAM microscopy (upper row) and corresponding H&E-stained histology (lower row). (D) SLAM microscopy of the tumor microenvironment of a living rat without and with CD45 staining for leukocytes. The blue channel was assigned to NADH before staining and assigned to CD45 after staining. (E) Correlation of normal breast cells before and after acridine orange labeling of freshly excised breast tissue. (F) Imaging and characterization of isolated EVs by multiphoton microscopy and NTA. Adapted from previous work using SLAM and full details can be found from the original papers (27,30). Scale bar: 50 μm.

To validate the SLAM’s performance on normal cell detection, we used acridine orange (a nuclear dye) for labeling the fresh normal breast tissue and obtained images before and after the acridine orange staining (Fig. 2E). Despite the unavoidable spatial misalignment (caused by the dipping procedure of acridine orange staining), a largely consistent match can be observed from the content between the label-free SLAM image and the acridine orange-labeled image – cell cytoplasm by label-free SLAM image (THG and 2PF channel) and yellow spindles in the acridine orange -labeled image (stroma, not well-resolved due to saturation of the 2PF/dye channel), which demonstrates the fidelity of SLAM detecting normal stromal cells. To evaluate the efficiency for detecting EVs from mammary cells of varying breast cancer status, our previous work isolated EVs from the conditioned media of different human breast cancer cell lines and quantitatively characterized the reliability of label-free detection of EVs by SLAM microscopy (30). With no exogenous markers, the majority of EVs appear as individual diffraction-limited (their size usually smaller than the optical resolution of the microscope) bright punctate pixels in multiphoton microscopy images, with the intensity of 2PF and 3PF representing the cargo concentrations of FAD and NAD(P)H within the EVs (Fig. 2F), respectively, and the THG being generated from the lipid layers/structures of each EV (optical heterogeneity at the aqueous-lipid interface) (24). Correlations of these label-free signatures with conventional analysis methods for EVs (electron microscopy, PKH26, and nanoparticle tracking analysis (NTA)) have been demonstrated in a previous study (30), showing high sensitivity (91%) and specificity (99%) of the proposed label-free detection method.

Multi-class segmentation of cells and EVs by a deep neural network

Relying on these validation approaches, we manually annotated the label-free multiphoton images and feed them to the neural network. Training pairs consisting of label-free multicolor images and their corresponding manually labeled pixel-registered segmentation masks were created to train the DNN to segment cells and EVs of interest at the single pixel level. The training datasets focused on major cell types that are present in breast tissue, esp. mammary cancer tissue, including tumor cells, fibroblasts, endothelial cells, lymphocytes, red blood cells, adipocytes, and EVs. With these training datasets, a supervised modified U-Net (32) model was trained to perform the task of nonlinear pixel-level classification (see more in Methods and Supplementary Figure S1). By applying the trained model to label-free nonlinear microscopy images, a probability map for each class was simultaneously produced, assigning a probability of that class for each pixel of previously unseen input images. Figure 3 shows representative examples of automatically generated multi-class segmentation masks of cancer (27) and normal tissue and Supplementary Table S1 reports the segmentation performance for each class in the test dataset. A highly heterogenous distribution of various cell subtypes was observed in the cancer tissue while a relatively homogeneous distribution of fibroblasts and adipocytes was observed to dominate the cell populations in normal tissue (Fig. 3A,B and Fig. 4A,B). To capture the spatial and metabolic heterogeneity of cancer tissue, the individual properties, spatial distributions, and neighborhood influence of cells and EVs were quantitatively characterized and compared with normal tissue in the following sections.

Figure 4. Analysis of EV and cell properties and distributions.

Component percentage of (A) cancer and (B) control tissue, which is computed using the area occupied by one cell type divided by the total positive areas of the multiclass mask. Heatmaps of cellular and EV properties of (C) cancer and (D) control tissue. Values are normalized per row to facilitate visualization and comparison. Scatterplots of the optical signatures of individual EVs in (E) cancer and (F) control tissue.

Metabolic and morphological analysis of single cells and EVs

With the multi-class segmentation mask generated by the DNN model, individual cells and EVs were readily characterized by their metabolic, structural, and morphological properties in the living tissue. For this study, three rats were imaged for the control group (normal/healthy) and three rats were imaged for the experimental group (cancer). From these rat models, images were acquired with a field-of-view of 1 mm x 1 mm (4 million pixels) from 7 different imaging sites in the experimental group and from 6 different imaging sites in the control group, at an optical resolution of 700 nm and a pixel size of 500 nm. Metabolic and molecular structural features were extracted by examining the autofluorescence signals from NAD(P)H by three photon fluorescence (3PF) and FAD (21,41) by two photon fluorescence (2PF), and the harmonic signals from phase-matched molecular structures, including second harmonic generation (SHG) from collagen fibers and third harmonic generation (THG) from micro-sized lipid-water interfaces (23,42). Morphological features were extracted by using common metrics for describing cells, including diameter, area, perimeter, roundness, and irregularity (43,44). To visualize these features associated with different biological entities, two heatmaps were generated (Figure 4C,D) to display these features for each type of biological entity in cancer tissue and normal tissue, respectively. The average values of these features and the corresponding confidence intervals (without normalization) are reported in Supplementary Table S2. One of the most obvious differences between the cancer and control groups is that the average size of lipid droplets decreased more than two times in the cancer tissue (Supplementary Table S2) (45,46). This was previously observed in living mice and human breast cancer, and is attributed to the overexpression of collagen VI and the extensive extracellular matrix near tumors (47). As expected, certain cell types demonstrated relatively consistent morphological features and optical signatures, regardless of cancer status. For example, in both cancer and normal tissues, endothelial cells exhibited the highest level of irregularity with their elongated shape, and fibroblasts exhibited a higher ratio value of FAD/(FAD+NAD(P)H) compared to all other cells and EVs. Interestingly, the NAD(P)H-rich EV-shedding stromal cells were observed to be abundant in cancerous tissues, while completely absent in normal tissues (Fig. 4A-D). Consistent with our previous study (30), EVs from cancer tissue appeared to have lower mean values of FAD/(FAD+NAD(P)H) than EVs from normal tissue. Individual EVs were mapped to the 2-D space of optical signatures featuring the metabolic information by autofluorescence and molecular structure by THG. A larger spread in both dimensions and formation of at least two clusters were observed in the EV scatterplot of cancer tissue, which suggests the higher heterogeneity of EV properties in the form of distinct subpopulations in the tumor microenvironment (Fig. 4E,F). This hypothesis was investigated in later sections. This higher concentration of NAD(P)H has been observed in cancer cells (21,41) previously as a result of the Warburg effect (48) (increased glycolysis), and also a potential result of the antioxidant defense mechanism in endothelial microvesicles (49). This is an intriguing observation that highlights the possibly unique metabolic functions and signatures of EVs in breast cancer (49-51), which can potentially serve as a biomarker and better our understanding of active metabolic role that EVs play in cancer progression.

Microenvironmental landscaping by cell-to-cell/EV concurrence mapping

In addition to single-cell and single-EV analysis, a higher-level spatial analysis was performed from the perspective of cell-to-cell/EV concurrence and proximity to characterize the microenvironmental landscape, which is known to involve the active and dynamic collaboration between different cell types and EVs. Inspired by previous work on deep profiling (37), the probability of the interaction between two cell types, or one cell type and EVs, was quantified by the ratio of the observed and the expected frequency of concurrence between every pair of cell type and/or EV. The heatmaps, which summarize the pairwise cell-to-cell/EV concurrence maps, revealed highly correlated patterns between cell types and EVs. For example, a consistently high degree of association was observed between endothelial cells and NAD(P)H-rich EV-shedding stroma cells (Fig. 5A,B, and Supplementary Figure S2D), which lends further support to the active roles of EVs in angiogenesis (52). In addition, in tumor tissue, smaller and isolated adipocytes are frequently observed at the tumor boundary (Fig. 5C and Supplementary Figure S2E), which could be a result of lipolysis induced by cancer cells (53). In contrast, the most likely neighbors of endothelial cells in normal tissues are fibroblasts (Fig. 5D,E, and Supplementary Figure S2A), which could be attributed to the reliance on fibroblast-derived matrix proteins in angiogenesis (54). The diagonal line of the heatmap matrix summarizes the likelihood of self-clustering of every cell type and EVs. It was found that in normal tissues, adipocytes (lipids) tend to cluster and pack together with their own type, in the company of fibroblasts (Fig. 5F and Supplementary Figure S2B,C). On the other hand, stroma cells are observed to have a relatively even probability distribution for all the potential neighbors, which is probably due to their prevalent and relatively homogeneous distribution in the stroma, compared to other cells. This frequency analysis of concurrence enabled a high-level view of the cell-to-cell and cell-to-EV interaction landscape with quantitative details, and indicated a significant non-random distribution of cells/EVs in the living tumor microenvironment.

Figure 5. Concurrence mapping of different cell types and EVs in cancer and normal tissue.

(A) Concurrence heatmap for every pair of cell type and EV in cancer tissue. (B) A representative post-segmentation image of endothelial cells (overlaid with green masks and pointed with green arrows) in cancer tissue surrounded by NAD(P)H-rich EV-shedding stroma cells (overlaid with cyan masks and pointed with white arrows). (C) A representative post-segmentation image of adipocytes (blue mask and blue arrow) at the tumor boundary (red mask and red arrow). (D) Concurrence heatmap for every pair of cell type and EV in normal tissue. (E) A representative post-segmentation image of endothelial cells (green mask and green arrow) in normal tissue surrounded by fibroblasts (yellow mask and yellow arrow). (F) A representative post-segmentation image of adipocytes (blue mask and blue arrow) in normal tissue that features accumulation of large lipid droplets surrounded by fibroblasts (yellow mask and yellow arrow). The co-occurrence frequency between cell type AB is computed by counting the frequency of having cell type B as the nearest neighbor of cell type A, divided by the frequency of cell type B at the imaging site. Here in the heatmap, the frequency ratio was then normalized by the row to facilitate visualization and comparison. The raw frequency ratio is reported in Supplementary Figure S2.

Optical signature of EVs depends on local cellular neighborhood

The metabolic profiles of EVs and cells are highly heterogenous, depending on their origin as well as their current surroundings (1,3,5,55,56). The metabolic heterogeneity of EVs was highlighted by the 2-D scatterplot of optical signatures in Fig. 4E,F, which shows subpopulations of cancer EVs that have dramatically different metabolic and structural signatures. Interestingly, a metabolic-focused spatial analysis in the form of EV-to-cell concurrence mapping suggests a profound influence of the cellular neighborhood on the metabolic products and structural properties of the EVs. Each EV was categorized by its closest cell neighbor, e.g., if the closest cell neighbor of an EV is an endothelial cell, then this EV counts as an endothelial cell associated EV (defined in this work for simplicity of reference). Thus, the influence of cell neighborhoods on EV properties can be visualized by mapping the optical signatures of individual EVs in combination with the spatial relationship between EVs and different cell types (Fig. 6 and Table 1). For instance, EVs in the proximity of endothelial cells (green) and EV-shedding stroma cells (cyan) are strongly correlated with high levels of NAD(P)H, while EVs neighboring tumor cells exhibit the opposite properties, suggesting potentially different metabolic roles of EVs in angiogenesis and tumor growth. In contrast to such distinct distributions, EVs neighboring fibroblasts display much more homogeneous optical properties in both NAD(P)H levels and THG signals, suggesting a weaker influence of fibroblasts on the NAD(P)H levels of neighboring EVs. These results shed light on the high metabolic, structural, and spatial heterogeneity of EVs, and highlight the importance of deep profiling of EVs in the context of various cellular neighborhoods for understanding this heterogeneity. The rise of NAD(P)H levels in the subpopulations of cancer associated EVs has been correlated with early and late stages of human breast cancer (30). The significantly positive correlation between NAD(P)H-rich EVs and vascular endothelial cells observed in this study (Table 1) indicate that the rise of NAD(P)H levels in cancer-associated EVs could be attributed to increased angiogenesis and the antioxidant defense mechanisms surrounding newly growing vessels (49). The specific roles of EVs in cancer progression have been of high interest but remain largely unknown. The proposed methodology provided the capability to map the relationships between cells and EVs in the context of authentic microenvironment. A large-scale follow-up study in cancer progression is warranted to further illustrate the significance of these observations.

Figure 6. Mapping of individual EV properties in relation to their cell neighbors.

Each dot corresponds to one EV, with the color indicating their spatial origin. For instance, for EVs that are adjacent to tumor cells, they are shown as red dots in the scatterplot and their intensity distribution is shown as the red curve in the histogram plot that is parallel to the axis, with the vertical plot corresponding to THG intensity and the lateral plot corresponding to FAD/(FAD+NAD(P)H) values. For interpretation, the density of datapoints along one dimension in the scatterplot corresponds to the distribution height of the histograms.

Table 1.

Pearson Linear correlation coefficient between the optical signature of EVs and their distance to different cell types

EV optical signature/ EV distance to cell type	Tumor cell	Fibroblast	Endothelial cell	Lymphocyte	EV-shedding stroma cell
THG	0.26***	0.11***	0.25***	0.19***	0.28***
FAD/(FAD+NAD(P)H)	0.33***	−0.16***	−0.32***	−0.26***	−0.39***

(EV number = 10,620

^***,P<0.001 (P-value for testing a non-zero Pearson’s correlation using a Student’s t distribution)).

Discussion

This study developed a computational framework for the systematic spatial analysis of label-free nonlinear microscopy images, with the ultimate goal of understanding the spatial heterogeneity of the tumor microenvironment from the perspective of single-cell/EV properties and cell-to-cell/EV influence within the unperturbed living tissue environment. To enable the subsequent single-cell/EV analysis, automatic segmentation of label-free nonlinear microscopy was achieved by a multiclass (tumors, cancer-associated stroma cells, and EVs) pixelwise DNN-based model. With these generated segmentation masks, the concurrence probability between every pair of cells and/or EVs was computed to provide a high-level view of the highly collaborative tumor microenvironment. By comparing cancer tissue with normal tissue, extensive tissue re-organization and the formation of a patterned cell-EV neighborhood was observed in the tumor microenvironment. The metabolic spatial analysis of individual EVs revealed how the metabolic and structural properties of EVs depend on their local cellular neighborhoods, which provided an important illustration and potential rationale for the essence of in situ imaging to capture and further understand the EV heterogeneity.

Previous work has shown that the metabolic status of tumor cells can be characterized by label-free two-photon fluorescence microscopy and has been demonstrated to be indicative of disease progression as well as the efficacy of treatment plans (16,21,57). The proposed pipeline of automatic segmentation and quantification can enable similar large-scale single-cell single-EV analysis in these efforts. On the other hand, building on the results from previous work of label-free multiphoton microscopy, this work highlights the inclusion of multiple cell types and EVs in the analysis of the tumor microenvironment, which is an authentic reflection of the heterogeneity and complexity of the tumor microenvironment. Part of our observations (e.g. clustering of tumor cells, and adipocytes) corroborate phenomena that have been shown using conventional methods (45,52,54,58), which demonstrates the feasibility of this label-free method. More importantly, because of the direct metabolic and structural contrast, we were able to monitor the metabolic status of individual EVs and cells and their neighborhoods, which could not be easily achieved by marker-based methods. For instance, we observed that EVs in the proximity of vascular endothelial cells are strongly correlated with high levels of NAD(P)H, while EVs neighboring tumor cells exhibit the opposite properties (Fig. 6 and Table 1), suggesting potentially different metabolic roles of EVs in angiogenesis and tumor growth. Such single-EV single-cell spatial analysis paints a picture that other methods (lipid biopsy, or marked-based EV imaging) could not, and thus is an attractive complementary method that may advance investigations and our understanding of the metabolic heterogeneity of the tumor microenvironment.

Furthermore, these new features can potentially be used in clinical settings to identify earlier changes in cancer progression as the reorganization of the microenvironment can happen long before the formation of a tumor mass or the actual event of metastasis (6,59). Due to the label-free and streamlined analytical capacity, the proposed computational pipeline can serve as a real-time analysis tool for point-of-procedure diagnosis. We have demonstrated that cancer classification at the point-of-procedure is possible with the combination of label-free multiphoton microscopy and a convolutional neural network (34). The pipeline proposed in this paper (quantitative single-cell single-EV analysis based on segmentation) will add interpretability and statistical confidence to the classification network, which is largely a black box with limited interpretability. In addition to interpretation, this method promises to give pathologists a full view of the metabolic spatial mapping of the tissue, which is complementary to the existing tissue assessment methods, and can provide new and personalized insights into disease/therapy monitoring (16,35).

The main bottleneck of the proposed methodology is the availability of training data for segmenting label-free intravital images. Although the validation approaches used in this work (histology, protein surface markers, dyes) are essential for generating pixel-level ground truth and subsequent proof-of-concept analysis for intravital label-free images, we want to emphasize that the user-annotated approach is labor-intensive and might not be a sustainable solution for large-scale biomedical studies. Previous work has elegantly addressed the validation problem by imaging the same sample using both label-free and marker-based methods, which directly provided pixel-coregistrated training data (38,39). This approach is much more challenging to implement for intravital imaging due to the limited options of markers for our animal models, the incompatibility between different markers, and potential contamination of the label-free fluorescence channels. These challenges will be investigated fully in the future to establish a more robust framework (e.g. utilizing genetic labeling techniques) for validating label-free multiphoton microscopy. Another opportunity for improvement is that this pipeline only considers two-dimensional images despite the imaging sample being a three-dimensional (3D) thick tissue. As multiphoton microscopy has the advantage of optical sectioning and deeper penetration, it will be an interesting future direction to extend this pipeline to 3D analysis to obtain a more comprehensive view of the 3D tumor microenvironment.

In summary, our study demonstrated a label-free profiling tool that enables simultaneous metabolic, structural, and spatial analysis of the cell-EV architecture within the context of the living tissue environment. The metabolic, structural, and spatial profiling capacity of the developed computational framework could open possibilities for label-free profiling of other tissue architectures, and enhances our understanding of the spatial heterogeneity involved in a broad array of tissue environments.

Significance.

The proposed computational framework allows label-free microscopic analysis that quantifies the complexity and heterogeneity of the tumor microenvironment and opens possibilities for better characterization and utilization of the evolving cancer landscape.