Multiplexed molecular imaging with surface enhanced resonance Raman scattering nanoprobes reveals immunotherapy response in mice via multichannel image segmentation

Abstract

Visualizing the presence and distribution of multiple specific molecular markers within a tumor can reveal the composition of its microenvironment, inform diagnosis, stratify patients, and guide treatment. Raman imaging with multiple molecularly-targeted surface enhanced Raman scattering (SERS) nanoprobes could help investigate emerging cancer treatments preclinically or enable personalized treatment assessment. Here, we report a comprehensive strategy for multiplexed imaging using SERS nanoprobes and machine learning (ML) to monitor the early effects of immune checkpoint blockade (ICB) in tumor-bearing mice. We used antibody-functionalized SERS nanoprobes to visualize 7+1 immunotherapy-related targets simultaneously. The multiplexed images were spectrally resolved and then spatially segmented into superpixels based on the unmixed signals. The superpixels were used to train ML models, leading to the successful classification of mice into treated and untreated groups, and identifying tumor regions with variable responses to treatment. This method may help predict treatment efficacy in tumors and identify areas of tumor variability and therapy resistance.

Introduction

Molecular imaging is key to precision medicine¹. Today, ever richer images are acquired by using multiple modalities to scan a patient² or by using multiplexed analysis methods to image many target molecules in a single sample.³ Multiplexed molecular imaging can prove especially beneficial for the advancement of new cancer treatments, such as immunotherapy. In particular, immune checkpoint blockade (ICB) therapy has great potential as cancer treatment, however, the benefit of therapy has been limited to a minority of patients.⁴ The ability to detect the presence of important immune cells and other crucial markers could predict treatment efficacy and elucidate immunotherapy mechanisms.^5–7 By imaging multiple markers in vivo, namely immune cell infiltration and activation markers directly in the tumor, it may become possible to overcome these challenges.⁸ A promising approach is multiplexed IHC, where immunity-associated targets (typically immune cell infiltration) are imaged in excised fixed tumors, and the spatial distribution and colocalization of the markers have been shown to predict treatment outcome.^9–11 Newer imaging techniques based on mass spectrometry, such as multiplexed ion beam imaging (MIBI), image hundreds of biomarkers on a sample.^{12, 13} However, these techniques require excised samples, limiting their potential for in vivo imaging. A method that provides comparable features (i.e., high-resolution imaging of multiple molecular markers) while being amenable to in vivo use, will be useful for preclinical research and patient care alike.

An emerging optical imaging method has shown great promise in preclinical oncology imaging, namely Raman spectroscopy with surface enhanced Raman scattering (SERS) nanoparticles.¹⁴

The phenomenon of SERS, was discovered in the 1970s^{15, 16} and has since proved beneficial in many fields.¹⁷ SERS nanoparticles typically used in biomedical imaging rely on a plasmonic metal core that enhances the light inelastically scattered by Raman reporter molecules adsorbed on its surface. The core is encapsulated within a silica shell, on which antibodies or other targeting moieties can be attached to produce a molecularly targeted nanoprobe. The multiplexing capacity of the technique has been known for years. Proof-of-concept imaging of 10 distinct nanoparticles (each with a distinct Raman reporter but no targeting) has been shown¹⁸ along with various examples of tagging and imaging of several targets in excised tissues.^19–22 Recently, SERS-based multiplexed imaging has enjoyed renewed interest. Multiplexed imaging of 5 untargeted nanoparticles at different ratios simultaneously in vivo has been reported,²³ as well as the spatial encoding of 26 distinct nanoparticles with targeting shown for 5 targets in vitro.²⁴

Here, we report a general framework for employing the technique for imaging the tumor microenvironment. Specifically, we present how to select Raman reporter molecules for SERS nanoprobe synthesis, a quantitative way of imaging formation by color-coding the multichannel pixel-wise spectra, an example application of 8-plexed imaging in a mouse model of cancer immunotherapy, and an algorithmic approach for parsing and classifying the multiplexed images based on machine learning (ML).

Results and Discussion

Raman reporters

The Raman reporter molecule is a critical component of the nanoprobe. Although a handful of commercial dyes can be employed, we pursued a systematic custom reporter synthesis approach, which can provide many new distinct Raman nanoprobes, greatly augmenting the multiplexing capacity of the platform. Chalcogenopyrylium infrared (ChPIR) dyes have been shown to be effective SERS reporter molecules.^{25, 26} The absorbance wavelength maximum of these synthetic molecules can be tuned, as it is proportional to the molecule’s length²⁷ and sequential bathochromic shifts are produced with the incorporation of heavier chalcogens in the order of oxygen, sulfur, selenium, and tellurium.²⁸ This versatility in tuning the absorption maxima allows for structural variety without sacrificing the resonance enhancement effect that arises from having the absorption wavelength close to the excitation wavelength.²⁹ Variability in structure enables different Raman signatures to be incorporated into different dyes, augmenting multiplexing capacity. An additional benefit to using chalcogenopyrylium-based Raman reporters is that substituents can be introduced at the 2- and 6-positions with high affinity for the gold and silver surfaces, which are used as typical SERS substrates.^30–32 With these tools at our disposal, we have produced a library of ChPIR Raman reporter molecules. The synthesis and characterization of dyes are described in the Supplementary Information (SI). Their structures and UV/Vis absorbance spectra are shown in SI Figure S1.

Raman nanoprobe characterization

To determine the best candidate dyes for the synthesis of SERS nanoprobes and produce a panel of SERS nanoparticle contrast agents, we followed a comprehensive screening approach using 20 ChPIR and 8 commercial NIR dyes – all using the same synthesis methodology^{33, 34} with minor modifications as described in SI Table S1. After synthesis, the nanoparticles were characterized to determine morphology, stability, and yield. Transmission electron microscopy (TEM) images of the synthesized nanoparticles and their absorbance spectra are shown in Figure S2(a). The wide absorbance profile of the nanoparticles allows plasmonic excitation with the 785 nm laser and intense signal, consistent with our previous work^{35, 36} and reveals that the plasmonic core contributes most of the extinction, while the absorbance of the reporter molecules is negligible.

The SERS signal of the nanoprobes is shown in Figure 1, after baseline subtraction, and the raw spectrum including the fluorescence background in Figure S2(b). The nanoparticles were placed in a plastic 1,536-well plate vertically in order of chemical similarity of the reporter molecules³⁷ (Figure 1a) and horizontally in increasing concentrations from left to right (Figure 1b), and Raman scanned to establish limits of detection. We observed that not all reporters provided strong SERS signals (Figure 1c). The signal intensity depends on many parameters, including plasmonic enhancement, the reporter affinity to the gold core, and colloidal stability of the core during silica encapsulation. Chemical similarity appears to be a predictor of signal intensity and also, it can be seen that chemically similar dyes exhibit related spectral features. The reporter dyes can be clustered into four classes, based on chemical similarity, namely, (I) commercial cyanine-type dyes with moderate to good intensities and complex spectra, (II) ChPIR dyes with high intensities and single peak in the high-wave region, (III) ChPIR dyes with weak to moderate intensities and more complex spectra, and finally (IV) ChPIR dyes with very low intensity, likely due to poor interactions with the gold substrate, e.g., zwitterionic compounds and compounds with aliphatic sidechains. Chemical similarity can help in directing the selection of reporter dyes for curating a multiplex panel, or for the rational design of new reporter molecules. The Raman intensity of the best performing nanoparticles can quantify the nanoparticle concentration over at least three orders of magnitude, at least down to about 200 femtomolar, as shown in Figure 1d.

Figure 1. SERS nanoparticle library based on 8 commercially available IR dyes and 20 ChPIR dyes as Raman reporters.

(a) Raman reporters are ordered by chemical similarity for a systematic approach to developing a SERS nanoparticle library, forming four classes with related signal intensity and complexity. (b) SERS nanoparticles synthesized with the 28 reporters and imaged in a standard plastic 1536-well plate at various concentrations. Compounds grouped by chemical similarity demonstrate similar spectral features and intensities. (c) Each nanoparticle type provides a uniquely identifiable spectrum, shown here after baseline subtraction and averaging of 98 point spectra. (d) Peak signal intensity for each of the 22 brightest nanoparticles at various concentrations shows the dynamic range of the SERS signal.

Image formation

Linear regression with a least-squares fit was used for spectral demultiplexing, as it can readily decompose a complex spectrum into its constituents based on a set of known reference spectra. Here, we use non-negative least squares (nn-LS) to assign a set of scores on each point spectrum from the image, using reference spectra derived from pure nanoparticle populations. By using the constrained (non-negative) fitting algorithm, we avoid having negative scores, as this would not be physically meaningful. This process yields a set of positive scores for every pixel indicating the contribution of each of the reference spectra as is demonstrated in Figure 2, where SERS nanoparticles with four distinct Raman reporters were imaged in a microwell plate, in different concentrations and ratios (Figure 2a). Here, the nn-LS scores on each point were calculated using the reference spectra shown in Figure 2c (see Methods for preprocessing). The point-wise derived scores (Figure 2d) show the distribution for each tag, clearly separated and quantified, as well as the background signal of the plastic plate. The intensity before spectral processing follows a clear increasing trend (Figure 2e); the derived nn-LS scores for each dye correspond to the maximum intensity of the analyzed spectrum after baseline subtraction (Figure 2f).

Figure 2. Spectral unmixing and visualization for multiplex Raman imaging.

(a) Four SERS nanoparticle configurations featuring distinct reporter molecules were placed in a well plate, at various concentrations and ratios. (b) Raman image showing the intensity at a single wavenumber cannot separate the signals. (c) Reference spectra from pure populations used for spectral unmixing. (d) The nn-LS score obtained for each reference at each pixel unmixes the signals spatially. (e) Raw Raman intensity of representative peaks. (f) The nn-LS score for each configuration (dashed line) matches the corresponding Raman signal intensity after baseline subtraction at the characteristic peak (dotted line). (g) Radial coordinate system where the references are spaced angularly on the circumference of a modified HSV color wheel. The color of each pixel is determined based on its projection onto the color wheel coordinates. Spectra from pure populations are projected on lines from the center (low intensity) towards the direction of their reference (high intensity), whereas mixtures lie in areas between the corresponding references (e.g., blue is a 1:1 mixture of purple A and cyan B). The signal corresponding to “plastic” was excluded from the coordinate system. (h) Single-channel images show the signal quantitation for each nanoparticle configuration. (i) Final image showing the distribution and intensity of the four nanoparticle configurations. In the central well, an equal mixture of all four dyes yields a small vector sum (assigned to black). Pixels with signal (from the included references) below the threshold are rendered transparent. For panels (e-f) the values reported are averages of 648 spectra, with error bars showing the standard deviation from the mean.

To visualize multiplex images in a standardized, consistent, and meaningful fashion, we have developed a quantitative and generalizable way of color-coding the Raman signatures of the nanoprobes after spectral unmixing, where each set of scores is assigned a pseudocolor based on radially projected coordinates on a color wheel. Our method was inspired by RadViz,³⁸ a method for visualizing similarities in samples after dimensional reduction, by projecting them onto a new radial coordinate system where basis vectors (anchors) are placed around the circumference of a circle. To generate a single image summarizing the measurement, the calculated scores for each pixel are projected onto a new coordinate system, where they are assigned a pseudocolor. The new coordinate system is circular, with radial basis vectors that correspond to any number of the reference spectra. In this way, the scores are now reduced to a 2D disk space, which can be superimposed on a color wheel (Figure 2g). Here, we opted for a modified HSV wheel with full saturation; we used a different hue to mark each channel and the intensity to encode the magnitude of the scores. This method can be used to generate single-channel images corresponding to each tag (Figure 2h) each with a unique color assigned to its maximum value and black for the minimum value or a single image with all the channels of interest (Figure 2i). As the coordinate system depends on the superposition of radial vectors, diametrically opposite components add to zero. To differentiate such points from points of actually low signal, we assigned a transparency threshold, based on an absolute summation of the score magnitudes. In this way, the well-plate scan is rendered as shown in Figure 2i, with the point projections shown on the color wheel (Figure 2g), and the corresponding pixels colored accordingly. Channels with irrelevant information, like the plastic plate here or paraffin in histology blocks, can be seamlessly excluded from the visualization. This method goes beyond traditional methods that show only the presence of a nanoprobe or the ratio of two, and it allows quantification and visualization of mixtures.

Imaging the tumor immune microenvironment

We selected a panel of targets that can reveal information about the immune profile, the state of the tumor cells, and their response to therapy, as well as non-specific nanoparticle uptake controls. The panel comprises SERS nanoprobes functionalized with antibodies (SI Figure S3) against the following targets: CD8a (cytotoxic T lymphocytes); CD4 (helper T-cells); GITR and 4–1BB (T-cell activation/costimulatory receptors); CD11b (myeloid phagocytes implicated in immunotherapy and nanoparticle opsonization); NKp46 (natural killer cells); PD-L1 (expressed by tumor cells and implicated in PD-1 mediated immune suppression); and a non-targeted IgG isotype control. All of these markers are actively investigated in various studies of ICB^39–41. SERS nanoprobes (7 targeted + 1 untargeted control) were synthesized with reporter molecules and targeting antibodies as stated in Figure S3 and Table S2. The Raman reporters were selected to yield high intensities when interrogated with the 785-nm laser, and to have sufficiently distinct spectral features to facilitate unmixing. The hydrodynamic diameter was measured via Dynamic Light Scattering (DLS) at 69 nm for the star core, at 146±16.5 nm for the silicated nanoparticles (all configurations), and 166±13.8 nm after antibody functionalization.

A mouse model of cancer immunotherapy was employed to test our imaging approach in a preclinical in vivo scenario. To induce the tumors, BALB/c mice were injected subcutaneously with CT26 (colon) and 4T1 (breast) murine cancer cells on opposite flanks. We selected this model because the BALB/c mice are immunocompetent and the cancer cell lines are widely available and have been used previously to assess the effects of immunotherapy.^42–45 The animals were randomly divided into two groups: the treatment group received ICB treatment (anti-CTLA-4 and anti-PD-1), and the control group received sham treatment (PBS), as described in Figure S4a. We elected to perform the imaging before any therapy-induced reduction in tumor size was observed, to discern the initial effects of the treatment. After only two rounds of treatment, we proceeded to nanoparticle administration and imaging. Tumors from another cohort of mice, subjected to the same conditions (treatment and control) were examined via flow cytometry. The results (SI Figure S4b) suggest that early, yet not robust, changes are occurring in the activation markers (GITR, 4-1BB) and tumor expression of PD-L1.

The nanoprobes were administered to the tumor-bearing mice as a cocktail bolus (SI Figure S4) and allowed to circulate overnight. The mice were subjected to Raman imaging of the tumor regions, simulating a non-invasive imaging scenario through the intact skin and an intraoperative scenario with the skin removed and the tumors imaged in situ (Figure 3a,d). Subsequently, the tumors were excised, fixed with 4% paraformaldehyde, and embedded in paraffin. The paraffin blocks were subjected to Raman imaging at high resolution to produce Raman multiplex images (Figure 3b, e). The SERS signal of the injected nanoprobes remained detectable in the excised tumors at least for several days after fixation. Single-channel images were compared to histologically stained sections from the same blocks (Figure 3c, f). Our first observation was that there was no detectable nanoprobe signal from tumor areas that showed necrosis or hemorrhage. Such areas were identified by the H&E stain and were typically in the central regions of 4T1 tumors (Figure S6). This observation was expected, as the necrotic areas are poorly vascularized preventing nanoprobe delivery via the bloodstream; which produces imaging contrast between viable and necrotic areas of the tumor, irrespective of targeting moiety. This creates a complication when comparing the multiplexed Raman images to IHC stains: IHC stains are applied to thin tissue sections of the tumor, and, as such, are naïve to blood flow, staining necrotic and viable areas alike. This observation is exemplified in SI Figure S5 where IHC for CD11b, for example, stains necrotic areas more strongly than viable positive areas, and for PD-L1 where practically no contrast appears between viable positive and necrotic areas. The physiological differences between the different tumor regions are clearly seen with the H&E stain. Conversely, the images obtained with our method reveal the presence of the target molecules in the viable regions alone, without confounding signals from necrotic areas.

Figure 3. Raman imaging of tumor immune composition with histological validation.

Tumors were scanned in different settings to simulate potential clinical scenarios such as non-invasive and intraoperative imaging (a, d) and histological analysis (b, e). Validation of imaging efficacy was performed by comparing subregions (indicated with the box in b, e) of single-channel Raman images (for the 7 targets + control) to histological stains of the same subregions with related antibodies (c, f). The histological stains were quantified as percentage of stained pixels for clarity. Raman imaging reveals the relative abundance of different markers in the tumors, in agreement with IHC. Scale bars signify 500 µm.

When examining the viable tumor areas, IHC staining corroborated the Raman images well, as the spatial patterns of all the markers are similar between the two techniques. Some spatial discrepancies were expected due to image sampling parameters. For example, in the IHC images, cells that appear in low numbers (such as CD8⁺ cells seen in Figure 3c, f) can only be detected at high magnifications and are not discernible in low magnification images; whereas in the Raman scan, the larger pixels allow for the signal to be detected even though their exact location of the cells within the pixel is not revealed. An additional hurdle to the direct comparison of the two techniques is the innate variability of IHC stains, as the affinity of different antibodies used in the immunostaining process yields different background levels and contrast for every IHC image. Overall good agreement between the Raman images and IHC was noted in areas of viable tumor tissue: both Raman imaging and IHC reveal high expression of some markers (CD11b, PD-L1), moderate (NKp46, CD4), and lower of others (CD8, GITR, 4-1BB). The non-targeted control NP (IgG) provided low signals in areas correlated to the expression of CD11b, which may indicate non-specific uptake by phagocytes, but in low numbers. Although CD8, GITR, and 4-1BB are detected at similar intensity levels as the non-targeted control, their spatial distribution is distinct, and it is likely not attributed to the same (non-specific) uptake mechanism.

Tumor treatment response assessment

The imaged tumors demonstrated heterogeneous morphology; typically, with a necrotic center and proliferative areas in the periphery or the vicinity of blood vessels, while immune cells infiltrated the tumor in different areas and to different extents. A few consistent changes in expression level and localization were observed between the treatment group versus control, which agree with the expected effects of ICB on the tumor microenvironment.^{5–7, 46, 47} For example, moderately increased levels of CD8 and GITR and higher levels of PD-L1 were observed in the periphery of 4T1 tumors of treated animals around areas of necrotic tumor, indicating T-cell infiltration and activation, as well as an increased immune-suppressing response from the tumor, respectively. Conversely, for the control group, these markers are dispersed throughout the viable tumor area and do not point to an active immune response. A similar pattern was noted for CT26 tumors but to a lesser extent. These qualitative differences can be seen in the tumors shown in Figures S6–S7, where tumors from animals that received immunotherapy exhibited, in certain areas, elevated PD-L1 (and moderately increased CD8 and GITR) compared to their PBS-treated counterparts. However, when averaged for the whole tumor without considering their spatial distribution, the unmixed scores of the markers were found to not show any statistically significant differences (Figure S8), similar to flow-cytometry analysis. A spatially aware approach (taking into consideration marker distribution and co-localization) can provide more useful information than simply averaging the signal of the markers in bulk.^{48, 49}

Segmentation and classification of tumor regions

To take into consideration morphological features of the imaged tissue, a suitable method must be employed to segment the tumor in subregions with similar features. We selected to follow a superpixel clustering algorithm (SLIC)⁵⁰, which groups pixels by minimizing the distance in the 8-dimensional space span by the nn-LS scores derived on the markers as well as in the two-dimensional space of the image. The scores on each channel were mean-centered and normalized to unit variance. The spatial coordinates were also scaled, as described in Methods, to control the superpixel compactness. The results of the superpixel segmentation are shown in Figure 4a, b.

Figure 4. Segmentation and classification of tumor regions.

(a) Superpixel clustering was used to group together adjacent pixels with similar nanoprobe uptake profiles into superpixels. (b) The superpixels colored based on their average Raman signal. (c) Example nanoparticle uptake profiles from the 4 superpixels indicated in (b). The bars show the mean signal on each of the markers from the constituent pixels, and the whiskers the standard deviation from the mean. Each superpixel contains about 100 pixels. (d) Classification of the superpixels using ANN. Tumors from treated mice are classified overwhelmingly as “ICB Tx” whereas control tumors as “No Tx”. Necrotic areas are classified as “No tumor”. (e) Mean expression profile from the 4T1 tumor shown in (d) according to class. The bars show the mean profile of the superpixels in each class, and the whiskers the standard deviation. A total of 1,100 superpixels were considered. (f) Same for the CT26 tumor shown in (d). (g) Expression profile of superpixels from (d) based on experimental condition. (h) Overall expression profile based on class for all superpixels representing all of the 20 tumors examined in this study.

After SLIC-based segmentation, the superpixels were used to train classification models based on different ML approaches, to classify the superpixels into classes. The complete analysis procedure from raw Raman spectra to classification is demonstrated in Figure S9, and described in more detail in Methods. Approximately 150 superpixels were generated per tumor (plus the adjacent paraffin region), as seen in Figure 4a. The nn-LS scores were averaged within each superpixel, to provide the mean superpixel nanoprobe uptake image (Figure 4b). Each superpixel is characterized by a nanoparticle uptake profile, four exemplary superpixel uptake profiles are shown in Figure 4c.

The superpixels were then used as training data to calibrate several models based on supervised machine learning (SML). Classes for calibration were assigned as ‘ICB’, ‘No treatment’, and ‘No tumor’ based on whether the superpixel was from a treated animal, a control animal, or an area without a tumor, respectively. We elected that intra-tumoral variation not be taken into consideration for model calibration, meaning that areas within a tumor (either from a treated or control animal) were assigned to the same class, regardless of being viable, necrotic, or distinct in any other way. This decision reduces the bias introduced to the model during training by limiting input from a human user but leads to decreased goodness of fit statistics because varied tumor regions are treated as the same class. Several SML methods were assessed, including artificial neural networks (ANN), support vector machines (SVM), partial least squares discriminant analysis (PLS-DA), and boosted gradient trees (XBG). In all methods cross-validation was performed, to avoid over-fitting of the data, by leaving a set of 4 tumors out of calibration and using it for validation. This was performed for all tumors in the dataset (in five iterations). The classification results are summarized in the SI, Figure S10.

The resulting classification is shown in Figure 4d for one set of tumors, and in Figure S10 for the whole dataset. Superpixels classified as ‘ICB’, ‘No treatment’, or ‘No tumor’ are shown in green, pink, and black, respectively. Almost all models classified areas from naïve and treated CT26 tumors correctly, but indicated that ICB-treated 4T1 tumors contained areas more closely resembling the “No Tx” group. This indicates that 4T1 tumors are more resistant to ICB therapy compared to CT26, consistent with reports in the literature.⁵¹

Overall, the ANN model performed adequately, with class error of cross-validation around 0.10, and root mean square error of cross-validation (RMSECV) around 0.26, while SVM and XBG performed better, with class error of cross-validation 0.13 and 0.07, respectively. As expected, most of the error stems from intra-tumoral variability, as necrotic areas provided negligible signal and were classified as ‘No tumor’. Even so, our methodology was able to classify the tumors appropriately based on the multiplexed SERS signal, and indicate if any (and which) tumor areas are not aligned with the expected outcome. We expect that by fine-tuning the calibration data, or using morphological image processing approaches, better models can be generated, where necrotic areas and areas of effective treatment or emerging resistance can be identified.

Other SML models corroborated the results of the ANN model, indicating that the approach is model-independent. PLS-DA identified the variables with the most descriptive power as PD-L1 and GITR with T² weights of 0.77 and 0.59, respectively, compared to CD4 and CD11b, which were the least important for classification with weights of 0.27 and 0.19, respectively. XBG also identified PD-L1 expression as the single most important predictor for classification, with a gain of 95, compared to 13 for the weakest predictor, 4-1BB.

Conclusions

New techniques with high multiplexing capacity, such as the one we present here, can greatly benefit the field of molecular imaging. Today, the detection and visualization of a single molecular target can help the physician reach a diagnosis and monitor disease progression. By detecting a panel of multiple relevant targets, the physician can gain access to synergistically more information, which also can be used for computer-aided diagnosis via ML.

In this work, we present a comprehensive methodology for multiplexed molecular Raman imaging with SERS nanoprobes, demonstrating the high potential of this modality. Initially, we examined how to discover and select new Raman reporter molecules. For in vivo applications, lasers in the IR are advantageous, and reporters with resonances close to the excitation wavelength are preferable as they provide higher signals, creating a need for good Raman reporters in the IR. Another important consideration is the affinity of the molecule to the gold core, as the plasmonic enhancement rapidly decays with distance. These effects may be predicted from the structure of a reporter molecule by using a chemical similarity metric. This approach can indicate whether a molecule will yield high-intensity SERS signal, as well as whether the spectrum will be distinct from other molecules in a multiplexed imaging panel.

Once we assembled a library of 8 distinct and bright nanoprobes, we were able to perform molecularly targeted multiplexed imaging. The data-rich images acquired, consisting of 8 channels or data, are not easily interpreted visually. It is established that ratiometric approaches can be used to show the relative prevalence of one probe vs. another. We generalized this philosophy to accommodate arbitrary numbers of markers, and provide the relative prevalence of one marker vs. all others based on a circular coordinate system. We then used this approach to assign colors based on these relative abundances. In this way, we were able to generate images that convey information from 8 channels (or more) in a single snapshot.

Our multiplexed imaging approach showcases that the immune microenvironment is not uniform throughout a tumor, with different areas exhibiting different profiles, that change drastically with certain treatments, as shown here for ICB. The local complexity within the tumor can only be effectively observed with a multiplexed imaging approach that highlights the variations in biomarker expression and immune infiltration. By visualizing this variability and identifying regions that do not fit the expected phenotype it may become possible to predict therapeutic efficacy and guide the treatment approach.

The multichannel images (Figures S6–S7) can serve as a first description of the tumor microenvironment, indicating the spatial abundance of immune cells or the expression of PD-L1. However, a single image is only a qualitative visual indicator of the cellular composition of the tumor, and does not convey the complete quantitative information obtained through the scan. The complete multi-channel dataset can be analyzed algorithmically to provide a guiding model encompassing all the relevant information. A pixel-by-pixel approach has several disadvantages, as it is prone to random noise stemming from the distribution of a small number of nanoparticles in the tumor. Subsampling the image and averaging the signal over many pixels can lead to better statistics. However, to preserve physiological features we must segment the multichannel image in a way that reflects the signal of the multiple nanoparticles as well as their relative positions. We did this by using a superpixel algorithm to segment the image, grouping pixels together in clusters, that are spatially cognate, and related in molecular expression. These clusters have improved signal statistics and correspond to physiologically similar regions within the tumor microenvironment, allowing us to classify them algorithmically into responsive and naïve regions, after ICB.

Multiplexed imaging with SERS nanoparticles has now matured, and provides a new way of thinking about molecular imaging. Even though nanoparticle contrast agents are still a long way from clinical use, this versatile method can be applied readily in preclinical studies as well as ex vivo, using excised tissues. Overall, multiplexed molecular imaging, with SERS nanoprobes or another agent with comparable multiplexing capacity, can usher in a new era of high-capacity, computer-aided, and personalized imaging.

Materials and Methods

ChPIR synthesis

All reagents were purchased from commercial sources and, unless otherwise stated, were used without further purification. Reactions run under anhydrous and/or inert conditions are noted in the procedure as such. All other reactions were performed exposed to atmospheric conditions. Tetrahydrofuran (THF) was distilled from a sodium benzophenone ketyl, and other anhydrous solvents were dried over 4 Å molecular sieves before use. Concentration in vacuo was performed on a rotary evaporator. NMR spectra were recorded on either a 300 MHz Gemini or a 500 MHz Varian Inova spectrometer for ¹H NMR spectroscopy and a 300 MHz Gemini spectrometer operating at 75.5 MHz for ¹³C NMR spectroscopy. Residual solvent signal was used as the internal standard. If a mixture of CD₂Cl₂ and CD₃OD was used, the residual peak for CH₂Cl₂ was used as the internal standard. UV/VIS-near-IR spectra were recorded in quartz cuvettes with a 1-cm path length on a Perkin Elmer Lambda 12 spectrophotometer. Melting points were determined with a Büchi capillary melting point apparatus and are uncorrected. ¹³C NMR spectra were not recorded for some pyrylium dyes due to limited solubility in common NMR solvents. Detailed synthesis for each is presented in the SI.

SERS nanoprobe synthesis

SERS nanoparticles were synthesized as reported previously, with a gold nanostar core encapsulated by a silica shell.³³ Briefly, gold nanostars were synthesized by quickly adding 10 ml of 20 mM tetrachloroauric acid (HAuCl₄) solution per 1 L of 60 mM ascorbic acid under vigorous stirring at 4 °C. The resulting nanostars were concentrated 160-fold (to approximately 1 nM) via centrifugation (20 min, 3,220 g) and dialyzed (MWCO 3.5 kDa) against DI water at 4 °C for at least three days. The silica shell was formed via a modified Stöber reaction, during which the Raman reporter molecules (commercial IR or ChPIR dyes) were also introduced to yield the various SERS spectra. In a 500 ml plastic bottle 90 ml isopropanol, 4.5 ml tetraethylorthosilicate, 1.8 ml DI, and the corresponding Raman reporter were mixed. All nanoparticle configurations were synthesized using the same procedure with different amounts of reporter molecule (at a concentration of 20 mM in dimethylformamide), as listed in Table S1. In a 50 ml conical tube, 27 ml ethanol, 10.8 ml nanostars from dialysis, and 1.8 ml ammonium hydroxide were mixed quickly and then poured into the plastic bottle under vigorous stirring. The reaction was allowed to proceed for 15 minutes and was then quenched by adding ethanol to a final volume of 400 ml. The nanoparticles were collected via centrifugation (20 min, 3,220 g), concentrated 250-fold to approximately 3 nM, and washed repeatedly via centrifugation (3 min, 11,000 g) followed by aspiration of the supernatant and redispersion (via ultrasonication) in fresh ethanol. For the antibody conjugation, the silicated nanoparticles were dispersed in a solution of 10% (3-Mercaptopropyl)trimethoxysilane and 5% DI in ethanol, for one hour at room temperature. Antibodies (as listed in Table S2) were reacted with 10× molar excess of a heterobifunctional PEG crosslinker ((poly(ethylene glycol)(N-hydroxysuccinimide 5-pentanoate) ether N′-(3-maleimidopropionyl) aminoethane (CAS: 851040-94-3)) in HEPES buffer (pH 7.1) for 30 min at room temperature. Excess crosslinker was removed by centrifuging the antibody-PEG solution using a centrifugal filter (MWCO 100 kDa). The thiolated nanoparticles were washed (2× in ethanol and 2× in DI) and finally resuspended in HEPES buffer (pH 7.1) and mixed with the antibody-PEG conjugate. The solution was allowed to react overnight at 4°C. The functionalized SERS nanoprobes were washed one last time to remove unreacted antibodies, and the different configurations were mixed prior to injection.

The nanoparticle morphology was tested at every step of the synthesis via TEM. A droplet of the suspension was placed on a carbon film-coated copper grid (300 mesh, Electron Microscopy Sciences) and allowed to air-dry. Images were acquired at 250,000× magnification using a JEOL JEM-1400 (JEOL USA, Inc.) operated at 100 kV. The concentration and size of the nanoparticles were monitored via nanoparticle tracking analysis (NanoSight NS500, Malvern Instruments). DLS was used to measure the size distribution, zeta-potential, of the nanoparticles before and after silication and functionalization, and establish successful conjugation and stability of each sample. For DLS measurements, the nanoparticle samples were diluted 1:1000 or 1:500 in HEPES buffer at pH 7.1.

SERS nanoprobe characterization

Aliquots of each of the synthesized nanoparticle configurations at varying concentrations were scanned in a 1,536-well microplate via Raman imaging. The scan results are shown in Figure 1, where the signal from the nanoparticles is detected using a non-negative Least Squares (nn-LS) fit (see Methods) (Figure 1b). The distinct spectra of each of the configurations at the highest concentration (Figure 1c) were used as references for the mathematical fitting model.

For in vivo administration, eight of the best performing nanoparticle configurations were selected and surface-functionalized with antibodies to yield 7 targeted (and one non-targeted control) SERS nanoprobes. The antibodies used (SI Table S2) were validated independently via flow-cytometry for these targets in live cells. Successful functionalization of the nanoparticles was confirmed by dynamic light scattering (DLS), revealing changes in size and ζ-potential for all nanoparticle-antibody pairs (SI Figure S3).

Cell culture

The murine cancer cell lines CT26 and 4T1 were obtained from ATCC and were maintained in RPMI medium supplemented with 7.5% FCS and penicillin with streptomycin. All of the cell lines were tested and found to be negative for mycoplasma contamination. The cell lines were not reauthenticated after their receipt from the original sources.

Animal model/treatment

All animal studies were conducted under a protocol approved by the Institutional Animal Care and Use Committee of Memorial Sloan Kettering Cancer Center following the NIH guidelines for the care and use of laboratory animals (NIH Publication No. 85-23 Rev. 1985). BALB/c mice (female, 4–6 weeks) were acquired from Jackson Labs. Bilateral flank CT26 and 4T1 tumors were established by subcutaneous implantation of 10⁶ and 3×10⁵ cells, respectively. The mice were randomized into a treatment group (n=5) and a control group (n=5). The treatment group received intraperitoneal injections of anti-CTLA-4 (9H10, 100 µg) and anti-PD-1 (RMP1-14, 250 µg) antibodies in PBS on days 8 and 11 post tumor inoculation, whereas the control group received intraperitoneal injections of pure PBS. For the 8-plex experiment, all animals were administered a bolus of the SERS nanoprobes mixture (200 µl, 3 nM each configuration, in HEPES buffer pH 7.1) on day 13 via tail vein injection. The mice were observed for an hour after nanoparticle injection, noting no abnormal behavior or signs of major discomfort. The animals were sacrificed approximately 12 to 16 hours after injection via carbon dioxide asphyxiation.

Flow cytometry

Each tumor was extracted, mechanically cut, enzymatically lysed and processed for surface labeling. The lysates were stained for CD8, GITR, 4-1BB, CD11b, NKp46, CD4, and PD-L1. Data were acquired using the LSRII Flow cytometer (BD Biosciences) and analyzed using FlowJo software (Treestar).

Raman imaging

Raman imaging was performed on a Renishaw InVia Raman imaging microscope, via the Wire software v4 interface. The mice were imaged twice, once through the intact skin and once after surgical exposure of the tumor area, with alternating order between control and treatment groups. One animal at a time was fastened on a platform and mounted on the microscope’s mechanized stage. Prior to each Raman scan, the focal plane was selected to intersect the tumors using white-light parfocal optics, and a widefield image was acquired. The tumor areas were selected, and scanned with the Raman laser (wavelength 785 nm, objective ×5, power 161 mW, acquisition 0.5 s, resolution 300 µm). The platform was then removed from the microscope, the tumors exposed surgically, and the imaging procedure repeated with the same settings. After imaging each animal, the tumors were excised and processed for histological analysis as described below. Prior to sectioning, the paraffin blocks were subjected to Raman imaging (wavelength 785 nm, objective ×20, power 134 mW, acquisition 100 ms, resolution 40 µm) to acquire the high-resolution images for histological comparison.

Histology

Tumors were excised, fixed in 4% paraformaldehyde in PBS overnight, processed, and embedded in paraffin. Sets of four tumors (from two animals) were placed in each block to represent all experimental conditions (4T1-treated, 4T1-control, CT26-treated, CT26-control). The blocks were trimmed to reveal the four tumors and subjected to Raman imaging as described above. Next, the blocks were sectioned (5 µm thick) and stained with H&E for histological comparison (Figure 3b,e). Additional sections were subjected to immunohistochemical staining against the same targets as the SERS nanoprobes, but using antibodies for different epitopes, independently titered and tested (Table S2). EDTA-based antigen retrieval was used on a Ventana Discovery XT (Roche diagnostics). Optical imaging of the stained sections was performed on a Mirax digital slide scanner (Zeiss). Image quantitation was performed in MATLAB (Mathworks). RGB histological images were converted to the L*a*b* colorspace and segmented to three channels (white, blue, and brown) using a K-nearest neighbor algorithm based on the a* and b* values alone. The image was divided in square subareas and the percentage of pixels classified as ‘brown’ was counted.

Computational – spectral processing

Data analysis was performed using MATLAB (Mathworks) and PLS Toolbox (EigenVector Research Inc.). Raman images were subjected to spectral unmixing using non-negative least squares (nn-LS) regression. Reference spectra for the regression analysis were acquired from suspensions of each nanoparticle configuration in a well plate. The reference spectra underwent baseline subtraction (Whittaker filter, λ=200) and normalization by the maximum value. The spectral data from the image were subjected to pointwise baseline subtraction (Whittaker filter, λ=200), but no normalization was performed. Regression analysis provided a score on each of the N reference spectra, corresponding to the unmixed signal of that spectral component for every point. Specifically, for Figure 1, N=29 (28 reference spectra of the various nanoparticles plus the spectrum of the plastic well-plate); for Figure 2, N=5 (four nanoparticle spectra plus for the plastic well-plate); for Figure 3 and Supplementary Figures S5–9, N=9 (8 nanoprobe spectra+paraffin). In all cases, as the reference spectra were normalized but the sample spectra were not, the scores retain their natural units and can be used as such for quantitative analysis, or normalized for visualization via pseudocolor assignment.

Computational –visualization

For pseudocolor representation, a color wheel was constructed as a 201×201×3 matrix using the HSV color model. Hue was set to change with the azimuth, saturation as an all-ones matrix, and value increasing radially, with 0 at the center. This matrix was then converted to a 201×201×3 RGB matrix to display the color wheel. For each visualization, the N radial reference vectors were placed evenly along the circumference of the wheel (r=100), and their cartesian coordinates (x_i, y_i) were calculated.

For each point spectrum, the color was computed as follows: the coordinates on the disk were calculated as $\left(x_p\cdot y_p\right)=\left(S_1{S}_2\cdots S_N\right)\cdot \left(\begin{array}{ll}{x}_1\hfill & y_1\hfill \\ ⋮\hfill & ⋮\hfill \\ x_N\hfill & y_N\hfill \end{array}\right)$ where S_i are the scores of the spectrum on each reference. Since all scores were constrained between 0 (non-negative fit) and 1 (normalization) the resulting coordinates were within the bounds of the color wheel (x_p, y_p between −100 and 100). The coordinates for each point were then rounded to the nearest integer, and assigned a color corresponding to the position of the point on the wheel. In this way, each point of the Raman scan was converted into a pixel on the pseudocolor image.

Computational – machine learning models

The images derived from nn-LS scores were used as input for the ML models. Initially, each of the 8 data channels was mean-centered and standardized (µ=0 and σ=1). Then, a SLIC algorithm was followed to group individual pixels from the Raman scans into superpixels. The same constraints and superpixel density were used for all images. The algorithm minimized the distance

$$d=a\left({\left(x_{s{p}_n}-x_{p_i}\right)}^2+{\left(y_{s{p}_n}-y_{p_i}\right)}^2\right)+{\sum _{k=1}^N\left(scor{e}_{s{p}_n}^k-scor{e}_{p_i}^k\right)}^2$$

for each pixel i and each superpixel n, with coordinates (x,y) and the corresponding normalized scores on each of the N=8 multiplexed image channels. The compactness parameter a, used for weighing physical distance (in 2D space) against score distance (in 8D space), was determined empirically as a=5×10⁻³ m⁻². The process was repeated for 20 iterations, each time recalculating new coordinates and mean scores for the n superpixels. Each of the resulting superpixels represented the average of similar pixels within its vicinity. To create the dataset, each superpixel was assigned to a class (Tx, C, or none) based on whether the physical location of its center was located within the tumor boundaries.

For the classification models, the physical location of the superpixels was ignored, and only the aggregate nn-LS scores and the assigned class of the superpixel were considered. For cross-validation, a set of 4 tumors was excluded from training and was used as validation. Training with cross-validation was performed 5 times to include all 20 tumors for validation. Several SML methods were tested. After classification, the predicted class (of calibration) with the highest probability (most likely class) was used to set the color of each pixel in the image, based on the superpixel classification. The main results are presented in Figures 4 and S10.

New concepts.

Imaging multiple molecular markers in vivo is crucial for revealing therapeutic mechanisms and effecting personalized medicine. Raman imaging with surface enhanced Raman scattering nanoprobes (SERS NPs) can provide data-rich spectral images of multiple markers, owing to the NPs’ distinct spectral signatures. Here, we present a comprehensive methodology for developing, employing, and visualizing multiple molecularly targeted SERS NPs to map the tumor microenvironment. We demonstrate this technique via the in vivo administration of SERS NPs and imaging in a mouse model of immune checkpoint blockade therapy. The multi-channel images are color-coded to produce molecular maps of the tumor microenvironment, and subsequently analyzed using a super-pixel segmentation approach with supervised machine learning, to classify tumor areas as responsive or nave to immunotherapy. This work provides a generalized guideline for multiplexed SERS imaging, that may be applied to preclinical studies and personalized treatment alike.