Skip to main content
NCBI home page
ACS AuthorChoice logoLink to ACS AuthorChoice
. 2023 Jul 14;95(29):10957–10965. doi: 10.1021/acs.analchem.3c00979

Weakly Supervised Identification and Localization of Drug Fingerprints Based on Label-Free Hyperspectral CARS Microscopy

Jindou Shi †,‡, Kajari Bera †,§, Prabuddha Mukherjee †,§, Aneesh Alex †,∥, Eric J Chaney †,§, Bradley Spencer-Dene ⊥, Jan Majer ⊥, Marina Marjanovic †,§,#, Darold R Spillman Jr †,§, Steve R Hood †,⊥, Stephen A Boppart †,‡,§,#,∇,*
PMCID: PMC10372874  PMID: 37450658

Abstract

graphic file with name ac3c00979_0010.jpg

Understanding drug fingerprints in complex biological samples is essential for the development of a drug. Hyperspectral coherent anti-Stokes Raman scattering (HS-CARS) microscopy, a label-free nondestructive chemical imaging technique, can profile biological samples based on their endogenous vibrational contrast. Here, we propose a deep learning-assisted HS-CARS imaging approach for the investigation of drug fingerprints and their localization at single-cell resolution. To identify and localize drug fingerprints in complex biological systems, an attention-based deep neural network, hyperspectral attention net (HAN), was developed. By formulating the task to a multiple instance learning problem, HAN highlights informative regions through the attention mechanism when being trained on whole-image labels. Using the proposed technique, we investigated the drug fingerprints of a hepatitis B virus therapy in murine liver tissues. With the increase in drug dosage, higher classification accuracy was observed, with an average area under the curve (AUC) of 0.942 for the high-dose group. Besides, highly informative tissue structures predicted by HAN demonstrated a high degree of similarity with the drug localization shown by the in situ hybridization staining results. These results demonstrate the potential of the proposed deep learning-assisted optical imaging technique for the label-free profiling, identification, and localization of drug fingerprints in biological samples, which can be extended to nonperturbative investigations of complex biological systems under various biological conditions.

Introduction

Detailed understanding of drug distribution, pharmacology, and toxicology is essential for drug discovery and development. To exert the desired pharmacology, drug molecules must reach biological targets at the site of action while minimizing the undesired accumulation of the drug or its metabolites in tissues, which can instigate toxicological outcomes.1,2 Currently, techniques that depend on the use of radiolabeled drugs, including quantitative whole-body autoradiography and microautoradiography, have been routinely used to determine the distribution of drugs in various types of biological samples.3,4 However, the use of exogenous radiolabels may lead to a misrepresentation of the drug distribution profile due to off-target labeling, nonspecific binding effects, false signals from radioactive metabolites, and potential perturbation of the drug metabolism.5 Therefore, label-free imaging techniques (e.g., mass spectrometry imaging), which have been used to elucidate the distribution of a drug and its metabolites based on endogenous compounds in the tissues, have gained acceptance in pharmaceutical research and development.6,7 In this study, we investigate the capability of a nonlinear optical imaging technique for noninvasive label-free identification and localization of a drug fingerprint in tissues based on their endogenous biomolecular properties.

Coherent Raman scattering (CRS) imaging is capable of providing noninvasive label-free imaging of chemical compounds in biological samples.8−12 By exploiting vibrations of chemical bonds within molecules, CRS can extract chemical information from a specimen without the need of exogeneous labeling.13−15 Several studies have been reported using CRS for the visualization of drug biodistribution in tissues and cells.16−19 Among them, hyperspectral coherent anti-Stokes Raman scattering (HS-CARS) microscopy, where spatially resolved images are acquired at multiple vibrational frequencies, enables the visualization of the spatial distribution of various biomolecular components in biological samples.20,21 In this study, we explored the potential of HS-CARS for the label-free identification and localization of a drug based on endogenous molecular vibrational optical signatures at single-cell resolution (Figure 1). Here, the drug fingerprint is defined as any molecular changes due to the presence of the drug, which may include the accumulation of the drug molecule and associated metabolites. These effects do not imply pharmacological action but rather an interaction of the molecule with the surrounding tissue.

Figure 1.

Figure 1

Open in a new tab

Experimental design and workflow. (a) Demonstrates the workflow of the investigation of drug treatment effect using HS-CARS microscopy and weakly supervised deep learning. An HS-CARS image of a murine liver sample is shown in (b). (c) Proposed weakly supervised deep learning model is trained on whole-image-level labels. (d) After training, the model is able to highlight informative regions that are highly relevant to the whole-image classification task. PMT, photomultiplier tube.

For studies using hyperspectral imaging techniques to investigate the drug (or its metabolites) distribution, the common analytical approach requires prior knowledge of either the spatial distribution17,22 or the spectral information16,23 of the targeted compound, that is, comparing the spectral profiles of selected regions of interest or visualizing the spatial distribution of the target molecule with its specific spectral signature. However, these approaches may unavoidably suffer from human bias and potentially miss information associated with the drug distribution. For instance, approaches that depend on prior spatial knowledge would only focus on human-defined regions in the tissue, while the remaining areas are assumed to show no evidence of drug treatment. Meanwhile, methods that depend on prior spectral knowledge may only focus on certain molecules (e.g., drug itself) while other molecules (e.g., drug metabolites and endogenous metabolites) could be missed. An effective and bias-free analytical approach is therefore needed for the investigation of a drug fingerprint and its localization in a heterogeneous chemical environment using HS-CARS microscopy.

With the growing applications of deep learning (DL) techniques in spectroscopic analysis, deep neural networks (DNNs) have achieved substantial success in various data processing and analysis tasks, including signal enhancement, segmentation, regression, and classification.24−26 Being able to extract complex spectral patterns directly from unprocessed spectral data, DNNs have been reported to outperform conventional chemometric analysis pipelines, which may unavoidably suffer from human bias caused by the selection of data preprocessing and spectrum unmixing methods.25,27,28 So far, the applications of DNNs in CRS image analysis have mainly focused on fully supervised tasks, where fine-grained annotations (e.g., pixel-level labels) are required for model development.29 Such a requirement can hardly be satisfied for the bias-free analysis of hyperspectral images when both spatial and spectral prior knowledge are unavailable. Instead, coarse-grained annotations, such as labels for the whole images, are often available more readily. To exploit such weak supervision, multiple instance learning (MIL) techniques have been proposed to tackle inexact supervision problems.30−32 In MIL, multiple instances are grouped into bags, while only bag-level annotations are available. By leveraging the coarse-grained label information, MIL models aim to make predictions on bags as well as instances.33

Therefore, a weakly supervised end-to-end DNN, the hyperspectral attention net (HAN), was developed (Figure 2). Here, each hyperspectral image is defined as a MIL bag, while its spectra are treated as MIL instances. Without requiring prior spatial and spectral knowledge, the HAN was trained on whole-image-level labels (i.e., positive for drug-treated images, negative for control images). During inferencing, the HAN can generate whole-image-level classification predictions as well as inform regions in the image that are highly relevant to prediction.

Figure 2.

Figure 2

Open in a new tab

HAN model architecture. The HAN consists of a wavenumber encoder, a one dimensional convolutional neural network (CNN), an attention module, and a multilayer perceptron (MLP)-based classifier. The input and output of each component are shown as 3D blocks, of which the dimensions are noted (H: image height, W: image width, L: spectrum length, N: number of spectra, F: length of the feature representation).

We demonstrated the usage of the proposed DL-assisted HS-CARS imaging technique for the investigation of antisense oligonucleotide (ASO) drug fingerprints in murine tissues. As an emerging area in pharmacological research, ASOs target the disease source at the RNA level.34,35 Based on the proposed imaging and analysis approaches, we studied the drug fingerprint of an ASO drug, namely, bepirovirsen (“ASO” for brevity). The drug is currently under investigation by GSK as a novel therapeutic agent for hepatitis B virus infection.36 In this study, we investigated the drug fingerprints in liver, which is the targeted delivery site of the ASO drug.

The proposed DL-assisted label-free HS-CARS imaging technique offers a new way to investigate drug fingerprints in biological samples, which provides researchers with nonperturbative chemical information about drug fingerprints in biological samples. Without special requirements on tissue selection and preparation, the usage of the proposed DL-assisted HS-CARS imaging technique can be extended to other investigations of various types of drugs and biological conditions in complex biological systems.

Methods

Animal Experiments and Tissue Preparation

The animals used in this study were female CByB6F1 (wild type) mice, which were injected with the ASO drug (i.e., bepirovirsen) or vehicle. Bepirovirsen is a chimeric 2′-O-(2-methoxyethyl)-modified phosphorothioate antisense oligonucleotide (2′-MOE ASO). During this study, bepirovirsen was given to mice (N = 4 per dose group) at 0 (control), 4 (low-dose), 12 (medium-dose), or 40 (high-dose) mg/kg/dose via subcutaneous injection on days 1, 3, 5, and 7, and then once weekly thereafter through day 91. From all animals, whole livers were collected within 36 h post dose on day 91. The study was conducted in accordance with the GSK Policy on the Care, Welfare and Treatment of Laboratory Animals and was reviewed by the Institutional Animal Care and Use Committee at GSK. Further information on tissue processing procedure can be found in Supplementary Note 1.

HS-CARS Imaging System

In this study, the murine tissue samples were imaged by a lab-built HS-CARS microscope. The HS-CARS microscope used a spectral focusing approach, where both pump and Stokes pulses were linearly chirped, and the relative timing between the two chirped pulses was modulated to collect the hyperspectral images.37,38 HS-CARS images were collected from tissue sections in a transmission configuration (Figure S1). The system collected data across a spectral range of 2700–3100 cm–1 based on the characterization of the spectral features of the drug (i.e., bepirovirsen) using the spontaneous Raman technique (Figure S2). The spectral information captured in this range comprised of the characteristic -CH2 and -CH3 vibrational peaks, which can be correlated with the lipid and protein components in the tissue, respectively. The imaging system setup is further described in Supplementary Note 2.

Synthetic HS-CARS Datasets

Synthetic HS-CARS images with tunable spectral differences were created to evaluate the classification and localization capability of the proposed DL model. The synthetic data generation program consists of the simulation of spectral and spatial components of hyperspectral images (Figure S3). Two classes of objects (i.e., type A and type B) with different spectral signatures were randomly placed in images without overlap. By gradually increasing the spectral differences between type A and type B objects (at a factor of 2), we evaluated the model performance on differentiating positive and negative images, as well as localizing the discriminative objects (type B objects). A detailed description of the HS-CARS data simulation program is in Supplementary Note 3.

HAN Model Architecture

In this study, the identification and localization of drug fingerprints were formalized to an MIL task. Each HS-CARS image was described as a bag X = {x1, ..., xN}, where the element xi, i = 1, ..., N represents the spectrum input at the i-th pixel of the image (i.e., an instance in MIL). Each bag was assigned a label Y.

The proposed HAN model is an end-to-end attention-based DNN. The HAN consists of four main components (Figure 2).

To encode the precise spectral pattern location information, the wavenumber encoder in the HAN pairs a spectrum vector si with a wavenumber vector wi (Figure S4a). The element wij is the normalized wavenumber (Raman shift) corresponding to sij. Paired si and wi form one instance Inline graphic:

graphic file with name ac3c00979_m002.jpg 1

where L is the number of elements in the spectrum vector. Min-max scaling was applied to the normalization of wi.

A one-dimensional convolutional neural network (CNN) was used to extract spectral features from the hyperspectral image (Figure S4b). The CNN-based spectrum feature extractor can be described as a function Inline graphic, that maps the input bag X (with N instances) to the corresponding instance-level feature vectors {hi} (Inline graphic). The number of convolution modules and parameters of each layer were determined via hyperparameter tuning.

The bag-level feature representation Inline graphic was obtained by integrating instance-level feature vectors {hi} via the attention mechanism (Figure S4c). A parametric gated attention module was adopted in the HAN.39 The weight parameters of the fully connected layers in the module can be denoted as Inline graphic, Inline graphic, and Inline graphic (E is the number of elements in the hidden representation vector, bias terms are omitted for simplicity). The attention score ai of instance xi was then calculated by

graphic file with name ac3c00979_m009.jpg 2

where ⊙ is the element-wise product, tanh is the hyperbolic tangent function, and sigm is the sigmoid function. Based on attention scores and instance-level feature representations, the bag-level feature representation hbag was generated via attention pooling:

graphic file with name ac3c00979_m010.jpg 3

Intuitively, the bag-level feature representation was the weighted average of instance-level representations, of which the weights were determined by the trainable attention module. Instances with higher weights (i.e., attention scores) were considered to be more relevant for the bag-level classification task.

Bag-level predictions Inline graphic (C is the number of classes) were generated by a multilayer perceptron (MLP)-based classifier (Figure S4d), of which weight parameters and bias terms are denoted as Wc and bc, respectively:

graphic file with name ac3c00979_m012.jpg 4

In our study, the whole-image-level drug treatment prediction was a binary classification task (C = 2). During training, given a bag X, the HAN generated the bag-level prediction Ŷ, which was then used for loss calculation. In the inference mode, in addition to bag-level predictions, HAN produced attention heatmaps, which informed the spatial region of interest (ROI) for the classification task. Notably, due to the end-to-end nature of the HAN, the pixel-level spectrum feature extractor was optimized simultaneously with the whole-image-level classifier using backpropagation and thus did not require separate pretraining.

In this study, the HAN model had 41,059 trainable parameters. The model development procedure, training and inferencing speed, as well as the hardware and software used in this study are further demonstrated in Supplementary Note 4. The source code of the HAN is available at https://github.com/Biophotonics-COMI/HyperspectralAttentionNet.

Attention-Weighted Class Activation Maps

In addition to spatial ROIs highlighted by the attention module, the highly relevant spectral regions (i.e., spectral ROIs) were informed by the class activation maps (CAMs), which exploited the inherent object localization ability of CNNs.40 By taking the attention scores into consideration, the attention-weighted CAM (A-CAM) technique was developed, which informs the importance of different spectral locations for the classification of an image to a given class. The CAM and A-CAM are further illustrated in Supplementary Note 5.

Results

HS-CARS ASO Murine Liver Dataset

In total, 172 HS-CARS murine liver images (each image has 400 × 400 × 60 pixels) were collected from four dose groups. Each HS-CARS image covers a field-of-view (FOV) of 100 × 100 μm2. Each pixel in the hyperspectral image was a CARS spectrum with 60 values, which corresponded to the 2700–3100 cm–1 vibrational range. From the HS-CARS images and adjacent hematoxylin and eosin (H&E) histology slides of the murine liver tissue, we identified the nuclei to be circular in shape with a relatively darker contrast, the sinusoids to be the dark elongated areas, and the lipid droplets as having a very strong signal, represented as bright point-like objects (Figure S5). To augment the dataset for model development, we implemented a sampling procedure on the original hyperspectral image. Through this procedure, 16 hyperspectral images were derived from the original image, each with dimensions of 100 × 100 × 60 pixels. This sampling procedure involved dividing the original image into nonoverlapping regions of size 4 × 4 pixels. From each of these regions, one pixel was selected for one sampled image. This sampling strategy ensured that the resulting images covered a similar FOV as the original image while substantially increasing the dataset size by a factor of 16. It is crucial to highlight that during the partitioning of the dataset, the sampled images originating from the same original hyperspectral image were exclusively allocated to either the training set, validation set, or test set, maintaining the integrity of the data segregation. A summary of the murine liver dataset can be found in Table S1.

Model Evaluation on Synthetic HS-CARS Images

To examine the classification and localization capabilities of the proposed HAN models, we first evaluated the model performance using synthetic HS-CARS datasets with known and tunable spectral differences. Three types of CARS spectral differences were generated (i.e., peak height, peak location, and peak width), which simulate the changes in the number of molecular oscillators, energy, and frequency of molecular vibration, respectively. In each classification experiment, HAN models were trained to differentiate positive hyperspectral images (with discriminative objects of which the spectra were manipulated) from negative ones. The evaluation results of the whole-image classification and localization predictions are reported in Figures 3 and 4, respectively.

Figure 3.

Figure 3

Open in a new tab

Classification evaluation on synthetic HS-CARS datasets. Three types of spectral differences, including changes in peak height (a, b), peak location (c, d), and peak width (e, f), were simulated. In (a, c, e), solid lines represent individual Lorentzian components in the synthetic CARS spectra, with gray lines being the unchanged components and rainbow-colored lines (dark red to purple) showing the changes of the tunable component. The final spectra (before introducing noise) are shown as dashed lines. The classification results of each experiment are shown in (b, d, f), where box-and-whisker plots show the distribution of area under the curve (AUC) scores. Each black dot in the plots represents the result of one cross-validation fold (4 folds in total) in every experiment.

Figure 4.

Figure 4

Open in a new tab

Localization predictions of discriminative objects with increased peak height in synthetic HS-CARS images. The results are shown in columns a–d, where the peak height of the tunable Lorentzian component was increased by 2, 8, 32, and 128%, respectively. The noise-contaminated spectra of pixels in background (BG), nondiscriminative (A), and discriminative (B) objects are shown in the top row, with the horizontal axis representing Raman shift (cm–1) and the vertical axis representing the CARS intensity (a.u.). The frames at the peak location of the tunable component in the positive images are shown in the second row. The ground truth (GT) masks in the third row show the locations of discriminative objects, while the min-max normalized attention heatmaps produced by the HAN models are shown in the last row.

Under the simulated spectral noise, an average area under the curve (AUC) of 1.0 was achieved in 4-fold cross validation when the peak height increased by 8% or more (Figure 3b). In addition, the HAN was able to accurately identify peak shift as little as 0.4 cm–1 (Figure 3d) and peak width increases greater or equal to 0.8 cm–1 (Figure 3f) in the synthetic HS-CARS images.

The localization predictions of spatial ROIs were evaluated based on the ground truth (GT) masks of discriminative objects. As shown in Figure 4, discriminative objects were highlighted in attention heatmaps when the peak height increases by 8% or more. With the increase in spectral differences between discriminative and nondiscriminative objects, the difference in attention scores between discriminative objects and the rest of the image became more obvious. However, when the HAN achieved low classification accuracy, attention heatmaps may fail to highlight the precise location of discriminative objects (Figure 4a). Similar trends was observed for simulated changes in the peak location and peak width (Figure S6). These results demonstrate the capability of the HAN in identifying and localizing subtle spectral differences when being trained on whole-image annotations.

Classification Results on ASO Murine Liver Dataset

When applied to the investigation of the ASO drug fingerprint in murine liver tissues, the HAN models were trained to differentiate drug-treated group images from untreated control group images. The ASO murine liver dataset was randomly partitioned into a training set, a validation set, and a test set for 10-fold Monte Carlo cross-validation. The classification evaluation results are reported in Figure 5. Among all dose levels, the highest AUC was achieved when the model was trained to classify images from the high-dose group and the control group, with an average AUC of 0.942 (Figure 5a). The AUC score declined with the decrease in drug dosage, indicating the increased similarity in the spectral information between lower dose-level drug-treated samples and control samples (Figure 5b,c). A similar trend of the dose-dependent classification performance was observed when using HAN models without wavenumber encoders (Figure 5a–c). Notably, when the wavenumber encoder was removed, the classification performance degraded for high-dose and medium-dose groups, where the decreased mean value and increased standard derivation of AUC scores were shown. Other classification evaluation metrics (i.e., accuracy, precision, recall, and F1-score) are reported in Figure 5d, where a similar trend of classification performance among dose levels was observed. Based on the drug treatment classification results and the observations from synthetic dataset experiments, the localization and spectral profiles of ASO fingerprints were further investigated based on the high-dose drug-treated group, which demonstrated the best overall classification performance.

Figure 5.

Figure 5

Open in a new tab

Classification evaluation on the ASO murine liver dataset. (a–c) ROC curves for each dose group. The average ROC curves for the 10-fold cross validation are shown as solid lines, while the interval regions show the standard derivation. Additional classification metrics, including accuracy, precision, recall, and F1 score, are shown in (d), with the error bar representing the standard derivation.

Attention-Informed Drug Fingerprint Localization

The predictive importance of various hepatic structures was informed by the attention module in HANs for the drug treatment classification (Figure 6). When examining the attention heatmaps and their correlated HS-CARS images, we observed that the highly attended regions in drug-treated murine liver images were mainly located in hepatocyte nuclei (the dark circular area in the hepatocytes) and sinusoidal areas (epithelial and Kupffer cells surrounding hepatocytes), indicating that these regions are more likely to show drug fingerprints. Lipid droplets (bright circular point-like objects in the HS-CARS image) and the cytoplasm of hepatocytes exhibited relatively low attention scores, suggesting that these regions were less informative for the ASO treatment classification.

Figure 6.

Figure 6

Open in a new tab

Highly informative hepatic regions predicted by the HAN. (a) H&E histology image of a murine liver sample from the high-dose group. (b) An ISH image showing the distribution of the ASO drug in the high-dose murine liver sample. (c) HS-CARS images (sum of all spectral frames) from six randomly sampled FOVs of high-dose treatment samples; their corresponding attention heatmaps are shown in the top and bottom rows. The HS-CARS images are overlaid with their attention heatmaps in the middle row.

To validate the drug fingerprint localization predictions, the location of the ASO molecule inside the drug-treated murine liver samples was visualized using the in situ hybridization (ISH) technique (Figure 6b). When comparing the attention heatmaps and ISH ASO staining results, a high degree of similarity can be observed, where the heaviest ISH ASO staining was located in the nuclei of hepatocytes as well as in nonparenchymal cells (i.e., sinusoidal endothelial cells and Kupffer cells).

CARS Spectral Signatures of the ASO Drug Fingerprint

To facilitate the understanding of the CARS spectral signatures of drug fingerprints, two methods, namely, the spectral profile visualization and the CAM-based method, were adopted to gain insights into model predictions.

The CARS spectral profiles of regions with high and low attention scores were visualized and compared. The scores in attention heatmaps were quantized to five levels: low, low-medium, medium, medium-high, and high. Each level consists of 20% pixels from all images. The spectral profiles of different attention levels are shown in Figure 7. It is observed that highly attended areas have a larger AUC in the 2900–3050 cm–1 range of the normalized spectral profiles, indicating that the protein peak broadens in the highly attended regions. Such CARS spectral patterns became more obvious with the increase in attention scores.

Figure 7.

Figure 7

Open in a new tab

CARS spectral profiles of regions at different attention levels. (a) By quantizing the attention scores to five levels, the attention heatmaps were divided into five attention-ranked regions (i.e., low, low-medium, medium, medium-high, and high). The normalized HS-CARS spectral profiles of regions at different attention levels are shown in (b).

In addition, the location of drug fingerprint-related spectral features in the CARS spectra (spectral ROI) can be informed by CAMs and A-CAMs. Since every spectrum in the hyperspectral image has a one-dimensional CAM, CAMs can then be visualized at their corresponding location in the image, resulting in a three-dimensional class activation array (Figure 8a). By taking the attention scores into account, the three-dimensional attention-weighted class activation array of the same image is visualized in Figure 8b, which informs the actual contribution of spatial and spectral regions in the hyperspectral image.

Figure 8.

Figure 8

Open in a new tab

CARS spectral ROIs informed by class activation maps. (a) three-dimensional class activation array of a predicted positive HS-CARS image from the high-dose group. When taking the attention scores into account, the resulting attention-weighted class activation array is shown in (b). (c) Attention scores (left) and class activation maps (right) of 10,000 randomly sampled spectra from HS-CARS images in the high-dose group. The class activation maps are sorted by their corresponding attention scores. (d) Four CARS spectra at different attention levels [(1)–(4)] are visualized on top of their class activation maps.

To further investigate the spectral ROIs informed by the HAN, CAMs of 10,000 randomly sampled spectra from high-dose predicted positive HS-CARS images were sorted based on their corresponding attention scores and visualized in Figure 8c. The highly attended spectra (top rows in Figure 8c) have broader high-activation regions in the protein region compared to spectra with lower attention values, whereas the low-activation regions (colored in blue) tend to expand with the decrease of attention score. CARS spectra with high-to-low attention scores are visualized on top of their CAMs in Figure 8d. It can be observed that high CAM values are predominately present in the protein region of CARS spectra (2900–3050 cm–1), indicating that the ASO treatment-related discriminative spectral patterns are mainly shown in the protein region, while the lipid region (from 2750 to 2900 cm–1) exhibits lower CAM values. The spectra of lipid droplets, which have constantly lowest attention values, have low activation values in both lipid and protein regions. These findings correspond to the observations gained from the first method, which highlight that the ASO drug fingerprint may be exhibited in the protein region of the CARS spectra.

Discussion

In this study, we present a DL-assisted label-free optical imaging technique to identify and localize a drug fingerprint in complex biological systems based on endogenous optical properties. The HS-CARS imaging technique bridges chemical profiling with spatially resolved imaging with submicron spatial resolutions, enabling the profiling of endogenous chemical composition of cellular and subcellular components in various biological systems. Such property makes HS-CARS a promising technique for the label-free and nondestructive investigation of drug fingerprint in complex biological systems, where labeling may significantly change the pharmacological properties of the drug or the biological processes of the cell and tissue microenvironment in which the drug resides.

Due to the chemical complexity of biological systems, identifying molecules of interest in a heterogeneous chemical environment remains challenging in biomedical and pharmacological research.41 We proposed a DL method as a bias-free analytical approach for the weakly supervised identification and localization of molecules of interest in hyperspectral images. Compared to conventional hyperspectral chemometric approaches, the HAN has the following advantages. First, the HAN was trained in a weakly supervised manner. Since no manual annotation is required (i.e., whole-image-labels can be acquired from metadata), the HAN can explore the rich biomolecular information in the HS-CARS images and extract relevant spectral patterns without human interference. Second, the HAN model, equipped with a CNN-based spectral feature extractor, can detect subtle spectral differences from the raw hyperspectral images. Studies have demonstrated the superior capability of CNNs in identifying complex spectral patterns.25,28 In addition, CNNs can act directly on the raw spectra by mimicking chemometric preprocessing methods, which greatly simplifies the analysis pipeline and further reduces human bias in the choice of preprocessing methods.42,43 However, due to the spatial invariance property, conventional CNNs are not able to encode the precise location of the pattern in the spectrum, which is essential for chemometrics. We addressed this challenge by adding a wavenumber encoder to the HAN, leading to improved classification performance. Third, due to the end-to-end design of the HAN, all learnable components (i.e., spectral feature extractor, attention module, and classifier) can be optimized simultaneously with the whole-image-level classification task. In addition to simplified training procedure, the end-to-end nature also encourages the HAN to learn spectral patterns that are directly related to the whole-image-level classification task. The performance comparison between the HAN and conventional clustering and classification methods is further illustrated in Supplementary Note 6. Combining those advantages, the HAN can be an effective and bias-free tool to identify and localize molecules of interest in heterogeneous hyperspectral images.

By leveraging the proposed imaging and analytical approaches, we investigated the fingerprints of bepirovirsen in murine liver samples. Attention heatmaps highlighted several cellular and subcellular structures in murine liver, including hepatocyte nuclei and sinusoidal regions. Related, the significant accumulation of the ASO drug in Kupffer cells, sinusoidal endothelia, and blood has been reported by previous studies.44 In addition, the observation that the nuclei of hepatocytes have high attention scores correlates with the fact that they are the main site of action of the ASO drug.45 Moreover, the similarity in spatial distribution revealed by ISH and HS-CARS imaging approach indicates the promising potential of the proposed DL-assisted label-free optical imaging technique for the investigation of drug fingerprint and its localization in complex biological systems. Notably, this approach focuses on the exploration of drug fingerprints inside tissues, which encompass any molecular changes resulting from drug presence. Consequently, the attention heatmaps may predominantly highlight the changes in the molecular composition of the tissues rather than directly pinpointing the drugs themselves.

One limitation of our CARS microscopy technique is the spectroscopic ability to distinguish closely lying resonances, known as the spectral resolution. The spectral resolution of our HS-CARS system was measured to be ∼20 cm–1. Improvements in spectral resolution may further improve the classification and localization performance of the DL model. In addition, advancements in HS-CARS technology that enable faster image acquisition from more sample locations are needed to address the throughput limitation and to facilitate the adoption of this DL-assisted imaging technology for drug biodistribution studies. Besides, due to the resource constraints, we demonstrated the capability of the proposed imaging and analysis approaches on one ASO drug (i.e., bepirovirsen). However, it is imperative that future research endeavors encompass a wider spectrum of drugs to fully comprehend the applicability and potential benefits of these approaches.

In conclusion, this study offers a DL-assisted optical imaging approach to investigate biomolecular changes induced by the drug treatment. By generating high-resolution spatially resolved images, the proposed label-free optical imaging approach can provide complementary information to the traditional methods for drug distribution investigations. Although we demonstrated the usage of HS-CARS and HAN for an ASO drug in murine liver, we believe that the proposed approach can be utilized for the investigation of other types of drugs in different biological systems. In addition, the usage of the HAN model can be extended to other hyperspectral imaging modalities, including stimulated Raman scattering, mass spectrometry imaging, and medical hyperspectral imaging. In general, we anticipate that this DL-assisted label-free high-resolution optical imaging approach could be used as a new tool for the nonperturbative investigation of various types of biological conditions in complex biological systems.

Acknowledgments

The authors would like to thank Michael Fazio and Punit Seth (Ionis Pharmaceuticals, Inc.) for supplying ASO samples used in this study. This research was supported, in part, by GSK through the GSK Center for Optical Molecular Imaging at the Beckman Institute for Advanced Science and Technology on the campus of the University of Illinois at Urbana-Champaign.

Supporting Information Available

The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acs.analchem.3c00979.

  • Additional experimental details for murine tissue preparation; ISH staining procedure; imaging system setup; dataset generation; DL model development; CAM and A-CAM; methods comparison; Supplementary tables and figures (PDF)

Author Contributions

S.A.B. and S.R.H. conceptualized the work. J.S. developed the code used in this work and analyzed the data. K.B. and P.M. acquired data. E.J.C. prepared biological samples. B.S.D. conducted the ISH staining. J.M. contributed to the data analysis. The manuscript was written through contributions of all authors.

The authors declare the following competing financial interest(s): The GSK Center for Optical Molecular Imaging, its personnel, and the projects that are pursued are supported financially through an academic-industry partnership grant between the University of Illinois at Urbana-Champaign and GSK. A.A., B.S.D., J.M., and S.R.H. are employees and shareholders of GSK. J.S., K.B., P.M., E.J.C., M.M., D.R.S., and S.A.B. declare no potential conflict of interest.

Supplementary Material

ac3c00979_si_001.pdf (2.5MB, pdf)

References

  1. Lanao J.; Fraile M. Drug tissue distribution: study methods and therapeutic implications. Curr. Pharm. Des. 2005, 11, 3829–3845. 10.2174/138161205774580679. [DOI] [PubMed] [Google Scholar]
  2. Mouton J. W.; Theuretzbacher U.; Craig W. A.; Tulkens P. M.; Derendorf H.; Cars O. Tissue concentrations: do we ever learn?. J. Antimicrob. Chemother. 2008, 61, 235–237. 10.1093/jac/dkm476. [DOI] [PubMed] [Google Scholar]
  3. Solon E. G. Use of radioactive compounds and autoradiography to determine drug tissue distribution. Chem. Res. Toxicol. 2012, 25, 543–555. 10.1021/tx200509f. [DOI] [PubMed] [Google Scholar]
  4. Solon E. G. Autoradiography techniques and quantification of drug distribution. Cell Tissue Res. 2015, 360, 87–107. 10.1007/s00441-014-2093-4. [DOI] [PubMed] [Google Scholar]
  5. Swales J. G.; Hamm G.; Clench M. R.; Goodwin R. J. Mass spectrometry imaging and its application in pharmaceutical research and development: A concise review. Int. J. Mass Spectrom. 2019, 437, 99–112. 10.1016/j.ijms.2018.02.007. [DOI] [Google Scholar]
  6. Prideaux B.; Stoeckli M. Mass spectrometry imaging for drug distribution studies. J. Proteomics 2012, 75, 4999–5013. 10.1016/j.jprot.2012.07.028. [DOI] [PubMed] [Google Scholar]
  7. Solon E. G.; Schweitzer A.; Stoeckli M.; Prideaux B. Autoradiography, MALDI-MS, and SIMS-MS imaging in pharmaceutical discovery and development. AAPS J. 2010, 12, 11–26. 10.1208/s12248-009-9158-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Zhang C.; Zhang D.; Cheng J.-X. Coherent Raman scattering microscopy in biology and medicine. Annu. Rev. Biomed. Eng. 2015, 17, 415. 10.1146/annurev-bioeng-071114-040554. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Pope I.; Langbein W.; Borri P.; Watson P. Live cell imaging with chemical specificity using dual frequency CARS microscopy. Methods Enzymol. 2012, 504, 273–291. 10.1016/B978-0-12-391857-4.00014-8. [DOI] [PubMed] [Google Scholar]
  10. Zumbusch A.; Holtom G. R.; Xie X. S. Three-dimensional vibrational imaging by coherent anti-Stokes Raman scattering. Phys. Rev. Lett. 1999, 82, 4142. 10.1103/PhysRevLett.82.4142. [DOI] [Google Scholar]
  11. Cheng J.-X.; Xie X. S. Coherent anti-Stokes Raman scattering microscopy: instrumentation, theory, and applications. J. Phys. Chem. B 2004, 108, 827–840. 10.1021/jp035693v. [DOI] [Google Scholar]
  12. Sun Y.; Chen E. W.; Thomas J.; Liu Y.; Tu H.; Boppart S. A. K-means clustering of coherent Raman spectra from extracellular vesicles visualized by label-free multiphoton imaging. Opt. Lett. 2020, 45, 3613–3616. 10.1364/OL.395838. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Marks D. L.; Vinegoni C.; Bredfeldt J. S.; Boppart S. A. Interferometric differentiation between resonant coherent anti-Stokes Raman scattering and nonresonant four-wave-mixing processes. Appl. Phys. Lett. 2004, 85, 5787–5789. 10.1063/1.1829162. [DOI] [Google Scholar]
  14. Tu H.; Boppart S. A. Coherent anti-Stokes Raman scattering microscopy: overcoming technical barriers for clinical translation. J. Biophotonics 2014, 7, 9–22. 10.1002/jbio.201300031. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Jones G. W.; Marks D. L.; Vinegoni C.; Boppart S. A. High-spectral-resolution coherent anti-Stokes Raman scattering with interferometrically detected broadband chirped pulses. Opt. Lett. 2006, 31, 1543–1545. 10.1364/OL.31.001543. [DOI] [PubMed] [Google Scholar]
  16. Sepp K.; Lee M.; Bluntzer M. T.; Helgason G. V.; Hulme A. N.; Brunton V. G. Utilizing stimulated Raman scattering microscopy to study intracellular distribution of label-free ponatinib in live cells. J. Med. Chem. 2020, 63, 2028–2034. 10.1021/acs.jmedchem.9b01546. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Fu D.; Zhou J.; Zhu W. S.; Manley P. W.; Wang Y. K.; Hood T.; Wylie A.; Xie X. S. Imaging the intracellular distribution of tyrosine kinase inhibitors in living cells with quantitative hyperspectral stimulated Raman scattering. Nat. Chem. 2014, 6, 614–622. 10.1038/nchem.1961. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. You S.; Liu Y.; Arp Z. A.; Zhao Y.; Chaney E. J.; Marjanovic M.; Boppart S. A. Intracellular imaging of docosanol in living cells by coherent anti-Stokes Raman scattering microscopy. J. Biomed. Opt. 2017, 22, 070502 10.1117/1.JBO.22.7.070502. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Saar B. G.; Contreras-Rojas L. R.; Xie X. S.; Guy R. H. Imaging drug delivery to skin with stimulated Raman scattering microscopy. Mol. Pharmaceutics 2011, 8, 969–975. 10.1021/mp200122w. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Wurpel G. W.; Schins J. M.; Müller M. Chemical specificity in three-dimensional imaging with multiplex coherent anti-Stokes Raman scattering microscopy. Opt. Lett. 2002, 27, 1093–1095. 10.1364/OL.27.001093. [DOI] [PubMed] [Google Scholar]
  21. Pope I.; Langbein W.; Watson P.; Borri P. Simultaneous hyperspectral differential-CARS, TPF and SHG microscopy with a single 5 fs Ti: Sa laser. Opt. Express 2013, 21, 7096–7106. 10.1364/OE.21.007096. [DOI] [PubMed] [Google Scholar]
  22. Yue S.; Cárdenas-Mora J. M.; Chaboub L. S.; Lelievre S. A.; Cheng J.-X. Label-free analysis of breast tissue polarity by Raman imaging of lipid phase. Biophys. J. 2012, 102, 1215–1223. 10.1016/j.bpj.2012.01.023. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Seidel J.; Miao Y.; Porterfield W.; Cai W.; Zhu X.; Kim S.-J.; Hu F.; Bhattarai-Kline S.; Min W.; Zhang W. Structure–activity–distribution relationship study of anti-cancer antimycin-type depsipeptides. Chem. Commun. 2019, 55, 9379–9382. 10.1039/C9CC03051D. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Manifold B.; Men S.; Hu R.; Fu D. A versatile deep learning architecture for classification and label-free prediction of hyperspectral images. Nat. Mach. Intell. 2021, 3, 306–315. 10.1038/s42256-021-00309-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Manifold B.; Thomas E.; Francis A. T.; Hill A. H.; Fu D. Denoising of stimulated Raman scattering microscopy images via deep learning. Biomed. Opt. Express 2019, 10, 3860–3874. 10.1364/BOE.10.003860. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Zhang X.; Lin T.; Xu J.; Luo X.; Ying Y. DeepSpectra: An end-to-end deep learning approach for quantitative spectral analysis. Anal. Chim. Acta 2019, 1058, 48–57. 10.1016/j.aca.2019.01.002. [DOI] [PubMed] [Google Scholar]
  27. Houhou R.; Barman P.; Schmitt M.; Meyer T.; Popp J.; Bocklitz T. Deep learning as phase retrieval tool for CARS spectra. Opt. Express 2020, 28, 21002–21024. 10.1364/OE.390413. [DOI] [PubMed] [Google Scholar]
  28. Liu J.; Osadchy M.; Ashton L.; Foster M.; Solomon C. J.; Gibson S. J. Deep convolutional neural networks for Raman spectrum recognition: a unified solution. Analyst 2017, 142, 4067–4074. 10.1039/C7AN01371J. [DOI] [PubMed] [Google Scholar]
  29. Luo R.; Popp J.; Bocklitz T. Deep Learning for Raman Spectroscopy: A Review. Analytica 2022, 3, 287–301. 10.3390/analytica3030020. [DOI] [Google Scholar]
  30. Dietterich T. G.; Lathrop R. H.; Lozano-Pérez T. Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 1997, 89, 31–71. 10.1016/S0004-3702(96)00034-3. [DOI] [Google Scholar]
  31. Zhou Z.-H. A brief introduction to weakly supervised learning. Natl. Sci. Rev. 2018, 5, 44–53. 10.1093/nsr/nwx106. [DOI] [Google Scholar]
  32. Foulds J.; Frank E. A review of multi-instance learning assumptions. Knowl. Eng. Rev. 2010, 25, 1–25. 10.1017/S026988890999035X. [DOI] [Google Scholar]
  33. Amores J. Multiple instance classification: Review, taxonomy and comparative study. Artif. Intell. 2013, 201, 81–105. 10.1016/j.artint.2013.06.003. [DOI] [Google Scholar]
  34. Bennett C. F. Therapeutic antisense oligonucleotides are coming of age. Annu. Rev. Med. 2019, 70, 307–321. 10.1146/annurev-med-041217-010829. [DOI] [PubMed] [Google Scholar]
  35. Dhuri K.; Bechtold C.; Quijano E.; Pham H.; Gupta A.; Vikram A.; Bahal R. Antisense oligonucleotides: an emerging area in drug discovery and development. J. Clin. Med. 2020, 9, 2004. 10.3390/jcm9062004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Yuen M.-F.; Heo J.; Jang J.-W.; Yoon J.-H.; Kweon Y.-O.; Park S.-J.; Tami Y.; You S.; Yates P.; Tao Y.; Cremer J.; Campbell F.; Elston R.; Theodore D.; Paff M.; Bennett C. F.; Kwoh T. J. Safety, tolerability and antiviral activity of the antisense oligonucleotide bepirovirsen in patients with chronic hepatitis B: a phase 2 randomized controlled trial. Nat. Med. 2021, 27, 1725–1734. 10.1038/s41591-021-01513-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Mukherjee P.; Aksamitiene E.; Alex A.; Shi J.; Bera K.; Zhang C.; Spillman D. R.; Marjanovic M.; Fazio M.; Seth P. P.; Frazier K.; Hood S. R.; Boppart S. A. Differential uptake of antisense oligonucleotides in mouse hepatocytes and macrophages revealed by simultaneous two-photon excited fluorescence and coherent Raman imaging. Nucleic Acid Ther. 2022, 32, 163–176. 10.1089/nat.2021.0059. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Zhang C.; Boppart S. A. Tracking the formation and degradation of fatty-acid-accumulated mitochondria using label-free chemical imaging. Sci. Rep. 2021, 11, 6671. 10.1038/s41598-021-85795-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Ilse M.; Tomczak J.; Welling M.. Attention-based deep multiple instance learning. In International conference on machine learning; PMLR: 2018; pp. 2127–2136. [Google Scholar]
  40. Zhou B.; Khosla A.; Lapedriza A.; Oliva A.; Torralba A.. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016; pp. 2921 −2929.
  41. Boorman D.; Pope I.; Masia F.; Langbein W.; Hood S.; Borri P.; Watson P. Hyperspectral CARS microscopy and quantitative unsupervised analysis of deuterated and non-deuterated fatty acid storage in human cells. J. Chem. Phys. 2021, 155, 224202 10.1063/5.0065950. [DOI] [PubMed] [Google Scholar]
  42. Yang J.; Xu J.; Zhang X.; Wu C.; Lin T.; Ying Y. Deep learning for vibrational spectral analysis: Recent progress and a practical guide. Anal. Chim. Acta 2019, 1081, 6–17. 10.1016/j.aca.2019.06.012. [DOI] [PubMed] [Google Scholar]
  43. Bjerrum E. J.; Glahder M.; Skov T.. Data augmentation of spectral data for convolutional neural network (CNN) based deep chemometrics. arXiv preprint arXiv:1710.01927 2017.
  44. Prakash T. P.; Graham M. J.; Yu J.; Carty R.; Low A.; Chappell A.; Schmidt K.; Zhao C.; Aghajan M.; Murray H. F.; Riney S.; Booten S. L.; Murray S. F.; Gaus H.; Crosby J.; Lima W. F.; Guo S.; Monia B. P.; Swayze E. E.; Seth P. P. Targeted delivery of antisense oligonucleotides to hepatocytes using triantennary N-acetyl galactosamine improves potency 10-fold in mice. Nucleic Acids Res. 2014, 42, 8796–8807. 10.1093/nar/gku531. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Liang T. J. Hepatitis B: a new weapon against an old enemy. Nat. Med. 2021, 27, 1672–1673. 10.1038/s41591-021-01512-5. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ac3c00979_si_001.pdf (2.5MB, pdf)

Articles from Analytical Chemistry are provided here courtesy of American Chemical Society

RESOURCES